Are you able to get began with cloud-native observability with telemetry pipelines?
This text is a part of a collection exploring a workshop guiding you thru the open supply challenge Fluent Bit, what it’s, a primary set up, and organising the primary telemetry pipeline challenge. Learn to handle your cloud-native information from supply to vacation spot utilizing the telemetry pipeline phases masking assortment, aggregation, transformation, and forwarding from any supply to any vacation spot.
Since Chronosphere acquired the capabilities for integrating telemetry pipelines, I have been digging into how this works, the use circumstances it solves, and having a whole lot of enjoyable with the idea, CNCF project Fluent Bit. This workshop is the results of my sharing the way to get began with telemetry pipelines and all that you are able to do with Fluent Bit.
This primary article within the collection gives an introduction to Fluent Bit the place we acquire an understanding of its position within the cloud native observability world. You will discover extra particulars within the accompanying workshop lab.
Earlier than we get began, let’s get a baseline for outlining cloud-native observability pipelines. As famous in a recent trend report:
Observability pipelines are offering real-time filtering, enrichment, normalization and routing of telemetry information.
The rise within the quantity of knowledge being generated in cloud-native environments has turn into such a burden for groups making an attempt to handle all of it, in addition to a burden to group’s budgets. They’re trying to find extra management over all this telemetry information, from amassing, processing, and routing, to storing and querying.
Knowledge pipelines have gained traction in serving to organizations take care of the challenges they’re dealing with by offering a strong technique to decrease ingestion volumes and assist scale back information prices.
One of many advantages is that telemetry pipelines act as a telemetry gateway between cloud-native information and organizations. They carry out real-time filtering, enrichment, normalization, and routing to low-cost storage. This reduces dependencies on costly and infrequently proprietary storage options.
One other plus for organizations is the flexibility to reformat collected information on the fly, typically bridging the hole between legacy or non-standards-based information constructions to present requirements. They will obtain this with out having to replace code, re-instrument, or redeploy present purposes and companies.
Telemetry Pipelines
This workshop focuses solely on Fluent Bit because the open-source telemetry pipeline challenge. From the project documentation, Fluent Bit is an open-source telemetry agent particularly designed to effectively deal with the challenges of amassing and processing telemetry information throughout a variety of environments, from constrained techniques to complicated cloud infrastructures. It is efficient at managing telemetry information from numerous sources and codecs could be a fixed problem, significantly when efficiency is a crucial issue.
Whereas the time period “observability pipelines” is thrown about to cowl all types of basic pipeline actions, the main focus on this workshop will probably be extra on telemetry pipelines. This is because of our deal with getting all various kinds of telemetry from their origins to the locations we need, and as famous within the beforehand referenced development report:
Telemetry pipelines present real-time filtering, enrichment, normalization, and routing of telemetry information.
Reasonably than serving as a drop-in alternative, Fluent Bit enhances the observability technique in your infrastructure by adapting and optimizing your present logging layer, in addition to metrics and hint processing. Moreover, Fluent Bit helps a vendor-neutral method, seamlessly integrating with different ecosystems similar to Prometheus and OpenTelemetry.
Fluent Bit may be deployed as an edge agent for localized telemetry information dealing with or utilized as a central aggregator or collector for managing telemetry information throughout a number of sources and environments. Fluent Bit has been designed for efficiency and low useful resource consumption.
As a telemetry pipeline, Fluent Bit is designed to course of logs, metrics, and traces at pace, scale, and with flexibility.
What About Fluentd?
First, there was Fluentd, a CNCF graduated challenge. It is an open-source information collector for constructing the unified logging layer. When put in, it runs within the background to gather, parse, rework, analyze, and retailer numerous varieties of information.
Fluent Bit is a sub-project inside the Fluentd ecosystem. It is thought of a light-weight information forwarder for Fluentd. Fluent Bit is particularly designed for forwarding the information from the sting to Fluentd aggregators.
Each tasks share similarities. Fluent Bit is absolutely designed and constructed on high of the most effective concepts of Fluentd structure and basic design:
Understanding the Ideas
Earlier than we dive into utilizing Fluent Bit, it is vital to have an understanding of the important thing ideas, so let’s discover the next:
- Occasion or Report:Â Every incoming piece of knowledge is taken into account an occasion or a document.
- Filtering: The method of altering, enriching, or dropping an occasion
- Tag: An inside string utilized by the router in later levels of our pipeline to find out which filters or output phases an occasion should cross by
- Timestamp:Â Assigned to every occasion because it enters a pipeline and is at all times current
- Match: Represents a rule utilized to occasions the place it examines its tags for matches
- Structured Message: The objective is to make sure that all messages have a structured format, outlined as having keys and values.
Pipeline Phases
A telemetry pipeline is the place information goes by numerous phases from assortment to last vacation spot. We will outline or configure every section to govern the information or the trail it is taking by our telemetry pipeline.
The primary section is INPUT, which is the place Fluent Bit makes use of enter plugins to collect data from particular sources. When an enter plugin is loaded, it creates an occasion that we will configure utilizing the plugin’s properties.
The second section is PARSER, which is the place unstructured enter information is changed into structured information. Fluent Bit does this utilizing parsers that we will configure to govern the unstructured information producing structured information for the subsequent phases of our pipeline.
The FILTER section is after we modify, enrich, or delete any of the collected occasions. Fluent Bit gives many out-of-the-box plugins similar to filters that may match, exclude, or enrich your structured information earlier than it strikes onward within the pipeline. Filters may be configured utilizing the supplied properties.
The BUFFER section is the place the information is saved, utilizing in-memory or file system-based choices. Be aware that when information reaches the buffer section it is in an immutable state (no extra filtering) and that buffered information just isn’t uncooked textual content, however in an inside binary illustration for storage.
The subsequent section is ROUTING, which is the place Fluent Bit makes use of the beforehand mentioned tag and match ideas to find out which output locations to ship information to. Through the enter section, information is assigned a tag; through the routing section, information is in comparison with match guidelines from output configurations. If it matches, then the information is shipped to that output vacation spot.
The ultimate section is OUTPUT, which is the place Fluent Bit makes use of output plugins to attach with particular locations. These locations may be databases, distant companies, cloud companies, and extra. When an enter plugin is loaded, it creates an occasion that we will configure utilizing the plugin’s properties.
For code examples for these phases and extra particulars about telemetry pipeline phases, see the workshop lab.
What’s Subsequent?
This text was an introduction to telemetry pipelines and Fluent Bit. This collection continues with the subsequent step on this workshop: putting in Fluent Bit in your native machine from the supply or utilizing container photos.
Keep tuned for extra hands-on materials that can assist you along with your cloud-native observability journey.
