Logs with Fluentbit

Overview

You can collect logs for Lumigo with Fluent Bit through integrating Fluent Bit with Lumigo. Integrating Fluent Bit with Lumigo enhances your observability by efficiently collecting, processing, and forwarding logs from your applications to Lumigo's monitoring platform.

Prerequisites

• Fluent Bit Installation: Ensure Fluent Bit is installed in your environment. For installation instructions, refer to the Fluent Bit documentation.

• Lumigo Account: Obtain a Lumigo account and ensure you have the necessary permissions to configure log ingestion.

Installation Steps

Follow the installation steps below to integrate Lumigo with Fluent Bit.

1. Configure Fluent Bit to Collect Logs

Set up Fluent Bit to collect logs from your applications. This involves configuring the appropriate input plugins based on your log sources. For example, to collect logs from files, you can use the tail input plugin:

[INPUT]
    Name        tail
    Path        /var/log/containers/*.log
    Tag         kube.*
    Parser      docker
    DB          /var/log/flb_kube.db
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

This configuration tells Fluent Bit to monitor log files in the /var/log/containers/ directory, commonly used in Kubernetes environments. Adjust the Path parameter to match the location of your log files.

2. Configure Fluent Bit to Output Logs to Lumigo

To send logs to Lumigo, configure Fluent Bit's output plugin to forward logs to Lumigo's HTTP endpoint. As of now, Fluent Bit does not have a dedicated Lumigo output plugin, so you can use the http output plugin to send logs to Lumigo's HTTP ingestion endpoint:

[OUTPUT]
    Name           opentelemetry
    Match          *
    Host           ga-otlp.lumigo-tracer-edge.golumigo.com
    Port           443
    tls            On
    tls.verify     On
    Header         Authorization LumigoToken <your_lumigo_token>

Replace <your_lumigo_token> with your Lumigo API token.

3. Deploy Fluent Bit Configuration

Deploy the Fluent Bit configuration in your environment. For example, in Kubernetes, you can create a ConfigMap with the Fluent Bit configuration and deploy it as a DaemonSet:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf

    [INPUT]
        Name        tail
        Path        /var/log/containers/*.log
        Tag         kube.*
        Parser      docker
        DB          /var/log/flb_kube.db
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [FILTER]
        Name        kubernetes
        Match       kube.*
        Merge_Log   On
        Keep_Log    Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

   [OUTPUT]
        Name           opentelemetry
        Match          *
        Host           ga-otlp.lumigo-tracer-edge.golumigo.com
        Port           443
        tls            On
        tls.verify     On
        Header         Authorization LumigoToken <your_lumigo_token>

Ensure the Header parameter includes your Lumigo API token.

4. Verify Integration

After deployment, verify that logs are being sent to Lumigo:

• Fluent Bit Logs: Check Fluent Bit's logs to ensure there are no errors in processing or forwarding logs.

• Lumigo Dashboard: Log in to your Lumigo account and navigate to the logs section to confirm that logs are being ingested and displayed correctly.