Logs with Fluentd

Overview

You can collect logs for Lumigo with Fluentd through integrating Fluentd with Lumigo. Integrating Fluentd with Lumigo enhances your observability by efficiently collecting, processing, and forwarding logs from your applications to Lumigo's monitoring platform.

Prerequisites

• Fluentd Installation: Ensure Fluentd is installed in your environment. For installation instructions, refer to the Fluentd documentation.

• Lumigo Account: Obtain a Lumigo account and ensure you have the necessary permissions to configure log ingestion.

Installation Steps

Follow the steps below to integrate Lumigo with Fluentd.

1. Configure Fluentd to Collect Logs

   Set up Fluentd to collect logs from your applications by configuring the appropriate input plugins based on your log sources. For example, to collect logs from files, you can use the tail input plugin:

   ini

   <source>
       @type tail
       path /var/log/containers/*.log
       tag kube.*
       <parse>
           @type json
       </parse>
   </source>

   This configuration instructs Fluentd to monitor log files in the /var/log/containers/ directory, commonly used in Kubernetes environments. Adjust the path parameter to match the location of your log files.

2. Configure Fluentd to Output Logs to Lumigo

   Fluentd does not natively support sending logs through OpenTelemetry. To send logs to Lumigo, format the logs as OpenTelemetry ones, then send them to Lumigo as a plain HTTP request, like this:

   ini

   <filter **>
     @type record_transformer
     <record>
       resourceLogs [{
         "resource": {
           "attributes": [
             { "key": "service.name", "value": { "stringValue": "fluentd-service" } }
           ]
         },
         "scopeLogs": [{
           "scope": { "name": "fluentd", "version": "1.0" },
           "logRecords": [{
             "timeUnixNano": "${time.to_i * 1000000000}",
             "severityNumber": 9,
             "severityText": "INFO",
             "body": { "stringValue": "${message}" },
             "attributes": [
               { "key": "log.file", "value": { "stringValue": "${file}" } }
             ]
           }]
         }]
       }]
     </record>
   </filter>
   <match **>
       @type http
       endpoint https://ga-otlp.lumigo-tracer-edge.golumigo.com
       http_method post
       serializer json
       <format>
           @type json
       </format>
       <buffer>
           @type memory
           flush_interval 5s
       </buffer>
       <header>
           Authorization LumigoToken <your_lumigo_token>
       </header>
   </match>
   

   Replace <your_lumigo_token> with your Lumigo API token.

3. Deploy Fluentd Configuration

   Deploy the Fluentd configuration in your environment. For example, in Kubernetes, you can create a ConfigMap with the Fluentd configuration and deploy it as a DaemonSet:

   YAML

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: fluentd-config
     namespace: logging
   data:
     fluentd.conf: |
       <source>
           @type tail
           path /var/log/containers/*.log
           tag kube.*
           <parse>
               @type json
           </parse>
       </source>
   
       <filter kube.**>
           @type kubernetes_metadata
       </filter>
   
       <filter **>
     @type record_transformer
     <record>
       resourceLogs [{
         "resource": {
           "attributes": [
             { "key": "service.name", "value": { "stringValue": "fluentd-service" } }
           ]
         },
         "scopeLogs": [{
           "scope": { "name": "fluentd", "version": "1.0" },
           "logRecords": [{
             "timeUnixNano": "${time.to_i * 1000000000}",
             "severityNumber": 9,
             "severityText": "INFO",
             "body": { "stringValue": "${message}" },
             "attributes": [
               { "key": "log.file", "value": { "stringValue": "${file}" } }
             ]
           }]
         }]
       }]
     </record>
   </filter>
       <match **>
           @type http
           endpoint https://ga-otlp.lumigo-tracer-edge.golumigo.com
           http_method post
           serializer json
           <format>
               @type json
           </format>
           <buffer>
               @type memory
               flush_interval 5s
           </buffer>
           <header>
               Authorization LumigoToken <your_lumigo_token>
           </header>
       </match>

   Ensure the Authorization header includes your Lumigo API token.

4. Verify Integration

   After deployment, verify that logs are being sent to Lumigo:

   • Fluentd Logs: Check Fluentd's logs to ensure there are no errors in processing or forwarding logs.

   • Lumigo Dashboard: Log in to your Lumigo account and navigate to the logs section to confirm that logs are being ingested and displayed correctly.

By following these steps, you can effectively integrate Fluentd with Lumigo to enhance your application's observability.