Fluent Bit log drain

Overview

This topic explains how to send logs from Fluent Bit to LaunchDarkly’s observability dashboards. Fluent Bit is a lightweight and high-performance log processor and forwarder that is commonly used in containerized environments such as Kubernetes and Amazon ECS.

By configuring Fluent Bit to forward logs to LaunchDarkly, you can centralize your application logs alongside your feature flag data for unified observability.

Prerequisites

To use the Fluent Bit integration, you need:

• A LaunchDarkly project key for your target environment
• Fluent Bit installed and running in your environment
• Network access to LaunchDarkly’s OpenTelemetry endpoint

To learn how to find and copy your LaunchDarkly project key, read Project keys.

Configure Fluent Bit

Fluent Bit uses the Forward protocol to send logs to LaunchDarkly’s OpenTelemetry collector endpoint.

Basic configuration

Add the following output configuration to your Fluent Bit configuration file:

1
[OUTPUT]
2
    Name          forward
3
    Match         *
4
    Host          otel.observability.app.launchdarkly.com
5
    Port          24224
6
    Tag           launchdarkly.project_id=YOUR_PROJECT_KEY

Replace YOUR_PROJECT_KEY with your LaunchDarkly project key.

Configuration with TLS

For production environments, enable TLS for secure communication:

1
[OUTPUT]
2
    Name          forward
3
    Match         *
4
    Host          otel.observability.app.launchdarkly.com
5
    Port          24224
6
    Tag           launchdarkly.project_id=YOUR_PROJECT_ID
7
    tls           on
8
    tls.verify    on

AWS ECS with AWS FireLens

If you’re running containers on Amazon ECS, you can use AWS FireLens to route logs through Fluent Bit to LaunchDarkly. FireLens is a container log router for Amazon ECS that gives you the ability to use Fluent Bit for log processing.

Here’s an example ECS task definition snippet:

1
{
2
  "containerDefinitions": [
3
    {
4
      "name": "your-app",
5
      "image": "your-app-image",
6
      "logConfiguration": {
7
        "logDriver": "awsfirelens",
8
        "options": {
9
          "Name": "forward",
10
          "Host": "otel.observability.app.launchdarkly.com",
11
          "Port": "24224",
12
          "Tag": "launchdarkly.project_id=YOUR_PROJECT_KEY"
13
        }
14
      }
15
    },
16
    {
17
      "name": "log_router",
18
      "image": "amazon/aws-for-fluent-bit:latest",
19
      "essential": true,
20
      "firelensConfiguration": {
21
        "type": "fluentbit"
22
      }
23
    }
24
  ]
25
}

Here is an example using Terraform:

1
resource "aws_ecs_task_definition" "app" {
2
  # ... other configuration ...
3
4
  container_definitions = jsonencode([
5
    {
6
      name  = "your-app"
7
      image = "your-app-image"
8
      logConfiguration = {
9
        logDriver = "awsfirelens"
10
        options = {
11
          Name = "Forward"
12
          Host = var.observability_logging.host
13
          Port = "24224"
14
          Tag  = "launchdarkly.project_id=${var.launchdarkly_project_id}"
15
        }
16
      }
17
    },
18
    {
19
      name      = "log_router"
20
      image     = "amazon/aws-for-fluent-bit:latest"
21
      essential = true
22
      firelensConfiguration = {
23
        type = "fluentbit"
24
      }
25
    }
26
  ])
27
}

Kubernetes with Fluent Bit DaemonSet

For Kubernetes environments, you can configure Fluent Bit as a DaemonSet to collect logs from all pods:

1
apiVersion: v1
2
kind: ConfigMap
3
metadata:
4
  name: fluent-bit-config
5
  namespace: logging
6
data:
7
  fluent-bit.conf: |
8
    [SERVICE]
9
        Flush         1
10
        Log_Level     info
11
        Daemon        off
12
        Parsers_File  parsers.conf
13
14
    [INPUT]
15
        Name              tail
16
        Path              /var/log/containers/*.log
17
        Parser            docker
18
        Tag               kube.*
19
        Refresh_Interval  5
20
        Mem_Buf_Limit     5MB
21
22
    [OUTPUT]
23
        Name          forward
24
        Match         *
25
        Host          otel.observability.app.launchdarkly.com
26
        Port          24224
27
        Tag           launchdarkly.project_id=YOUR_PROJECT_KEY
28
        tls           on
29
        tls.verify    on

Configuration options

The following table describes the key configuration options for the Fluent Bit Forward output:

View logs in LaunchDarkly observability dashboards

After you configure Fluent Bit, logs begin flowing to LaunchDarkly within a few minutes.

To view your logs:

1. Navigate to Observability in LaunchDarkly.
2. Select the environment associated with your project.
3. Navigate to the Logs tab.
4. Use the search and filter options to find specific log entries.

Your logs are automatically associated with your LaunchDarkly project, allowing you to correlate log data with feature flag evaluations and other observability metrics.

Troubleshooting

If logs are not appearing in LaunchDarkly:

1. Verify network connectivity: Ensure your Fluent Bit instance can reach otel.observability.app.launchdarkly.com on port 24224.

2. Check the project key: Verify that the launchdarkly.project_id in your tag matches your actual LaunchDarkly project key.

3. Review Fluent Bit logs: Check Fluent Bit’s own logs for connection errors or warnings:

$
# For standalone Fluent Bit
$
fluent-bit -c /path/to/fluent-bit.conf -v
$
$
# For Kubernetes DaemonSet
$
kubectl logs -n logging -l app=fluent-bit

4. Test connectivity: Use a tool like nc or telnet to verify port accessibility:

$
nc -zv otel.observability.app.launchdarkly.com 24224

Related content

• Observability overview
• Logs
• Fluent Bit documentation
• AWS FireLens documentation