Django Instrumentation, logs not flowing in Graphana alloy OTel collector

[overide]

Hi guys,

I'm exploring Grafana Cloud and trying to instrument my Django app and send the Otel data to the Grafana Cloud using Grafana Alloy. I could not get logs on the Graphana dashboard, even in the Alloy debug console; I don't see the logs flowing.

I had a quick chat with the Alloy developer in the Graphana community, and he confirms that my Alloy config is correct, and the issue could be either the way I instrumented the Django app or in Django logging

Following is my app setup -

• Django app running inside Docker
• Gunicorn serving my Django app (Instrumented it as mentioned here - https://grafana.com/docs/opentelemetry/instrument/python/)
• Alloy is running inside Docker as well

What's working -

• Traces
• Metrics

What's not working -

• Logs (app and access logs)
• Postgres instrumentation

Python and package version

Python 3.9.19
opentelemetry-distro = {extras = ["otlp"], version = "^0.59b0"}
opentelemetry-exporter-otlp = "^1.38.0"
opentelemetry-instrumentation-django = "^0.59b0"
opentelemetry-instrumentation-psycopg2 = "^0.59b0"
opentelemetry-instrumentation-logging = "^0.59b0"

Adding the following config below -

1. Gunicorn config
2. Django Log config
3. Graphana Alloy config

Gunicorn Config

"""
The OpenTelemetry Python SDK uses the Global Interpreter Lock (GIL), which
can cause performance issues with Gunicorn spawn multiple processes
to serve requests in parallel.

To address this, register a post fork hook that runs after each worker
process is forked. Gunicorn post_fork hook initializes OTel inside each
worker which avoids lock inheritance/deadlocks and the perf hit.

Read more - https://grafana.com/docs/opentelemetry/instrument/python/?pg=blog&plcmt=body-txt#global-interpreter-lock
"""

import os
import logging
import multiprocessing
from uuid import uuid4

from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc._log_exporter import (
    OTLPLogExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)

# support for logs is currently experimental
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.resources import SERVICE_INSTANCE_ID
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from opentelemetry.instrumentation.django import DjangoInstrumentor
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor

# your gunicorn config here
wsgi_app = "spectra.wsgi:application"
bind = "0.0.0.0:8000"
name = "spectra-backend"

# Multi-process concurrency (bypasses the GIL)
workers = multiprocessing.cpu_count()  # adjust via WEB_CONCURRENCY if needed
threads = 2
worker_class = "gthread"

# Performance & stability
timeout = 300
keepalive = 5
max_requests = 1000
max_requests_jitter = 100
worker_tmp_dir = "/dev/shm"

# Logging
loglevel = "debug"
accesslog = "-"
errorlog = "-"
capture_output = False

# https://github.com/open-telemetry/opentelemetry-python/blob/main/docs/examples/logs/example.py

def post_fork(server, worker):
    server.log.info("Worker spawned (pid: %s)", worker.pid)

    collector_endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")

    resource = Resource.create(
        attributes={
            # each worker needs a unique service.instance.id to distinguish the created metrics in prometheus
            SERVICE_INSTANCE_ID: str(uuid4()),
            "worker": worker.pid,
        }
    )

    tracer_provider = TracerProvider(resource=resource)
    tracer_provider.add_span_processor(
        BatchSpanProcessor(OTLPSpanExporter(endpoint=collector_endpoint))
    )
    trace.set_tracer_provider(tracer_provider)

    metrics.set_meter_provider(
        MeterProvider(
            resource=resource,
            metric_readers=[
                PeriodicExportingMetricReader(
                    OTLPMetricExporter(endpoint=collector_endpoint)
                )
            ],
        )
    )

    logger_provider = LoggerProvider(resource=resource, )
    logger_provider.add_log_record_processor(
        BatchLogRecordProcessor(OTLPLogExporter(endpoint=collector_endpoint, insecure=True))
    )
    logging.getLogger().addHandler(
        LoggingHandler(level=logging.NOTSET, logger_provider=logger_provider)
    )

    # This links the LoggerProvider to the Python logging system.
    # LoggingInstrumentor().instrument(
    #     set_logging_format=True,
    #     logger_provider=logger_provider,
    #     log_level=logging.DEBUG,
    # )

    # Instruments incoming HTTP requests handled by Django/Gunicorn
    DjangoInstrumentor().instrument()

    # Instruments database calls made through psycopg2
    Psycopg2Instrumentor().instrument()

    server.log.info("OTel initialized in worker pid=%s", worker.pid)

Django Log settings

LOGGING = {
    "version": 1,
    "disable_existing_loggers": False,
    "formatters": {
        "verbose": {
            "format": "{levelname} {asctime} {module} {message}",
            "style": "{",
        },
        "simple": {
            "format": "{levelname} {message}",
            "style": "{",
        },
    },
    "handlers": {
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "stream": sys.stdout,
            "formatter": "verbose",
        },
    },
    "root": {
        "handlers": ["console"],
        "level": "INFO",
    },
    "loggers": {
        "django": {
            "level": "INFO",
            "propagate": True,
        },
        "celery": {
            "level": "INFO",
            "propagate": True,
        },
        "gunicorn.error": {
            "level": "INFO",
            "propagate": True,
        },
        "gunicorn.access": {
            "level": "INFO",
            "propagate": True,
        },
    },
}

Alloy configuration

// 1. OTLP Receiver: listens for data from your Django app
otelcol.receiver.otlp "default" {
  http {}
  grpc {}
  output {
    traces  = [
      otelcol.processor.batch.default.input,
      otelcol.connector.host_info.default.input,
	  ]
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
  }
}

// 2.1 Host info
otelcol.connector.host_info "default" {
  host_identifiers = ["host.name"]

  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

// 2.2 Processor: batches data to improve efficiency before exporting
otelcol.processor.batch "default" {
	output {
		logs = [otelcol.exporter.otlphttp.grafana_cloud.input]
		traces  = [otelcol.exporter.otlphttp.grafana_cloud.input]
		metrics = [otelcol.exporter.otlphttp.grafana_cloud.input]
  }
}

// 3 OTLP Exporter Authentication: use Instance ID (username) and API Key (password)
otelcol.auth.basic "grafana_cloud_auth" {
  username = sys.env("GRAFANA_CLOUD_OTLP_INSTANCE_ID")
  password = sys.env("GRAFANA_CLOUD_OTLP_API_KEY")
}

// 4. Export all OTLP signals to Grafana Cloud
otelcol.exporter.otlphttp "grafana_cloud" {
  client {
    endpoint = sys.env("GRAFANA_CLOUD_OTLP_ENDPOINT")
    auth  = otelcol.auth.basic.grafana_cloud_auth.handler
  }
}

// Exporter: Gathers metrics from the mounted host directories
prometheus.exporter.unix "host_metrics_source" {
  enable_collectors = ["cpu", "memory", "disk", "network"]
  include_exporter_metrics = true
  procfs_path = "/host/proc"
  sysfs_path  = "/host/sys"
  rootfs_path = "/rootfs"
}

// Scraper: Scrapes the built-in exporter component itself
prometheus.scrape "host_metrics_scrape" {
  targets         = prometheus.exporter.unix.host_metrics_source.targets
  scrape_interval = "30s"
  forward_to      = [prometheus.relabel.host_metrics_source.receiver]
}

// Rename job and label
prometheus.relabel "host_metrics_source" {
  rule {
    target_label = "instance"
    replacement  = "spectra-backend"
  }

  rule {
    target_label = "nodename"
    replacement  = "spectra-backend"
  }

  rule {
    target_label = "job"
    replacement  = "host-infra"
  }

  forward_to = [prometheus.remote_write.prometheus_receiver.receiver]
}

// Export to Grafana cloud prometheus
prometheus.remote_write "prometheus_receiver" {
  endpoint {
      url = sys.env("GRAFANA_CLOUD_PROMETHEUS_WRITE_ENDPOINT")
      basic_auth {
        username = sys.env("GRAFANA_CLOUD_PROMETHEUS_INSTANCE_ID")
        password = sys.env("GRAFANA_CLOUD_PROMETHEUS_API_KEY")
      }
    }
}


livedebugging {
  enabled = true
}

[overide]

Anything looking for solution, this works - #4294