Missing Observability Across Distributed Tasks
warningTasks execute on different workers/containers without unified tracing, making it impossible to understand end-to-end latency, trace failures to root cause, or see downstream impact of errors across the pipeline.
Lack of distributed tracing context between tasks; inability to correlate task failures with downstream data quality issues; fragmented logs across workers preventing root cause analysis; missing visibility into task-to-task latency breakdown.
Enable OpenTelemetry tracing in Airflow 2.7+ via traces.otel_on config; implement trace context propagation through XCom between tasks; add custom instrumentation inside operators to trace internal operations; use traced_task decorator pattern to automate context propagation; export traces to centralized backend for unified view.