Ingestion Source Backpressure from Sink Write Throttling
warningDataHub ingestion pipelines stall when sink write rate (sink_workunits_write) lags significantly behind source production rate (source_workunits_produced). This indicates downstream storage systems (Elasticsearch, database) cannot keep pace with ingestion volume.
Monitor source_workunits_produced continuously increasing while sink_workunits_write flatlines or grows slowly. Check kafka_message_queue_time increasing as messages queue. Correlate with Elasticsearch thread_pool.write rejections or database connection pool exhaustion.
Scale Elasticsearch cluster horizontally to increase write throughput. Increase database connection pool size (SPRING_DATASOURCE_HIKARI_MAXIMUM_POOL_SIZE). Tune ingestion batch sizes to match sink capacity. Implement backpressure limits in ingestion pipelines to prevent overwhelming storage. Consider splitting large ingestion jobs into smaller batches.