Confluent PlatformApache Kafka

Async processing in worker threads breaks offset commit guarantees

warning
configurationUpdated Mar 5, 2026(via Exa)
How to detect:

When offloading heavy processing to separate worker thread pool, application polls quickly and hands records to BlockingQueue. If enable.auto.commit is enabled, offsets commit before worker threads complete processing. If worker thread crashes, messages are lost since offset was already committed. Standard auto-commit assumes synchronous processing within poll loop.

Recommended action:

Disable enable.auto.commit when using async worker pattern. Implement centralized Offset Manager that tracks completion acknowledgments from worker threads. Only commit offsets after worker confirms successful processing. Poller thread polls, enqueues to BlockingQueue, and immediately polls again. Worker pool processes from queue and signals completion to Offset Manager for manual commit.