Async processing in worker threads breaks offset commit guarantees
warningWhen offloading heavy processing to separate worker thread pool, application polls quickly and hands records to BlockingQueue. If enable.auto.commit is enabled, offsets commit before worker threads complete processing. If worker thread crashes, messages are lost since offset was already committed. Standard auto-commit assumes synchronous processing within poll loop.
Disable enable.auto.commit when using async worker pattern. Implement centralized Offset Manager that tracks completion acknowledgments from worker threads. Only commit offsets after worker confirms successful processing. Poller thread polls, enqueues to BlockingQueue, and immediately polls again. Worker pool processes from queue and signals completion to Offset Manager for manual commit.