Amazon Kinesis insights

Event Source Mapping Metric Blind Spots Without Opt-In

Kinesis event source mappings for Lambda provide detailed metrics (PolledEventCount, FilteredOutEventCount, FailedInvokeEventCount, etc.) only when explicitly enabled via EventSourceMappingMetricsConfig, leaving operators blind to processing failures and filtering effectiveness by default.

GetRecords.IteratorAgeMilliseconds measures how far behind consumers are from the latest data. Growing iterator age indicates consumers cannot keep up with incoming data rate, risking data loss when age exceeds retention period.

WriteProvisionedThroughputExceeded indicates producers are being throttled when exceeding 1MB/sec/shard or 1000 records/sec/shard limits. Without proper retry logic in producers, throttled records are permanently lost with no downstream indication.

Uneven data distribution across shards (hot shards) occurs when partition keys don't distribute records evenly, causing individual shards to hit throughput limits while others remain underutilized. Requires enhanced shard-level metrics to detect.

A significant drop in IncomingRecords or IncomingBytes metrics below expected baseline indicates producer application failure, network issues, or upstream data source problems. Unlike throttling, this shows successful writes decreasing rather than rejected writes increasing.

When Kinesis Firehose delivery streams fail to send events to destinations (like Splunk HEC), failed events accumulate in S3 backup buckets, but stream-level Kinesis metrics remain normal, hiding downstream ingestion failures.

Producers exceeding 1 MB/second/shard or 1,000 records/second/shard trigger WriteProvisionedThroughputExceeded errors, causing data ingestion failures and requiring application-level retries.

ReadProvisionedThroughputExceeded indicates GetRecords calls are throttled when multiple consumers exceed 5 reads/sec/shard or 2MB/sec/shard aggregate limit. Standard iterators share throughput, creating contention as consumer count increases.

Lambda Poison Message Creates Duplicate Processing

Lambda processes Kinesis records in batches and checkpoints only on success. A single poison message that causes processing failure triggers retry of the entire batch, creating duplicates for successfully processed records before the failure point.

When consumer processing logic can't keep pace with incoming data, IteratorAgeMilliseconds climbs continuously, risking data expiration and eventual loss. This creates a feedback loop where backlogged consumers fall further behind.

When consumers restart or fall behind while producers continue ingesting, accumulated records cause each GetRecords call to fetch larger batches, increasing GetRecords.Latency even after IncomingBytes normalizes.

Uneven partition key distribution concentrates write traffic on specific shards, causing per-shard throughput limits to be hit while aggregate stream capacity remains underutilized, leading to WriteProvisionedThroughputExceeded and processing delays.

Enabling server-side encryption on Kinesis streams introduces additional latency to GetRecords operations due to encryption/decryption overhead, increasing GetRecords.Latency baseline.

Lambda Consumer Falling Behind Stream

Lambda functions consuming Kinesis events can accumulate processing delays when function Duration, Errors, or Throttles prevent keeping pace with incoming stream data, manifesting as increased IteratorAge.

GetRecords.Latency measures time for each GetRecords operation. High latency indicates consumer processing logic (not network) is the bottleneck. For KCL consumers, this correlates with ProcessTask.Time metric showing slow per-record processing.

When consumers exceed Kinesis service limits of 5 GetRecords calls/second/shard or 2 MB/second/shard, ReadProvisionedThroughputExceeded throttling occurs, causing processing delays and increasing iterator age.

Applications using Kinesis enhanced fan-out (dedicated 2 MB/second per consumer per shard) can experience SubscribeToShard.RateExceeded errors when exceeding call limits or when multiple active subscriptions conflict, causing consumer processing delays.