AWS Kinesis Integration
Monitor shard-level metrics, iterator age, throughput, and consumer lag across your Kinesis Data Streams. Get AI-powered hot shard detection and iterator age prediction before consumers fall irreversibly behind.
How It Works
Enable Enhanced Shard-Level Monitoring
Enable enhanced monitoring on your Kinesis streams to unlock per-shard metrics. Use the TigerOps CloudFormation template or AWS CLI to enable IncomingBytes, OutgoingBytes, and IteratorAgeMilliseconds at shard granularity.
Deploy CloudWatch Metric Streams
Run the TigerOps CloudFormation stack to stream the AWS/Kinesis namespace. Stream-level and shard-level metrics begin flowing to TigerOps within minutes of deployment.
Configure Consumer Application Tracking
Tag your Kinesis Enhanced Fan-Out consumers with application and team labels. TigerOps uses these tags to track per-consumer iterator age and throughput separately.
Set Iterator Age and Throughput Alerts
Define SLOs on GetRecords.IteratorAgeMilliseconds per stream and per consumer. TigerOps fires predictive alerts when iterator age trends upward and correlates with shard write throttling.
What You Get Out of the Box
Shard-Level Throughput Metrics
IncomingBytes, IncomingRecords, OutgoingBytes, and OutgoingRecords per shard. Identify hot shards that are approaching throughput limits and causing write rejections.
Iterator Age Monitoring
GetRecords.IteratorAgeMilliseconds per stream and per Enhanced Fan-Out consumer. High iterator age indicates consumer processing lag that risks losing real-time event streaming guarantees.
Consumer Lag Analysis
Per-consumer lag tracking for Enhanced Fan-Out consumers. TigerOps identifies which downstream consumers are falling behind and correlates with their CPU and processing metrics.
Throttling Detection
WriteProvisionedThroughputExceeded and ReadProvisionedThroughputExceeded per shard. Detect producers being throttled before they start dropping records or backing off.
Put Record Success Rate
PutRecord.Success and PutRecords.Success rates with failed record counts. Track producer health and identify record rejection spikes caused by shard capacity limits.
AI Hot Shard Detection
TigerOps detects uneven partition key distribution causing hot shards. When one shard handles disproportionate throughput, TigerOps recommends partition key rebalancing strategies.
CloudFormation Stack for Kinesis Metric Streams
Deploy the TigerOps CloudFormation stack and enable enhanced shard monitoring for complete Kinesis visibility.
# TigerOps CloudFormation — Kinesis Metric Streams
# aws cloudformation deploy \
# --template-file tigerops-kinesis-streams.yaml \
# --stack-name tigerops-kinesis \
# --capabilities CAPABILITY_IAM
Parameters:
TigerOpsApiKey:
Type: String
NoEcho: true
Resources:
TigerOpsKinesisStream:
Type: AWS::CloudWatch::MetricStream
Properties:
Name: tigerops-kinesis-stream
FirehoseArn: !GetAtt TigerOpsDeliveryStream.Arn
RoleArn: !GetAtt MetricStreamRole.Arn
OutputFormat: opentelemetry0.7
IncludeFilters:
- Namespace: AWS/Kinesis
- Namespace: AWS/Firehose
TigerOpsDeliveryStream:
Type: AWS::KinesisFirehose::DeliveryStream
Properties:
HttpEndpointDestinationConfiguration:
EndpointConfiguration:
Url: https://ingest.atatus.net/api/v1/cloudwatch
AccessKey: !Ref TigerOpsApiKey
RequestConfiguration:
CommonAttributes:
- AttributeName: service
AttributeValue: kinesis
RetryOptions:
DurationInSeconds: 60
# Enable enhanced shard-level monitoring on each stream:
# aws kinesis enable-enhanced-monitoring \
# --stream-name my-stream \
# --shard-level-metrics IncomingBytes IncomingRecords \
# OutgoingBytes OutgoingRecords WriteProvisionedThroughputExceeded \
# ReadProvisionedThroughputExceeded IteratorAgeMillisecondsCommon Questions
What Kinesis metrics does TigerOps collect?
TigerOps collects all AWS/Kinesis CloudWatch metrics including IncomingBytes, IncomingRecords, OutgoingBytes, OutgoingRecords, WriteProvisionedThroughputExceeded, ReadProvisionedThroughputExceeded, GetRecords.IteratorAgeMilliseconds, PutRecord.Success, and SubscribeToShardEvent.MillisBehindLatest per stream and shard.
Does TigerOps support Kinesis Data Firehose monitoring?
Yes. Kinesis Firehose metrics are published to the AWS/Firehose namespace. TigerOps supports a separate Metric Stream filter for Firehose delivery streams, covering DeliveryToS3.Success, DeliveryToRedshift.Success, BackupToS3.Success, and delivery latency metrics.
How do I monitor Enhanced Fan-Out consumers with TigerOps?
Enhanced Fan-Out consumer metrics are published with the ConsumerName dimension in CloudWatch. TigerOps uses this dimension to display per-consumer MillisBehindLatest and SubscribeToShardEvent counts, so you can compare consumer lag across all registered consumers on a stream.
Can TigerOps alert on Kinesis shard splitting and merging events?
Yes. Kinesis UpdateShardCount operations emit CloudTrail events that TigerOps captures via EventBridge. When a shard split or merge occurs, TigerOps creates a timeline annotation so you can correlate resharding events with throughput and iterator age changes.
How does TigerOps help with Kinesis Data Streams capacity planning?
TigerOps tracks write throughput utilization as a percentage of provisioned capacity per stream and per shard over time. By analyzing peak utilization trends and growth rates, TigerOps generates capacity recommendations for when to increase shard count before throttling begins.
Stop Discovering Kinesis Hot Shards and Iterator Age Issues at 3 AM
Shard-level throughput, iterator age tracking, and AI hot shard detection. Deploy in 5 minutes.