Amazon MSK (Managed Streaming for Apache Kafka)

Amazon Managed Streaming for Apache Kafka (MSK) runs open-source Apache Kafka as a managed service. MSK provides the control plane — creating, updating and deleting clusters, replacing failed brokers, applying patches — while producers and consumers use the ordinary Kafka data-plane protocol. Because it is Kafka rather than a Kafka-compatible API, existing applications, tooling and community plugins work unchanged.

Cluster types

MSK Provisioned is a cluster whose brokers are chosen and sized explicitly. It offers two broker types: Standard brokers, and Express brokers, which trade some configurability for faster scaling and recovery. Metadata is managed by KRaft controllers, included at no extra cost, which replaced the ZooKeeper ensembles earlier clusters ran.

MSK Serverless removes broker sizing entirely: capacity is provisioned and scaled automatically at the cluster level, and billing is for throughput and storage rather than broker hours. It is fully compatible with Apache Kafka, and suits spiky or unpredictable workloads where a provisioned cluster would be idle much of the time.

MSK Connect runs Kafka Connect connectors as a managed service, to stream data into and out of a cluster without operating connector workers yourself.

MSK Replicator replicates data between MSK clusters, in the same Region or across Regions, for disaster recovery or to give consumers a local copy.

Security and logging

1. Integrates with AWS KMS for server-side encryption
2. Encryption at rest is on by default
3. TLS 1.2 encrypts traffic in transit between brokers in a cluster
4. Broker logs can be delivered to Amazon CloudWatch Logs, Amazon S3 and Amazon Data Firehose
5. Metrics are published to CloudWatch
6. All Amazon MSK API calls are recorded in AWS CloudTrail

MSK or Kinesis Data Streams

Both are ordered, replayable, partitioned streams, and either will carry most workloads. The choice is usually decided by ecosystem and operating model rather than capability.

Choose MSK when the application already speaks Kafka, when the design depends on the Kafka ecosystem (Kafka Connect connectors, Kafka Streams, ksqlDB, Schema Registry, Debezium), when consumer groups and offset management are part of the design, or when portability between clouds and on-premises matters.

Choose Kinesis Data Streams when the smallest possible operational surface matters, when the consumers are Lambda functions or other AWS services with native integrations, or when on-demand capacity that scales itself is worth more than the Kafka feature set.

Choose MSK Serverless over MSK Provisioned when traffic is uneven or hard to forecast, or when nobody wants to own broker sizing. Choose provisioned when throughput is steady and predictable enough that reserved capacity is cheaper, or when broker-level configuration that serverless does not expose is required.