What's been your expiriance with regard to producer latency with Kinesis? My limited experimentation with it so far seems to point to that as a potential pain point for high throughout low latency systems.

In my experience, Kinesis' performance makes it a nonstarter for low-latency systems and use cases with many consumers. Shards are limited to 5 reads/sec, so consumers are heavily throttled. Producers also have significant put latency. Kafka's latency and throughput are OOM better but of course come with the operational overhead. Depends on what your needs are, but Kafka is a better choice for "real-time" systems.

Interesting; I didn't know that Kafka had order-of-magnitude-better latency. Thanks for the data point.

I wonder if this is something inherent to Kinesis's design, or if Amazon will magically make it faster at some point in the future. Do you have a suspicion/indication either way?

Well, for one, there is no streaming socket API, just HTTP. I suspect the 5 reads/sec limit on shards is mainly due to multitenancy, but I could see this increasing in the future. Amazon has already improved the latency of Kinesis a fair amount, but it still has a ways to go before it matches Kafka.

We also have cases where we need many consumers reading a stream, so having that effectively limited to 5 concurrent consumers is a pretty tough limitation. Their solution to this is to just have consumers sleep on a failed poll, but that doesn't really help if you want to scale out your ingestion. You don't have that problem with Kafka.

Kinesis has gotten better about it's latencies, but it is inherently an HTTP protocol, which already gets you in pretty big trouble for serious real-time applications.

I only ran some perf testing and haven't used Kinesis in production yet but perf testing I ran end to end was < 1 sec end to end for a couple of thousand records per second using 3 shards. The reads per second are a lot lower than the writes per second because readers are expected to batch reads - you get 1000 puts/sec/shard vs. 5 reads/sec/shard, but 1 MB/sec read throughput and 2 MB /sec write throughput in shard, so you can read faster than you can write (and catch up if you fall behind). You won't get low millisec or microsec responses because Kinesis is replicating messages to multiple stores for resiliency.

Generally 10s of seconds to low minutes. Two caveats: 1) we batch/queue writes on the producer side, and 2) our systems are more near-line than true low-latency, e.g. we scale the cluster up/down instead of keeping machines running to keep latency as low as possible. So YMMV.

I'd be interested in your results, if you were shooting for a specific number and couldn't hit it, just for future reference.