Object store is quickly becoming the new core data substrate. Lets build kafka, but on s3. Lets build github, but on s3. It feels like we going to see more and more "object-store first" systems in the next few years.
I am excited about this future. Give me stateless servers and a storage bucket over having to manage systems with disks any day.
I do wonder if we will see an expansion of the s3 api to support more of these use cases. S3 added a janky file append operation to their new express-one-zone bucket type, and limited to 10k total file append operations. I wonder what else we will get in the next few years.
show comments
pcthrowaway
Rather than requiring consumers to ack the batch, why not just have them submit the ID of the batch tail on `consume` requests?
show comments
necubi
I'm the author of the post and tech lead for K2. Happy to answer any questions!
show comments
max8539
Cloudflare is catching up to be a full AWS/GCP/Azure. At start it was just a few services around but they are adding everything else
vira28
I wonder how many of data infra startups are wrapper on top of S3.
Also, the boundary between OLTP and OLAP is blurring every day.
Congrats on the launch! Stream/event-based system are really powerful, but they are also just pretty complex, in no small part because the modeling of streams for most people today is really modeling Kafka topic/partitions which has a whole bunch of foot-guns and complexity. Making the individual stream really cheap and easy is a big simplification, especially if flexibly consuming a stream for both ordered and unordered use-cases is made simple. This looks to work for unordered, be curious to see how the managed dividing the work for the mentioned key-based ordering.
show comments
wg0
Has anyone built anything on top of D1 + Durable objects for multi tenancy? How has been the experience?
eivindga
Congratulations on the launch! Looks very interesting indeed!
As someone who enjoys writing Kafka streams applications I am also looking forward to the day you support the Kafka APIs.
Having a cost efficient fully serverless Kafka compatible service would be great, and something I think many businesses would find useful.
Great work!
kirillkosolapov
How does this solution differ from AutoMQ and WarpStream? I’ve worked with one of them, and it is indeed a serverless solution built on top of S3.
As far as I know, Kafka itself already supports offloading some data to S3 for long-term storage.
Based on the articles—which I didn't fully grasp—I’m wondering if there are additional benefits mentioned, such as multi-region distribution (though I find it hard to imagine how that would be implemented).
show comments
sarkarghya
i think nats jetstream is superoverlooked in this space. the amount of throughput is more then enough for any medium scale enterprise as long as you know what you are doing
senderista
Not a single mention of AWS Kinesis? That seems like the closest competitor.
show comments
senderista
Minor correction: it is not true that no major object stores support appends--Azure Blob Storage has from the beginning.
loufe
If I was a serious Cloudflare customer I would be seriously concerned about the security of my infrastructure with them. Yes LLMs can code fast but this is an almost frenetic pace of releasing new products, all with fewer staff.
show comments
thepaulmcbride
This sounds like a super interesting product, but I'm always reluctant to build on anything that isn't a portable industry standard.
show comments
theredsix
What's the benefits of this over traditional GKE pub/sub, kafka or another queuing service?
show comments
ipkstef
i was trying to understand why i would use this over their current offerings of queues, and im really sick so my brain isn't working. so i ran it through ai
You need... | Use
-------------------------------------------------------|-------
“Make sure this job gets done” | Queue
retries / dead-letter handling | Queue
delayed jobs | Queue
distribute jobs among workers | Queue
“Record that this event happened” | K2
multiple independent systems reading the same events | K2
replay old events | K2
ordered event streams | K2
Kafka-like architecture | K2
Object store is quickly becoming the new core data substrate. Lets build kafka, but on s3. Lets build github, but on s3. It feels like we going to see more and more "object-store first" systems in the next few years.
I am excited about this future. Give me stateless servers and a storage bucket over having to manage systems with disks any day.
I do wonder if we will see an expansion of the s3 api to support more of these use cases. S3 added a janky file append operation to their new express-one-zone bucket type, and limited to 10k total file append operations. I wonder what else we will get in the next few years.
Rather than requiring consumers to ack the batch, why not just have them submit the ID of the batch tail on `consume` requests?
I'm the author of the post and tech lead for K2. Happy to answer any questions!
Cloudflare is catching up to be a full AWS/GCP/Azure. At start it was just a few services around but they are adding everything else
I wonder how many of data infra startups are wrapper on top of S3.
Also, the boundary between OLTP and OLAP is blurring every day.
For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed
(Disclaimer: I am one of the committers)
Congrats on the launch! Stream/event-based system are really powerful, but they are also just pretty complex, in no small part because the modeling of streams for most people today is really modeling Kafka topic/partitions which has a whole bunch of foot-guns and complexity. Making the individual stream really cheap and easy is a big simplification, especially if flexibly consuming a stream for both ordered and unordered use-cases is made simple. This looks to work for unordered, be curious to see how the managed dividing the work for the mentioned key-based ordering.
Has anyone built anything on top of D1 + Durable objects for multi tenancy? How has been the experience?
Congratulations on the launch! Looks very interesting indeed!
As someone who enjoys writing Kafka streams applications I am also looking forward to the day you support the Kafka APIs.
Having a cost efficient fully serverless Kafka compatible service would be great, and something I think many businesses would find useful.
Great work!
How does this solution differ from AutoMQ and WarpStream? I’ve worked with one of them, and it is indeed a serverless solution built on top of S3.
As far as I know, Kafka itself already supports offloading some data to S3 for long-term storage.
Based on the articles—which I didn't fully grasp—I’m wondering if there are additional benefits mentioned, such as multi-region distribution (though I find it hard to imagine how that would be implemented).
i think nats jetstream is superoverlooked in this space. the amount of throughput is more then enough for any medium scale enterprise as long as you know what you are doing
Not a single mention of AWS Kinesis? That seems like the closest competitor.
Minor correction: it is not true that no major object stores support appends--Azure Blob Storage has from the beginning.
If I was a serious Cloudflare customer I would be seriously concerned about the security of my infrastructure with them. Yes LLMs can code fast but this is an almost frenetic pace of releasing new products, all with fewer staff.
This sounds like a super interesting product, but I'm always reluctant to build on anything that isn't a portable industry standard.
What's the benefits of this over traditional GKE pub/sub, kafka or another queuing service?
i was trying to understand why i would use this over their current offerings of queues, and im really sick so my brain isn't working. so i ran it through ai