Low level how does this compare to Victoriametrics, you can get the high cardinality with clickhouse and other columnar dbs but the tradeoff is more ram usage, slower queries, io, etc
Our architecture is built around columnar design, and we use Apache Arrow for in-memory columnar processing and Apache Parquet for durable columnar storage on S3-compatible object storage. In Parseable, every labels stay as columns in the data instead of becoming a large long-lived per-series index like many TSDBs.
Also, one thing we’ve been thinking about a lot is how observability changes as agents become part of day-to-day engineering workflows. They're not just another service, they produce traces, tool calls, prompts, intermediate decisions, errors, costs, and sometimes sensitive business context.
Observing them matters just as much as observing any other system. But it is equally important to decide where that telemetry data should reside. Our view is that teams should be able to keep these observability data close to them: in their own object storage, under their own retention, access, and compliance controls.
show comments
simonw
The open source version doesn't accept protobufs, but does accept JSON.
Low level how does this compare to Victoriametrics, you can get the high cardinality with clickhouse and other columnar dbs but the tradeoff is more ram usage, slower queries, io, etc
Hi HN,
This is Yash, founding team at Parseable (https://github.com/parseablehq).
We've built an open source observability data lake using Rust, that handles high-cardinality data at around 100M time series in production (https://www.parseable.com/blog/how-parseable-handles-100-mil...)
Our architecture is built around columnar design, and we use Apache Arrow for in-memory columnar processing and Apache Parquet for durable columnar storage on S3-compatible object storage. In Parseable, every labels stay as columns in the data instead of becoming a large long-lived per-series index like many TSDBs.
Also, one thing we’ve been thinking about a lot is how observability changes as agents become part of day-to-day engineering workflows. They're not just another service, they produce traces, tool calls, prompts, intermediate decisions, errors, costs, and sometimes sensitive business context.
Observing them matters just as much as observing any other system. But it is equally important to decide where that telemetry data should reside. Our view is that teams should be able to keep these observability data close to them: in their own object storage, under their own retention, access, and compliance controls.
The open source version doesn't accept protobufs, but does accept JSON.
I found this out because I set Codex the task of running this locally agains another of my apps and it worked around the limitation by running this proxy: https://gist.github.com/simonw/b0e61a0aa8e3f7d30f27ce2f747c9...
... but it turns out my stack can emit JSON just fine, so I switched to that instead. Here's me TIL write-up of getting Parseable running locally https://til.simonwillison.net/datasette/datasette-parseable-...
How’s this any different then Iceberg?