We've found DuckDB 2.0 to be way faster on high core count machines on EC2. Not just reading data from S3, but the actual computations too. Much more likely to see 100% utilisation up to about 128vCPUs or 192vCPUs, and that CPU usage to translate into workloads completing faster.
Still don't usually see a speedup past about 192vCPU, but prior to 2.0, speed would top out at more like 64vCPU, so this is a big win.
That has enabled us to scale Splink (FOSS record linakge software) to process 1bn records with 10bn record comparisons in 10 mins, which as far as I'm aware makes it the fastest linkage engine of this kind. Most of this performance comes from simply delegating the big calculations to the DuckDB engine! https://moj-analytical-services.github.io/splink/topic_guide...
Thanks to the team for all their work!
show comments
nicoritschel
It's very fast, especially on a decent sized cloud VM and when querying Iceberg. I've been running the alpha in production as a centralized analytics server for a small organization.
I built a DuckDB extension so you can connect via Postgres wire protocol, or Arrow flight. DuckDB also has its own Quack protocol, but client support for Quack is very limited right now, whereas many data tools support Postgres wire protocol or ADBC (which you can use with Arrow Flight).
The fact that can now be used as a client-server architecture is a game changer. It will for sure replace many data stacks that now run on many servers with tons of cpu and ram
show comments
datadrivenangel
Very excited for Triggers. Now DuckDB really is close to being a data warehouse in a box.
show comments
xnx
Absolutely fantastic piece of software. Even beyond being used as engine for analysis, it's amazingly versatile for ingesting and reshaping data. Great job.
wodenokoto
Is it possible to save DuckDB UI notebooks to files?
show comments
skybrian
I wonder how terrible would it be to use DuckDB as the main database (running transactions) for a small project?
mikkelam
I really want to use DuckDB. But I have struggled to find a use case in my company.
What are you guys using it for?
show comments
pletnes
What is the benefit of using async io for csv and parquet files?
show comments
fragebogen
>DuckDB 2.0 introduces asynchronous I/O for reading Parquet and CSV files.
Sorry, does this mean that their own duckdb format already supports async or doesn't?
show comments
jiggawatts
It’s a Swiss Army knife for data!
I just used DuckDB 2.0 last week to compress 7 terabytes of Azure logs (NDJSON) to 2.5% of the original size in Parquet format in under two hours with a medium sized VM!
Those logs were then small enough to run ad-hoc queries in tens of seconds directly from the archive blob storage.
I created a “frozen DuckLake” and sent the signed blob URL to the security analyst.
No need for a server, no need for Splunk!
It’s easy to use, fast, can be plopped anywhere on a network, and it’s free!
If it just had KQL syntax support, it would be perfect…
We've found DuckDB 2.0 to be way faster on high core count machines on EC2. Not just reading data from S3, but the actual computations too. Much more likely to see 100% utilisation up to about 128vCPUs or 192vCPUs, and that CPU usage to translate into workloads completing faster.
Still don't usually see a speedup past about 192vCPU, but prior to 2.0, speed would top out at more like 64vCPU, so this is a big win.
That has enabled us to scale Splink (FOSS record linakge software) to process 1bn records with 10bn record comparisons in 10 mins, which as far as I'm aware makes it the fastest linkage engine of this kind. Most of this performance comes from simply delegating the big calculations to the DuckDB engine! https://moj-analytical-services.github.io/splink/topic_guide...
Thanks to the team for all their work!
It's very fast, especially on a decent sized cloud VM and when querying Iceberg. I've been running the alpha in production as a centralized analytics server for a small organization.
I built a DuckDB extension so you can connect via Postgres wire protocol, or Arrow flight. DuckDB also has its own Quack protocol, but client support for Quack is very limited right now, whereas many data tools support Postgres wire protocol or ADBC (which you can use with Arrow Flight).
https://github.com/sidequery/duckflight-extension
The fact that can now be used as a client-server architecture is a game changer. It will for sure replace many data stacks that now run on many servers with tons of cpu and ram
Very excited for Triggers. Now DuckDB really is close to being a data warehouse in a box.
Absolutely fantastic piece of software. Even beyond being used as engine for analysis, it's amazingly versatile for ingesting and reshaping data. Great job.
Is it possible to save DuckDB UI notebooks to files?
I wonder how terrible would it be to use DuckDB as the main database (running transactions) for a small project?
I really want to use DuckDB. But I have struggled to find a use case in my company.
What are you guys using it for?
What is the benefit of using async io for csv and parquet files?
>DuckDB 2.0 introduces asynchronous I/O for reading Parquet and CSV files.
Sorry, does this mean that their own duckdb format already supports async or doesn't?
It’s a Swiss Army knife for data!
I just used DuckDB 2.0 last week to compress 7 terabytes of Azure logs (NDJSON) to 2.5% of the original size in Parquet format in under two hours with a medium sized VM!
Those logs were then small enough to run ad-hoc queries in tens of seconds directly from the archive blob storage.
I created a “frozen DuckLake” and sent the signed blob URL to the security analyst.
No need for a server, no need for Splunk!
It’s easy to use, fast, can be plopped anywhere on a network, and it’s free!
If it just had KQL syntax support, it would be perfect…