I think this is confusing Planetscale's primary objective: to make it incredibly easy and efficient to scale a database up and out.
There's no mention of sharding whatsoever. Without that this has very little to do with Planetscale and is much closer to your average managed DB (RDS etc.). There's also no mention of a bouncer/gateway/reverse proxy, which is necessary for zero downtime.
I get that Planetscale hosts "vanilla" Postgres instances but naturally those are limited by single instance size limits. I imagine this is predominantly a marketing strategy for them, acting as a funnel for their sharding products.
But perhaps that's the goal with this project, to not be Planetscale at all, and to focus on the single node. If that's the case, then great, best of luck, but the roadmap is missing some important pieces for me to take this seriously. In either case I find drawing comparison with Planetscale to not be very helpful or illustrative of the project and its goals.
show comments
xolox
The conceptual relationship to PlanetScale seems somewhat dubious to me (as mentioned in other comments) but the idea of implementing snapshots and branching below the database server level, in the storage layer using Ceph, is neat and potentially quite elegant! Kudos for that idea :-). I hope implementing it works out to the same elegant API envisioned in the initial concept. I could see this being very useful in a development workflow like the example (dev-db/feature-login).
I have to say that my experiences with running virtualized relational database servers on top of Ceph centralized storage have shown somewhat disappointing performance, so I think that could become a real challenge if performance is or becomes a goal. However, I've encountered multiple mentions that Ceph performance is supposed to increase significantly as the number of storage nodes increases (from a handful to a dozen to much larger clusters) and while I cannot corroborate this from personal experience, it stands to reason that "throwing more (hardware) resources at the problem" can make a big difference.
show comments
maherbeg
A few of us built nearly the exact same thing for a Hackathon which was fun. This definitely can work. There are a couple of other approaches too that are interesting like
- xata - https://xata.io/blog/xatastor-zfs-nvme-of-for-millions-of-postgres-databases
- neon - which has a more sophisticated architecture that builds abstractions at the Postgres layer
But separating compute and storage sucks and the performance you get out of EBS and friends is mediocre. The elasticity is nice, but if you have High Availability and can move instances around, you can still expand your cluster relatively easily, just not easily in an emergency scenario.
show comments
Shalomboy
OP's interview with the F1 team sounds super cool, I'd actually love to hear more about their experience and the vibes they got from potentially working a dev job for a sports team. I had a close encounter with the analytics department of an MLB Team not too long ago and found that pocket of the tech world beyond fascinating. I just wish I had more exposure to the folks working in it.
nullbio
You did the easy part. Now do the managed database part, and at scale, whereby I don't have to worry about any chance of data loss. Otherwise this isn't "building PlanetScale" - it's building 1/100th of it.
It annoys me when people claim they've "easily and quickly" built something that took many developers many months or years worth of work and optimization to build a solid product.
It's like someone who generates a pretty looking HTML page with an LLM and claims they've built a customer-facing product. So much slop these days...
show comments
KoleSeise1277
Nice breakdown of the COW model. How do you plan to clean up snapshots once branches get a few generations deep?
show comments
lewi
Awesome. I've been playing exploring PITR stuff recently in my homelab. Will give it a go and try to contribute if I spot any issues
show comments
androiddrew
Lolz, just forward to my friend who works at Planetscale. Looking forward to his reaction
I think this is confusing Planetscale's primary objective: to make it incredibly easy and efficient to scale a database up and out.
There's no mention of sharding whatsoever. Without that this has very little to do with Planetscale and is much closer to your average managed DB (RDS etc.). There's also no mention of a bouncer/gateway/reverse proxy, which is necessary for zero downtime.
I get that Planetscale hosts "vanilla" Postgres instances but naturally those are limited by single instance size limits. I imagine this is predominantly a marketing strategy for them, acting as a funnel for their sharding products.
But perhaps that's the goal with this project, to not be Planetscale at all, and to focus on the single node. If that's the case, then great, best of luck, but the roadmap is missing some important pieces for me to take this seriously. In either case I find drawing comparison with Planetscale to not be very helpful or illustrative of the project and its goals.
The conceptual relationship to PlanetScale seems somewhat dubious to me (as mentioned in other comments) but the idea of implementing snapshots and branching below the database server level, in the storage layer using Ceph, is neat and potentially quite elegant! Kudos for that idea :-). I hope implementing it works out to the same elegant API envisioned in the initial concept. I could see this being very useful in a development workflow like the example (dev-db/feature-login).
I have to say that my experiences with running virtualized relational database servers on top of Ceph centralized storage have shown somewhat disappointing performance, so I think that could become a real challenge if performance is or becomes a goal. However, I've encountered multiple mentions that Ceph performance is supposed to increase significantly as the number of storage nodes increases (from a handful to a dozen to much larger clusters) and while I cannot corroborate this from personal experience, it stands to reason that "throwing more (hardware) resources at the problem" can make a big difference.
A few of us built nearly the exact same thing for a Hackathon which was fun. This definitely can work. There are a couple of other approaches too that are interesting like
But separating compute and storage sucks and the performance you get out of EBS and friends is mediocre. The elasticity is nice, but if you have High Availability and can move instances around, you can still expand your cluster relatively easily, just not easily in an emergency scenario.OP's interview with the F1 team sounds super cool, I'd actually love to hear more about their experience and the vibes they got from potentially working a dev job for a sports team. I had a close encounter with the analytics department of an MLB Team not too long ago and found that pocket of the tech world beyond fascinating. I just wish I had more exposure to the folks working in it.
You did the easy part. Now do the managed database part, and at scale, whereby I don't have to worry about any chance of data loss. Otherwise this isn't "building PlanetScale" - it's building 1/100th of it.
It annoys me when people claim they've "easily and quickly" built something that took many developers many months or years worth of work and optimization to build a solid product.
It's like someone who generates a pretty looking HTML page with an LLM and claims they've built a customer-facing product. So much slop these days...
Nice breakdown of the COW model. How do you plan to clean up snapshots once branches get a few generations deep?
Awesome. I've been playing exploring PITR stuff recently in my homelab. Will give it a go and try to contribute if I spot any issues
Lolz, just forward to my friend who works at Planetscale. Looking forward to his reaction
MoonScale