You can "mount" your filesystem (or anything else) as a SQL database, wtf. That's amazing.
This sounds like it could be extremely useful.
show comments
XJ6w9dTdM
Yes !
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
show comments
djoldman
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
1. ELF file to SELF file
2. modify SELF file
3. SELF file to ELF file
show comments
inigyou
> I realized something that bugged me. ELF is already a database.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
show comments
garganzol
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
show comments
catlifeonmars
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
(And that’s totally ok)
show comments
unified101
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
Very cool hacking! My hats off to the author.
show comments
djhworld
This was a fantastic read thanks for putting it together. I'm always fascinated by binfmt_misc, I remember fooling around with steganography a few years ago and putting executables inside images that could then be executed using a wrapper tool + binfmt_misc.
I think this is awesome though, feels like a lot of things on OS's could just be represented by databases - where does it end?
SmasherEpilepti
> ELF MECHANISM: .strtab / .dynstr
> The database primitive it reinvents: string interning
> Surprisingly a lot more falls out as well: .dynstr is gone, because name is TEXT and SQLite already interns strings
What's with this claim? SQLite does not intern strings, as far as I can find anywhere, and a quick test shows that duplicate strings are actually duplicated in a database file. You can intern strings in SQLite manually with an intern string table, but it doesn't happen automatically.
skavi
This was super fun. Dude’s been on an absolute tear recently.
athrowaway3z
These are the kind of posts i love.
One nit-pick;
> The preload table is a list of objects to map last, so their exports win.
made it sound like the 'map last' is a consequence of this new perspective, but it’s really still a loader convention: by definition, the symbol-resolution query needs to give entries in the preload table precedence over ordinary dependencies.
JSR_FDED
I appreciate the inventiveness of this idea. But I don’t find myself thinking I must have this.
show comments
luciana1u
an executable that is also a database means my bugs get relational now. instead of a segfault i get a foreign key constraint violation.
ale42
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
show comments
didip
A lot of kernel filesystem features should be inside a database as well.
show comments
vlovich123
The lack of mmap is a problem, especially given how dynamic link heavy Linux distros are. It’s going to cause a lot of extra RAM usage and the kernel won’t know to dump those pages efficiently and will swap them to disk. The extra RAM usage will probably be recovered - the kernel typically these days has deduplication capabilities - but the lack of efficient swap is a problem.
eqvinox
NB: ELF executables and libraries don't need to have sections, or a section header table. The program headers are what matters. You can use sstrip to remove the SHT.
I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
show comments
kjuulh
Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
show comments
howerj
I've wanted to do something like this for quite a while! Of course I wanted to make my own OS, language and DB from scratch in order to do it, so naturally I've never gotten around to it. It goes to show that there is nothing new under the sun.
cobbzilla
This is really creative and clever, nicely done!
The “Inception” graphic was hilarious and made me wonder if perhaps the site itself is served by some SELF binary (is it?? that would be insane)
colinhb
The handling of dynamic libraries and closing over them into single dbs made this a fun read.
simonw
The word "Your" was dropped from the start of the title for some reason.
show comments
mongol
So where does this lead to? Is it leading us towards SQLite as some kind of alternative container image format? Because the benefit seems to only occur when it incorporates dependencies
enygmata
How many layers of "SQLite for everything" are we at now?
miketery
As far as agent workflows are concerned whats needed is the other direction. Store the files wherever but let an agent interact with it as if its on local FS via bash or other languages using their file io.
Whats worked for others?
alex7o
What if, we put the sqlite driver in kernel space? So this is your only filesystem like a unikernel
show comments
__m
not sure if the people exploring binaries prefer sql over command line tools
show comments
self_awareness
> Two processes running the same SELF binary do not share text pages the way a normally-mmap‘d ELF does, because the bytes are copied out of the b-tree rather than mapped.
Isn't this the whole point of executable file formats? To avoid loading everything all at once, share immutable segments with other processes, be extremely fast when loading, and integrate with OS page fault handlers to load data on demand?
I don't understand this project at all.
Even if it's possible to query the executable using SQL, we can always just use the SQL frontend to interface with executable data. I mean, ELF as a backend format for the "executable database". Like osquery interfaces with the system using SQL language. Osquery doesn't convert all OS data structures to sqlite :)
Btw, I think SELF is already "coined" by Signed ELF executables - SELF files, existing for example on Playstation 3.
HackerThemAll
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
(author) I'm enjoying the comments. When I published a short-paper with this idea in academic circles, the feedback wasn't so kind
This whole article is fantastic, but already at the start, the SQLite virtual tables thing is blowing my mind.
https://www.sqlite.org/vtablist.html
You can "mount" your filesystem (or anything else) as a SQL database, wtf. That's amazing.
This sounds like it could be extremely useful.
Yes !
This has been on the back of my mind for a while. And, as other commenters have noted, it would be great for the file to contain the (self-modifiable) Lisp image, a builtin virtual file system, and whatever the application want to use as (runtime modifiable) extra tables.
I find SQLite dynamic linking being basically compatible with ELF dynamic linking to be very impressive, I can imagine that if well done, it cloud replace most uses of AppImages with a much more efficient format, like the author suggests.
How about an option for compressing section contents within the SQLite blobs, since the author mentioned you can't mmap directly the text pages and have to copy anyway ?
There are two extensions that I was thinking would make a SQLite executable truly unique.
First, a linking extension that would allow patching in functions and hooks more directly to allow for a very powerful plug-in system. Imagine the plug-in SQLite defining a BEFORE/AFTER/REPLACE hook for some symbol the host SQLite defines as extensible.
Second, re-linking at runtime. This would require application author cooperation because you won't be able to do that from anywhere, but imagine changing a dependency or loading a plugin at runtime through editing the db, and the interpreter just maps that on demand/automatically in the background, now next time your web server accept(), it calls the new version of the handling function.
> The format itself is incredibly terse, designed for a world where disk space and network bandwidth was at an extreme premium. Modifying the format is hard, you often have to zero out sections and add new ones since it is packed so tightly. There is also no self-describing schema. ELF itself is a very generic format that supports sections of data that by convention are interpreted in specific ways but the format does not enforce it.
Sounds like a great use case for:
> I realized something that bugged me. ELF is already a database.
Even more broadly applied: Every base of data is already a database (that's what the compound word means). Programs like sqlite and postgres used to be called by the more precise term "Relational Data Base Management Software" or "RDBMS" until their use became so widespread that they were colloquially called databases instead.
The very first data base structures were more like, each geometric sector of a hard drive is a record, and each head is a column. This was a straightforward translation of punch-card workflow onto a magnetic disk.
The copied vs. mapped memory situation is the only deal breaker in this experiment. Otherwise, file format unification would be a big step forward. The PE/COFF executable file format used by Windows (and some older Unix systems) is a relational database as well. The same goes to .NET assembly format - it's a relational database too. The wheel gets reinvented over and over again.
I can buy that an object file can be viewed as a relational db. Why SQLite though? Why not SQL query engine over the object file using a virtual table abstraction? I’m not seeing how most the SQLite features, with the exception of a subset of the query engine would translate over.
If the author wants to make a case for including schema metadata in an object file, again why SQLite? This strikes me a lot as someone who is trying to find uses for their favorite hammer (a very useful hammer I might say) rather than a serious exploration of what a new, improved object file format would look like.
(And that’s totally ok)
I don't think this goes far enough! Make the actually app store be the same file itself. so it's a living application and the file is constantly updated to how you use it. Copy it around, and you carry your data wiht you as well.
Let's go deeper. it's a webserver app + the server code + application code + db, so pocketbase++ where it's also the deployment target.
Then combine it with APE liek system, and the same file loads and stores things on every platform. evil laugh
Very cool hacking! My hats off to the author.
This was a fantastic read thanks for putting it together. I'm always fascinated by binfmt_misc, I remember fooling around with steganography a few years ago and putting executables inside images that could then be executed using a wrapper tool + binfmt_misc.
I think this is awesome though, feels like a lot of things on OS's could just be represented by databases - where does it end?
> ELF MECHANISM: .strtab / .dynstr > The database primitive it reinvents: string interning
> Surprisingly a lot more falls out as well: .dynstr is gone, because name is TEXT and SQLite already interns strings
What's with this claim? SQLite does not intern strings, as far as I can find anywhere, and a quick test shows that duplicate strings are actually duplicated in a database file. You can intern strings in SQLite manually with an intern string table, but it doesn't happen automatically.
This was super fun. Dude’s been on an absolute tear recently.
These are the kind of posts i love.
One nit-pick;
> The preload table is a list of objects to map last, so their exports win.
made it sound like the 'map last' is a consequence of this new perspective, but it’s really still a loader convention: by definition, the symbol-resolution query needs to give entries in the preload table precedence over ordinary dependencies.
I appreciate the inventiveness of this idea. But I don’t find myself thinking I must have this.
an executable that is also a database means my bugs get relational now. instead of a segfault i get a foreign key constraint violation.
I was thinking that someone managed to put an ELF header in an SQLite file while keeping it compatible with SQLite... but no, "just" a new binary format. Very interesting nevertheless!
A lot of kernel filesystem features should be inside a database as well.
The lack of mmap is a problem, especially given how dynamic link heavy Linux distros are. It’s going to cause a lot of extra RAM usage and the kernel won’t know to dump those pages efficiently and will swap them to disk. The extra RAM usage will probably be recovered - the kernel typically these days has deduplication capabilities - but the lack of efficient swap is a problem.
NB: ELF executables and libraries don't need to have sections, or a section header table. The program headers are what matters. You can use sstrip to remove the SHT.
Reminds me of the DBOS, everything's a database, ethos. https://www.dbos.dev/
Congratulations, you’ve rediscovered mainframes.
I wonder how flexible is the SQLite binary format to allow to design a tool that would take a SELF binary/db and rewrite it to make BLOB values more mmap-able (OS page aligned) in order to allow shortcuts in the self-exec loader.
Great read. I always find these turn a feature of a computer into a database an interesting read/idea. It certainly has upside, and downsides. For example having an entire filesystem as a sqlite database etc, or in this case an executable. It would be nice to have a structured way of talking to our utilities, rather than interpreting various commands as json through a few layers of tools to then get the output in a common format. That said I'd probably rather have a split up tool-set, than everything having to be in a single database for it to interoperate.
I've wanted to do something like this for quite a while! Of course I wanted to make my own OS, language and DB from scratch in order to do it, so naturally I've never gotten around to it. It goes to show that there is nothing new under the sun.
This is really creative and clever, nicely done!
The “Inception” graphic was hilarious and made me wonder if perhaps the site itself is served by some SELF binary (is it?? that would be insane)
The handling of dynamic libraries and closing over them into single dbs made this a fun read.
The word "Your" was dropped from the start of the title for some reason.
So where does this lead to? Is it leading us towards SQLite as some kind of alternative container image format? Because the benefit seems to only occur when it incorporates dependencies
How many layers of "SQLite for everything" are we at now?
As far as agent workflows are concerned whats needed is the other direction. Store the files wherever but let an agent interact with it as if its on local FS via bash or other languages using their file io.
Whats worked for others?
What if, we put the sqlite driver in kernel space? So this is your only filesystem like a unikernel
not sure if the people exploring binaries prefer sql over command line tools
> Two processes running the same SELF binary do not share text pages the way a normally-mmap‘d ELF does, because the bytes are copied out of the b-tree rather than mapped.
Isn't this the whole point of executable file formats? To avoid loading everything all at once, share immutable segments with other processes, be extremely fast when loading, and integrate with OS page fault handlers to load data on demand?
I don't understand this project at all.
Even if it's possible to query the executable using SQL, we can always just use the SQL frontend to interface with executable data. I mean, ELF as a backend format for the "executable database". Like osquery interfaces with the system using SQL language. Osquery doesn't convert all OS data structures to sqlite :)
Btw, I think SELF is already "coined" by Signed ELF executables - SELF files, existing for example on Playstation 3.
Yeah, I have been amazed for an entire life how many tools the IT industry invented during last 50 years to just disguise a database.
OK, in '70s, '80s or '90s when compute and storage resources were limited and every bit counted, specialized formats did make sense.
But nowadays we'd save enormous efforts by just packaging stuff in SQLite databases. Microsoft's proprietary file formats (Office, Power BI etc.), OpenOffice/LibreOffice OpenDocument format, or almost everything else would fit perfectly.
Documents, files, are all data (called "data files", aren't they?), including executables, which as can be seen in this article are also databases.
https://sqlite.org/aff_short.html
https://sqlite.org/appfileformat.html
ha ha ha
ha ha ha
yes