UNIX signals *do not* queue. If two or more signals with the same number are sent faster than the receiving thread handles them (due to the signal being blocked and/or the thread not being scheduled), all but the last will be lost irrevocably. There is no mechanism to prevent this.
"Yes, we built a message broker using nothing but UNIX signals and a bit of Ruby magic. Sure, it’s not production-ready, and you definitely shouldn’t use this in your next startup (please don’t), but that was never the point.
"The real takeaway here isn’t the broker itself: it’s understanding how the fundamentals work. We explored binary operations, UNIX signals, and IPC in a hands-on way that most people never bother with.
"We took something “useless” and made it work, just for fun. So next time someone asks you about message brokers, you can casually mention that you once built (or saw) one using just two signals. And if they look at you weird, well, that’s their problem. Now go build something equally useless and amazing. The world needs more hackers who experiment just for the fun of it."
show comments
foofoo12
This has nothing to do with Kafka and it's not not really a functioning message queue except for theoretically speaking.
The article is fine, but call it what it is: abusing the Unix signal system for shit and giggles. Nothing wrong with that.
m-hodges
This was a really fun article and many in the comments seem to have forgotten you're allowed to have fun with the computer.
bradleybuda
This is awesome. Does POSIX guarantee the order of signal delivery? And I'm dying to see what the bandwidth / throughput of this channel is...
Such titles should be flagged and banned to protect the innocent.
bcrl
This is the kind of article that deserves to be posted on April 1st, preferably with an accompanying RFC published at the IETF.
donatj
We run a very simple filesystem based queue that processes around 1 billion events a day. Makes use of XFS for it's better handling of large numbers of files.
Corporate tried to push us to replace it with SQS and it could not keep up / costs with through the roof
show comments
florians
I enjoyed reading the article and found it interesting. Had no prior interest in the topic but it was an entertaining read. Last time I used binary and calculated with it was in high school. Didn’t know about UNIX signals and now I understand how processes are terminating.
sheepscreek
I challenged ChatGPT the other day to design a bidirectional process interop in *nix and this was one of the suggestions. Until then I had only ever thought of pipes as unidirectional. I still thought it was bonkers. This looks like a neat prototype though.
lifthrasiir
Initially I assumed that the author was about to use something like shared memory for bulk transfer. Boy I was totally wrong...
briandw
You have problem with too much traffic that your server can't handle. You add a queue. Now you have 2 problems :)
ape4
I hope this is a joke / hack.
A named pipe (like Postfix sendmail uses) seems slightly more sane.
show comments
hajimuz
Great article for learning UNIX Signal
kryword
I liked a lot your article, hacking with Unix signals is impressive, and was faster than I expected. I would have expected signals to be slower.
hshdhdhehd
Can those two Unix signals run Doom?
president_zippy
Glad you had fun doing this!
If you choose to take this experiment further and go deeper, you will discover something even more fun: reentrancy.
... That, and how software interrupts work at the kernel level. Happy hunting!
EDIT: In anticipation of an eventual response, I just realized how condescending this sounds at first glance. I meant it in good faith.
gleenn
Fun article but the title is definitely overstating the huge amount of functionality lost if you replace Kafka. Immediately on my mind would be durability and broadcasting.
up2isomorphism
The author doesn’t know why people use things like Kafka in the first place
nijave
Obligatory: Kafka is a log, not a message queue (although it turns out there's a lot of overlap)
bawolff
> And if you came here just because of the clickbait title, I apologize and invite you to keep reading. It’ll be fun, I promise.
I kind of hate this trend of making clickbait and then apologizing for it. I think its more annoying then just making clickbait.
Is it really that hard to just accurately title your blog posts?
UNIX signals *do not* queue. If two or more signals with the same number are sent faster than the receiving thread handles them (due to the signal being blocked and/or the thread not being scheduled), all but the last will be lost irrevocably. There is no mechanism to prevent this.
https://ldpreload.com/blog/signalfd-is-useless
Before y'all go nuts with the criticisms...
"Yes, we built a message broker using nothing but UNIX signals and a bit of Ruby magic. Sure, it’s not production-ready, and you definitely shouldn’t use this in your next startup (please don’t), but that was never the point.
"The real takeaway here isn’t the broker itself: it’s understanding how the fundamentals work. We explored binary operations, UNIX signals, and IPC in a hands-on way that most people never bother with.
"We took something “useless” and made it work, just for fun. So next time someone asks you about message brokers, you can casually mention that you once built (or saw) one using just two signals. And if they look at you weird, well, that’s their problem. Now go build something equally useless and amazing. The world needs more hackers who experiment just for the fun of it."
This has nothing to do with Kafka and it's not not really a functioning message queue except for theoretically speaking.
The article is fine, but call it what it is: abusing the Unix signal system for shit and giggles. Nothing wrong with that.
This was a really fun article and many in the comments seem to have forgotten you're allowed to have fun with the computer.
This is awesome. Does POSIX guarantee the order of signal delivery? And I'm dying to see what the bandwidth / throughput of this channel is...
Why use signals when POSIX defines a complete message queue interface: https://www.opensourceforu.com/2023/06/a-deep-dive-into-posi...
Wow, I didn't know! I will work on replacing my Kafka cluster handling 10 million msg/sec with this right away!
you can literally just attach data[1] to signals... you don't need to do this.
[1]: https://man7.org/linux/man-pages/man3/sigqueue.3.html
Such titles should be flagged and banned to protect the innocent.
This is the kind of article that deserves to be posted on April 1st, preferably with an accompanying RFC published at the IETF.
We run a very simple filesystem based queue that processes around 1 billion events a day. Makes use of XFS for it's better handling of large numbers of files.
Corporate tried to push us to replace it with SQS and it could not keep up / costs with through the roof
I enjoyed reading the article and found it interesting. Had no prior interest in the topic but it was an entertaining read. Last time I used binary and calculated with it was in high school. Didn’t know about UNIX signals and now I understand how processes are terminating.
I challenged ChatGPT the other day to design a bidirectional process interop in *nix and this was one of the suggestions. Until then I had only ever thought of pipes as unidirectional. I still thought it was bonkers. This looks like a neat prototype though.
Initially I assumed that the author was about to use something like shared memory for bulk transfer. Boy I was totally wrong...
You have problem with too much traffic that your server can't handle. You add a queue. Now you have 2 problems :)
I hope this is a joke / hack.
A named pipe (like Postfix sendmail uses) seems slightly more sane.
Great article for learning UNIX Signal
I liked a lot your article, hacking with Unix signals is impressive, and was faster than I expected. I would have expected signals to be slower.
Can those two Unix signals run Doom?
Glad you had fun doing this!
If you choose to take this experiment further and go deeper, you will discover something even more fun: reentrancy.
... That, and how software interrupts work at the kernel level. Happy hunting!
EDIT: In anticipation of an eventual response, I just realized how condescending this sounds at first glance. I meant it in good faith.
Fun article but the title is definitely overstating the huge amount of functionality lost if you replace Kafka. Immediately on my mind would be durability and broadcasting.
The author doesn’t know why people use things like Kafka in the first place
Obligatory: Kafka is a log, not a message queue (although it turns out there's a lot of overlap)
> And if you came here just because of the clickbait title, I apologize and invite you to keep reading. It’ll be fun, I promise.
I kind of hate this trend of making clickbait and then apologizing for it. I think its more annoying then just making clickbait.
Is it really that hard to just accurately title your blog posts?
Click bait