I've written a lot of low level software, BSPs, and most of an OS, and the main reason to not write your own OS these days is silicon vendors. Back in the day, they would provide you a spec detailed enough that you could feasibly write your own drivers.

These days, you get a medium-level description and a Linux driver of questionable quality. Part of this is just laziness, but mostly this is a function of complexity. Modern hardware is just so complicated it would take a long time to completely document, and even longer to write a driver for.

John describes exactly what I'd like someone to build:

"To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers."

As a thought experiment:

* Pick a place where cost-of-living is $200/month

* Set up a village which is very livable. Fresh air. Healthy food. Good schools. More-or-less for the cost that someone rich can sponsor without too much sweat.

* Drop a load of computers with little to no software, and little to no internet

* Try reinventing the computing universe from scratch.

Patience is the key. It'd take decades.

Was at Oculus post acquisition and can say that the whole XROS was an annoyance and distraction the core technology teams didn’t need. There were so many issues with multiple tech stacks that needed fixing first.

Mind you, this XROS idea came after Oculus reorged into FB proper. It felt to me like there were FB teams (or individuals) that wanted get on the ARVR train. Carmack was absolutely right, and after the reorg his influence slowly waned for the worse.

The problem that is kind of glossed over here is that Meta hired a bunch of folks from Microsoft who were primarily interested in writing operating systems, and set them to work on XR - obviously they wanted to write a custom operating system

> They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad

I've only seen John Carmack's public interactions, but they've all been professional and kind.

It's depressing to imagine HR getting involved because someone's feelings had been hurt by an objective discussion from a person like John Carmack.

I'm having flashbacks to the times in my career when coworkers tried to weaponize HR to push their agenda. Every effort was eventually dismissed by HR, but there is a chilling effect on everyone when you realize that someone at the company is trying to put your job at stake because they didn't like something you said. The next time around, the people targeted are much more hesitant to speak up.

I was at Google when the Flutter team started building Fuchsia.

They had amazing talent. Seriously, some of the most brilliant engineers I've worked with.

They had a huge team. Hundreds of people.

It was so ambitious.

But it seemed like such a terrible idea from the start. Nobody was ever able to articulate who would ever use it.

Technically, it was brilliant. But there was no business plan.

If they wanted to build a new kernel that could replace Linux on Android and/or Chrome OS, that would have been worth exploring - it would have had at least a chance at success.

But no, they wanted to build a new OS from scratch, including not just the kernel but the UI libraries and window manager too, all from scratch.

That's why the only platform they were able to target was Google's Home Hub - one of the few Google products that had a UI but wasn't a complete platform (no third-party apps, for example). And even there, I don't think they had a compelling story for why their OS was worth the added complexity.

It boggles my mind that Fuchsia is still going on. They should have killed it years ago. It's so depressing that they did across-the-board layoffs, including taking away resources from critically underfunded teams, while leaving projects like Fuchsia around wasting time and effort on a worthless endeavor. Instead they just kept reducing Fuchsia while still keeping it going. For what?

Mechanisms for getting the linux kernel out of the way is pretty decent these days, and CPUs with a lot of cores are common. That means you can isolate a bunch of cores and pin threads the way you want, and then use some kernel-bypass to access hardware directly. Communicate between cores using ring buffers.

This gives you best of both worlds - carefully designed system for the hardware with near optimal performance, and still with the ability to take advantage of the full linux kernel for management, monitoring, debugging, etc.

Back in mmm like 2002 or 2003 or 2004 while at Microsoft I read an internal paper from a few OS guys who hackathoned something for Bill Gates's Think Week (which is when he used to go to some island in San Juans or somewhere similar and just read curated papers and think, it was a huge prestige to get such a paper to him) and that something was an OS written from scratch with GC and memory management on top of something very .NET framework'y (which was released a couple of years ago. They had it booting on all kinds of hardware and doing various neato things. One of explicitly called design principles was 0 compatibility with anything Windows before. Which is why it didn't go anywhere I assume. I remember it was just a handful of engineers (presumably OS folks) hacking for like a month. . It was awesome to read about.

> my old internal posts... got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad

That jives with my sense that META is a mediocre company

What would be the real advantage of a custom OS over a Linux distribution?

The OS does process scheduling, program management, etc. Ok, you don’t want a VR headset to run certain things slowly or crash. But some Linux distributions are battle-tested and stable, and fast, so can’t you write ordinary programs that are fast and reliable (e.g. the camera movement and passthrough use RTLinux and have a failsafe that has been formally verified or extensively tested) and that’s enough?

Huawei seem pretty committed to building their own OS and uncoupling from the Western technology stack in total

https://en.wikipedia.org/wiki/HarmonyOS_NEXT https://www.usenix.org/conference/osdi24/presentation/chen-h...

My objection is that there is no universe in which Meta can be trusted with direct access to your raw gaze tracking data.

The only thing I can imagine that would be more invasive would require a brain implant.

Late 2019 I had a short conversation with Abrash about a new OS for the next set of glasses and my immediate reaction was “why?” He was adamant that there was a security need which Linux could not fill (his big concern was too much surface area for exploits in the context of untrusted 3rd party code). I remember thinking that this would be a surprise to cloud engineers at the big hosters, but chose not to continue the argument. He didn’t get where he is by being dramatically wrong very often, after all, but it still struck me as a waste. Note I did not work at Meta so he may have had stronger justifications he chose not to expose.

I think people have forgotten about Google Fuchsia which I guess is a good sign for a new OS. They’ve done quite well in deploying it seamlessly to their consumer devices.

I worked on a completely different hardware project within meta and while they didn't want a custom OS, they used an off the shelf rtos with the intention of modifying it and it was a shit show. They had a million justifications for why they needed it, but they had no performance tests or metrics to verify to actually justify it. They incurred a huge development overhead for no verifiable performance improvements.

None of the code they wrote couldn't have just been written as a kernel module in Linux. It would've also been so much easier due to all the documentation and general knowledge people have about Linux both within the company and outside the company.

You could write a book on why it's practically impossible to create a new OS these days. Love Carmack for stating it so clearly. I also love that called out TempleOS, I also have a weird respect for it. Plan 9 is the probably the best example of a totally new OS and I hope someday it becomes viable because it's really a joy to use.

But ultimately it just makes sense to adapt existing kernels / OS (say, arch) and adapt it to your needs. It can be hair wrenchingly frustrating, and requires the company to be willing to upstream changes and it still takes years, but the alternative is decades, because what sounds good and well designed on paper just melts when it hits the real world, and linux has already gone through those decades of pain.

What if instead of writing the entire OS, a company were to pick up an existing “hobby” OS and refine it?

For example any of the systems listed in Carmack’s post. Or perhaps Serenity OS, RedoxOS, etc.

Whom the gods would destroy, they first persuade to design an OS :)

Does SteamOS count as something Carmack would discourage as well? Yes it's a Linux-based system and yes even based on an existing distro, but it is a purpose-specific OS and it seems like it's working well for Valve and people using it to play Windows games without Windows...

To be fair, monastic order of engineers is absolutely what the world could use about now.

The engineers were right. “If not us then who”.

Meta has the talent and the balance sheet to pull this off. Worst case scenario we end up with one more open sourced operating system. Who knows what happens 20 years down the line.

Is the difficulty in theoretical complexity of operating systems, or in project scoping/scope creep?

It's probably not that hard to write bare metal code for a modern CPU that runs and crashes. It's obviously insurmountably hard to compete with Android in features with scratch built bare metal code. An "OS" can be anything between the two. But it's very easy to imagine an "XR OS" project snowballing quickly into the latter, and Carmack's concerns would be spot on(as always is, and as proven). Is it then an inherent difficulty in "designing a new operating system", or is it technically something else?

> To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers.

I mean, I'd give a fair shake to an OS from the SQLite team [1].

1. https://sqlite.org/codeofethics.html

Why bother making a new OS when you can make a new user interface for an existing OS?

The drivers are the hard part. It takes a lot of inter-industry collaboration to get driver compatibility

But what should you be running on an XR headset? The OS has to be real time. Linux can sort of do that. Probably a stripped down Linux. About 90% of Linux is irrelevant or undesirable in this application.

Where can I find Jonathan Blows “Why can’t we even conceive of writing a new OS today” post? No luck when searching for it

This is completely right from a product point of view, which is Carmack's argument.

But I have wondered why one of these companies with billions of dollars to burn hasn't tried to create something new as a strategic initiative. Yes, there wouldn't be any ROI for years, and yes, the first several products on the platform would probably be better off on something more traditional.

But the long term value could potentially be astronomical.

Just another case of quarterly-report-driven decision making, I suppose. Sigh.

When you're at a certain scale it makes sense.

That scale is when creating an OS gives you a clear advantage over licensing or working with an open source OS.

Every other scale below that it's for knowledge, growth, research, or fun.

Sigh. It's really depressing how a technical discussion on merits of a solution keeps getting reported to HR. I've seen it many times.

Someone said your preferred design won't work, and you go to HR.

I gladly throw my idea under the bus when I hear why it's bad.

Now offering any critique of a thing in order to help the company comes with a career risk.

T

> I wish I could drop (so many of) my old internal posts publicly, since I don’t really have the incentive to relitigate the arguments today – they were carefully considered and prescient. They also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad, but I expect many of them would acknowledge in hindsight that the Meta products would not be in a better place today if the new OS effort had been rammed into them.

So someone at Meta was so sensitive that being told their behemoth of a project was ill advised ended up getting reported to HR?

I'm surprised at the comments here. Linux's days as the sole hegemon are numbered.

I wonder whether an unarticulated desire of Meta's was to avoid any license issues that could incur by using Linux or any other existing OS.

I'd love a truly new OS, but I just don’t know what it would look like at this point? "New OS" ideas tend to converge on the same trunk.

Building a hobby OS taught me how little is just "software". The CPU sets the rules. Page tables exist because the MMU says so. Syscalls are privilege flips. Task switches are register loads and TLB churn. Drivers are interrupt choreography. The OS to me is just policy wrapped around fixed machinery.

> To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers. Which was sort of Plan 9…

Roll call!

Brace! Here comes “I said don’t build it, but they did anyway” comments. They should have did what you said, you were right, don’t worry

I was there when they wanted to do the custom XROS. I remember asking them in a Q&A session exactly why they would build this and I recall the reasoning behind it totally fell flat. Fundamentally it became clear these guys just wanted to write a new OS because they thought it would be cool or fun.

Much of the scenarios they tried to address could have been done with Mach or some realtime kernel or with fuchsia. I recall later on they did consider using fuchsia as the user space for the os for some time.

On another note, there was similarly an equally “just for fun” language effort in the org as well (eg “FNL”). That was also conceived by a bunch of randos who weren’t a bunch of compiler guys that had no product vision and just did it for fun.

Well when the era of efficiency arrived all of this stuff ended.

The fact that Facebook, a company far richer than Bell labs ever was, like all FAANG, has a culture of expensive employees baby sitting broken software products (rather than lab researcher vs field technician separation), and cannot be bothered to do the long term investment in a new OS, is why I think the industry actually doesn't deserve these R&D tax breaks HN was bemoaning had gone away until this year.

The point of R&D is the time horizon is long, and the uncertainty is high. Making JS slop that then has to be constantly babysat is opex, not capex.

I lost a lot of respect for carmack when he joined meta.

The company is a black hole of wasted talent.

Jonathan Blow is the world’s most successful hobbyist programmer. His whole thing is doing projects from scratch. Every game he made could be done in Unity with far less effort.

Most opinions of this man exists in a vacuum space isolated from the real world software industry. Building an OS from scratch is one of those examples.

It’s never seems like there’s a significant reason behind them other than………”I made dat :P”

I'd love to hear John's more detailed take on TempleOS

How about ReactOS?

I love this part: "To make something really different, and not get drawn into the gravity well of existing solutions, you practically need an isolated monastic order of computer engineers"

And yet, Sony did it, Nintendo did it, both have been pretty succeesful.

We also need to be clear what an OS is. Is it "darwin" or "macOS" - they have different scopes.

Things I'd want from an OS for an XR device.

1. Fast boot. I don't want to have to wait 2-3-4-5 minutes to reboot for those times I need to reboot.

I feel like Nintendo figured this out? It updates the OS in the background somehow and reboot is nearly instant.

2. Zero jank. I'm on XR, if the OS janks in any way people will get sick AND perceive the product as sucking. At least I do. iOS is smooth, Androind is jank AF.

Do any of the existing OSes provide this? Sure, maybe take an existing OS an modify it, assuming you can.

Did they have people who have built an OS before?

I’ve seen this firsthand. These giant tech companies try to just jump into a massive new project thinking that because they have built such an impressive website and have so much experience at scale they should just be able to handle building a new OS.

In my case it wasn’t even a new OS it was just building around an existing platform and even that was massively problematic.

The companies that build them from scratch have had it as one of their core competencies pretty much from the start.

I’m unsurprised meta had issues like this.

The problem when working for Meta is that if you do a good job, you've helped make the world worse... so the real heroes are the people wasting money and reducing efficiency

If you're at all competent, go work somewhere else

Lost me and TempleOS

The problem with this guy is that it’s hard to criticize him, whether at work or in this forum. For example, I am going to be downvoted for mocking the fact that this guy thinks it’s some genius move to say “No” to making an operating system, whatever making an operating system means.

https://xcancel.com/ID_AA_Carmack/status/1961172409920491849

I like that the top reply to Carmack's wall of text is a screenshot of TempleOS with a doodle of an elephant lmao. And ironically, that meme reply is on topic and it says a thousand words with just one photo.

Another point I would add in support of that meme comment, is Google's recent rug-pull of Android not allowing sideloading apps from unsigned developers anymore starting this autumn, after over a decade of conquering the market with their "go with us, we're the open alternative to iOS" marketing.

The conclusion is to just never EVER trust big-tech/VC/PE companies, even when they do nice things, since they're 100% just playing the long game, getting buddy-buddy with you waiting till they smothered the competition with their warchest, and then the inevitable rug-pull comes once you're tied to their ecosystem and you have nowhere else to go.

Avoid these scumbags, go FOSS form the start, go TempleOS. /s but not really

He's acting like their VR UX is top notch when it's as bad as it gets. Just yesterday I dusted off my Meta Quest 2 to play a bit, and spent around an hour trying to pair up my left controller to the helmet after replacing the battery.

You can't do it without going through their fucking app, that asks for every permissions under the sun, including GPS positioning for some reason. After finally getting this app working and pairing it with my headset, I could finally realize the controller was just dead and their was nothing to do.

> they also got me reported to HR by the manager of the XROS effort for supposedly making his team members feel bad

This is madness. The safe space culture has really gone too far.

tbh linux has quite a bit of cruft in it these days at the syscall and interface layer.

if youre apple, it does make sense to do stuff from scratch. i think in a way, software guys wind up building their own prisons. an api is created to solve problem X given world Y, but world Y+1 has a different set of problems - problems that may no longer be adequately addressed given the api invented for X.

people talk about "rewrite everything in rust" - I say, why stop there? lets go down to the metal. make every byte, every instruction, every syscall a commodity. imagine if we could go all the way back to bare metal programming, simply by virtue of the LLM auto-coding the bootloader, scheduler, process manager, all in-situ.

the software world is full of circularities like that. we went from Mainframe -> local -> mainframe, why not baremetal -> hosted -> baremetal?