Incident with multple GitHub services

257 points123 comments20 hours ago
embedding-shape

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally.

This is probably the first time I felt vindicated with my self-hosting move literally the day after I finished the migration, very pleasant feeling. Usually it takes a month or two before I get here.

show comments
LorenDB

https://mrshu.github.io/github-statuses/ says they are down to 88.15% uptime. Even when you consider uptime of individual components, their best is 99.78%, so two nines.

show comments
alansaber

How much of this github downtime is due to git actions?

hedayet

Is Github losing any significant business from all these outages?

Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days.

Happy to be corrected about my perception too.

show comments
dankobgd

Don't worry, status page says that it's 100% working - green color, all good. even though i can't access a static page

eshack94

At this point, I feel like there should be a HN post whenever there ISN'T an issue with some GitHub service. Otherwise, it's business as usual...

cjonas

It would be wild if they dropped below the "two 9's" metric. I think they would need an additional ~16hr of outage in the 90 day rolling period.

show comments
frereubu

We have pretty basic needs - git repos + actions - and a bit of downtime here and there doesn't really affect us too much because we're not constantly committing and deploying, but even we're looking around for alternatives now.

Also, looks like people might be pummelling the SourceHut servers looking for an alternative: https://sr.ht/ is down. (Edit: was down when I wrote that, back up now).

show comments
agartner

There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf

> We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC.

We had ~8 commits get entirely reverted on our default branch during this time. I've never seen a github incident quite this bad.

show comments
x0ruman

We lost about a day of git history across several repos on Bitbucket a while back-not an outage, a data issue on their side. Local clones saved most of it, but issues and PRs from that window were just gone. That’s roughly why I started building gitbacker as a side project. Turns out the ‘back up the repo’ part is easy; the metadata is where it gets interesting.

Groxx

So... three incidents today, all of them ~1h or longer, and everything's green for the day with "no recorded downtime".

These don't really look any different than past incidents which have red bars on their respective days, except maybe that those tended to be several hours.

What do the green bars even mean? Are they changed to non-green retroactively if people complain enough or something? So far as I can tell, literally none of the previous green days have any incident shown in the mouse-over, but there are multiple for today only, so I kinda have to assume the mouse-overs are conveniently "forgotten" or all incidents become non-green and they just don't bother informing anyone on the same day. Either seems intentionally misleading.

delusional

Once that 10x developer velocity from AI kicks in, I'm sure github stability improves. Did you know AI finally makes it economical to fix all the little bugs?

CamouflagedKiwi

I wondered. We'd seen for most of today that Actions were slow to trigger, I had at least one that was just missed, it felt like something was definitely off but the status was green all day until this.

BhavdeepSethi

I'd really like to read the Post Mortem doc for this. I'm still shocked how they allowed this to happen in production.

jonnonz

Well I suppose they are finding out if you lay off too many people the IP of how the system works goes out the door with them.

throwatdem12311

Seems like they just can’t deal with the absolute deluge of AI vomit being uploaded every day.

Good riddance I hope it completely destroys them.

show comments
argee

I moved to Gitlab a while ago. It's a whole new level of freedom not having to pay for self-hosted CI runners.

deferredgrant

These outages are also a good test of how much local resilience teams actually built. My guess is most shops are much more dependent on GitHub than they like to admit.

bakies

I definitely have better uptime hosting my own gitea instance. It's faster too. It's basically a knock off GitHub. Plus with privacy concerns, I'm just happier overall. Easy setup, all I did was deploy the helm chart.

jasoncartwright

Just cancelled my GitHub Copilot Pro+ year subscription. Removal of Opus 4.6 stung, but the repeated continued downtime makes it unusable for me. Very disappointed.

No fuss instant refund of my unused subscription (£160) appreciated.

show comments
redwood

Guessing the sheer volume of pull requests related to AI code jitter is leading to instability of this Microsoft product.

fishgoesblub

At this point it'll be better to have alerts for when GitHub is online, rather than offline.

AnkerSkallebank

Some of my jobs are completing, some are failing. Seems to be random. Kind of wish they would just fail outright, instead of running for 10 minutes and then failing.

surya2006

even vercel also have more downtime nowadays

supakeen

I mean; this is the normal mode of operation for GitHub at this point.

show comments
surya2006

what are the good alternatives available for github i find some alternative but as long as widely people use github i cant use other service right like i cant share my alternative to other developer and force him to use this for me. so i feel like i locked in even i want to move i can't

show comments
tomjen3

At this point it should almost be news when it works.

ChrisArchitect

Multple 9s

buildbot

Anyone also seeing Active Directory/Entra issues?

chrisweekly

mods: title tupo "multple"

josefritzishere

Microslop is destroying Github

nhhvhy

If the day ends in Y…

linhns

Business as usual.

0xbadcafebee

I am this > < close to just running Gogs or Forgejo on some Hetzner boxes, quit my job, charge people for access. Why aren't there like 10 startups doing this yet? Please? I want to give you my money. Just give me a git host that doesn't suck. (All the current ones suck)

show comments
poplarsol

Azure webapp deploys are also trash right now. Microsoft needs to stop slathering h1b copilot slop and get basic things like Windows patches working.

shevy-java

Microsoft again.

I think it is time that Microsoft lets go of GitHub. They are handling it too poorly.

ossa-ma

Seems like outages are increasingly more frequent nowadays. Obviously, this is not the best state of affairs, and developers should not be limited by services. In the meantime I've been experimenting with building third spaces for people to chill while they wait for the services they are dependent on to go back up.

The first one I've built is a little ASCII hangout for Claude @ https://clawdpenguin.com but threads like this make me want to build it for Github too.