GitHub is down again

458 points379 comments6 hours ago
mrshu

GitHub no longer publishes aggregate numbers so here they are parsed out. It looks like they are down to a single 9 at this point across all services:

https://mrshu.github.io/github-statuses/

show comments
showerst

If you'd have asked me a few years ago if anything could be an existential threat to github's dominance in the tech community I'd have quickly said no.

If they don't get their ops house in order, this will go down as an all-time own goal in our industry.

show comments
mholt

Of course they're down while I'm trying to address a "High severity" security bug in Caddy but all I'm getting is a unicorn when loading the report.

(Actually there's 3 I'm currently working, but 2 are patched already, still closing the feedback loop though.)

I have a 2-hour window right now that is toddler free. I'm worried that the outage will delay the feedback loop with the reporter(s) into tomorrow and ultimately delay the patches.

I can't complain though -- GitHub sustains most of my livelihood so I can provide for my family through its Sponsors program, and I'm not a paying customer. (And yet, paying would not prevent the outage.) Overall I'm very grateful for GitHub.

show comments
stefankuehnel

You can literally watch GitHub explode bit by bit. Take a look at the GitHub Status History; it's hilarious: https://www.githubstatus.com/history.

show comments
petetnt

GitHub has had customer visible incidents large enough to warrant status page updates almost every day this year (https://www.githubstatus.com/history).

This should not be normal for any service, even at GitHub's size. There's a joke that your workday usually stops around 4pm, because that's when GitHub Actions goes down every day.

I wish someone inside the house cared to comment why the services barely stay up and what kinds of actions are they planning to do to fix this issue that's been going on years, but has definitely accelerated in the past year or so.

show comments
adamcharnock

We've migrated to Forgejo over the last couple of weeks. We position ourselves[0] as an alternative to the big cloud providers, so it seemed very silly that a critical piece of our own infrastructure could be taken out by a GitHub or Azure outage.

It has been a pretty smooth process. Although we have done a couple of pieces of custom development:

1) We've created a Firecracker-based runner, which will run CI jobs in Firecracker VMs. This brings the Foregjo Actions running experience much more closely into line with GitHub's environment (VM, rather than container). We hope to contribute this back shortly, but also drop me a message if this is of interest.

2) We're working up a proposal[1] to add environments and variable groups to Forgejo Actions. This is something we expect to need for some upcoming compliance requirements.

I really like Forgejo as a project, and I've found the community to be very welcoming. I'm really hoping to see it grow and flourish :D

[0]: https://lithus.eu, adam@

[1]: https://codeberg.org/forgejo/discussions/issues/440

PS. We are also looking at offering this as a managed service to our clients.

show comments
MattIPv4

Status page currently says the only issue is notification delays, but I have been getting a lot of Unicorn pages while trying to access PRs.

Edit: Looks like they've got a status page up now for PRs, separate from the earlier notifications one: https://www.githubstatus.com/incidents/smf24rvl67v9

Edit: Now acknowledging issues across GitHub as a whole, not just PRs.

show comments
cedws

Screw GitHub, seriously. This unreliability is not acceptable. If I’m in a position where I can influence what code forge we use in future I will do everything in my power to steer away from GitHub.

show comments
jbpadgett

3 outages in 3 months straight according to their own status history. https://www.githubstatus.com/history

show comments
RomanPushkin

Looks like AI replacement of engineering force in action.

show comments
razwall

They're overwhelmed with all the vibecoded apps people are pushing after watching the Super Bowl.

show comments
pimpl

What are good alternatives to GitHub for private repos + actions? I'm considering moving my company off of it because of reliablity issues.

show comments
feverzsj

Seems Microsoft goes downhill after all in AI.

show comments
mikert89

pretty clear that companies like microsoft are actually terrible at engineering, their core products were built 30 years ago. any changes now are generally extremely incremental and quickly rolled back with issue. trying to innovate at github shows just how bad they are.

show comments
JamesTRexx

Sorry, my fault. I tried to download a couple of CppCon presentations from their stash. Should have known better than to touch anything C++. ducks

show comments
mentalgear

Seems like MS copilot is vibe-ing it again ! Some other major cloud provider outages come to mind that never happened before the "vibe" area.

ecshafer

Well its a day that ends in Y.

Github is down so often now, especially actions, I am not sure how so many companies are still relying on them.

show comments
romshark

GitHub is slowly turning into the Deutsche Bahn of git providers.

tapoxi

Is it really that much better than alternatives to justify these constant outages?

show comments
MattIPv4

Looks like they've got a status page up now for PRs, separate from the earlier notifications one: https://www.githubstatus.com/incidents/smf24rvl67v9

Edit: Now acknowledging issues across GitHub as a whole, not just PRs.

danelski

I wonder what's the value of having a dedicated X (formerly Twitter) status account post 2023 when people without account will see a mix of entries from 2018, 2024, and 2020 in no particular order upon opening it. Is it just there so everyone can quickly share their post announcing they're back?

twistedpair

In the age of Claude Code et al, my honest biggest bottleneck is GH downtime. I've got a dozen PRs I'm working on, but it's all frozen up, daily, with GH outages.

Are the other providers offering much better uptime GitLab, CircleCI, Harness? Saying this as someone that's been GH exclusive sicne 2010.

byte_surgeon

Just remove all that copilot nonsense and focus on uptime... I would like to push some code.

trollbridge

Fortunately, git is quite resilient and you can work offline and even do pull requests with your peers without GitHub.

porise

Take it away from Microsoft. Not sure how this isn't an antitrust issue anyway.

show comments
ZpJuUuNaQ5

It's a funny coincidence - I pushed a commit adding a link to an image in the README.md, opened the repo page, clicked on the said image, and got the unicorn page. The site did not load anymore after that.

petterroea

When I was a summer intern 10 years ago I remember there without fail always being a day where GitHub was down, ever summer. Good times.

CamT

It feels like GitHub's shift to these "AI writes code for you while you sleep!" features will appeal to a less technical crowd who lack awareness of the overall source code hosting and CI ecosystem and, combined with their operational incompetence of late (calling it how I see it), will see their dominance as the default source code solution for folks using it to maintain production software projects fade away.

Hopefully the hobbyists are willing to shell out for tokens as much as they expect.

1vuio0pswjnm7

I am able to access github.com at 140.82.112.3 no problem

I am able to access api.github.com at 20.205.243.168 no problem

No problem with githubusercontent.com either

koreth1

The biggest thing tying my team to GitHub right now is that we use Graphite to manage stacked diffs, and as far as I can tell, Graphite doesn't support anything but GitHub. What other tools are people using for stacked-diff workflows (especially code review)?

Gerrit is the other option I'm aware of but it seems like it might require significant work to administer.

show comments
jnhbgvjkb

If you were looking for a signal to leave github, then this is it.

zurfer

to be fair, i think usage has increased a lot because of coding agents and some things that worked well for now can't scale to the next 10x level.

show comments
BhavdeepSethi

I wonder if GH charges for the runners during their downtime. Last week lot of them would retry multiple times and then fail.

0xbadcafebee

List of company-friendly managed-host alternatives? SSO, auditing, user management, billing controls, etc?

I would love to pay Codeberg for managed hosting + support. GitLab is an ugly overcomplicated behemoth... Gitea offers "enterprise" plans but do they have all the needed corporate features? Bitbucket is a joke, never going back to that.

alfanick

Oh! It's not my GitLab@Hetzner that's not working, it's GitHub. Just when I decided to opensource my project.

show comments
patrick4urcloud
show comments
hkt

Churlish of me to say, but wasn't GH so much more reliable before the acquisition by Microsoft?

jablongo

It looks like one of my employees got her whole account deleted or banned without warning during this outage. Hopefully this is resolved as service returns.

Culonavirus

Azure Screen of Death?

show comments
bovermyer

Meanwhile, Codeberg and Worktree are both online and humming along.

Codeberg gets hit by a fair few attacks every year, but they're doing pretty well, given their resources.

I am _really_ enjoying Worktree so far.

show comments
aqme28

The saddest part to me is that their status update page and twitter are both out of date. I get a full 500 on github.com and yet all I see on their status page is an "incident with pull requests" and "copilot policy propagation delays."

show comments
Tade0

I don't know if it's related, but for the past week I've been getting pages cut off at some point, as if something closed the connection mid-transfer.

Today, when I was trying to see the contribution timeline of one project, it didn't render.

bigbuppo

On the plus side, it's git, so developers can at least get back to work without too much hassle as long as they don't need the CI/CD side of things immediately.

ilikerashers

Yeap, getting this for the last 20 minutes. Everything green on their status pages.

edverma2

Anyone have alternatives to recommend? We will be switching after this. Already moved to self-hosted action runners and we are early-stage so switching cost is fairly low.

show comments
ascendantlogic

So what's the moneyline on all these outages being the result of vibe-coded LLM-as-software-engineer/LLM-as-platform-engineer executive cost cutting mandates?

huntertwo

Microslop strikes again!

tigerlily

So, what're people's alt stack for replacing GitHub?

show comments
kachapopopow

I made this joke 10 hours ago: "I wonder if you opened https://github.com/claude in like 1000's of browsers / unique ips would it bring down github since it does seem to try until timeout"

coincidence I think not!

thewhitetulip

Has anyone noticed that in the past year we have seen a LOT of outages?

show comments
davidfekke

I guess Bill Gates has a virus.

canterburry

I wonder if the incident root cause analysis will point to vibe coding?

rileymichael

the incident has now expanded to include webhooks, git operations, actions, general page load + API requests, issues, and pull requests. they're effectively down hard.

hopefully its down all day. we need more incidents like this to happen for people to get a glimpse of the future.

show comments
yoyohello13

Azure infra rock solid as always.

elcapitan

Maybe we should post when it's up

parvardegr

Damn, I was also trying to push and deploy a critical bug fix that was needed within minutes.

show comments
altern8

One reason for the reduction in global downtime could be that with time they add more and more services that can go down and affect the stats.

Just saying.

an0malous

I think this is an indicator of a broader trend where tech companies put less value on quality and stability and more value on shipping new features. It’s basically the enshittification of tech

jcdcflo

We replaced everything except the git part because of reliability issues. Pages…gone Actions…gone KB…gone. Tickets…gone.

Maybe they need to get more humans involved because GitHub is down at least once a week for a while now.

zingerlio

I was wondering why my AUR packages won’t update, just my luck.

GeneralGrevous

Fix this or I will send my droid army. #greenpurplelifesmatter #Imcocoforcocoapuffs #ihatejedi

CodingJeebus

Do they publish proper post-mortems? I feel like that's gotta be the bare minimum nowadays for such critical digital infrastructure.

The new-fangled copilot/agentic stuff I do read about on HN is meaningless to me if the core competency is lost here.

semiinfinitely

They put too much AI in it bot enough engineering rigor

EToS

sorry all, i took a month off and then opened github.com

nusaru

I look forward to the day that jjhub becomes available...

gpmcadam

> Monday

Beyond a meme at this point

peab

Is it just me, or are critical services like GitHub, AWS, Google, etc., down more often than they used to be these days?

simianwords

Monolith looking like a good now?

unboxingelf

1 engineer, 1 month, 1 million lines of code.

dbingham

Github's two biggest selling points were its feature set (Pull Requests, Actions) and its reliability.

With the latter no longer a thing, and with so many other people building on Github's innovations, I'm starting to seriously consider alternatives. Not something I would have said in the past, but when Github's outages start to seriously affect my ability to do my own work, I can no longer justify continuing to use them.

Github needs to get its shit together. You can draw a pretty clear line between Microsoft deciding it was all in on AI and the decline in Github's service quality. So I would argue that for Github to gets its shit back together, it needs to ditch the AI and focus on high quality engineering.

thinkindie

it's Monday therefore Github is down.

arnvald

GitHub is the new Internet Explorer 6. A Microsoft product so dominant in its category that it's going to hold everyone back for years to come.

Just when open source development has to deal with the biggest shift in years and maintainers need a tool that will help them fight the AI slop and maintain the software quality, GitHub not only can't keep up with the new requirements, they struggle to keep their product running reliably.

Paying customers will start moving off to GitLab and other alternatives, but GitHub is so dominant in open source that maintainers won't move anywhere, they'll just keep burning out more than before.

wrxd

Copilot, what have you done again?

seneca

GitHub has a long history of being extremely unstable. They were down all the time, much like recently, several years ago. They seemed to stabilize quite a bit around the MS acquisition era, and now seem to be returning to their old instability patterns.

properbrew

Now it seems Actions has broken - https://www.githubstatus.com/incidents/lcw3tg2f6zsd

hit8run

They should have just scaled a proper Rails monolith instead of this React, Java whatever mixed mess. But hey probably Microslop is vibecoding everything to Rust now!

show comments
blibble

presumably slophub's now dogfooding GitHub Agentic Workflows?

sama004

3 incidents in feb already lmao

ilovefrog

when im on w2 this is good but when im contracting this is bad

thesmart

Can we please demand that Github provide mirror APIs to competitors? We're just asking for an extinction-level event. "Oops, our AI deleted the world's open source."

Any public source code hosting service should be able to subscribe to public repo changes. It belongs to the authors, not to Microsoft.

show comments
thesmart

It's really pathetic for however many trillions MSFT is valued.

If we had a government worth anything, they ought to pass a law that other competitors be provided mirror APIs so that the entire world isn't shut off from source code for a day. We're just asking for a world wide disaster.

guluarte

vibe coding too much?

run-run-forever

I bet Microsoft did this...

ChrisArchitect

Related incidents:

Incident with Pull Requests https://www.githubstatus.com/incidents/smf24rvl67v9

Copilot Policy Propagation Delays https://www.githubstatus.com/incidents/t5qmhtg29933

Incident with Actions https://www.githubstatus.com/incidents/tkz0ptx49rl0

Degraded performance for Copilot Coding Agent https://www.githubstatus.com/incidents/qrlc0jjgw517

Degraded Performance in Webhooks API and UI, Pull Requests https://www.githubstatus.com/incidents/ffz2k716tlhx

show comments
Hamuko

I get the feeling that most of these GitHub downtimes are during US working hours, since I don't remember being impacted them during work. Only noticed it now as I was looking up a repo on my free time.

iamleppert

Good thing we have LLM agents now. Before this kind of behavior was tolerable. Now it's pretty easy to switch over to using other providers. The threat of "but it will take them a lot of effort to switch to someone else" is getting less and less every day.

show comments
GeneralGrevous

fix it or I will send robot to your house blud #greenpurplelifesmade #Imcocoforcocoapuffs

ruined

tangled is up B]

gamblor956

MS is now all in on agentic coding.

Github stability worse than ever. Windows 11 and Office stability worse than ever. Features that were useful for decades on computers with low resources are now "too hard" too implement.

Coincidence?

retinaros

migrating to azure kills businesses

dmix

Welcome to Microsoft Github

guluarte

And now actions are down... great. https://www.githubstatus.com/incidents/lcw3tg2f6zsd

show comments
run-run-forever

Now Github pages are down

DetroitThrow

GitHub downtime is going from once a month (unacceptable) to twice a month (what the fuck?)

iamsyr

The next name after Cloudflare

charles_f

That pink "Unicorn!" joke is something that should be reconsidered. When your services are down you're probably causing a lot of people a lot of stress ; I don't think it's the time to be cute and funny about it.

EDIT: my bad, seems to be their server's name.

show comments