> Railway owns our vendor choices, and we ultimately own this one. Your customers don't care whether the failure was Google or Railway; they see your product. Your uptime is our responsibility, and we'll keep delivering on it.
What if the reason for their stuff being shut down was a payment issue like an expired credit card or maxed credit account? Unless I missed it skim reading their post I don’t see any information anywhere about their communications with Google.
If you have an account manager and a contract, there's zero excuse for automated suspension. That's literally the whole point of having a dedicated person. From the report:
> May 19, 22:22 UTC - P0 ticket filed with Google Cloud. Railway's GCP account manager engaged directly.
They have every right to sue, and if they did sue they almost certainly would win. This is clear breach of contract. The only argument Google could make is "they did something to violate our agreement" but they'd have to prove that, and then have a damn good explanation for why they were in the right to suspend the account without any outreach. Unless Railway did something egregious, Google clearly made an error.
But that's not what will happen. Google will offer an apology (perhaps even a public one), a giant pile of account credit, and a pinky promise not to do it again. Railway will accept it and hmmm and haw internally about whether to decrease their reliance on GCP, and then when they calculate the cost of going in on other clouds more heavily (or their own metal), they'll just think harder about weird failure modes.
I'm sure their contract explicitly states that their account can be suspended or terminated immediately without prior notice upon violating some TOS. And, most TOS are incredibly wide and vague, it wouldn't necessarily be hard to find something they violated.
This is sort of the problem with these new-age internet companies. The contracts are incredibly hostile. Most TOS you see amount to "you have no rights and we can fuck you up the ass"
Google is a B2C company so I'm sure some of that culture transfers over to B2B relations, but I'm speculating. Maybe the contracts are more normal for B2B.
If Railway's account was suspended due to an error, not a TOS violation, I doubt they'll pull such a card. If Google were sued, such a blatant lie would be found in discovery pretty quickly and I doubt a court would look upon it favourably. They'd also be starting a public PR battle that they'd almost certainly lose (assuming there was indeed no real TOS violation) which would make them look completely untrustworthy. So I doubt Google will do that.
bastawhiz is probably right in that Google will offer some credits and an apology, and Railway will reduce their dependence (probably), rather than a lawsuit. And I doubt Railway wants to be on the bad side of one of the few big cloud providers. But I'd be surprised if Railway didn't have a good argument to make for compensation or a lawsuit.
One company I worked at is highly reliant on Google Cloud, but at one point we moved some services to Azure.
Azure noticed, and immediately hit us with a discount offer in the hopes of getting more of our business.
Google noticed, and immediately hit us with a discount offer in the hopes of keeping more of our business.
This is just a reminder that your multi-cloud strategy doesn't have to be 'deploy everything across multiple clouds'; it can even just be 'make it obvious that you have leverage'.
1. This isn't self service GCP, this is an enterprise contract. You don't build your medium sized business on a provider without one. Once you're a certain size, you don't get binding arbitration, you go to court in Santa Clara.
2. The TOS for GCP is very clear about an injunctive relief carve out:
> 14.11 Equitable Relief. Nothing in this Agreement will limit either party's ability to seek equitable relief.
3. This (according to the incident report) wasn't Railway being targeted specifically, it was a platform wide automated action. That's gross negligence, and the limitation of liability caps on Google's side of the agreement may not even be enforceable.
I don't think you understand the purpose of the courts. They are not the first resort, they are the last resort, after you have tried all other (non violent) methods to resolve any grievances with another party.
I'm sure there are/will be discussions privately between the two to figure out a resolution.
Railway say the incident is resolved but many are still down (returning 502): on our side, we had to manually trigger a redeploy to fix it but I believe it should have been triggered automatically by Railway and I can't understand how they can mark this as resolved while many are still down.
>Read Trump's unsigned AI executive order
>POLITICO has obtained a copy of the executive order on artificial intelligence oversight that the president had been expected to sign Thursday.
https://news.ycombinator.com/item?id=48237953
reply