A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.
I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.
They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up".
I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do that, it would appear that the code has to be written much more defensively than what they're doing it right now.
Yea agree.. This is the same discussion point that came up last time they had an incident.
I really don’t buy this requirement to always deploy state changes 100% globally immediately.
Why can’t they just roll out to 1%, scaling to 100% over 5 minutes (configurable), with automated health checks and pauses? That will go along way towards reducing the impact of these regressions.
Then if they really think something is so critical that it goes everywhere immediately, then sure set the rollout to start at 100%.
Point is, design the rollout system to give you that flexibility. Routine/non-critical state changes should go through slower ramping rollouts.
For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout?
Blue/green and temporarily ossify capacity? Regional?
That's ok but doesn't solve issues you notice only on actual prod traffic. While it can be a nice addition to catch issues earlier with minimal user impact, best practice on large scale systems still requires a staged/progressive prod rollout.
If there is a proper rollout procedure that would've caught this, and they bypass it for routine WAF configuration changes, they might as well not have one.
The update they describe should never bring down all services. I agree with other posters that they must lack a rollout strategy yet they sent spam emails mocking the reliability of other clouds
Apparently somehow this had never been how Cloudflare did this. I expressed incredulity about this to one of their employees, but yeah, seems like their attitude was "We never make mistakes so it's fastest to just deploy every change across the entire system immediately" and as we've seen repeatedly in the past short while that means it sometimes blows up.
They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear.
Blameless post mortems should be similar to air accident investigations. I.e. don't blame the people involved (unless they are acting maliciously), but identify and fix the issues to ensure this particular incident is unlikely to recur.
The intent of the postmortems is to learn what the issues are and prevent or mitigate similar issues happening in the future. If you don't make changes as a result of a postmortem then there's no point in conducting them.
The aviation industry regularly requires certifications, check rides, and re-qualifications when humans mess up. I have never seen anything like that in tech.
Sometimes the solution is to not let certain people do certain things which are risky.
Agree 100%, however using your example, there is no regulatory agency that investigate the issue and demand changes to avoid related future problems. Should the industry move towards this way?
However, one of the things you see (if you read enough of them) in accident investigation reports for regulated industries is a recurring pattern
1. Accident happens
2. Investigators conclude Accident would not happen if people did X. Recommend regulator requires that people do X, citing previous such recommendations each iteration
3. Regulator declined this recommendation, arguing it's too expensive to do X, or people already do X, or even (hilariously) both
4. Go to 1.
Too often, what happens is that eventually
5. Extremely Famous Accident Happens, e.g. killing loved celebrity Space Cowboy
6. Investigators conclude Accident would not happen if people did X, remind regulator that they have previously recommended requiring X
7. Press finally reads dozens of previous reports and so News Story says: Regulator killed Space Cowboy!
8. Regulator decides actually they always meant to require X after all
As bad as (3) sounds, I'll strongman the argument: it's important to keep the economic cost of any regulation in mind.*
On the one hand, you'd like to prevent the thing the regulation is seeking to prevent.
On the other hand, you'd have costs for the regulation to be implemented (one-time and/or ongoing).
"Is the good worth the costs?" is a question worth asking every time. (Not least because sometimes it lets you downscope/target regulations to get better good ROI)
*Yes, the easy pessimistic take is 'industry fights all regulation on cost grounds', but the fact that the argument is abused doesn't mean it doesn't have some underlying merit
I think conventionally the verb is "to steelman" with the intended contrast being to a strawman, an intentionally weak argument by analogy to how straw isn't strong but steel is. I understood what you meant by "strongman" but I think that "steelman" is better here.
There is indeed a good reason regulators aren't just obliged to institute all recommendations - that would be a lot of new rules. The only accident report I remember reading with zero recommendations was a MAIB (Maritime accidents) report here which concluded that a crew member of a fishing boat has died at sea after their vessel capsized because they both they and the skipper (who survived) were on heroin, the rationale for not recommending anything was that heroin is already illegal, operating a fishing boat while on heroin is already illegal, and it's also obviously a bad idea, so, there's nothing to recommend. "Don't do that".
Cost is rarely very persuasive to me, because it's very difficult to correctly estimate what it will actually cost to change something once you decided it's required - based on current reality where it is not. Mass production and clever cost reductions resulting from the normal commercial pressures tend to drive down costs when we require something but not before (and often not after we cease to require it either)
It's also difficult to anticipate all benefits from a good change without trying it. Lobbyists against a regulation will often try hard not to imagine benefits after all they're fighting not to be regulated. But once it's in action, it may be obvious to everyone that this was just a better idea and absurd it wasn't always the case.
Remember when you were allowed to smoke cigarettes on aeroplanes? That seems crazy, but at the time it was normal and I'm sure carriers insisted that not being allowed to do this would cost them money - and perhaps for a short while it did.
> it's very difficult to correctly estimate what it will actually cost to change something once you decided it's required - based on current reality where it is not. Mass production and clever cost reductions resulting from the normal commercial pressures tend to drive down costs
Difficult, but not impossible.
What are calculable and do NOT scale down is cost for compliance documentation and processes. Changing from 1 form of documentation to 4 forms of documentation has measurable cost, that will be imposed forever.
> It's also difficult to anticipate all benefits from a good change without trying it.
That's not a great argument, because it can be counterbalanced by the equally true opposite: it's difficult to anticipate all downsides to a change without trying it.
> Remember when you were allowed to smoke cigarettes on aeroplanes?
Remember when you could walk up to a gate 5 minutes before a flight, buy a ticket, and fly?
The current TSA security theater has had some benefits, but it's also made using airports far worse as a traveler.
I mean, I'm pretty sure there was a long period where you could walk up 5 minutes before, and fly on a plane where you're not allowed to smoke. It's completely unrelated.
The TSA makes no sense as a safety intervention, it's theatre, it's supposed to look like we're trying hard to solve the problem, not be an attempt to solve the problem, and if there was an accident investigation for 9/11 I can't think why, that's not an accident.
As to your specific claim about enforcement, actually we don't even know whether we'd increase paperwork overhead in many cases. Rationalization driven by new regulation can actually reduce this instead.
For a non-regulatory (at least in the sense that there's no government regulators involved) example consider Let's Encrypt's ACME which was discussed here recently. ACME complies with the "Ten Blessed Methods". But prior to Let's Encrypt the most common processes weren't stricter, or more robust, they were much worse and much more labour intensive. Some of them were prohibited more or less immediately when the "Ten Blessed Methods" were required because they're just obviously unacceptable.
The Proof of Control records from ACME are much better than what had been the usual practice prior yet Let's Encrypt is $0 at point of use and even if we count the actual cost (borne by donations rather than subscribers) it's much cheaper than the prior commercial operators had been for much more value delivered.
You provided an example of where arguing against regulation was ill-conceived in hindsight. I offered an obvious example of the opposite (everyone against plane hijacking -> regulation -> air travel is made worse for everyone without much improvement for the primary issue).
> Rationalization driven by new regulation can actually reduce [paperwork] instead.
Ha! Anything is possible, I suppose.
I'd point out that the TBM were not ratified by committee (much less a government) and were rammed through by unilateral Mozilla fiat.
The CA/B Forum did in fact ratify these rules, which is why they're in its "Baseline Requirements".
They (I would say deliberately) stalled the process to actually make those rules binding on the CAs and that is where Mozilla used their fiat powers to just require this anyway, knowing that all the other trust stores would come for the ride and actually nobody at CA/B offered a legitimate reason not to do this, they just (again I would say deliberately) allowed it to get procedurally bogged down so that nothing would happen for a period of time.
> They have blameless post mortems, but maybe "We actually do make mistakes so this practice is not good" wasn't a lesson anybody wanted to hear.
Or they could say, "we want to continue to prioritise speed of security rollouts over stability, and despite our best efforts, we do make mistakes, so sometimes we expect things will blow up".
I guess it depends what you're optimising for... If the rollout speed of security patches is the priority then maybe increased downtime is a price worth paying (in their eyes anyway)... I don't agree with that, but at least it's an honest position to take.
That said, if this was to address the React CVE then it was hardly a speedy patch anyway... You'd think they could have afforded to stagger the rollout over a few hours at least.
It's just poor risk management at this point. Making sure that a configuration change doesn't crash the production service shouldn't take more than a few seconds in a well-engineered system even if you're not doing staged rollout.
Mentioning React Server Components in the status page can be seen as a bad way to shift the blame. Would have been better to not specify which CVE they were trying to patch. The issue is their rollout management, not the Vendor and CVE.
True, thanks for sharing. Worth mentioning that's on the "full-stack" part of the framework. It doesn't impact most React website while it impacts most next.js websites.
Thanks, that's what I acknowledged in the message you just replied to.
I'm not blaming anyone. Mostly outlining who was impacted as it's not really related to the front-end parts of the framework that the initial comment was referring to.
That makes no difference: while I am not a layer and you absolutely should contact your lawyer before doing something like this to get your ducks in a row (and generally you would thereby be a fool to use what I say as "legal advice"), the relevant fair use here comes from legal precedent and even explicit exemptions required to ensure "software interoperability", not some concept surrounding "non-commercial use" (which I think is highly unlikely to protect you in almost any situation).
Those about:config and chrome:flags fixes are only useful for us hackers. You cannot ask your user base to change the deep settings of your browser. For apps it either works out of the box or it does not.
He doesn't force anybody to use the Epic Store. Unless you want to play Fortinite (which is made and published by Epic Games) or games that have an exclusivity contract with the Epic Store.
This is much, much different than the position of Apple or Google. They don't have to pass an exclusivity contract with anybody since they have a de-facto monopoly on their device. Want you game to be playable on iPhone ? You need to pay Apple rate, you can't just decide to go an publish your game on steam or any other platform.
Nobody is complaining that you can't have iTune on another platform. But for third-party developer, it means that you have no choice but to agree with Apple and Google rates.
Granted, the situation with Google is a bit different since you can technically install another store, but remember that Microsoft was forced to stop pre-installing IE on Windows and offer user a choice of browser. I hardly see how this is any different than the current situation of Apple / Google.
> He doesn't force anybody to use the Epic Store. Unless you want to play Fortinite (which is made and published by Epic Games) or games that have an exclusivity contract with the Epic Store.
So you're not forced, except when you are... I think the GP meant people is forced to use their store to play certain games. Games not developed by Epic, which were meant to be released in other stores and operating systems. You can choose not to play those games. Many people do in fact just that. But that was not the point.
Of course Tim Sweeney does not force random people at gunpoint to use their store. That's never going to be the case because there's laws forbidding him!
There's clearly a difference between a company having exclusive control over how people access a particular game on a particular platform (eg Fortnite on Windows) and a company having control over how people access all software on a platform (eg everything on iOS and Apple).
I don’t think there’s that much of a difference, unless you want to legally classify a platform differently. Today no such legal classification exists.
The problem is exactly those exclusivity contracts and how they were made.
Epic literally bribed devs to do deals that went back on promises, people are not just upset about exclusivity, they are upset about exclusivity of stuff they PAID FOR on other platforms, there were kickstarted games, games pre-sold for Steam, etc... that suddenly became Epic exclusive, and devs went and explicitly said they wouldn't even return the money.
It became an obvious pattern, whenever a game "became" Epic Exclusive, someone would be screwed by it, it is why people are so mad at Epic.
And before someone says that this is their way to compete... well, yes, a crappy one, their store suck, doesn't provide basic features, has no reviews for example, has a ton of dark patterns, it is super hard to get rid of your account or fix your account and so on.
Depends. You bought your groceries there? You paid their price.
You offered extra money to the grocer to sell to you something that was already sold to someone else that had already paid? I would count that as a bribe, and yes, it is a corrupt thing to do.
Epic is literally paying people to go back on their word, or in some jurisdictions, literally commit a crime by not delivering on purpose something that was paid for, in some places that is called interference, in others it is called fraud.
You can just install all game stores on Windows and there is no downside to doing so, and no real difference between installing the Epic store and purchasing a game there and buying the game on the game publisher's website.
Also currently Steam has a dominant position and 30% cut, and if the Epic store succeeds it will result in a more competitive situation with at least two game stores probably taking a 5-15% cut, which is a drastic improvement for everyone except for Valve, which makes Steam.
Same thing on mobile if Epic succeeds in the antitrust lawsuit with Google/Apple stores instead of Steam.
How does that benefit me, Joe Customer? I’m not getting a lower price by going to EGS. I’m just getting more friction in finding a game I want to play.
There are now more store-wide sales available ("Fall Sales" and whatnot). While at the developer's choice a game may have the same retail and sale price at different stores, since the sale dates are different, I the consumer am more likely to find and buy the game at the lower sale price. If you the consumer only want to purchase from one store it is still your choice to wait (just like timed exclusives).
Others have complained about Steam's lack of curation causing discoverability problems. It was at the height of this controversy that Epic launched their store promising better curation. Again, this helps consumers because not all consumers are alike. Some consumers do not want arbitrary moral decisions to block game availability, while others want this so that they do not have to wade through "garbage". Others, still, want to make that decision for all consumers.
I get free games from EGS, just like I got free games from Amazon App Store. For the same reason that the Amazon App Store exists, I can also get open source apps from F-Droid, including apps like Newpipe that aren't allowed on the primary app store on the device.
Do you wish that the Mac App Store were the only way to get apps on your Mac?
Free games are a marketing practice, nothing more. It's not a sustainable business practice, and it's paid for by Epic's exploitative Fortnight lootboxes. The "free games" will become part of a paid monthly subscription (or just go away entirely) when Epic feels like they have the audience they need.
And me? The value those free games might provide me is not worth the cost - being marketed to by a company whose values are quite the opposite of my own.
> Free games are a marketing practice, nothing more.
It's a marketing practice that would not happen without competing stores. The end result is user benefit.
> it's paid for by Epic's exploitative Fortnight lootboxes
Why does it matter to me how it's paid for? Does not allowing competing stores change where Epic or Amazon get their money? The end result is that I benefit from Epic and Amazon getting to compete.
> being marketed to by a company whose values are quite the opposite of my own.
I find Apple's values to be opposite to my own (selling low privacy devices and lying about it, preventing me from doing general computing on my devices, etc.). With competition, I would at least have a choice of finding an app distributer like F-Droid whose values are closer to mine were I to be stuck with an iOS device.
You still haven't answered the question about whether you would prefer if your Mac didn't allow you to install apps outside of it's app store.
If the game store takes a lower cut, then either games will cost less, or they will be better because at least some of the game developers will choose to use the extra money to pay more employees to work on the game.
Also, if there are more stores it's less likely that entire categories of games will be unavailable to you (or available but worse due to reduced budgets) due to all game stores having rules banning them, and in general game stores are less likely to be able to enact policies that serve them at your expense.
They don't. The prices, between MTX and multiple release versions, have gone up if anything.
> they will be better
Pretty close to unuantifiable. What quantification we have, critic scores, aren't showing any movement in the ~decade since Steam got competition (Origin, GoG, etc).
The only benefit seems to be for the shareholders and corporate executives thus far.
That’s not inconsistent. Users can choose to use the Epic store or not. If they choose not to use it, they can’t buy games from that store, just like I can’t buy Trader Joe’s food if I decide to not use the TJ’s store.
Exactly, people are having a hissy fit over exclusive content that has no further price than downloading a free store app. It's like whingeing about driving to another store to get the brand of ice cream that you like. I have no problem with that as long as the device manufacturer doesn't limit me to a single store. Preferably I'd like to be able to choose any store or independent content. More choices, more freedom, more competition.
It is in fact those three things. Different stores compete with each other providing different value, both users and devs having the freedom to choose to support one or the other (or all) based on their personal criteria.
It's up to the developers whether they want to sign an exclusive-deal with Epic (and of course up to Epic to offer one, because it turns out this is a very sweet deal which takes a lot of risk out of a game project, and especially independent developers would be foolish not to accept it).
Apart from Epic there are many other distribution opportunities on the PC platform. This is not the case on iOS. You either make a deal with Apple - and a very lousy deal at that, or you're not on iOS (the same problem exists on game consoles).
A change made to how Cloudflare's Web Application Firewall parses requests caused Cloudflare's network to be unavailable for several minutes this morning. This was not an attack; the change was deployed by our team to help mitigate the industry-wide vulnerability disclosed this week in React Server Components. We will share more information as we have it today.
https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q