This is probably one of the best summarizations of the past 10 years of my career in SRE. Once your systems get complex enough, something is always broken and you have to prepare for that. Detection & response become just as critical as pre-deploy testing.
I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles all the way down!
One of the question I frequently get is "do you automatically rollback". And I have hide in the corner and say "not really". Often, if you knew a rollback would work, you probably could also have known to not roll out in the first place. I've seen a lot of failures that only got worse when automation attempted to turn the thing on and off again.
Luckily from an automation roll-out standpoint, it's not that much harder to test in isolation. The harder parts to validate are things like "Does a Route 53 Failover Record really work in practice at the moment we actually need it to work?"
Usually the answer is yes, but then there's always the "but it too could be broken", and as you said, it's turtles all the way down.
The nice part is realistically, the automation for dealing with rollout and IaC is small and simple. We've split up our infrastructure to go with individual services, so each piece of infra is also straight forward.
In practice, our infra is less DRY and more repeated, which has the benefit of avoiding complexity that often comes from attempting to reduce code duplication. The ancillary benefit is that, simple stuff changes less frequently. Less frequent changes because less opportunity for issues.
Not-surprisingly, most incidents comes from changes humans make. Where the second most amount of incidents come from assumptions humans make about how a system operates in edge conditions. If you know these two things to be 100% true, you spend more time designing simple systems and attempting to avoid making changes as much as possible, unless it is absolutely required.
Iac is definitely a failure point, but the manual alternative is much worse! I’ve had a lot of benefit from using pulumi, simply because the code can be more compact than the terraform hcl was.
For example, for the fall over regions (from the article) you could make a pulumi function that parameterizes only the n things that are different per fall over env and guarantee / verify the scripts are nearly identical. Of course, many people use modules / terragrunt for similar reasons, but it ends up being quite powerful.
I think some people are going to scream when I say this, but we're using mostly CloudFormation templates.
We don't use the CDK because it introduces complexity into the system.
However to make CloudFormation usable, it is written in typescript, and generates the templates on the fly. I know that sounds like the CDK, but given the size of our stacks, adding an additional technology in, doesn't make things simpler, and there is a lot of waste that can be removed, by using a software language rather than using json/yaml.
There are cases we have some OpenTofu, but for infrastructure resources that customer specific, we have deployments that are run in typescript using the AWS SDK for javascript.
It would be nice if we could make a single change and have it roll-out everywhere. But the reality is that there are many more states in play then what is represented by a single state file. Especially when it comes to interactions between—our infra, our customer's configuration, and the history of requests to change the configuration, as well as resources with mutable states.
One example of that is AWS certificates. They expire. We need them expiring. But expiring certs don't magically update state files or stacks. It's really bad to make assumptions about a customer's environment based on what we thought we knew the last time a change was rolled out.
IMO Pulumi and CDK are an opportunity to simplify your infra by capturing what you’re working with using higher-level abstractions and by allowing you to refactor and extract reusable pieces at any level. You can drive infra definitions easily from typed data structures, you can add conditionals using natural language syntax, and stop trying to program in a configuration language (Terraform HCL with surprises like non-short-circuited AND evaluation).
You still end up having IaaC. You can still have a declarative infrastructure.
That's how we use CDK. Our CDK (in general) creates CloudFormation which we then deploy. As far as the tooling which we have for IaC is concerned, it's indistinguishable from hand-written CloudFormation — but we're able to declare our intent at a higher level of abstraction.
> and stop trying to program in a configuration language
Many people don't program with a configuration language like HCL. We use it as what it is - a DSL - that covers its main use case in an elegant manner. Maybe I never touched complex enough infra that twists a DSL into a general-use language, but in my experience there are simply no real benefits when using something like CDK (I never tried Pulumi to be fair).
Absolutely, the best case is it's much better, safer, readable etc. However, the worst case is also worse. From the perspective of someone who provides devops support to multiple teams, terraform is more "predictable".
Agreed, it is much too easy to fall into bad habits. The whole goal of OpenTofu is declarative infrastructure. With CDK and pulumi, it's very easy to end up in a place where you lose that.
But if you need to do something in a particular way, the tools should never be an obstacle.
If you do use terraform, for the love of god do NOT use Terraform Cloud. Up there with Github in the list of least reliable cloud vendors. I always have a "break glass" method of deploying from my work machine for that very reason.
I'm a HOA president and while HOAs can be very extreme, the flip side is if homeowners are breaking rules designed to protect property or common areas (pool, lawns, playground, etc) a $100 is not enough to stop people. Thankfully our HOA focuses on our common areas and is responsible for all exteriors and lawns (it's all townhomes), so the lines are a bit clearer.
We've had all sorts of wild issues such as building scaffolding on top of balconies (not attached), ripping up common area plants, parking issues (we all have garages, street parking is guest only), drying food on the pool deck (really), dumping garbage bags outside in the common area and more. If we can only levy a $100 fine there's little incentive for some people to stop doing things that impact the community.
I do cringe when I hear about these crazy HOAs of what are usually a collection of single family homes. I think a better approach would be some kind of limitations of the what HOAs can have rules about vs the penalties. Interiors of homes should be generally off limits (aside from townhomes that are all technically 1 building, so you should not be doing anything structural without approval). For single family homes with private property surrounding them I'd rather there be limits that are purely for safety, legal reasons or impacting common areas.
As a permanent structure or for temporary renovations?
> ripping up common area plants
Just for fun? Were they drunk? Or is the border between the "common area" and "their property" somewhat hazy? Are you not able to simply forward the invoice for repairs to the resident? That's not a fine and doesn't seem like it would be covered?
> parking issues (we all have garages, street parking is guest only)
This impacts property values? What about tow to impound?
> drying food on the pool deck (really)
> dumping garbage bags outside in the common area
A $100 fine is not adequate for these relatively petty issues?
It might just be me. I don't have kids and I don't spend a lot of time around home. I don't understand HOAs at all.
Yes, as a non-American, HOAs seem so strange to me. It seems like most of those issues could be resolved by the existing legal system (destroying other people's property, dumping stuff in public areas, etc.) or by the city's regulations and codes.
When you buy a house you know whether there there is an HOA, so there shouldn't be any surprises.
HOAs are interesting for cities as they cordon off certain parts for which the city pays no street maintenance, no park maintenance, yet it collects full taxes.
For people living in an HOA it can provide amenities like more private parks, pools etc.
The city doesn't govern what happens in HOA common areas, because the HOA owns that property. Destruction of other people's property - clearly yes, but destruction of the HOA's property is different, because the homeowner is part of the HOA and thus it's their own property (but shared among all the other homeowners). Thus the HOA has to come up with a set of rules to govern its own property from its own homeowners.
> homeowners are breaking rules designed to protect property or common areas
Fines are administrative. If someone is causing property damage, that’s liability—indemnification (where the homeowner pays the HOA’s legal fees) should be sufficient.
No, I think they meant protect property from damage. Lawsuit is a high bar for action.
If I go into a private gym and start smashing things, they would want to fine me and kick me out, but it may not be worth several thousand dollars to sue me for the damages.
You make some fair points, but it’s also worth some self-reflection as an HOA president to understand why so many people resent these institutions. I’ve given two HOAs an honest try, and both ended up reinforcing the same patterns of pettiness and overreach that give them their reputation. The structure itself seems to attract a small group exerting outsized control over others’ property. Hopefully, over time, communities can move toward simpler, more democratic systems that preserve shared spaces without breeding unnecessary conflict.
I get it, really I do. But do the HOAs really need financial enforcement mechanisms intended to seriously harm people, and to punish them as judge, jury and executioner? A HOA’s legal job is to maintain the common-interest property and enforce the CC&Rs. It is not a HOA’s job to extract enormous sums of money out of its members, even annoying ones. The right lever to pull to get some rich person partying at 4am and trashing the place (for example) to stop is for the HOA to file for a court injunction after repeated violations; once a judge orders “no loud music 10 pm - 7 am”, the next 4 am party will become contempt of court, which is a problem for the cops, not the HOA. Hell, 4 a.m. noise is a municipal nuisance and probably a crime; people should be calling the cops every time it happens. Individual members could even sue the owner in small-claims court for private nuisance, where judges can issue even more injunctions or award damages.
All this to say, you don’t need to take people’s money to get them to stop doing bad stuff. But you do need to take people’s money to get rich, and to hurt people. This new legislation should be deeply concerning to people interested in the latter, and IMO shouldn’t really be a concern to people interested in the former.
I don't know where you live, but calling cops over noise nuisance has not worked in most cities in the US for a long time. E.g. with LAPD you will be lucky if cops will show up in 4 hours and if they show up they are not going to ticket anybody. And there is nothing you can do about it. "Petty" crime is free-for-all in any city with a "restorative justice" DA. So we need to use other means to slow down our degrading quality of life.
>But do the HOAs really need financial enforcement mechanisms intended to seriously harm people, and to punish them as judge, jury and executioner?
No, they don't. But to be fair, your local enforcement agencies have the same power to unilaterally fine people insane amounts of money. So in a technical sense it makes sense that HOAs would have the same unilateral power to screw people.
1) Governments are often much easier to sway. You can get a newspaper or TV station involved. You can show up to open meetings. You can campaign against the incumbents. While you can porbably technically do some of that against rogue HOA boards, it's going to be a lot harder.
2) Governments are usually large enough not to make things a personal vendetta. That's clearly not always true; I'm only talking about trends. Meanwhile, the HOA members are your neighbors, by definition. Get on the wrong side of them and they can easily get involved in everything you do.
Ah, got it. You were saying neither part should do that. I interepreted that as HOAs should also be allowed to do that. I see what you're saying now, though.
You have to phrase it properly. One time when a neighbor had a school-/work-night party that lasted until after midnight, I went over and asked them to wrap it up. When they didn't, I called the police non-emergency line and asked them to go break it up. When we were still awake from noise an hour later, I called the police again, and told them that in 15 minutes I was going back over there myself. They asked me to please not do that, and took care of it within the next 10 minutes.
They were ambivalent about dealing with noise, but were happy to stave off a riot.
It wouldn't surprise me if it's still a net positive, even with the downsides. Also the article says there are carveouts for health and safety, I wonder if excessive noise at night counts as a health issue (there's more than enough research on how important sleep is where it wouldn't be absurd to have it as part of the health carveout).
I took a Waymo that drove on an 'expressway' which had a speed limit of 40mph and it was definitely a different feeling. I did feel a bit scared, at 25mph it feels like a gentle theme park ride, at 40mph it's beyond that and feels dangerous.
I worked at Yahoo in 2008 when they laid off thousands and yes every single person got a calendar invite and met in a meeting room 1:1 with a manager. It was difficult but they did it. Times definitely have changed.
Wow, just the logistics of that is impressive. I feel like I would watch a 60-minute documentary on pulling that together because it no doubt took dozens or hundreds of people weeks of logistics to do that, and unlike almost any other major project, literally no one involved was happy about any part of it.
Not explicitly, but there were rumors a few days before. Also the signs were there: every single meeting room was booked, meeting rooms all had water & tissues, etc.
I agree, I've been at places that can tie alerts at a host level to an automated task runner. Basically a workflow system that gets kicked off on an alert. Alert fires, host is rebooted or terminated. Helpful for things like this.
Under $100 - Ember mug. No more reheating coffee a few times/day or trying to drink cold coffee. I was surprised at how much I enjoy it. A nice napkin holder that is easy to pull napkins from. Vertical magazine/tablet organizer. Whiteboard next to my desk. Anything simple that makes a frequent chore or task 50% easier.
Under $1000 - Twice/week 'house helper' who does dishes and laundry. My wife travels a few weeks/month and we have 2 kids. Having someone do all the dishes and laundry saves me at least 5hrs/week.
It keeps the drink at a specific temperature indefinitely.
With an insulated mug, you aren't putting more heat into the drink but just trying to keep it for longer. In my opinion, insulation works great for cold drinks (I use a giant insulated cup for my cold drinks) but not so much for warm drinks.
The old (maybe incorrect) advice I was always told was that continuing to heat the coffee fouled the taste, hence the recommendation to use an insulated mug/thermos as opposed to a Mr. Coffee style warmed carafe. What makes this gadget different/better?
Much of coffee's flavour comes from a balance of acids and oils. These, particularly the oils, are sensitive to temperature. But unless you're drinking your coffee all day, you're unlikely to notice it. Real spoiling sets in after several hours at the kinds of heat you're probably running your Ember mug at. Most drip brewers with heating elements for the pot keep the temperature too high (often just below boiling). A steady heat is much better for the flavour than reheating. (This is what I remember from working at a coffee shop way, WAY back when I was at university.)
That makes sense. The mug version of Ember (https://ember.com/products/ember-mug-2) is open on top, which seems like a fancy Bluetooth version of an old-school mug heater.
I could see the sealed one (the "Cup") as working better though.
It will hold my coffee at the perfect temperature all day. Also if the battery runs out I can put it on its charger and after ~5 mins it will start heating my now cold coffee back up to temperature. I thought it was kinda gimmicky too before I bought it but I am surprised at how great it is to have perfect temp coffee all the time.
Everything can go in the dishwasher at least once. ;)
But, on a more serious note. Over the years I've replaced everything in my kitchen that isn't dishwasher friendly with something that is. I do not want to waste the small amount of free time I have doing any dishes.
I've ridden a few times and their prices seem close to Lyft. I would happily pay a premium for the service they are providing right now. New, clean cars that drive very smoothly and lower rates of accidents than an 'average' driver. No weird smells, distracted drivers, inexperienced drivers, etc.
I think in a few more years with the amount of training their AI will have Waymo will be a truly incredible taxi service. It can only get better!
> New, clean cars that drive very smoothly.. No weird smells.
One concern I had is that once self-driving cars are widespread people might take advantage and treat them poorly with no human driver watching. Where a human would likely notice and deal with weird smells or things added by the past customer. Waymo-type service probably rely on reporting by passengers once the car already arrives, no? (I've never driven in one)
People might be treating them nice today because Waymo is a fancy new service in Jaguars that people treat as a novelty. Once those e-taxi services are under aggressive economic demands I'm curious to see how it plays out.
Waymo's have interior cameras on their vehicles. Any rider that leaves mess will get a strike and probably be banned for repeated infractions. You're right, some problems will only be noticed by the next rider.
I hate DMs in Slack for this and many reasons. One thing I do try is to ask people to move to a public channel unless it's a personal issue. Copy/paste their question and cc their handle.
Cosmetically maybe, but Hertz has very good incentives to do minimal maintenance and only fix things when very broken. The renter has little incentive to report issues like weird noises or anything mechanically broken, especially if it may have been caused by them.
I do worry about all the automation being another failure point, along with the IaC stuff. That is all software too! How do you update that safely? It's turtles all the way down!