I'm right with you. My dad was a farmer and taught me: First deal with the irregular, annoying parts on the outside. Create rectangles with a long side and turning space on each shorter side. Then deal with each rectangle by going back and forth along the long edge. The turnarounds are also a great place for some wheelbarrows to collect the clippings closest to the Komposthaufen.
Hence why I went for the irregular strange part on the lower left first, and then the more regular part on the right. That's the weird rain collection, herb garden, thing we have at the back.
> People are, in heavy LLM systems, realising that the most valuable commodity is engineers knowing WTF is going on, and that loss of understanding of your codebase is the #1 blocker to actually getting things done. Any senior engineer knows this all too well
I've been pushing against this "maximum LLM driven speed" in our infrastructure as well and been working on a middle ground:
We can generate a change to Ansible or terraform code in half or a third of the time, sure. But we don't use this to make three times the changes in the same time frame. We rather use the freed up time to discuss the change, the context and the affected systems with 2-3 engineers maintaining them. And yes, at times this means us three are sitting around half a day discussing and drawing diagrams about our systems.
I'm just happy to work in a company understanding the value of speeding up less than we could in the feature direction, but investing this time in the direction of control and understanding of the infrastructure.
Yup, this is how I see teams who care about quality, design and architecture seems to use LLMs, not to "produce more and faster" but to retain same speed but with a lot more confidence and reliability. Hoping this will spread eventually, some companies seem to take a more... hazardous approach to the whole thing.
This is one of the unintuitive parts when you get into operations: If you go to lower layers in the stack, production expands towards dev:
To the product developers and operators, customer-facing systems are production.
To us in infra-operations, dev and testing are actually production as well. Maybe with a lower SLA and easier maintenance scheduling, but if we fry dev or testing, a hundred developers can't work and start screaming.
Within the infra-ops team, our config management tests and the deployment pipelines are production. If those don't work, infra operators cannot test or roll out changes to the infrastructure.
This was recently discovered by a dev team providing a cross-cutting service: Their testing environment can halt work for a lot of other teams, so they have to be really careful with their testing environment.
I think the question there is how do you make test actually test? Allowing them to break their test env so they can actually fix something they broke, imo. It's enabling and it's educational. Too often I see it so abstracted away that devs don't even know what their environment is made out of, so how can you expect them to make performant decisions that align with the infra?
Maybe the cost you pay is in the complexity of security. And then where does that land? Is it ops or security when some ephemeral rotating key or token that's controlled by some test service account doesn't work?
IMO having a test environment is a bit of a failure: you should be able to spin up and stop a full test environment easily, ideally (and feasibly for most systems) on the developer's local machine.
Depends on the type of testing. If you're doing integration testing between systems, then it's not just a matter of spinning it up on a developer's local machine.
You need a testing environment, especially if there are other physical elements that are being integrated (embedded and other equipment).
It's bizarre to me that we live in a world where whole machines and their networking stack can be spun up and virtualized and yet integration testing STILL actually has to go out and touch a real resource running on a different machine.
Yeah, obviously this isn't gonna work for embedded development, but not that many people have that specific problem. Is it really so hard to spin up ephemeral testing clusters on a single machine? Everywhere that I've seen it done, it's a horrible kludge of scripts and hacks that's almost as difficult to understand as the code under test.
When you are integrating to other people's stacks, then you need a test facility where people can bring systems together, sometimes there are also physical elements that are required.
I work in Automatic Fare Collection (think railway passenger gates, ticket vending, contactless readers, bus driver consoles and readers etc as well as the backends that deal with the fare calculation and payments).
There are usually multiple vendors involved, and even with the best API and interface documentation and individual system testing against the specifications, you still need to bring the entire environment, including the hardware, together.
These ITFs (Integration Test Facilities) tend to be in existence for the life of the systems, including after hardware refreshes etc.
Yes you can test performance with jmeter etc, but sometimes you need 20 people walking in a circle between an entrance gate and an exit gate.
Fair enough, but IMO even a system like this one should have good mock implementations of the system hardware, down to and including a simple simulator for the physical layout of people near the machine.
In past years I wouldn't say that, but these days it's so easy to get an LLM to write something like this plus a quick and dirty visualization.
I'm not saying this can be a replacement for a physical test setup, but it's so much easier to make changes when the tests can just give a clear cut "yes, this acts like you expect" or "no, this is broken."
Docker has made this much much easier than it used to be, and the ease of publishing images has led to many SaaS companies providing "dockerized" versions of their services.
I write a lot of Terraform to manage internal infrastructure with dependencies on other internal and external infrastructure that's outside of my control. I define a module parameter for each dependency (usually with a default value that points to the real service URL) and then create a second module that uses Docker to spin up and configure local instances of each dependency and overrides the corresponding parameter.
This makes local testing very straightforward: `terraform -chdir=tests apply`
Sure, for integration with some specific hardware, there may be some limits (though I would ask if that could be simulated for most testing, or if multiple could be made available). But it's a goal that should be aimed for and is achievable for most software, if some effort is spent on it. I see a lot of places seem to cargo-cult a dev/staging/prod setup (each of which inevitably winds up slightly different and hard to reproduce for incidental reasons) and it seems like an approach with a lot of headaches that you probably don't want to use unless you're forced to.
When you have a lot of interdepended systems it is not always possible. In our case our test environment cannot be replicated locally on developer machines because it integrates to various 3rd parties to which we only have one set of test credentials.
In our case, "testing" as an environment has the goal to be a fully integrated setup of the entire service landscape. Each team is responsible for the uptime of their service, just like in production.
The teams are running automated end-to-end tests against this environment. On top, pre-sales uses it to build demo and sample environments, sales shows demos on it, solution engineers use it to test/try out the important parts of projects.
That generates a good production-like experience for the team: You have customers using the system, and possibly screaming if you break it. And it has been pretty successful at catching problems missed or integration failures.
This is correct, but there's additional nuance here, which is the security of your development pipeline plays a part in the production system too.
Way back when, you'd see a lot of people stuff their jenkins in their dev account. There's no way that the system that builds, publishes and deploys your application should be treated as anything but as sensitive as the production system itself.
( To be fair there are mitigations such as reproducible builds, but you're now doing a bunch of engineering to tie in your deploy system. )
Agreed. I've started calling our non-production infrastructure environments 'engineering' rather than dev, and nobody's invited. We'll pull in platform teams that are exposed to this shame environmental offset issue, but no customers.
In my experience, business software has narrow and predictable use cases. Note taking: You need to add, delete and update notes and the real meat of the product is usually in the note organization. Even SAP has use cases simple in nature: Account for milk cartons sold. The use cases just grow complex once you work in 30 countries or more. "Pay the right taxes on milk cartons sold" becomes tricky.
However, this is parallelizable: You can work on all 30 countries at once and ship when ready. You can work on taking and organizing notes in parallel. You can work on scaling in parallel. You can streamline UX in parallel while introducing tagging, or trees, or groups, or anything.
A game, and especially a narrative game, has an intrinsic linear component, the plot. You can work on a certain amount in parallel, sure: Once the rough shape of the plot is laid out, you can start working on assets, necessary engine features, sounds, dialog in parallel.
But you still have a strong need that a player or play tester has a good progression from act one to act two to act three, and a good progression from puzzle one to puzzle two to puzzle three. This takes a lot of very slow iteration.
This involves human testing by nature. You get a playtest every few months, if you have the money. You can easily throw more CI hardware to run your integration tests more often for more countries. Playtesters, especially those who don't know the game, are very, very limited.
Then you also want characters and settings to be consistent across the plot. If done well, this takes a lot of adjustments and refinement both back and forth. Sometimes the setting of Act three requires a character to have a certain trait, and that trait should be developed, so you now go back to Acts one and two to see if you can build up the character -- without knocking down the entire house of cards already in place.
And once you get to polish and atmosphere, it becomes even harder and more time consuming, because then you need new playtesters even more.
From a past job and working on some features, I spent a day or two presenting like 6-8 different options, and then spent half a day in different testing groups to figure out which procedural generation option "feels the best" for what the game designer wants. With a better spec, that could've been an hour. This made players more excited. Probably.
Back when we were working in the office, this was honestly a preferred pair programming setup for us. Two people, and each just had their keyboard and mouse.
Interestingly enough, it seemed that the inputs from multiple mice onto the same cursor is summed up by default. So you could fight cursor movement you disagreed with by moving the mouse in the other direction.
You jest, but companies have sent people home fully paid, because power or internet were down and would be down for an extended period of time. Technically a small number could have kept working on white-boarding design topics, but without access to documentation, existing tickets, it would have been minimally effective.
It's not clear if or when these tools become as essential as power or internet to a developers or admins work, but if they do? Giving everyone a quarter day off at least generates some good-will, unlike forcing them to sit around unproductively because of working hours.
Some of my deepest architectural work honestly happened... in a hammock on the balcony, while alternating between staring at a cloud, the sun, and two magpies building a nest in the birch tree while yelling at me occasionally.
Then I came back to my desk, saw that I had missed a call or two from my boss and he had a concerned message of where I was. I ignored that and sent him an architecture document an hour later. After years, it is still the backbone of our hosting infrastructure and will go into certification audits later this year.
One of my earlier startups, I was a junior/mid level software engineer. I took a skive day to WFH, but was actually travelling to a gig on the train. I sketched out something to pretend I was working that day, and sent him a photo of it. We ended up shipping that design and I only really made one major mistake that we fixed in v2.
I think the "But this is not intelligence because it is known math" is not a correct argument. It is unknown how the overall higher intelligence of humans works.
What I do notice however is that LLMs are becoming capable of doing an increasing part of the intellectual work I can do, and usually a lot faster.
Just today I presented an agent framework that can take an informal incident statement and propose infrastructure changes to fix it, all evidence backed. This did nothing I could not to, but it did all 5 test cases in 6 - 12 minutes each. I would have found all of the monitoring indications it did, but it would have taken me a day per test case. The LLM also included sass to silly tickets. ("This is not even worth spending monitoring resources on. It's obviously a configuration problem.")
That's how this is reading to me as well. It's just fast at slogging through a certain level of "simple" transformations.
But how do you conclude then that they are less safe? They are neither proven to be more, less, or equally safe. So any statement of "more injuries" is as unfounded as "less injuries" or "just as many injuries".
That seems like the same statement, unless we apply stochastic class level subtlety.
If a driver doesn't get into a crash after a single mile, you can't conclude that they are safer than the average driver. If a driver does get involved in a crash after a single mile, it's very likely that they are not safer than the average driver. It could of course be extraordinarily bad luck, but that's unlikely.
> If a driver does get involved in a crash after a single mile, it's very likely that they are not safer than the average driver. It could of course be extraordinarily bad luck, but that's unlikely.
This is not that simple. Emergency service vehicles are at a very high risk of collision when they pull out of the station and crossings close to the station, or when leaving a situation under time pressure. Busses are at an increased risk of collision when pulling away from a bus stop. Construction vehicles are at a high risk of collision when they pull away from a construction site at a busy road.
Hence why you need more data, after which average accidents per mile are a much stronger indicator.
Hence why I went for the irregular strange part on the lower left first, and then the more regular part on the right. That's the weird rain collection, herb garden, thing we have at the back.
reply