I am constantly getting LLMs to change features and fix bugs. The key is to micromanage the LLM and its context, and read the changes. It's slower that vibe coding but faster than coding by hand, and it results in working, maintainable software.
The comments explain the nuance there pretty well:
> This study had 16 participants, with a mix of previous exposure to AI tools - 56% of them had never used Cursor before, and the study was mainly about Cursor.
> My intuition here is that this study mainly demonstrated that the learning curve on AI-assisted development is high enough that asking developers to bake it into their existing workflows reduces their performance while they climb that learing curve.
Giving people a tool, that have no experience with it, and expecting them to be productive feels... odd?
That's a good point. Myself is the easiest person to fool.
I knocked together a quick analysis of my commit graphs going back several years, if you're interested: https://mccormick.cx/gh/
My average leading up to 2023 was around 2k commits per year. 2023 I started using ChatGPT and I hit my highest commits so far that year at 2,600. 2024 I moved to a different country, which broke my productivity. I started using aider at the end of 2024 and in 2025 I again hit my highest commits ever at 2,900. This year is looking pretty solid.
From this it looks to me like I'm at least 1.4x more productive than before.
As a freelancer I have to track issues closed and hours pretty closely so I can give estimates and updates to clients. My baseline was always "two issues closed per working day". These are issues I create myself (full stack, self-managed freelancer) so the average granularity has stayed roughly constant.
This morning I closed 8 issues on a client project. I estimate I am averaging around 4 issues per working day these days. I know this because I have to actually close the issues each day. So on that metric my productivity has roughly doubled.
I believe those studies for sure. I think there is nuance to using these tools well, and I think a lot of people are going backwards and introducing more bugs than progress through vibe coding. I do not think I have gone backwards, and the metrics I have available seem to agree with that assessment.
I'm sure that would work just like how phone apps asking for permission to do each thing has resulted in no phone user ever getting pwned and doxxed by their apps - phone apps are completely safe now, yay!
I think we're talking about fundamentally different things. :) You're talking about the UX of granting permissions, I'm talking about how permissions get implemented at the technical level, irrespective of how you arrived at them.
Surely your proposed solution is not "Don't implement a permission system to begin with"?
I guess what I am saying is at the end of the day you need the program to do the thing. Whatever mutation it needs to do to accomplish the task, that's what you're going to allow. That's exactly what happens with phone app permissions. Everybody just lets Facebook use their microphone (not me of course, but most people).
What you describe would be super cool though. If every program let you know ahead of time what it was going to try to read and write in the world. That does indeed sound useful!
What do you mean? You can force JSON with structured output.
It was just an example though, in real-world scenarios, sometimes I have to tell the AI to respond in a specific strict format, which is not JSON (e.g. asking it to end with "Good bye!"). Claude is the one who is the worst at following those type of instructions, and because of this it fails to return to correct answer in the correct format, even though the answer itself is good.
i agree that is annoying but seems like anthropic's stance is that the task/agent should be provided an environment to write the file in the output you provide or provided a skill.md description on how to do that specific task.
personally it's a blurry line. most times i'm interacting with an agent where outputting to a file makes sense but it makes it less reliable when treating the model call as a deterministic function call.
There's definitely many ways to improve the output of the AI, and provide it extra hints. Also, some AIs are made for a specific use-case. Maybe I should rephrase it and say that those benchmarks are more about the single-reply intelligence of a model, and more like an AGI test then for specific use-cases.
reply