Yes, correct. Essentially every single industry and tool which rents out capacity of any system or service does this. Your ISP does this. The airline does this. Cruise lines. Cloud computing environments. Restaurants. Rental cars. The list is endless.
Exploring a codebase tells you WHAT it's doing, but not WHY. In older codebases you'll often find weird sections of code that solved a problem that may or may not still exist. Like maybe there was an import process that always left three carriage returns at the end of each record, so now you got some funky "lets remove up to three carriage returns" function that probably isn't needed. But are you 100% sure it's not needed?
Same story with data models, let's say you have the same data (customer contact details) in slightly different formats in 5 different data models. Which one is correct? Why are the others different?
Ultimately someone has to solve this mystery and that often means pulling people together from different parts of the business, so they can eventually reach consensus on how to move forward.
Adding that this just gets worse when databases are peppered with direct access by vibe-coded applications that don’t look at production data or gather these insights before deciding “yeah this sounds like the format of text that should go in the column with this name, and that’s the column I should use.”
And now there’s an example in the codebase of what not to do, and other AI sessions will see it, and follow that pattern blindly, and… well, we all know where this goes.
How is an AI supposed to create documentation, except the most useless box-ticking kind? It only sees the existing implementation, so the best it can do is describe what you can already see (maybe with some stupid guesses added in).
IMHO, if you're going to use AI to "write documentation," that's disposable text and not for distribution. Let the next guy generate his own, and he'll be under no illusions about where the text he's reading came from.
If you're going to write documentation to distribute, you had better type out words from your own damn mind based on your own damn understanding with your own damn hands. Sure, use an LLM to help understand something, but if you personally don't understand, you're in no position to document anything.
Whats with this assumption that there's no human involvement? I dont just say "hey scan this 2m loc repo and give me some docs'... that would be insane. T
he AI is there to do the easy part; scan a giant spaghetti bowl and label each noodle. The humans job is to attach descriptions to those noodles.
Sometimes I forget that people on this site simply assume the worst in any given situation.
I don't find this surprising. Code and data models encode the results of accumulated business decisions, but nothing about the decision making process or rationale. Most of the time, this information is stored only in people's heads, so any automated tool is necessary blind.
This captures succinctly the one of the key issues with (current) AI actually solving real problems outside of small "sandboxes" where it has all the information.
When an AI can email/message all the key people that have the institutional knowledge, ask them the right discovery questions (probably in a few rounds and working out which bits are human "hallucinations" that don't make sense). Collect that information and use it to create a solution. Then human jobs are in real trouble.
Until that AI is just a productivity boost for us.
The AI will also have to be trained to be diplomatic and maybe even cunning, because, as I can personally attest, answering questions from an AI is an extremely grating and disillusioning experience.
There are plenty of workers who refuse to answer questions from a human until it’s escalated far enough up the chain to affect their paycheck / reputation. I’m sure that the intelligence is artificial will only multiply the disdain / noncompliance.
But then maybe there will be strategies for masking from where requests are coming, like a system that anonymizes all requests for information. Even so, I feel like there would still be a way that people would ping / walk up to their colleague in meatspace and say “hey that request came from me, thanks!”
i love the assumption by default that "ai generated" automatically excludes "human verified".
see, i actually read and monitor the outputs. i check them against my own internal knowledge. i trial the results with real trouble shooting and real bug fixes/feature requests.
when its wrong, i fix it. when its right, great we now have documentation where none existed before.
dogfood the documentation and you'll know if its worth using or not.
Literally several times a week, I have to close PRs with docs that clearly no one read because they are blatantly wrong. This happened after LLMs. If what you're claiming is happening, I'm not seeing it anywhere.
AI is incapable of capturing human context that 99.999% of the time exists in people's brains, not code or context. This is why it is crucial that humans write for humans, not an LLM that puts out docs that have the aesthetics of looking acceptable.
I am similarly interested, but mostly because my memory is awful and I'd like to actually remember what people tell me without having to ask repeatedly.
Write your own tools. Dont use something off the shelf. If you want it to read from a database, create a db connector that exposes only the capabilities you want it to have.
This is what I do, and I am 100% confident that Claude cannot drop my database or truncate a table, or read from sensitive tables.
I know this because the tool it uses to interface with the database doesn't have those capabilities, thus Claude doesn't have that capability.
It won't save you from Claude maliciously ex-filtrating data it has access to via DNS or some other side channel, but it will protect from worst-case scenarios.
This is like trying to fix SQL injection by limiting the permissions of the database user instead of using parameterized queries (for which there is no equivalent with LLMs). It doesn't solve the problem.
It also has no effect on whole classes of vulnerabilities which don't rely on unusual writes, where the system (SQL or LLM) is expected to execute some logic and yield a result, and the attacker wins by determining the outcome.
Using the SQL analogy, suppose this is intended:
SELECT hash('$input') == secretfiles.hashed_access_code FROM secretfiles WHERE secretfiles.id = '$file_id';
And here the attacker supplying a malicious $input so that it becomes something else with a comment on the end:
SELECT hash('') == hash('') -- ') == secretfiles.hashed_access_code FROM secretfiles WHERE secretfiles.id = '123';
> the tool it uses to interface with the database doesn't have those capabilities
Fair enough. It can e.g. use a DB user with read-only privileges or something like that. Or it might sanitize the allowed queries.
But there may still be some way to drop the database or delete all its data which your tool might not be able to guard against. Some indirect deletions made by a trigger or a stored procedure or something like that, for instance.
The point is, your tool might be relatively safe. But I would be cautious when saying that it is "100 %" safe, as you claim.
That being said, I think that your point still stands. Given safe enough interfaces between the LLM and the other parts of the system, one can be fairly sure that the actions performed by the LLM would be safe.
This is reminding me of the crypto self-custody problem. If you want complete trustlessness, the lengths you have to go to are extreme. How do you really know that the machine using your private key to sign your transactions is absolutely secure?
What makes you think the dbcredentials or IP are being exposed to Claude? The entire reason I build my own connectors is to avoid having to expose details like that.
What I give Claude is an API key that allows it to talk to the mcp server. Everything else is hidden behind that.
Unclear why this is being downvoted. It makes sense.
If you connect to the database with a connector that only has read access, then the LLM cannot drop the database, period.
If that were bugged (e.g. if Postgres allowed writing to a DB that was configured readonly), then that problem is much bigger has not much to do with LLMs.
No issue at all. There is a place for stored procs and functions in cases where you need to do things an ORM is not capable of. It is an exception, not a rule. Managing procs/functions is overhead and has the same if not more maintenance headaches than raw SQL strings in code.
reply