> It's better then incompetents code, worse then a motivated average dev...
That's well put.
> But good enough hence the real question is value aka time& money invested/quality.
There's time invested SO FAR and time that will have to be invested to maintain it. In my experience, even with Fable, it's not there yet. It's the reason why it's easy to vibe code an app from scratch, but at some point when complexity significantly increases, the codebase becomes a mess.
> I think "absolute crap but still useful to me" is a pretty high value and not worth neglecting
If I extrapolate this example to my professional life, this code now manages millions of dollars, a single mistake can wipe it all out, it has to be maintained by 5 other engineers and understood by 5 other domain experts.
That's a real risk, yes. There's definitely a difference between cases where the code is the artifact being built, and cases where the code is being built to generate an artifact that is independently valuable and assessable. I'm happily in the latter case -- but I definitely share your fear of those who think current LLMs are valuable in the former case.
AI is great at reducing the search space and using human-like reasoning (in a brute-force way) to carry out the brute-force search. I'm not surprised by this result. This is exactly what AI should excel at, with human guidance.
"using human-like reasoning (in a brute-force way)"
that's self-contradictory -- what brute force means is doing an exhaustive search of a search space (brute forcing it)
using human-like(?) reasoning means cutting down the search space by having some sort of insight or intuition which allows you to prune branches from the entire tree
That's not what happened here. This isn't a proof; it's a counterexample. The model was perfectly capable of verifying its correctness. You could have verified it by hand if you wanted; the verification is trivial. Finding it was the hard part.
Are you doing the "LLMs don't know how many R's are in 'Raspberry'" thing here?
A bunch of people on the original thread about the Conjecture were like "we need to wait and see if real mathematicians verify this proof it's probably just LLM psychosis", because they don't understand that the counterexample is a trivial calculation. Checking it isn't hard; any AP calc student can do it quickly. It's finding the counterexample that's the challenge.
It's as if someone presented a SHA-2 collision, which anyone could just feed to `openssl sha256` to see, and then naysayers were like "we need to wait for independent verification because the LLM can't know if that collision was real".
Checking, it how? I mean can you describe the mechanism? How does an LLM "check" that something is (or isn't) a counterexample? Does that involve pattern matching, formal verification or model checking, some other approach?
In this case, the same way I would -- calling out to SymPy.
Once you verify (with SymPy or another CAS, like Mathematica) that the given function has the claimed Jacobian determinant (which involves only taking partial derivatives and taking a determinant) and the given inputs map to the same output (which involves only evaluating polynomials with some inputs), you're done.
>> using human-like(?) reasoning means cutting down the search space by having some sort of insight or intuition which allows you to prune branches from the entire tree
So according to the OP there's a search of a tree and it also uses pruning btw, so I want to know what search they mean. Why are you asking?
Sorry, that's just egregious abuse of terminology. Either you're pruning the branches of a tree (or other graph, potentially) when you're doing a tree search, or you don't do pruning. Specifically advice taking (if that's what you mean) is not pruning.
It's a direct quotation from your comment, not a "scare quote". How does "implicit search" do pruning? Can you explain?
Edit: Turns out "implicit search" is actually a thing in the literature, albeit introduced in a single paper I could find that claims Diffusion Modelling does it:
who claimed that? Are you suggesting that the rewrite did not introduce any new bugs? The correct answer is, by the way, that no one knows since it's millions of lines of code no one has properly read.
> smaller, faster
I thought this has already been debunked. You could just write better zig and make it smaller, faster (and have fewer bugs!)
It's not even about requirements. It's about responsibility. Someone has to take responsibility for the code and the product. Someone has to hotfix a bug that's costing millions of dollars an hour and someone has to be blamed for a bug that's consting millions of dollars an hour.
Not really. The secret of our society is it wouldn't work if people with a controlling interest in things ever faced personal responsibility. Who got blamed for the Iraq war? How much did that cost? The theory is if you tried to blame someone then some future president wouldn't be able to declare war. You don't know how the law works apparently. Corporate law is pretty much the same. The economy doesn't boom if CEOs can actually be blamed for things. You don't blame a "someone" that's what incorporation does. If some code does a booboo just put a $ amount on it, withdraw the funds and move on. The media will focus on the humans because of the drama, but they are largely irrelevant. It doesn't come out of their pockets. A human is already window dressing in this status quo. A LLM has no reason to do the sorts of things that violate such legal protections, be it of a president (none really apply) or a CEO (good luck getting a CEO to take personal responsibility for anything short of outright fraud). If it suddenly goes haywire the law already has a concept of vis major.
The secret is individual humans are already largely irrelevant to our society. That's how we got here. The trend will only continue.
The infinite monkey theorem assumes random distribution of symbols*.
Given the tokenizers have a vocabulary in the 10k-100k range, "a million attempts" will generally still only get the first token of the answer correct.
Even really rubbish models, e.g. talkie, the "what if we only use pre-1930s data to train a model?"** model, had to be almost all the way to the right answer to reach the really low HumanEval pass@100 score of ~0.04 (I'm only eyeballing the relevant chart).
* Actual monkeys not being like this is, while amusing, irrelevant
Even if every atom in the universe were a supercomputer generating a trillion trillion random characters every second since the Big Bang, the chance of producing Hamlet would still be essentially zero.
People will downvote you because this comment is "not appropriate" for HN, but there were countless conversations on HN about how important these benchmarks are.
Indeed, much of the scariness is how fearlessly and confidently it writes them with little regard to their actual usefulness or value. When I find it adding a lot of tests, I often say something like: "audit each test carefully, and consider whether the test is testing a meaningful boundary or is more ceremonial. delete low-value tests and add new tests to cover meaningful boundaries not exercised by the gaps you identify". Without fail, this always produces some decent results.
Having said that, in truth, I almost never read the unit tests. Before AI, we had almost none (see: several person game studio) so the tradeoff is not "AI-generated tests" vs "human written ones", it's whether we have tests at all. So, I take them for what they're worth - not much - but if it catches an extra regression before it ships every now and then, it was worth it for the price (~free).
I notice that the tests are very often for its own benefit. Like, you’ll ask it to stop doing something, literally to just remove code, and it will write a test to verify the behavior is not there.
I can’t imagine a more useless test, but I get that it wants to verify that it actually made the change. I just delete the test when it’s done.
That's well put.
> But good enough hence the real question is value aka time& money invested/quality.
There's time invested SO FAR and time that will have to be invested to maintain it. In my experience, even with Fable, it's not there yet. It's the reason why it's easy to vibe code an app from scratch, but at some point when complexity significantly increases, the codebase becomes a mess.
reply