The limitation is efficiency and efficacy. If you have to add an additional layer of inference to any request you’re negatively impacting your bottom line so the companies, which are compute bound, have a strong incentive to squeeze everything into a single forward pass. It’s also not clear that a separate model that is smaller than the main model will perform better than just training the main model to detect prompt injection. They are both probabilistic models that have no structural way of distinguishing user input from malicious instructions.
If you’re getting prompt injected, you have skipped right passed thinking critically about what you’re doing and into the same level of intellectual dishonesty as cheating, ie, not learning the thing and then attempting to still attain a grade for work not done.
Agreed, but already in this same comment section there are people speculating on ways to defeat this, like a small model just to detect prompt injections. Students will catch on quick, and any novel trick you deploy will be killed by word-of-mouth once the first round of grades come back. I understand the need to do something, but it feels like a band-aid solution on a hemorrhaging gash. I don't think 'AI traps' are a viable solution moving forward for education
But debugging mutable state is much easier than debugging a distributed system. Even in C if some global gets mishandled I can just use: gdb, dtrace, strace or even just look at a core dump and know that whatever caused the problem will be discoverable. I have no such guarantee debugging an issue across a distributed systems service boundary.
It seems entirely logical that if an llm allows each ic to do the work of a 2-3 person team (debatable, but assume it’s true for the sake of argument) then you’ve effectively just added a layer to the org chart, meaning any tool that was effective for the next scale up of org becomes a requirement for managing a smaller team.
What should give anyone pause about this notion is that historically, by far the most effective teams have been small teams of experts focusing on their key competencies and points of comparative advantage. Large organizations tend to be slower, more bureaucratic and less effective at executing because of the added weight of communication and disconnect between execution and intent.
If you want to be effective with llms, it seems like there are a lot of lessons to learn about what makes human teams effective before we turn ourselves into an industry filled with clueless middle managers.
That's a very good point. Although I think maybe there are some crucial differences with LLMs here.
First, the extra speed makes a qualitative difference. There is some communication overhead when you're instructing an LLM rather than just doing the work directly, but the LLM is usually so fast that it doesn't necessarily slow you down.
Second, the lack of ego is a big deal. When reviewing LLM code, I have remind myself that it's okay to ask for sweeping changes, or even completely change the design because I'm not happy with how it turned out. The only cost is extra tokens -- it doesn't take much time, nobody's ego gets bruised, team morale doesn't suffer.
This might be an area where LLMs are able to follow human best practices better than humans themselves can. It's good to explore the design space with throwaway prototypes, but I think people are often too reluctant to throw code away and are tempted to try to reuse it.
3.5-plus was also only available via api. I don’t know what the long term business model for open weights is, I hope there is one, but it seems foolish to assume that companies will be willing to spend millions of dollars of compute on an asset worth zero in perpetuity.
To a certain extent, I do wonder if just letting claude do everything and then using the bug reports and CVE’s they find as training data for an RL environment might be part of the plan. “Here’s what you did, here’s what fixed it, don’t fuck up like that again"
I’ve heard this take before, but if you’ve spent any time with llm’s I don’t understand how your take can be: “I should just let this thing that makes mistakes all the time and seems oblivious to the complexity it’s creating because it only observes small snippets out of context make it’s own decisions about architecture, this is just how it does things and I shouldn’t question it.”
I don’t know that those people were exactly out of a job though, they didn’t do that job, but I find it hard to believe that any of the people solving orbital mechanics by hand wound up with nothing to do but twiddle their thumbs for the remainder of their lives. Similarly, I don’t know that there’s any realistic prospect, even if ai winds up writing all the software, that there wont also be incentive to have people that also understand it.
Once upon a time, I heard someone tell me a fairytale about this thing called a ‘law' and they said that laws could be used to enforce compliance with standards across an entire country. Pure fantasy I know, but a man can dream.
reply