Kind of interesting that LLMs are basically being sold as having “human-like” reasoning capabilities, but in this case when “obamawhitehouse” asked to have it’s password reset sent to bob12345667@gmail.com the LLM didn’t question it and just triggered the process that happened to have a bug.
Humans support agents certainly fall prey to social engineering all the time, but I can’t think of a case where it was done on this scale so easily.
It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.
For sure. Social engineering attacks on human support staff are common and well known, but the skill floor is non-trivial; you need to actually be able to convince a human of your ruse.
Having a support agent likely made it easier to enumerate the vuln, and certainly made it easier to scale out exploitation once it was discovered.
