I'm not familiar with the Brits so can't comment on the specifics there.
However, as a kid, I had a similar experience in a completely different country when we moved cities. My accent wasn't "posh" or "higher class" in any way, it was just from a different region. Kids would give me a hard time for it. But the exact same would happen in reverse form in the other region.
In the UK people from Liverpool and Manchester are rivals until they meet someone from London when it becomes a North Vs South thing. That all changes again when they meet someone from Glasgow when it becomes England Vs Scotland and yet again when the British meet someone French. There is always a more foreign foe.
The concept of the everchanging {in-group vs. out-group} is especially present in Japanese culture. It strongly affects wordchoice, too. Within a family, father might be chichi or, in more formal families, otousan, but it speaking of fathers to someone outside your family, you might refer to your father as chichi, and any other father as otousan.
However, as a kid, I had a similar experience in a completely different country when we moved cities. My accent wasn't "posh" or "higher class" in any way, it was just from a different region. Kids would give me a hard time for it. But the exact same would happen in reverse form in the other region.
Guess people just don't like "outsiders".