I would argue that it is a very popular film released by a major studio, and directed by one of the most popular directors in history. Mulholland Drive is not a film that only "film buffs" saw. I'm not sure if I've every met anyone who hasn't seen a David Lynch film, and I rarely meet anyone who hasn't enjoyed a David Lynch film. He's a multi-millionaire in high demand that entirely dictates his own work conditions.
Of the people I know who have seen a David Lynch movie, I haven’t met anyone who enjoyed them. Speaking for myself, Mulholland Drive is definitely in the running for worst movie I’ve ever seen.
Count me in your second group. I've seen several David Lynch movies, never enjoyed a single one of them. There's something about his film aesthetic but also about his view of what the world really is that just doesn't sit right with me. When I do see them it's because one or more film nerds I respect has raved about it, and I think I should give it a chance. But ... nope, same film aesthetic, same worldview that I just can't enjoy.