I think you may be begging the question when you assume that there must be "actual stuff" that is forever beyond physical explanation.
I suspect that one is more likely to understand "green" by realizing that it is a word (originally, a sound) that you have come to associate with memories of visual experiences. To say something is green is to do a comparison between an aspect of the experience of it, and aspects of some of those memories of prior experiences.
I don't assume that the stuff is forever beyond physical explaination, and like I said we can perform science on the behavior of it. But ultimately the scientific method will always conclude that its ultimately an illusion and that there is only the word green, or the recognition of 532nm light. A word doesn't even need be assigned to it, its witnessed either way.
I use color as an example because its very distant and theoretically could be many different ways (e.g. your experience of green may not be made of the same stuff my green is). However geometry seems far less arbitrary. Visualizing a square is more specific, the way the lines relate to space in the 2D plane if misrepresented will be wrong. I doubt we visualize such things with 100% accuracy, but we must visualize them to a certain common accuracy, and thats important, common. Basically, as far as experiencing the visual concept of a square, we all experience roughly the same object in our conscious experience. Thats where things can be studied a bit, because the behavior of consciousness given certain logic is likely absolute.
I think the difficult part of all this is that "experience" has been irreducible so far and extremely difficult to isolate as some kind of physical substance or variable. We can analyze the "actual stuff" in every dimension we know how, and still come no closer to explaining why it feels like something to experience green (borrowing from Thomas Nagel here). A starcraft-playing neural net can make a comparison between some of its earlier "memories" and what it's seeing now, but that says nothing about its Awareness.
Any more than it is begging the question to assume that there cannot be anything that is beyond physical explanation?
Yes, words are symbols. But when we're talking about consciousness, we're talking about the experience itself, not the association between the word and the experience. And we have no understanding of why or how the experience exists in us.
A willingness to search for physical explanations should not be mistaken for claiming that there is one simply because there must be one, which would be as fallacious as claiming that there isn't one simply because there can not be one. This is a distinct fallacy from the one of claiming that there isn't one because I can't imagine that there could be one, or that it can't be done because it hasn't been done so far.
As for having an opinion, on the other hand, that usually depends on which remarkable circumstance one thinks is less implausible.
I suspect that one is more likely to understand "green" by realizing that it is a word (originally, a sound) that you have come to associate with memories of visual experiences. To say something is green is to do a comparison between an aspect of the experience of it, and aspects of some of those memories of prior experiences.