You could, but there's two big reasons why this isn't useful:
1. The translation system would still have to implement the underlying ideas. For example, your spec might explain "key frames are made by doing a wavelet transform ABC, dropping frequency XYZ, applying Heuristic Alpha and then doing LZ encoding on the result" or something like that. You still need implementations of those specific operations. Having to explain all that stuff in the spec would mean each english language doc would be book length tretsie on encoding. As a result, you don't really gain much by doing this kind of meta-explaination.
2. Video encoding/decoding is something where optimization is extremely important. Writing code that is specific to a single purpose is generally going to be faster than generic code that could be applied to any theoretical format. For example, by knowing the exact size of a transformation matrix, you can decide whether to build a lookup table or calculate on the fly - you can use hardware level features that require knowing exact sizes at build-time etc.
Using good ol' ISO specs (or written specs in general) is such a fun game to play. What the spec says and what hardware/software actually do are not always the same. I worked on a project for a TV manufacturer that provided us the specs for what their chips in their new TV could do. Naturally, we pushed the encoding to the limits of the specs at first, and then had to start dialing things back to what was actually happening. If the spec says max bitrate of Xmpbs, it turns out that it was closer to .85Xmpbs to get decent playback. Or a cable provider's new digital STB that couldn't handle large changes in VBV even thought it was "within spec"
