It's been a few years since abstract algebra, but might it have something to do with finite fields? I seem to recall that modulo primes held special significance (i.e. you can't have a finite field size other than a power of a prime).