I also use OpenWRT with SQM (using cake queuing discipline) on a CM4 Pi to address buffer bloat. I don’t see any red blips, my latency is consistently between 20-40ms on Wi-Fi.

What hardware do you have? SQM is pretty CPU intensive. Mine is just powerful enough to be able to use SQM. Try checking if your cpu usage spikes from softirq interrupts.