I haven't experienced this. Rust's build system allows you to automate most of the pain in building/bundling those dependencies, so it's just up to the builder to have the right headers (if they're not bundled). That makes it no worse than meson C/C++ builds.
In Go, it doesn't have any of that so it depends on the end builder of the binary to be aware of all the C dependencies and manually manage integrating the cgo configuration into their build process.
Its highly dependent on the kind of project. A few years back libxml and libxmlsec1 were a consistent pain in the neck for me. I don't know about now, I simply moved on from the project that needed me to interact with them.
I don’t hate the C language. I hate the C compiler (and the rest of the toolchain). Anything that helps me not interact with the C compiler is a huge win.