Because organizations rely on internal communication, and remote/on-site/hybrid each require very different processes to manage work and internal communications.
If all your organization processes are set up to work well for on-site, then the effectiveness of the remote people in your organization will hurt; if all your organization processes are set up for remote interactions, then the overheads of that don't make sense for the on-site people sitting next to each other; and if some units are fully on-site and some are fully remote, each with different style of working, then you might as well have two separate organizations with vendor/contractor relationship, that would be more efficient as their collaboration anyway have to be managed that way.
If all your organization processes are set up to work well for on-site, then the effectiveness of the remote people in your organization will hurt; if all your organization processes are set up for remote interactions, then the overheads of that don't make sense for the on-site people sitting next to each other; and if some units are fully on-site and some are fully remote, each with different style of working, then you might as well have two separate organizations with vendor/contractor relationship, that would be more efficient as their collaboration anyway have to be managed that way.