It is rather well-known that, with VGA-compatible PC video adapters, in black-and-white text modes video memory is available at linear address 0xB0000, in colour text mode at address 0xB8000, while in graphics mode, it appears at address 0xA0000. Ralf Brown’s has a whole table showing which video modes use which segment bases.
It seems it would have been simpler to always place the video memory at the same address. Indeed, this design created some complications when a protected-mode video ROM interface was created; an operating system had to allocate three selectors to video memory to use it, instead of just one.
What is the reason for the changing memory mapping that we ended up with? Was it a deliberate design that served a useful purpose, or purely an accident, a compromise between an attempted do-over and backwards compatibility?