I am one of those people (and they are thousands of us if not millions) who always know where north is, which means where every other compass point is. Blindfold me, drive me for days to somewhere I've never been, spin me around until I'm dizzy and ask me to point to the north and I do it every time. I don't know how.
What bugs me is not so much colour. Yeah, white text on yellow is just stupid but it's not really about colour. It's about contrast. And online restaurant menus are among the worst!
At least in the EU and I think still the UK there is legislation to ensure website are visually accessible. I am not sure how well that is enforced, however.