My skin color means I can't possibly "know" what it's like for police to follow you and question you without cause, for clerks to be rude to you, or for taxis that you hail to drive by you... except, you know, I can and do. However, I still get talked at like I didn't live for over two decades in a place in which about .56% of the population was not visibly Asian in appearance and I was treated like an alien who'd just stepped off the mother ship on a daily basis.
The truth is that, having been snubbed by businesses in Japan, I know what it is like to be on the other side. To the extent that I don't get treated that way now, I can understand the concept of "white privilege", though I quibble with the vocabulary choice. I think that what it really is is "minority disadvantage", but the words were changed at some point to make white folks feel like they're getting something they don't deserve and didn't earn instead of focusing on the fact that other people aren't getting something they deserve and shouldn't have to earn. It's meant to make one group feel guilty for what it has, rather than make that same group feel sorry for those who don't have something they should. Somewhere along the line, somebody figured making the majority feel bad created a better prospect for change than relying on their empathy and compassion for those who were being denied basic human rights and dignity.
Because of my 23 years in Japan, I know what it's like to be treated like dirt because of your skin color, and the taxis in Tokyo that drove by when we tried to flag them down were one of the better indicators of that. I don't miss the way in which it made me feel like a lesser human being.