To be honest, it does sound racist when you blatantly say it out loud.
But on the other hand, it's not wrong either. You have your own preference, and are entitled to that. So it's your freedom of choice. I'm Asian, but even I have some people I'm not 'fond of'. For example, eventhough I like 'white men', but there some types from a selective state/region whom I probably want to avoid as best as I could. In a way, that might make me racist too, right?
Just try not to offend people of other races just because you do. And probably opt to use a more subtle manner of speech to convey the message. Something like "Sorry, you're not my type" instead of "Sorry, I don't like Black/Asian/POC". Be considerate of other's feelings, that's all that matters.