So the jist of the replies is, have your preference, but don't say it because that would racist.
I'm wondering if the same people would say the same thing if a black guy said: "black only".
I wouldn't because I would think of a history of repression and violence there and think that a black gay person might feel safer with another black guy and there isn't history of black men oppressing white men. I don't think the same applies to white men, just like the same things don't apply to straight people as they do gay people. I don't know if I'm explaining it well. I would think it was a shame that the black guy was limiting himself, just like a would the white gay guy.
However I'm just now remembering that there are in fact other races! So thinking a step further I would think that was shitty too. Maybe not as shitty (I'm not black so I would not pretend I would automatically know all the reasons for him saying "black only") but still kinda shitty towards asians etc.