I look at it this way: if you had an annoying friend, an abusive boyfriend or spouse, someone with a severe personality disorder (who doesn't try to deal with it), someone who mentally/emotionally abuses you, who constantly belittles you, etc., I think there would be near-universal agreement that you should disengage, stop trying to be friends, divorce them, do what you need to do to protect yourself, both physically and emotionally.
So why should it be different just because it's a family member. Life is too short to put up that shit and you're just not going to be able to "fix" them or the situation. Set boundaries, disengage, even terminate all contact if that's what it takes. Your life will be significantly improved.
