Doctor here (LOL, sorry for my English, in advance)
If you have a cold, with fever (more common with influenza), your erectile function could be impaired (more impaired as higher is the fever). This is a "top" problem, of course ;).
Of course, common cold is highly contagious. And, of course, no one cares. Anyway, you can equally infect you partner kissing him, sleeping with him, touching him, coughing and sneezing, so sexual transmission is not really a significant concern.
However, depending on your subjective perception (and tollerance) of symptoms (runny nose and congestion, sore throat, headache), you can be uncomfortable. Oh, well, I was forgetting to tell that exercise is a natural decongestive. I think is more up to your comfort, than the concern about your partner health.
Take care