@eobox91103 swarm simply refers to the collective of all peers who are active in the distribution of a torrent regardless of whether they are downloading or uploading. in this sense, there is only one swarm.
what you described can happen, still. that is because the tracker, who is responsible for peer discovery, often limits the number of peers that can be discovered, and the peer list you get is determined by the tracker's algorithm. it is possible to end up with a peer list that is not very useful.
but it's worth noting that peer discovery occurs every time your client re-announces itself to the tracker, and you might get another subset of peers.
this is not a protocol rule, each tracker has its own rules. some trackers might report all peers in very specific scenarios, some trackers might report only partial peers in all scenarios.
i will soon write a topic to help elucidate this better, along with other words from torrenting terminology.