At the end of tazria, the passuk in 13:46 says "All the days he has the Nega, he will surely be impure, he shall sit isolated, outside of the camp will he sit."
Rashi on the passuk says that he has to be isolated even from other people who are tamei sitting outside the camp. Even outside the camp he is by himself. Rashi asks what is different about the tzoraas that he cannot even be with other tamei people?
Rashi brings from Chazal that because he spoke Lashon Ha'Ra, thereby causing a division between husband and wife (and between person and friend), he therefore has to sit isolated, separate from others.
A husband and wife are meant to be together. Even if they do not always agree on everything, and they will not, there should not be a division among them. Despite different opinions on some matters, they need to work together for a unified purpose. They are one.
Anything that comes between that, anything that causes a division between them and makes a break in their unity, anything that causes a rift between them, has to be rejected.
And that is why Rashi says a Metzora is so bad. Because he caused a rift between man and wife. He broke that unity that is so essential for marriage, for Jewish continuity.