If it's wrong to kill fetuses, then why isn't it wrong to destroy sperm?

A lot of pro-lifers will say that it's immoral to kill a fetus because eventually they will grow up into normal adult human beings. Pro-lifers will say that it's wrong to deny a fetus of that future. But why is it wrong to destroy a sperm which also has the potential to impregnate an egg and then develop into a fully functioning adult human?