Yesterday I donwvoted this answer (and upvoted a comment highlighting the problem). Today I saw I received -1 rep, with a link to this answer (which had received another downvote since mine). I really don't care about 1 point of rep, but what happened?
There is a cost to downvoting answers so that people do not go about downvoting willy-nilly. If that -1 weren't there, I think there would be a lot less working with answerer in the comments. There would also be more "I am going to downvote every other answer to this question so mine will rise to the top"--which thankfully, I don't see very often.
BTW, I believe you will get your reputation back eventually if that question is deleted. Thus there is no permanent reputation cost to downvoting spam or racist tirades.