First: For spam, you can cast a flag instead, there is a specific flag for 'spam - Exists only to promote a product or service'.
For downvotes, the user being downvoted gets -2 reps, the person voting down gets -1. If the answer / post is then deleted, all reputation is given back.
The reasoning behind the person voting down losing a point is somewhat given for example here. This is designed to make people think twice before downvoting, to not just do it on reflex or as a "revenge".
Also, from here:
The motivation behind it is to put emphasis on up-voting or not voting at all. This way, down votes will carry more weight and it will also prevent users from abusing the system by down-voting excessively.
According to what Jeff/Joel discussed on the SO podcast, they wanted to find a way to discourage users from down-voting for less legitimate reasons (say a pro-Java developer down-voting everything remotely related to .NET or the like).