It can inhibit participation. A lot of people automatically downvote anyone with a negative score and upvote anyone with a positive score because of the peer pressure or something. It’s how the brigading and triple digit downvotes often get started.

To be fair, it could have a negative impact on whether or not new users or lurkers get involved.