People would use personal opinion rather than common sense when banning people. Although some are annoying as hell, they don't deserved to be banned. With that system in place, people would be getting banned for not doing fucked up shit but just because one person doesn't like them.
Also, what if that person just went on a banning frenzy and you were banned for absolutely no reason. That would certainly suck.