I guess everyone has forgotten the northern states (as well as every other country) had slaves too.
As others have stated, they didn't have as many due to the lack of agriculture, but you're kidding yourselves if you think there weren't slaves in the north!