Dedicated servers are faster by nature than virtual servers with thousands of little sites bogging them down. Google seems very concerned with page-load times lately, so you can't write it off as unimportant.
As far as IPs, that only matters if you have lots of small sites. If you link all of your sites together on the same IP address, you're going to have an uphill battle trying to get them all to rank. However, it's not that hard for Google to figure out algorithmically if you use the same linking schema for all your sites that those sites are all by the same webmaster.
Worry about your on-site SEO, and worry about link building. When you have those under control, worry about hosting.
