I lived in victoria for around 10 years... nice place. We used to say that victoria was for "the newly wed and the nearly dead", because of the visibly obvious age groups there.
It's a little too sleepy for my tastes, though. Seems like they roll up the streets at 9pm, and after hours entertainment you can find is a bit of a crap shoot. If you're looking for world class night life, don't expect to find it on the island. I moved to vancouver and it was much more my speed. Still, living in victoria means that vancouver (and whistler) are just a few hours away by ferry, so you can get the best of both. Good luck getting on the ferry when symphony of fire or a major holliday happens tho.
It's a beautiful place to live... very very green, and generally lacks that brown haze on the skyline. Crime is low even by canadian standards and there's a surprisingly diverse collection of ethnic food and cultural offerings. And yes, it is ground zero for the best chronic on the planet, so if you're down with the sickness it's the #1 place to be.
