It really depends on how sensitive you are to rubbing shoulders with the Hoi-Polloi. I actually prefer to live near the 'red-light' district to the high-rent Seefeld area, because I see different coloured faces, shop in multi-culti stores and enjoy the rough with the smooth. I guess I'm in the minority
As for raising a family, I can't comment with personal experience but know a few people even in Kreis 4 and 5 (city quarters 4 and 5) who are quite happy for their kids to see the druggies, the pimps and the destitute as a balance to the Ferraris and expensive wives of the Bahnhofstrasse.
I grew up in a rough area of Liverpool but went to a decent school, and value the education learnt on the street above the Latin I learned at school! All the luxury here makes you soft if you have no comparison.
So where to live? Traditional dilemmas, really: cosmopolitan or rural? House or appartment? Where ever you settle, you'll find almost all neighbourhoods full of ammenities and good transport links. Also a good mix of the 'classes', for what they are here. It's a great place to live. Personally I'd be as close to the centre as possible, as I find the outer suburbs - the 'Ausser Sihl' (area beyond the river Sihl) - and the semi-rural areas deadly boring.