There's not many professions where the source of your degree makes a difference. Especially after a few years in the work place. I can't recall it ever being an issue when I was applying for jobs, nor when I was a hiring manager. I've worked with a few Oxford and Cambridge graduates - they've been no worse and no better than anyone else particularly. There were some from other older universities who were a bit snobby, but I just put it down to the inate insecurity they had from not making to Oxbridge. The redbrick graduates couldn't give a toss.
Go where you're happy to go, where you'll enjoy the culture, and where you can get the funding. If you're smart, you'll go to a Swiss university and ask mum and dad for some of the $$$$$ they're saving...