I think it would be foolhardy to travel outside Switzerland without your passport...
The residency permit is helpful for indicating your permission to live in Switzerland, but if we had a problem the first place we would be running to would be the Australian embassy, with our Aussie passports...