So my question is, if I have a valid Schengen visa/permit, can I enter Bulgaria (I do need a visa to enter Bulgaria otherwise)?

I found some references, but not so sure about the wording.

On 25 January 2012 the Council of Ministers of the Republic of Bulgaria adopted a decision according to which by the date of Bulgaria’s accession to the Schengen area, our country will unilaterally apply a visa-free system for holders of valid Schengen visas. They will have the right to enter and reside in the Republic of Bulgaria for a period of no more than three months in any six-month period from the date of the first entry, without needing to have a Bulgarian short-stay visa. The decision entered into force on 31.01.2012.

http://www.mfa.bg/en/pages/109/index.html

Yes. Just take your passport and Swiss residence permit with you.