Angelina, don't worry. I didn't mean to be rude, but this sort of chauvinistic behaviour he had towards you makes my blood boil. Going down the 'accusations route' against the noble local swiss man would be very hard and if you have no witness, it makes things even worse. He is a coward as far as I can see and prey on the vulnerable.
Go through the legal route then, swissbob and miniMia gave you great advise about the renters association, plus what I said previously. Gather information, proof, notes he left you, txt messages, etc. Don't let him get you down.