I'm a non-EU passport holder and now hold a B permit. Can I go to Germany with B permit but without passport?

Short answer: No.
Foreign nationals from non-EU/EFTA states – travelling in the Schengen area Foreign nationals from non-EU/EFTA states in possession of a valid Swiss foreign national permit B, C, L or Ci do not require a visa to visit a country in the Schengen area (up to 90 days within a 180-day period). However, you must carry a valid Swiss residence permit and valid travel documents (your passport or a Swiss travel document for foreign nationals).
https://www.ch.ch/en/travelling-abroad/

Hi! I am a non-EU citizen and hold a B permit, same as you. I go to Germany (by train) every month and always carry with me my passport and my B permit. I have never been requested to show my documents, I just go and come back without problems. Hope it helps!

Yes..But it may still be advisable to carry your B permit and passport. Just in case.

You need to check the German government for acceptable ID in the Schengen area for non-EU/EEA nationals. Each country can be different, for example US Service Personnel can get by with their Militiary ID for instance. The permit is more for indicating you dont require a visa as a Schengen area resident, as CH is non-EU.

As it is written on the paper you received together with your B-Permit , your permit is only good with a valid passport at the customs.