What is legally possible is one thing. As MF said, a family reunification permit is to be together in Switzerland.

Nasty neighbours might notice that you are always gone for a week or more every month and your tax office will notice that you have income from work abroad. So make sure that you are very sure you are ok to do this.