To be clear, the visa you will apply for is a Type D visa for family reunification. This will be stamped in your passport, and it allows you to enter for the purpose of staying in Switzerland long-term (as opposed to a short-term tourist visa).
Once you are here, you will register with the local authorities and apply for a residence permit. This doubles as your authorization to work. After you have the residence permit, you can enter and exit the country as you wish and the D-visa is no longer relevant.
Hope that helps.