Yep, it's hard to take IR off your back if you're british.
Maybe your concern now, as we have established UK taxes are unavoidable, should be not paying anything to the Swiss? If I were you I would confirm with an accountant it is perfectly legal not to register or file for taxes in Switzerland in your situation, but again that's just my opinion. I'm not sure what will your wife's situation be if she's registered and you aren't but I think, I repeat, I'm not sure, allowances such as husband's support are non-taxable in Switzerland so she won't have to pay anything either?