I think the answer depends upon whether or not you are classified as being domiciled in the UK. If you are are british citizen and have family, bank accounts and plan on returning there long term in the future then the chances are you are.
If you are working in the UK for a sufficient number of days a year you will be tax resident in the UK. If you are domiciled and tax resident in the UK then the UK can tax you on your worldwide income, i.e. the Swiss rental income will be subject to UK tax. You should receive a tax credit if the rental income is also taxable in the Switzerland. So in effect you are taxed at the higher UK tax rate.
If you are not domiciled in the UK then I do not think they have the UK has a right to tax as the asset from which the income is derived is not situated in the UK.
I could of course be entirely wrong