Assuming you originally had a GB licence then my understanding is that in this case you are not exchanging a foreign driving licence for a GB one. You are simply asking for your GB licence to be returned.
So far as I know if you meet the requirement about having a UK address then they should simply return it to you. Assuming you are under 70 then it is yours & you paid for it. I do not see why you shoudl have to hand in a foreign licence.
There is this cosy arrangement that you give your GB licence to the Swiss in exchange for a Swiss licence and they then send your GB licence to DVLA.
Question then is "Do the DVLA have any right to keep it & on what legal basis?" I have tried to find the answer to this but with no success so far.

Marton

Misleading, you haven't qualified this partially true statement sufficiently. This only applies if you are already a holder of an EU/EEA license. So for example as you don't need to exchange a Liechtenstein/FL license (EEA) and it remains valid until 70 as per UK, for A & B caegories, you should wait 6 months before exchanging UNLESS you wish to use C, D and other professional categories. In this case you have two options:

1. keep your EU/EEA license and apply for a local UK counterpart
2. exchange your EU/EEA license.

As Switzerland is not in the EU/EEA the six month ruling does not apply.

3 years ago I moved to Germany where a non-EU/EEA (e.g. Swiss) license expires after 6 months, so I called up the UK-DVLA as I had originally exchanged my Swiss license for a UK-GB one. Under the following conditions the exchange back to a UK-GB license is free if:

1. You are returning to the address previously on your UK-GB license
2. Your categories haven't changed since in CH
3. You have no change in medical/optical conditions
4. You exchanged a photocard license and it is still valid (<= 10 years from issue)

I don't know if this applies for a UK-NI license, as they have their own licensing authority in Coleraine with a different driver number range.

Bear in mind it is possible to receive a copy of your driving record upon leaving the UK recording your entitlements. Handy if you lose your UK license before exchanging it in CH. Conversely a similar document is available from the Strassenverkehrsamt for approx. 30CHF here, useful before exchange elsewhere/in case of loss of Swiss license abroad.

In many EU countries exchanging a Swiss license to a local with C, D categories requires a medical. In Germany entitlement of these categories is limited until 50 years of age even after passing the medical which costs €150.

No but it is in Welsh too