Basically it varies a bit by canton, until the harmonisation ever gets agreed. In VD, it is basically 4eme-5eme (age approx. 9-10) officially, although have heard most schools have already made up their minds by this time based on previous evaluations and this period is more for confirmation.
In Geneva, there is a website on public schools which states that from the 3eme evaluations are made with grades and based on this decisions are made for each student on whether they continue higher education. The website also says they are put in a welcome class if they do not have French. Do not have any personal experience in GVA, but this seems the norm for Romande area.
Here is the link for public education in Geneva (in French)
http://www.ge.ch/dip/welcome.asp
Not sure if they are already well established in an IB programme why you would want to switch to Swiss system, as they are quite different. You may want to think about this.
Also, please also note that if your goal is for them to learn the language, there are many ways of doing this. Many people also just often consider public versus international schools, but you should think beyond this. Please be aware that there are many other private schools, esp. in GVA, where many paths can be chosen. You can follow the Swiss maturity in a private school with bilingual education; you can follow the French baccalurate stream, with or without other languages; German school, etc...