There is an organisation Benevol, and they need people whom are semi to fluent in the local language.
http://www.benevol.ch/ but the vacancies are mostly in the east.
I cannot say for sure, but I believe the asylum seekers are kept tucked away by the professional social workers, they don't want them wandering off into the community, they need them to be controlled.
However if your French is OK you could visit people in hospital. You could visit an old people's home. You could also ask at your commune where you can hekp out.