My memories of a very cold January visit are similar to those of NatsBrit. Brrrr.
If you do go, wear warm shoes because it was my feet that suffered the most and make plans to go straight to a warm cafe afterwards.
On the upside, visiting Chillon in the middle of winter will make you appreciate the creature comforts of modern day life. I remember thinking about how awful it would to have go to the bathroom with the cold winter wind blowing. If you've already visited Chillon and remember the WC in the castle, you'll understand what I mean.