Two words: plastic backed. I'll bet you're experiencing evaporative cooling from the shell moisture on a cloth-like outer evaporating. I've noticed this taking place, but hadn't considered the effect this could have in the winter, having never used those in the winter. The low humidity would make it happen really quickly, and the low temps would make it really unpleasant.
If your preferred product doesn't have a plastic backed variant, as
@Wetshisbed stated, putting plastic pants over the top will have a similar effect.