Rashi cites a Midrash that if one treats clothes with contempt, ultimately they will do him no good. David mistreated garments when he cut Saul's cloak (in I Samuel 24). This is why the power of clothes to warm a person later deserted him.