Musannaf Ibn Abi Shayba 16:406

Abu Bakr narrated to us, he said: Waki’ narrated to us, from ‘Uyaynah ibn ‘Abd al-Rahman, from ‘Ali ibn Zayd, who said: I sat with Sa’id ibn al-Musayyib while I was wearing a silk (khazz) jubbah. He took hold of the sleeve of my jubbah and said, "How excellent this jubbah of yours is!" I said, "What do you mean? They have already ruined it for me." He asked, "Who ruined it?" I said, "Salim." He said, "When your heart is righteous, then wear whatever appears to you (i.e., whatever you wish)." He said: So I mentioned their statement to al-Hasan. He said: "Indeed, part of the righteousness of the heart is abandoning silk (khazz)."