Musannaf Ibn Abi Shayba 16:505

Abu Bakr narrated to us, he said: Ibn ‘Ulayyah narrated to us, from Ayyub, from Tamim al-Khuza’i, who said: An old woman of ours narrated to us, she said: I used to see ‘Umar, when he saw a man wearing a saffron-dyed garment, he would strike him and say: "Leave these glittering garments for the women."