Musannaf Ibn Abi Shayba 16:491

Abu Bakr narrated to us, he said: Abu Usama narrated to us, from Ibn 'Awn, from Muhammad: He did not see any harm in a man wearing a garment dyed with safflower or saffron.