Musannaf Ibn Abi Shayba 3:1089

Waki’ narrated to us, from Mansur, from ‘Adi ibn Thabit, from ‘Aisha, who said: Whoever hears the caller [to prayer] and does not answer, he does not desire good, or good is not desired for him.