Musannaf Ibn Abi Shayba 2:250

Abu Bakr narrated to us, he said: Ibn ‘Ulayyah narrated to us, from al-Jurayri, from ‘Abdullah, who said: It is from rudeness that you hear the Adhan saying "La ilaha illa Allah" and "Allahu Akbar," then you do not respond to it.