Musnad Abu Ya'la al-Mawsili 110:273

Zuhayr narrated to us, Jarir ibn Abd al-Hamid narrated to us, from Ata ibn al-Sa’ib, from Tawus, from Ibn Abbas, who attributed it to the Prophet (may Allah bless him and grant him peace). Jarir and others did not attribute it [to the Prophet], but said: "Circumambulating the House (the Kaaba) is like prayer, except that you speak during it. So whoever speaks during it, let him not speak except with good."