Musnad Ahmad 30:3763

'Abd al-Wahhab al-Khaffaf narrated to us, from Sa'id, from Qatadah, from Abu Rafi', from Abu Hurayrah, from the Prophet, may Allah bless him and grant him peace, who said: "If one of you is invited and he comes along with the messenger, then that is his permission [to enter]."