Musannaf Ibn Abi Shayba 3:976

Abu Khalid al-Ahmar narrated to us, from Yahya ibn Sa'id, from Nafi', from Ibn 'Umar, who said: "When we would miss a man at the Isha prayer and the Fajr prayer, we would think ill of him."