Musnad Abu Ya'la al-Mawsili 153:211

Abu al-Rabi' al-Zahrani narrated to us, he said: Isma'il bin Zakariyya narrated to us, he said: al-A'mash narrated to us, from Abu Salih, from Ibn 'Umar, who said: The Messenger of Allah (ﷺ) said: "If you are three, then let not two whisper privately excluding their companion, for that would grieve him." He (the narrator) said: "What if they are four?" He said: "It does not harm you."