Musnad Ahmad 27:574

Muhammad ibn Ja'far narrated to us, Shu'bah narrated to us, from Sulayman, from Dhakwan, from Abdullah ibn Umar, who said: The Messenger of Allah (peace and blessings of Allah be upon him) said: "If you are three, then two of you must not whisper to each other excluding the one." He said: So I said to Ibn Umar: "What if they are four?" He said: "There is no harm in that."