Musnad Abu Ya'la al-Mawsili 6:32

Abu Khaythama narrated to us, Muhammad ibn Abd Allah al-Asadi narrated to us, Isra'il narrated to us, from Abu Ishaq, from Muhammad ibn Sa'd, from his father, who attributed the hadith [to the Prophet], he said: "It is not lawful for anyone to forsake his brother for more than three [days]."