Musannaf Ibn Abi Shayba 35:477

Waki' narrated to us, from Shu'bah, from al-Hakam, from Ibn Abi Rafi', that the Prophet (peace and blessings of Allah be upon him) sent a man from Banu Makhzum to collect the charity (Zakat). Abu Rafi' wanted to accompany him, so he asked the Prophet (peace and blessings of Allah be upon him). He said, "Do you not know that charity is not lawful for us, and that the freed slave of a people is one of them?"