Musannaf Ibn Abi Shayba 9:353

Muhammad ibn Fudayl narrated to us, from Mutarrif, from Abu al-Jahm, from Abu al-Akhdar, from Ammar, who said: Allah has not forbidden anything of the forbidden things except that He has also forbidden it regarding slave women, except that a man may gather [have] as many slave women as he wishes.