Musannaf Ibn Abi Shayba 9:427

Narrated to us Abdullah ibn Idris, from Hisham, from al-Hasan, regarding a man who has an adulteress slave-girl under him: He said: She is like the honor of his wealth; he may have intercourse with her.