Musannaf Ibn Abi Shayba 9:1757

Ghundar narrated to us, from Shu'bah, from al-Hakam, who said: I heard Dhakwan narrating from a freed slave of 'Amr ibn al-'As, that he [Amr] sent him to 'Ali to ask permission to enter upon Asma bint Umays, and he [Ali] gave him permission. When he had finished his need, the freed slave asked 'Amr about that. He said: "The Messenger of Allah (may Allah bless him and grant him peace) forbade us from entering upon women without the permission of their husbands."