Musannaf Ibn Abi Shayba 13:1107

Abu Bakr narrated to us, he said: Abu Usamah narrated to us, from Hisham, from al-Hasan, who said: If a man restricts his slave regarding the people of his marketplace, then it is not permissible for him.