Musnad Ahmad 791:4

'Affan narrated to us, Hammad ibn Salamah narrated to us, Abu Qaza'ah Suwayd ibn Hujayr al-Bahili informed us, from Hakim ibn Mu'awiyah, from his father, that his brother Malik said: "O Mu'awiyah, Muhammad has taken my neighbors, so go to him, for he knows you and has spoken to you." He said: So I went with him, and he said: "Release my neighbors for me, for they had already accepted Islam." He (the Prophet) turned away from him. He stood up, pulling at his clothes, and said: "By Allah, if you do that, the people will claim that you command a matter and then act contrary to it." He continued to pull at his clothes while he was speaking. The Messenger of Allah (peace and blessings of Allah be upon him) said: "What is he saying?" They said: "He says: 'By Allah, if you do that, the people will claim that you command a matter and then act contrary to it.'" He said: "Did they say that, or did their speaker say it? If I were to do that, it would only be against me, and there would be no blame upon them for that. Release his neighbors to him."