Musannaf Ibn Abi Shayba 36:162

Abu Usama narrated to us, from Sulayman ibn al-Mughirah, from Thabit, who said: The companions of Abu Jahl said to him while he was marching toward the Messenger of Allah, may Allah bless him and grant him peace, on the day of Badr: "Do you see your marching toward Muhammad? Do you know that he is a prophet?" He said: "Yes, but when have we ever been followers of Abd Manaf?"