Musnad Ahmad 27:1613

Muhammad ibn Ja‘far narrated to us, Shu‘bah narrated to us, from Mansur, from Sa‘d ibn ‘Ubaydah, who said: I was sitting with ‘Abdullah ibn ‘Umar, then I came to Sa‘id ibn al-Musayyib and left a man from Kindah with him. The Kindi came back terrified, so I said: "What is behind you [what happened]?" He said: "A man came to ‘Abdullah ibn ‘Umar just now and said: 'I swear by the Ka‘bah.' He [Ibn ‘Umar] said: 'Swear by the Lord of the Ka‘bah, for ‘Umar used to swear by his father, and the Prophet (peace and blessings of Allah be upon him) said to him: "Do not swear by your father, for whoever swears by other than Allah has committed shirk (polytheism).""