Musnad Ahmad 1153:1420

Muhammad ibn Ja'far narrated to us, Shu'bah narrated to us, from Abu Ishaq, who said: Ibn al-Zubayr said to al-Aswad: "Tell me about the Mother of the Believers, for she used to confide in you." He said: "She informed me that the Prophet of Allah, peace and blessings of Allah be upon him, said to her: 'Were it not that your people are newly removed from the Age of Ignorance (Jahiliyyah), I would have demolished the Ka'bah, and then I would have made two doors for it.'" When Ibn al-Zubayr became ruler, he demolished it and made two doors for it.