Musnad Ahmad 33:618

Musa narrated to us, Ibn Lahi'ah narrated to us, from Abu al-Zubayr, from Jabir, that he informed him that Umar ibn al-Khattab informed him that he heard the Prophet, peace and blessings of Allah be upon him, saying: "The people of Makkah will depart from it, then it will not be inhabited—or it will not be inhabited—except for a little. Then it will be inhabited, filled, and built up. Then they will depart from it, and they will never return to it ever."