Musnad Ahmad 2:71

Hassan narrated to us, Ibn Lahi'ah narrated to us, Abu al-Zubayr narrated to us, from Jabir, that Umar ibn al-Khattab, may Allah be pleased with him, informed him that he heard the Messenger of Allah, peace and blessings of Allah be upon him, say: "The people of Makkah will depart, then it will not be inhabited—or it will not be inhabited except by a few—then it will be filled and built up, then they will depart from it and will never return to it ever."