Musannaf Ibn Abi Shayba 3:5173

Abu Mu'awiyah narrated to us, from al-A'mash, from al-Ma'rur ibn Suwayd, who said: We went out with Umar during a Hajj he performed. He recited to us in the Fajr [prayer]: "Have you not seen how your Lord dealt with the companions of the elephant?" and "For the accustomed security of the Quraysh." When he finished the Hajj and returned, the people were rushing [towards a place]. He said: "What is this?" They said: "A mosque in which the Messenger of Allah (peace and blessings of Allah be upon him) prayed." He said: "It is in this way that the People of the Book were destroyed; they took the traces of their prophets as places of worship. Whoever among you is presented with the prayer while in it, let him pray, and whoever among you is not presented with the prayer while in it, let him not pray."