Musannaf Ibn Abi Shayba 3:2954

Waki’ narrated to us, from al-Awza’i, from ‘Amr ibn Shu’ayb, from ‘Umar ibn al-Khattab, who said: "The Friday [prayer] was four [rak'ahs], then it was made two rak'ahs because of the khutbah (sermon). So whoever misses the khutbah, let him pray four."