Musannaf Ibn Abi Shayba 2:228

Abu Bakr narrated to us, he said: Yazid and Waki’ narrated to us, from Isma’il, he said: Qays said: ‘Umar said: "If I were able to perform the Adhan along with the Caliphate, I would have performed the Adhan."