Musannaf Ibn Abi Shayba 33:1191

Yahya ibn Yaman narrated to us, from Hamza al-Zayyat, from Humran ibn A'yan, from Abu Harb ibn Abi al-Aswad al-Du'ali, who said: "Indeed, a man may be detained at the gate of Paradise for a sin he committed a hundred years ago, while he is looking at his spouses and his servants."