Musannaf Ibn Abi Shayba 33:109

Muhammad ibn Bishr narrated to us, he said: Isma'il ibn Abi Khalid narrated to us, he said: My brother Nu'man narrated to me, from Mus'ab ibn Sa'd, from Hafsa bint 'Umar, she said: She said to her father, "O Commander of the Faithful, what would it harm you if you wore softer than this garment of yours, and ate better than this food of yours? Allah has opened the lands for you and expanded the provision for you." He said, "I will dispute with you against your own self. Do you not know what the Messenger of Allah, peace and blessings of Allah be upon him, used to encounter of the hardship of life?" And he began to remind her of some of what the Messenger of Allah, peace and blessings of Allah be upon him, used to encounter, until he made her weep. He said, "I have told you that I had two companions who trod a path, and if I tread other than their path, I will be led to other than their path. By Allah, I will surely share with them in their severe life, so that perhaps I may attain with them their easy life." He meant by his two companions: the Prophet, peace and blessings of Allah be upon him, and Abu Bakr, may Allah be pleased with him.