Musannaf Ibn Abi Shayba 7:194

Abu Usama narrated to us, from Abdullah ibn Umar, from Nafi, from Ibn Umar, regarding the man who says, "I am obligated to walk to the Kaaba." He said: "This is a vow, so he should not walk."