Musannaf Ibn Abi Shayba 7:195

Hammad ibn Khalid narrated to us, from Makhlad ibn Hilal, who heard Sa’id ibn al-Musayyib saying: Whoever says, "It is upon me to walk to the Ka'bah," then it is nothing, unless he says, "It is a vow upon me to walk."