What is the ruling if the husband changes his travel destination mid-journey after traveling with the wife selected by lot?

Primary text

If the husband travels with one wife by lot, and then decides, after commencing the journey (for example, traveling to Jerusalem, and then deciding to proceed to Egypt), he may take her along with him, because it is considered one journey for which the lot was cast.