Allah describes the way the disbelievers answered His Prophet Shu`ayb and those who believed in him, by threatening them with expulsion from their village, or with forceful reversion to the disbeliever's religion.
Allah tells,
قال المﻼ الذين استكبروا من قومه لنخرجنك يا شعيب والذين آمنوا معك من قريتنا أو لتعودن في ملتنا
The chiefs of those who were arrogant among his people said: "We shall certainly drive you out, O Shu`ayb and those who have believed with you from our town, or else you (all) shall return to our religion.''
The chiefs spoke the words mentioned here to the Messenger Shu`ayb, but intended it for those who followed his religion too.
The statement,
قال أولو كنا كارهين
He said: "Even though we hate it!''
means, would you force us to do that, even though we hate what you are calling us to!