Conservative Muslim Forum founder Lord Sheikh said demands from the PM for an apology were not enough.
He has called for Mr Johnson to have the whip removed - meaning the MP would no longer represent the Tory party.
Mr Johnson said Muslim women wearing burkas "look like letter boxes" and compared them to "bank robbers". A source said he stood by the comments.
Conservative peer Baroness Warsi - the first Muslim woman to sit in a British cabinet - said comments like Mr Johnson's could lead to a rise in hate crime in the UK.
Culture Secretary Jeremy Wright said there was no reason not to have a "robust conversation", but added: "We're not talking to our friends in the pub, we are public figures and we have an additional obligation to be careful."
Conservative Party chairman Brandon Lewis told BBC Radio 4's Woman's Hour it was right to have a debate on the issue but that Mr Johnson should say sorry for the "language that he used".
And a former Tory chairman, Lord Pickles, said Mr Johnson, who was foreign secretary until resigning last month over Brexit, risked "closing down" the debate with his "illiberal language".
He said that if someone made a complaint to the party, an independent panel would look into it, but it was "pretty much inconceivable" that the former foreign secretary would face disciplinary action.
A source close to Mr Johnson has said: "We must not fall into the trap of shutting down the debate on difficult issues.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
In his Daily Telegraph article, Mr Johnson said full-face veils should not be banned but that it was "absolutely ridiculous" women chose to "go around looking like letter boxes".
Theresa May has backed calls for an apology. The prime minister said the remarks had "clearly caused offence".
Lord Sheikh, a former adviser to David Cameron, "Take the whip from him. Why not? He's not a super human being, he's a member of the party. The party chairman, the prime minister has the right to take the whip.
"It's not out of order and that's the thing I'd like to see."
Mr Johnson, who is a former mayor of London and the current MP for Uxbridge and South Ruislip, has long been seen as a potential candidate for the leadership of the Conservative Party.
He fronted the successful Leave campaign in the 2016 EU referendum, and resigned as foreign secretary last month in protest at Theresa May's Brexit plans.
His comments have seen him accused of using the issue of Muslim women's veils as a "political football" to further his leadership ambitions.
No comments: