Senior Conservatives – including a cabinet minister – say their party is likely to campaign to leave the European Convention on Human Rights (ECHR) at the next election, if Rwanda flights continue to be blocked.
There is frustration at the role of a European court in stopping flights for asylum seekers taking off last year.
The minister told the BBC the UK was "probably" being punished for Brexit.
The government's official position is that the UK will remain in the ECHR.
A spokesperson said: "The government has been clear that it will abide by its international treaty obligations.
"As we've set out previously – we believe our Stop the Boats Bill will deliver the changes necessary to reduce the incentives for people to risk their lives through illegal crossings while remaining party to the ECHR."
However, immigration minister Robert Jenrick would not rule out withdrawing from the convention, telling Times Radio the government would do "whatever is necessary ultimately to defend our borders".
The comments by senior figures in government are likely to reignite a debate in the Conservative Party about the ECHR – which splits opinion among Tory MPs.
Sir Bob Neill, the Conservative chairman of the Commons Justice Committee, said it would be "a completely foolish idea and absolutely wrong" for the UK to leave the ECHR.
He told the BBC that being outside the ECHR would leave the UK in the company of Belarus and Russia, and there would be a danger Britain's international reputation would be "very seriously damaged".
The ECHR was established in 1950 by a number of countries including the UK.
The treaty, which sets out the rights and freedoms people are entitled to in the 46 signatory countries, is overseen by the European Court of Human Rights.
It is separate to the European Union – so the UK remained part of both after Brexit.
Ministers are desperate to show they can deliver on tackling illegal migration to the UK before an election – likely next year.
But so far the Rwanda scheme, which would see the UK send some asylum seekers to the east African country, has not got off the ground and the timetable is slipping.
The first flights were stopped by a European judge in a last minute intervention last year – despite being cleared by UK courts.
Since then, the scheme has become bogged down in legal action.
The UK's Court of Appeal ruled against the plan last month, to the surprise of the Home Office.
There will now be an appeal in the Supreme Court in the autumn, but senior ministers are privately unsure about whether they can overturn the ruling.
They fear that even if the government does win, it could take many months for deportations to begin because individual legal challenges will take place too.
Alternative plans are being considered, but they would also face legal challenges and take time to deliver.
That is likely to lead to further debate about whether the UK needs to take further action to ensure the policy is delivered.
Legislation passed in recent weeks will put a legal obligation on the government to remove people who have entered the country illegally.
But without comprehensive returns agreements – and with the Rwanda policy stuck in the courts – it is unclear when ministers will trigger the implementation of that responsibility.
The government also now has the power to ignore certain ECHR interim injunctions relating to border security.
But some Conservative MPs wanted the government to go further and remove itself from other ECHR obligations.
A cabinet minister said events could make it "inevitable" the Conservatives end up backing leaving the ECHR.
Labour's shadow home secretary Yvette Cooper accused the Conservatives of going after headlines and "failing to tackle the crisis".
She told the BBC: "All of this rhetoric is making it harder to get the kinds of agreements that we need with France, with other European countries, to actually tackle this problem and prevent these dangerous crossings in the first place."
Some backbenchers have said the Conservatives should campaign to leave at the next election regardless of what happens with the Rwanda scheme.
Jonathan Gullis said: "Time and again we see the quasi-legislative European Court of Human Rights continue to undermine the government's plan to stop the boats."
He added the electorate should be given a choice over ECHR withdrawal at the next election.
But a former cabinet minister – strongly opposed to withdrawal – described the idea as "knee jerk nonsense".
They said those who backed quitting were a "minority" and there would be significant opposition in cabinet.
European court at odds with UK values – Braverman
Why does the UK want to send asylum seekers to Rwanda?
Florida shooter, 21, had racist manifestos – police
Prigozhin confirmed dead in plane crash – Moscow
China’s summer of climate destruction
Prigozhin and Putin: How a long friendship turned ugly
The K-pop girl group with no Korean members
Who owns OnlyFans? The elusive man at the top of a porn empire
A fond farewell to an uneasy South Africa
Do clean-air zones like London's Ulez work?
The indiscriminate devastation of Canada's wildfires
Andy Malkinson's battle for justice: What happens next?
World Cup kiss unleashes social tsunami in Spain
'We dangled for 14 hours': Cable car survivors recall harrowing ordeal
The new Napoleon film stirring debate
Stunning photos of a new African wonder
Why electric crops may soon be on the menu
© 2023 BBC. The BBC is not responsible for the content of external sites. Read about our approach to external linking.