The Prime Minister is being urged to postpone a decision in Parliament on gay marriage until after the next election.
According to a report in the BBC, the call has come from 20 Conservative Association chairmen.
In a letter being delivered to No 10 today, they express fear that "if enacted, this bill will lead to significant damage to the Conservative Party in the run-up to the 2015 election".
The warning comes ahead of the second reading in Parliament next week of legislation to legalise gay marriage.
The Marriage (Same Sex Couples) Bill will allow civil and religious marriage ceremonies for gay couples in England and Wales, where a religious institution has approved this.
The bill is opposed by the Church of England and the Catholic Church, who say that the Government has no mandate to change the law on marriage and that religious freedom may be compromised.
A petition in support of the traditional definition of marriage has been signed by more than 600,000 people.
The Telegraph reports that a survey of 2,000 people by ComRes suggested the Tories may lose enough votes over gay marriage to cost them power.
In the poll, 20% of those who said they voted Conservative in 2010 agreed that they would not do so again because of the gay marriage issue.
David Burrowes, Tory MP for Enfield, spoke out against the legalisation of gay marriage on the BBC on Saturday.
He said it would create a "divided party, a divided society and a divided church and state".
"It damages us all, not just our party," he said.
"Most importantly, it damages the institution of marriage. We don't have the mandate to redefine marriage."
