Malaysia's highest court has ruled that legal cases involving non-Muslims cannot be decided by sharia courts, drawing a line after a spate of high-profile cases that left many in legal limbo.

The Federal court, in a landmark judgement, held that disputes between a Muslim and a non-Muslim on family and Islamic matters should be settled in a civil court, the New Straits Times reported on Thursday.

"They (non-Muslims) can't be present to defend themselves in the sharia courts," Judge Abdul Hamid Mohamad was quoted by the daily as saying.

The ruling came amid a bitter debate on whether the mainly Muslim nation is an Islamic state. The polemic has exposed religious and racial faultlines ahead of a widely expected early general election.

Race and religion are touchy issues in multi-racial Malaysia, where ethnic Malay Muslims form about 60 percent of a population of roughly 26 million, while Hindus, Buddhists and Christians dominate the ethnic Indian and Chinese minorities.

Malaysia's legal system has been bogged down by the conflicting jurisdictions between civil and sharia courts.

"Both courts have to grapple with this problem," Abdul Hamid, the Federal court judge, said. "While a judgement settles the case before the court, it creates other problems in subsequent cases."

In May, the country's best-known Christian convert, Lina Joy, lost a battle in the Federal court to have the word "Islam" removed from her identity card.

In delivering judgment in that case, the chief justice said the issue of apostasy was related to Islamic law, and civil courts could not intervene.

Last month, a Malaysian woman accused Islamic religious police of intimidation and mental torture during her six-month detention for renouncing Islam in favour of the Hindu religion.

In practice, sharia courts do not allow Muslims to formally renounce Islam, preferring to send apostates to counselling and, ultimately, fining or jailing them if they refuse to desist. Such people often end up in legal limbo, unable to register their new religious affiliations or legally marry non-Muslims. Many keep silent about their choice or emigrate.