The actions of the Rev Geraint ap Iorwerth, rector of St Peter ad Vincula Church, Pennal, Gwynedd, who conducted the burning of some pages of the Bible is now being investigated by the Church in Wales, which is part of the Anglican Communion.
Unveiling his "artwork" at a recent church event, the Welsh clergyman said it revealed a "cruel and vile God".
The Bishop of Bangor, Andy John, said destroying the Bible, or passages from it, would cause offence to many.
He said: "I have therefore written to the Rev Geraint ap Iorwerth and will be investigating the matter further.
"There are parts of the Bible that we struggle to understand today because culturally our life is so far removed from that period in which the Bible was written.
"However, it is not given to us to pick and choose - sometimes the most challenging parts are those which we need to wrestle with most of all."
Mr ap Iorwerth told BBC Wales that he had burnt scraps of cut up the passages at the public event because he had been making a statement as part of an art experiment.
He said he had had "nothing but support" from people at the historic church near Machynlleth, close to the Gwynedd-Powys border.
The Welsh clergyman revealed his controversial piece of art at an event to celebrate the 400th anniversary of the King James Bible.
He said: "I find it highly offensive that people would think I have given my life to serving that type of God and that I would regard the words of the King James Bible as sacred truth.
"I cut out all the nasty bits of that Bible, the language of which is being celebrated all over the place this year, because I don't think you can separate that language from the God it is representing.
"I cut out all the nasty bits of that Bible, the language of which is being celebrated all over the place this year."
According to BBC Wales, he added, "I was gobsmacked [shocked] that no programs or articles are representing the cruelty, revenge and hatred of this version of God."
The "artwork," which also contrasts the language of the Bible and the festive greetings of Christmas cards, is mounted on a 9ft by 7ft board.
Mr ap Iorwerth wants to show it in a gallery and also promote it as an alternative Christmas card, because he thinks it will provoke thought and discussion.
He said it is the "most-popular" version of God as cruel that he takes exception to, whereas he thinks more attention should be paid to God's life-work and the view that "God is love."
"People have told me they turned away from the real message of Jesus because of this God - that this version put lots of people off him as children. My version of God is Jesus, who was pure compassion and unconditional love," he said.
BBC Wales says that the "artwork" created by the Reverend ap Iorwerth includes cut-out passages and Christmas cards.
"The King James Bible should be praised for its language but not for the God it represents - the two need to be separated," he said.
He said he burnt the pages, which were the remaining scraps of those he cut up to make the display, as a "symbol of all the suffering in the world."
The rector went on to say, "The point being that some people are more concerned about destroying a few bits of pages than about those who have died after suffering."
He said he had not yet heard anything of an investigation against him and will be pursuing his own "evangelical" investigation after the summer into "how such a cruel God has so prominent a place in national life."
He said he also planned to create a "wall of shame" at the church naming all of God's "cruel actions."
What has made this Bible burning so surprising to many is that this took place in the land of the Welsh Revival. Of all the names that are linked with the 1904 revival, Evan Roberts is the most well-known.
At 26 years of age in the autumn of 1904, he was to become an instant spiritual celebrity in Wales as the local and national newspapers of the day chronicled the spreading religious revival.