Speaking to BBC Radio Solent, Ron Heather said he felt “shock” and “horror” at the slogan on the adverts when he turned up for work last Saturday.
The adverts have been put up by the British Humanist Association on buses across the UK with the support of prominent atheist Professor Richard Dawkins. They proclaim “There’s probably no God. Now stop worrying and enjoy your life.”
Mr Heather returned to work on Monday with bus operators, First Bus, who said they would try to ensure he did not have to drive buses carrying the adverts.
Mr Heather told BBC Radio Solent, "I was just about to board and there it was staring me in the face, my first reaction was shock horror.
"I felt that I could not drive that bus, I told my managers and they said they haven't got another one and I thought I better go home, so I did.
"I think it was the starkness of this advert which implied there was no God."
The posters are the atheist answer to a number of high profile Christian advertising campaigns on buses and billboards, notably the Alpha Course, whose posters ask, "Is this it?" and "If God did exist, what would you ask him?".
Christians have generally welcomed the adverts, saying they provide an opportunity to talk about God.
The Methodist Church's Spirituality and Discipleship Officer, the Rev Jenny Ellis, said: "This campaign will be a good thing if it gets people to engage with the deepest questions of life … Christianity is for people who aren't afraid to think about life and meaning.”
The director of Faith-based think tank Theos, Paul Woolley, said the adverts were “hardly going to be a great comfort for those who are concerned about losing their jobs or homes in the recession”.
"And what does it tell us to do when we stop worrying?” he continued. “Volunteer overseas? Give money to charity? Campaign for the environment? No. It tells us to enjoy ourselves. It would be hard to come up with a more self-centred message than this."