Billy Graham's influence stretched to the very top of America's political tree – but, says Grant Wacker in his biography America's Pastor, his friendship with presidents was characterised by just that – friendship.

Graham knew all 11 presidents from Harry Truman to George W Bush, and met Barack Obama as well. He did not get on with Truman, whom he criticised over his conduct of the Korean War, but he was close friends with Lyndon Johnson, Richard Nixon, Ronald Reagan and George HW Bush. Wacker quotes Time journalist Nancy Gibbs, who wrote that no other person enjoyed such access to the White House: 'He came with the office like the draperies.'

On the face of it, some of these friendships were unlikely. Lyndon Johnson, for instance, was completely unlike Graham in all sorts of ways, given to bawdy language and extra-marital affairs. But, says Wacker, 'at the end of the day there can be little question that the two men admired and loved each other'. Johnson feared death acutely, and asked Graham to preach at his funeral. He did so, describing how Johnson tried to 'harness the wealth and knowledge of a mighty nation to assist the plight of the the poor and underprivileged'. He summed up his friend as 'a mountain of a man with a whirlwind for a heart'.

He was also friends with Richard Nixon, on the face of it more likely pairing; Graham's leanings were Republican and Nixon's public face was much more presentable than Johnson's. However, it was probably an unbalanced relationship, with the affection and respect on Graham's side more than on Nixon's. He continued to defend Nixon long after most people had lost faith in him. When the transcripts of the Oval Office conversations in the wake of Watergate were released, he said: 'The thing that surprised and shook me most was the vulgar language he used ... I felt physically sick.' To many, this seemed to underplay the seriousness of Nixon's real offences. However, to Graham it revealed depths of deceitfulness in his character which he found deeply wounding.

On the face of it, the president with whom he would have most in common was Jimmy Carter, a fellow Southern Baptist evangelical. There does seem to have been a considerable amount of warmth between them, though this does not seem to have ripened into the sort of friendship he had with Johnson. In the 1980 presidential election Graham worked behind the scenes for Ronald Reagan. With Reagan the connections were deeper. Wacker says they had a lot in common – both were born leaders and fine communicators who knew that there was no point in saying something if people didn't understand it.

With George HW Bush and Bill Clinton, the friendships also went deep – and his loyal support for Bill Clinton cost him friends with his more conservative supporters. In 2005 he said at his crusade in Flushing Meadows, with Bill and Hillary Clinton beside him, 'I told him, when he left the presidency, he should become an evangelist, because he had all the gifts.' He was roundly criticised by right-wing commentators.

One of the things that has to be faced about Billy Graham is that while he was first and foremost an evangelist, he was also a patriotic American who had views about how his country should be governed and led. His relationship with presidents and senior politicians was always primarily pastoral, not political. He sought to avoid taking sides in public and always claimed never to have publically endorsed a presidential candidate.

However, at the same time he had favourites, and did not always succeed in maintaining a political neutrality – though on the other hand, he was a world away from the sort of overt campaigning which has tarnished the image of many conservative evangelicals in the US. It can't be denied that he crossed a line in some of the things that he said and did – but given his level of access to the highest in the land and the influence that he might have wielded if he chose, the startling thing is not that he sometimes got things wrong, but that he usually got them right.

Follow Mark Woods on Twitter: @RevMarkWoods