The cancelled Christian TV reality show "19 Kids and Counting" might have presented the Duggar family in a happy and wholesome light, with "traditional" parents Jim Bob and Michelle maintaining a nurturing atmosphere in their home in Arkansas. But sources are now claiming that it's a whole different picture when the cameras stopped rolling.

Two sources talked to PEOPLE, saying that Jim Bob keeps his family in check by exerting authority over his wife and children.

"Jim Bob's favorite word is 'no,'" the Duggar family source said. "He's like a dictator. He once saw a guy kissing a girl before marriage and confronted him about it. And the guy said, 'Jim Bob, show me that verse in the Bible that says we can't kiss. Get out your Bible.' And Jim Bob was like, 'Uhhhh ...' because no one ever confronts Jim Bob. No one."

The second source added, "Jim Bob calls the shots. Always has and always will."

And now that it has come to light that their eldest son Josh had admittedly been unfaithful to his wife Anna, as indicated by his uncovered two paid accounts in the cheating website Ashley Madison, Jim Bob is reportedly exerting his authority more than ever by keeping Anna away from the rest of the world.

"The scary thing is, I don't know who is talking to Anna, outside of the people in Duggarville," the source said.

One person who was allowed to see Anna though was fun-loving Duggar cousin Amy, who is set to marry her long-time boyfriend Dillon King on Sept. 6 this year.

"I gave Anna the biggest hug on the planet," Amy said. "I just wrapped my love around her."

Josh has now checked into rehab while his parents are caring for Anna and their four kids—Mackynzie, Michael, Marcus, and newborn daughter Meredith Grace.

"As parents we are so deeply grieved by our son's decisions and actions. His wrong choices have deeply hurt his precious wife and children and have negatively affected so many others. He has also brought great insult to the values and faith we hold dear," they said in a statement.