Hosts Declan Donnelly and Holly Willoughby announced the news this evening, leaving the rest of the camp shocked.
Noel's departure came after the celebrities competed in the Immunity Games for the last two days.
Nick Knowles' team the Galahs were first to win Immunity after triumphing in the Rancid Race earlier in the episode.
Then after the Velodrome task, John Barrowman and Fleur East also saved themselves from the first public vote.
This left Noel, Anne Hegerty, Harry Redknapp, Emily Atack, James McVey and Rita Simons to go head to head.
Speaking to Dec and Holly after he had left the camp, Noel said he was "disappointed and relived in equal measure" and couldn't wait to be reunited with his wife, Liz Davies.
He added: "I am going to carry this experience for the rest of my life.
"The whole thing is such a weird experience. Since I came in here and I think I say, I was just going to be myuself, open and honest.
"This is in the top 10 life experiences for me."
When Holly asked why the 69-year-old had decided to do the show, Noel replied: "Mainly it was a combination of things, our boy Harrison said I've got to do it.
"I had a convo with you [Dec] and you said 'I've got to do it' so that stuck and I wanted to do something different.
"Please believe me, this is about the finest production I've ever been part of."