There have been suggestions the Syco label boss and pop mogul would appear in the UK as a guest judge or as a 'Voice of God' on Skype while he is working on the US version of the show.
But speaking to the Guardian newspaper, Thames managing director Holloway said that Cowell could now be appearing as a hologram.
"Simon will appear in some guise somewhere," Holloway said, but he refused to reveal any further details about Cowell's involvement.
"That would be giving it away, wouldn't it... That's all I'm prepared to say... It's not a hologram," he said.
The producer also said he was not concerned about the recent drop in ratings The X-Factor suffered during its launch show the other week, and said he had plans to improve the show in future.
"I've got some thoughts in terms of that in the future, especially with all the technology we have and YouTube and Google and Skype. There are ways we should be exploring making these shows even more personal," he said.
"Bringing the people at home into the show, there's a rich vein there."
X-Factor auditionee Lucy Spraggan's performance on the show on Saturday has seen her independently released 2011 album 'Top Room At The Zoo' soar up the iTunes chart as a result.
Olympic diver Tom Daley has signed up along with Jessica Ennis and Sir Chris Hoy to appear on the show in the coming months, which is shown on ITV1 on Saturday nights.
It was also revealed today that N-Dubz' Fazer is to host a new BBC X-Factor rival show named Urban Orchestra.
See below for Gigwise's gallery of contestants kicked off X-Factor.