Mr Justice Wilkie said the man – given four life sentences in the 1970s – should be immediately released.
The judge heard that the man was re-called to jail less than three weeks before the start of the 2012 games.
He concluded that then Conservative Justice Secretary Ken Clarke had made an “unreasonable” decision.
Mr Justice Wilkie did not identify the man or give details of his crimes, in a written judgment handed down at the High Court in London today.
But the judge said the man had been convicted of murder, attempted murder, causing an explosion with intent to endanger life and possession of firearms with intent to endanger life – and said the offences had been committed “in the context of Middle East terrorism”.
The man claimed that he had been unfairly recalled to jail, four years after being freed on licence, at a High Court hearing in London earlier this month.
Mr Justice Wilkie said the man had been given four life terms with a recommendation that he should not be considered for parole until he had served at least 20 years.
He had been released on licence in 2008 then recalled to prison in July 2012.
Mr Justice Wilkie was told that the man had attended a demonstration in Downing Street during the run-up to the games – which started on July 27.
The judge was told that the man was recalled to jail after probation officials decided that he had breached conditions of his release by “becoming involved in political events”.
But the judge said that “upon reflection” officials had changed their minds, decided that the “risk” posed could be “managed in the community” and asked Mr Clarke to “rescind” the decision to recall the man to prison.
Mr Justice Wilkie concluded that Mr Clarke’s initial decision to recall the man to jail was fair.
But he said the decision not to rescind the prison recall order following the request from probation officials was unreasonable.
Mr Justice Wilkie said the man had made “progress” in the four years he had been free on licence and on reflection probation officials had decided that a return to prison would damage his “successful rehabilitation”.
And the judge said in the “particular circumstances” of the case Mr Clarke’s decision to refuse to rescind the recall order was not reasonable.
He said officials had reconsidered and thought their “original view” wrong.
“(The) view was that recall would be disproportionate and that the risk could be managed in the community,” said the judge.
“The request to rescind had been properly made as a result of a proper review conducted by senior management.”
He added: “The very clear, and immediate, view expressed by probation was that their initial judgment was erroneous and that recall would cause great damage to the purposes of release on licence, namely the successful rehabilitation within the community of the offender.”
A High Court official said later that a judge sitting at an earlier High Court hearing had made an order preventing publication of the man’s identity.
A Ministry of Justice spokesman said later: “Public protection is our top priority. Serious offenders on licence are subject to a strict set of controls and conditions and can be recalled to custody if they breach them. Sometimes they will receive a formal written warning for a breach, where they have not re-offended.
“When released from prison, all terrorist offenders on licence are managed through multi agency public protection arrangements (MAPPA), will be risk-assessed, monitored and supervised by the Probation Service, police and other agencies.
“Terrorist offenders subject to probation may also be required to comply with additional licence conditions to better manage any risk they pose.”