Inmates at a jail for teenagers represent 48 different gangs, inspectors have discovered, fuelling a “constant juggling” by prison officers to keep the boys apart.

There were high levels of “unpredictable and reckless” violence among the 240 boys aged 15 to 18 at Feltham Young Offenders’ Institution (YOI), a significant amount of which was gang-related, Her Majesty’s Inspectorate of Prisons (HMIP) said in a report.

Around a quarter of the boys – most of whom are from the London area – were placed on so-called “keep apart” lists to tackle gang-related violence, inspectors said.

The report prompted the head of the prison officers’ union to claim that jail staff were managing on a “wing and prayer” and he feared one of his members could be murdered if overcrowding and staffing shortages were not addressed.

The number of fights and assaults had reduced since the previous HMIP inspection, but remained “too high”, with 262 incidents in the six months before the visit, including 79 assaults on staff.

Some incidents were deemed “very serious” by inspectors and involved gangs of boys attacking a single boy or member of staff in a “very determined way”.

Feltham is divided into two sites, with boys aged 15 to 18 held in site A, separated from 400 young adults aged 18 to 21 in site B.

Chief Inspector of Prisons Nick Hardwick said: “The welcome reduction in the number of boys in custody means that those who remain are a more concentrated mix of very troubled boys who sometimes display very challenging and violent behaviour.

“As at other YOIs for this age group, staff in Feltham A still struggled to manage this behaviour in a safe and secure way.

“Staff need more help to do this and I repeat my call for the Youth Justice Board (YJB) to initiate an independent expert review of its policies and resources for managing behaviour, reducing bullying and supporting victims across all YOIs.”

Use of force had increased dramatically at the site for boys, inspectors found, although supervision of, and accountability for, use of force was deemed good.

Around a quarter of boys spent 23 hours a day locked up in their cell, which critics said amounted to solitary confinement.

The report calls for “new thinking” about how to tackle the “debilitating and seemingly intractable” problem of gangs, violence and anti-social behaviour among young people.

The site for young adults still faces significant challenges, inspectors said, particularly a high level of staff vacancies.

However, the report did flag significant progress at site B, with a drop in fights and assaults and a decrease in use of batons against inmates.

During the inspection, eight foreign national prisoners were being held beyond the end of their sentence – including one who had been inside for two years after the end of his sentence.

Inspectors labelled this delay “completely unacceptable”.

Peter McParlin, chairman of the Prison Officers’ Association, told the BBC: “There’s a distinct lack of resilience in the prison system now. That’s been caused by a number of factors from outside, the increasing levels of violence outside translating inside the prisons.

“Add in the chronic overcrowding, the staffing levels we have, the acute staffing problems and so on and the use of synthetic cannabinoids, the drugs of choice, we have a very dangerous prison system.

“The prison population has increased to 85,000 and predictions are it might be as high as 100,000 by 2020 but let’s remember we have now got institutionalised overcrowding, not just overcrowding but overcrowding to save money.”

Mr McParlin also said 7,000 prison officers had left the service since 2010 and 18 jails had closed, adding that, if something was not done, “it’s going to get worse. I wake up every day thinking one of my members of staff is going to die today – that’s a worrying situation to say the least.”

Andrew Neilson, director of campaigns at the Howard League for Penal Reform, said: “One in four boys in Feltham prison is spending 23 hours a day locked up in his cell in conditions which amount to solitary confinement. This is unacceptable.

“Despite some improvements, violence is endemic, including 79 assaults on staff in six months. It is particularly concerning that the use of force on children has increased dramatically since the last inspection.

“However, the problems outlined in this report are not confined to Feltham – they mirror the findings of a series of inspections conducted across the country in recent months.

“Staff are rightly praised by the inspectorate, but they are working in challenging environments where they are being asked to achieve the impossible.

“It is high time we stopped locking up children in large, violent institutions and invest instead in what works in rehabilitating children whilst keeping them safe.”

A previous inspection of site B in March 2013 called for a radical overhaul of the way the jail was run. Inspectors said it was one of the most concerning visits they had conducted.

Michael Spurr, chief executive of the National Offender Management Service (Noms), said: “As this report makes clear, the challenge faced by the governor and staff at Feltham should not be under-estimated.

“Many of the young people in their care have strong gang affiliations and a history of violence.

“Managing their behaviour and supporting them to change and turn their lives around is a difficult and complex task. In this context the improvements achieved over the last 12 months are particularly impressive and the governor and his staff deserve huge credit for what has been achieved.

“There is no easy answer to the challenges presented by the young men in Feltham but we are committed to working positively with our partners in the YJB and in the wider community to reduce violence, prevent victims and support effective rehabilitation.”

Juliet Lyon, director of the Prison Reform Trust, said: “Improvements in Feltham B show the importance of providing dedicated accommodation and regimes for sentenced young adults which take account of their offending and level of maturity.

“However, despite the best efforts of staff, Feltham A remains an extremely challenging environment with extremely challenging children.

“Despite welcome falls in both violence and the use of segregation, there are still too many children under the age of 18 locked in their cells and sleeping through their sentences.

“Most alarmingly the Inspectorate found that some children were effectively kept in solitary confinement, spending as long as 23 hours a day locked up, closing the door on any chance of rehabilitation.”

The chief executive of the Youth Justice Board, Lin Hinnigan, said: “It is clear from the report that, since the last inspection in 2013, improvements to Feltham A have been made, and we are grateful to the inspection team for recognising the dedication of the staff who work there. However, and as the report also acknowledges, the young people placed in Feltham are some of the most challenging of the cohort we have across the youth secure estate.”

She added: “Improving Feltham A is very much a work in progress, and will take time because of the multiple challenges that need to be addressed. But it is a commitment that the YJB and Noms intend to see through successfully.”