The three-year project will focus on HMP Berwyn, the largest prison in England and Wales and the second largest in Europe.
Researchers from the Universities of Bath and Leicester will study factors such as prison design, leadership and the relationship between staff and inmates.
HMP Berwyn in Wrexham, North Wales, was opened in February 2017 and is a Category C jail for 2,106 men.
Attempts have been made to improve the physical environment at the prison, such as filling empty wall space with large photographic images of the local Welsh landscape.
The study will look at whether such initiatives, which researchers say are small and cheap to implement, lead to positive effects on prisoners.
Cultural changes have also been implemented at the jail, including the language used by prison officers.
Offenders are referred to as “men” rather than prisoners, and are housed in “communities” as opposed to blocks.
They are locked up in “rooms”, not cells.
Professor Yvonne Jewkes, from the University of Bath, said: “Some of these changes at Berwyn are relatively simple but they might be having an important effect when it comes to making prisoners feel that they are treated with respect and decency.
“And if, as a society, we are serious about rehabilitation and stopping re-offending then we need to look seriously at how these sorts of steps could be helping.
“It’s about striking the right balance between the competing aims of prisons – punishment but also rehabilitation.”
The £600,000 study has been funded by the Economic and Social Research Council (ESRC).
Researchers say that if the initiatives at Berwyn are successful, it will be a flagship model for the next phase of prisons to be constructed by 2012.
Prof Jewkes said: “We are particularly interested in looking in detail at how rehabilitation currently works in prisons in England and Wales.
“This is an age-old debate which is impacted by a range of factors – such as how prisons are designed, prison leadership and staff-prisoner relationships.
“We’re interested in learning from the case of HMP Berwyn so that improvements can be made nationwide.”
The initiatives, made by Berwyn’s management, are aligned more with prison cultures in small Scandinavian projects, researchers say.
Jails in England and Wales house on average 600 inmates – with some accommodating more than 2,000 – whereas average capacity for Norwegian and Danish prisons is about 80, they added.
The research will build on a three-year project by Prof Jewkes which ended last year.
This looked at prison architecture, design and technology in England and Wales, Scotland, Norway and Denmark.
Prof Jewkes has also advised prison governors in the UK and Ireland on how to improve custodial environments.
Mark Leech, Editor of The Prisons Handbook for England and Wales said he welcomed it.
Mr Leech said: “A prison is still a prison even when it does not look like the traditional idea of a jail, and calling people by their names isn’t being soft on crime its about decency and respect.
“I welcome this and hope that it will inform future designs and operations, convicts and dungeons have no place in a modern reforming prison system, and should be consigned to the Dickensian era where they belong.”