Demehl Thomas and Moysha Shepherd are thought to have captured footage of themselves on a banned mobile phone before it was uploaded on to the internet.
Bosses at HMP Birmingham have launched an investigation.
Thomas, 25, was serving a sentence for aggravated burglary but is believed to have been freed on licence earlier this year before being returned for breaching the terms of his release, according to the Sun.
By the time he was recalled in June he had compiled an album under the stage name Remtrex which was released on iTunes earlier this month, the newspaper said.
Shepherd, also 25, is serving a sentence for involvement in a plot to spring a criminal from a prison van in 2012.
The expletive-filled video shows the pair performing while dressed in black vests, taking it in turns to rap.
At one point Thomas says: “I want a mansion and I want my kids to live good. I don’t want to get locked up or killed in the hood.”
A second mobile phone appears to be charging in the background in the footage.
It is understood that the two do not share a cell and it is unclear in which room the film was shot.
Prison officials were said to have been tipped off last week but when the inmates’ cells were searched no mobile phone was found.
Birmingham Prison is a Victorian jail holding adult male inmates with a capacity of 1,450. It has been run by G4S since 2011.
The firm confirmed that mobile phones are banned.
Prison director Pete Small said:”Like every other prison in the country, it is a constant challenge to detect and seize contraband items such as mobile phones.
“Our prison custody officers are trained to look for contraband and we conduct regular and targeted cell searches to remove mobile phones, chargers and sim cards.
“In this instance, searches had already been carried out based on intelligence gathered and, as a result of the information received, further searches will be conducted.”
Figures show that in 2013 a total of 7,541 illicit mobile phones or sim cards were discovered in jails around the country.