Environment lighting without the environment

Hello, please bear with me. I want to do something very specific. I want to create an environment scene BUT I want the scene to be interchangeable between the characters that can appear in it.

For for example: I want to have 1 character in Spot X in the environment scene but with 3 different characters able to fill Spot X. So basically, I want to be able to render the environment and the 3 different characters in Spot X (i.e. 4 different renders, environment, char. 1, char. 2, and char. 3) and then be able to photoshop 3 different images by just making the character renders visible. I do not want to create 3 different renders with the full environment and the 3 different characters.

However, the problem is, the ways I've tried to render the characters only by lowering the environments opacity, etc. but it messes with the lighting of the character render. So if I try to put the character into the scene the lighting is wrong vs as if I did a single render WITH the character inside of the environment.