Combining separate Canvas renders?
SnowSultan
Posts: 3,773
in The Commons
Just have a quick question, when I render two things separately (such as a character and a background) using Canvases and then try to combine them into one image, there's a transparent edge around the whole character that can't really be removed. If I turn off Alpha in the canvas rendering, it comes in against a black background, which I know can be hidden by switching blending modes in Photoshop to Lighten, but that leaves a few odd edges too.
Is there a recommended method for combining elements that are rendered as separate canvases? Thanks.

Comments
Quickest fix would be to control click the layer with the character or whatever that you rendered thats not part of the BG render. Then go to the select menu > modify > contract. Start with 1 px, see if it's good enough, repeat until you have it selected properly. Maybe feather it by 1 px at the end. Once you got it done good enough, go to layer menu > layer mask > reveal selection. Not perfect, but it's quick, dirty and nondestructive. This is PS, I am not sure how to use gimp, but I assume the process would be similar.
I have not experienced this issue before so I may not have any info that helps but are you able/willing to attach your canvases so that I can download to try/test and see what you mean? Or attach some screenshots or use a dropbox link or something for the files etc? Are you using LPE's for your canvases or just the standard ones built in?
TheKD, I actually tried that but because the background also has transparency where the figure was, trimming the figure just adds more transparancy to the background.
Shane: I just re-rendered the two elements and also made an LPE render, will experiment further and see if I can share them if I still have no luck. Are you suggesting I work with the HDR canvas renders or the basic PNGs (I've really only tried with the PNGs so far)? Thanks in advance.
OK, I did a bunch more tests with canvas renders, normal renders, whatever LPE renders are, cutting, pasting, and trying Blend If... options and can't fix this problem. Here are two small cropped examples of the transparency halo that I can't seem to eliminate. Renders are of my character in a room (character and room rendered separately). You can clearly see the large transparency problem with the fluffy hair.
Any advice is appreciated.
Have you tried the Remove Matte commands in PS? Though you should not get a halo, since DS does not use a pre-multiplied alpha.
I have experienced this problem too. The solution I found was to do a 3rd render of the enviroment without the figures, then, in PS, putting it as a layer under the background (enviroment) layer.
By the way, I find easyer, instead of using canvas, doing separate renders of the figures and the enviroment (set a camera, save the scene, delete background, render characters, then, load scene, delete characters and render background), then putting all toghether in PS, and, if needed, adding shadows through PS.
I have gotten the halo every time I have tried this. I have also used the method of rendering the background without the figure at all once to fill in the gaps. It isn't ideal if there are shadows being cast that are visible around the figure, but I would touch up the rest manually. If there is a better way to not get the effect you are showing here I am also interested in it.
Richard: Remove matte doesn't work in this case because the background has a transparent area where the figure should be. The halo around the figure could be removed with postwork, but it will just enlarge the transparency that's 'behind' the figure, rendered into the background layer.
I see I'm not the only one with this issue. The advantage of rendering with canvases is that shadows are still rendered as if all objects were visible, but this issue with compositing makes that feature useless. I still plan on rendering the whole scene in one normal render and set up 3Delight masks to select everything individually, but I was hoping this might provide a more modern solution.
I can't get the white Halo whatever I do.
For me, from your picture, it's either that the transparency information is not good enough or misinterpreted by Photoshop
I'm using Gimp, so I don't know if there is a problem on the photoshop side
Do you use Iray alpha or are you using MaterialID or something else ?
Takeo, that's very strange that you can't actually duplicate this problem. I've tried using the standard Iray alphas and the actual canvas renders and the transparent area is always slightly too large when rendered as canvases. Are you using the default render settings? I'm going to re-render my test scene with the default settings to see if maybe one of my changes is causing this issue.
OK...this is weird. I rendered two canvases (character and background) with default settings and saved each as a TIF instead of PNGs. When loaded in Photoshop and the character render was pasted onto the background one, the halo was still visible. However, if a solid black layer is added beneath the background one, the halo completely disappears - and the strangest part - the new black layer is not visible where the halo was. It *is* visible if you follow these same steps with PNGs; a clear black outline appears around the figure and the hair looks awful.
Could this have something to do with how TIFs handle transparency? I honestly have no idea why adding the black background does *not* create the outline effect when used on TIFs when all logic says it should.
I am assuming that both the layers in photoshop are at 100% opacity? If so then something definately seems to be going on with the objects or masks from iray. I have not been able to replicate the issue. Have you tried putting the black background in for the png as well? I used to use a black background when I would change the opacity of a layer to affect its overall contribution before I found other ways and that is what made me think to ask this.
It is also possible that some lighting is not being rendered to a canvas and therefore missing some lighting info for the scene from the individual contributions you have selected. Maybe one of the canvases is not set correctly for the light source or is not included at all. Just a thought.
Shane: Yes, I tried putting a black background layer with PNGs and the halo remains. That's what is so strange, what are TIFs doing that PNGs are not?
As for the canvas renders, I'm doing normal Beauty renders with only HDRI lighting (no scene lighting).
Ah ok. I will try a scene with just beauty passes and hdri only to see if i can replicate the halo effect. I think everything else I had compared with was all multiple light sources.
Possibly entirely unrelated, but: is there by any chance a shaped point or spotlight behind the hair, serving as a backlight?
This is only a guess, but I suspect the halo issue for just the hair may be due to the position of the lights behind the hair. Iray has a known issue that shaped point or spotlights (disc, cylinder, rectangle, whatever, anything that's not just a point of light as the source) makes the transmaps in hair go very strange indeed. The only solutions I've ever found are either to make the lights into points, or to move them.
Of course, that will only help with the hair issue, not whatever is happening with the legs, which is very odd.
Lol...well I am getting dark artifacts instead of Halo's:
I am investigating...
**UPDATE**
Found the culprit...using exr-io there is an option that was checked causing the issue. Make sure that if you are using exr-io when you open an exr image to make sure that "Fix transparent light emitters" is unchecked:
Now everything looks right:
For these I just opened the Beauty.exr canvas in photoshop and used an exposure layer to bring it back to my tone map setting (negative value) and all was as it should be, as long as I had unchecked that option for light emitter from exr-io. I hope something from here helps.
VWrangler: No, there's just an HDRI (environmental) light in my scenes, no scene lights at all. Maybe the light's direction has something to do with it, I'll have to try one more set of renders and check.
Shane: That makes makes this whole thing even weirder. :) I don't use EXR-io, and I got the same transparent halo when I tried combining EXR renders as well. I guess what I'm really trying to figure out now is why the halos don't appear with TIFs, since that does seem to solve the problem.
In fact I've tried everything from leaving default settings to trying to mess everything up but still, no problem with transparency. I've opened every EXR file to check, as well as TIFF and PNG files and still no problem. I've even opened the PNG file with a simple editing software and all PNG or TIFF file generated were correct.
That's very weird so you should try Gimp 2.10 for EXR files or an other software that can handle these. You may have something wrong with your Photoshop. Saving to TIFF or PNG shouldn't make any difference regarding transparency. TIFF is a more capable file format and has some advantage like handling HDR information but that shouldnt play any role here
Info : With Gimp 2.10, after opening the EXR files, if you want to see something, do Color->Auto->Stretch Contrast
Is it just that scene that is having the issue? Don't know if rebuilding it is an option but if other scenes do not have the issue there must be something in that one that is wonkers. If you find anything out please do post as I would like to know in case it ever happens for me what to do.
Takeo: Thanks, but I'm not going to install GIMP just for this. I'm using the latest Photoshop CC and I can't imagine GIMP doing much of anything better than Photoshop except being easier on the wallet. :)
Shane, Dawn, I will make a new scene with different lighting, render as TIFs and PNGs, and try Linear Dodge on them and let you know.
Alright, results of second, all new scene: exactly the same as the first.
(note: I forgot to add the little gold rings around the horns to the character's canvas node, that's why they're transparent. That's an error on my end.)
First one is the default look when a character canvas render (TIF) and a background canvas render (TIF) are layered in Photoshop CC. Layers shown in next image.
Second one shows what happens when a solid black layer is added beneath both rendered images. Layers also shown.
This 'fix' does not appear to work with PNGs, only TIFs. I'm going to render a normal non-canvas image later and compare it to the 'fixed' composite shown. Also tried every blending mode on both renders, nothing removes the halo because the halo isn't caused by bright pixels; it's actually transparent areas that are larger than they should be.
Perhaps try in PS: Layer-> Matting-> defringe, you can adjust the pixel size but usually 1 or 2 pix will do the trick, orOther way is using remove black matte or white matte,
I'm curious, are the results different if you use fibermesh hair?
Maybe it's the material transparency not playing nice.
Oh, another possible factor:
When converting 32 to 16 bit, are you using the default HDR toning in Photoshop, or the Exposure based? Because the default plays all sorts of localized adjustments to get a cool consistent look that might make stuff not line up right.
Until SnowSultan does a test with an other software we don't know if the issue is on DS or Photoshop side . In your case that was Photoshop and I think that for Snowsultan that is the same. If saving as TIFF solves his problem, that may be enough unless he wants to dig further
I was able to reproduce. For me it was simply putting a black background below everything else. I had 4 separate canvases using 4 separate node selections, all alphas. Below are the 2 different results, one with black background and one without...This was with the 32bit exr's:
NO BLACK BACKGROUN:
WITH BLACK BACKGROUND:
AlmightyQuest is right, as I said before, this isn't an issue with white pixels, it's that Studio is rendering the transparency one or two pixels too large when using canvases. I'd like to know why placing a black layer behind the layers makes the transparency on TIFs go away - it's possible that has something to do with how Photoshop reads TIFs - but yeah, I don't want to install another program just to test that.
edit: Oso, I get the same results with EXRs regardless of how I try and convert them. I don't think I've tried putting the black background layer behind two converted EXRs, but since it works with TIFs, I haven't bothered.
To be honest, I'm not sure why we need to render elements separately on canvases unless your PC can't handle rendering a whole scene at once. Masking gives more options because you can mask each surface instead of entire objects, and it isn't like you can animate characters from canvas renders because the background will retain the transparency where the character should be. This really is just a curiousity that I'd like to understand, but probably isn't worth all this trouble. Thanks to everyone for your help and suggestions so far though. :)