Frustrated with 256 color-like renders

I'm having a new problem that I can't pinpoint the cause of, but it's been noticable in the last two images I've made and I do not remember it happening before updating both Studio (to 4.12) and Photoshop (to CC 2020).

When even very minor color or brightness adjustments are made to my latest renders, parts of them begin looking very patchy and low-res, as if the images was converted to 256-color. I always save the original renders as TIFs from Studio, then take them to Photoshop and immediately re-save them as PSDs, so compression shouldn't be to blame. I checked my video card settings and they all seem normal (32-bit, RGB, etc), and Photoshop doesn't have any unsual settings either. 

The only thing I can think of is that I do use Studio's built-in denoiser, but I let it run for 5000 iterations in the latest image and it looks worse than any I've done before once I make just a few small adjustments to brightness and curves in Photoshop. I've been doing that forever (and usually with the denoiser on) and never noticed this amount of banding and blotchiness before.

 

Does anyone happen to have any suggestions? Thanks in advance.

Comments

  • nonesuch00nonesuch00 Posts: 18,830

    I think your guess is correct - it is the denoiser. You can see the more you 'denoise' the more it looks like those 'toon' filters that for example, 6r, uses in the Art Studio section of his threads in the forum. It's a matter of taste, but I tend to like my renders better in the early stages, at 25 - 100 iterations and they look more 'real and interesting' to me but I let them finish anyway to at least 500+ up to 2000 iterations. 

  • The longer it renders the more detail will - usually - be preserved.

    How do the histograms look? If they are fairly smooth you should have enough tonal spread. Where are you reviewing the images? It might be a change in colour management or something making them look posterised.

  • SnowSultanSnowSultan Posts: 3,803

    Thanks for the replies. Yeah like Richard said, letting the denoiser go a long time will cause more details to be retained. I might redo this render without the denoiser (when I have two hours to kill) and compare with all of the same adjustment layers as my current one.

    Richard, the scene is supposed to be at night, so the histogram is mostly tilted towards the blacks (left side), but not extremely so. I'm checking them in Photoshop CC 2020 and double-checked the color management settings, that hasn't been changed.

    Here are two cropped areas of the pic I'm working on that show the posterization pretty well. They might be really hard to see against a white background though.

    shoulder.png
    208 x 81 - 16K
    hip.png
    114 x 132 - 13K
  • kimhkimh Posts: 401

    I had that happen with one of my recent renders and I was able to fix it by adjusting the lighting levels in Corel Paintshop Pro which is my photo editing software of choice. 

  • outrider42outrider42 Posts: 3,679

    The denoiser can vary a bit at times. Sometimes it can work as you might expect it to, other times it can really wash out the detail. It depends on your scene setup, resolution, and lighting. I think higher resolutions work better with it. The detail can work itself back in the longer it renders.

    That's why some people love the denoiser, while others hate it with a passion. I think it just comes down to how they use Studio as to how well the denoiser works for them. I'm guessing there must be something different about these recent renders that cause the denoiser to behave that way.

    I'd take the suggestion to render the image with the denoiser off, and then photoshop the best parts of the two images together. I've done that a couple times.

    We also have to think about how the denoiser supposedly works. It is supposed to be trained by AI to be able to "guess" what pixels are missing. In order to train the AI, finished images must be fed to the AI so it has the library to pull these from. And that is the issue. We have no way to train this AI on the images we will be making in Daz Studio.

  • Matt_CastleMatt_Castle Posts: 3,097

    Richard, the scene is supposed to be at night, so the histogram is mostly tilted towards the blacks (left side), but not extremely so. I'm checking them in Photoshop CC 2020 and double-checked the color management settings, that hasn't been changed.

    Here are two cropped areas of the pic I'm working on that show the posterization pretty well. They might be really hard to see against a white background though.

    If you're trying to do a dark image without colour banding, then running with the denoiser on in an 8bpc format is going to put you on the back foot. (I believe that although PNG and TIFF can support 16bpc, DS will only ever export these formats in 8bpc). Some time ago, I created a fix for colour banding in Source Filmmaker renders, which actually explicitly worked by adding a noise overlay to the render in order to break up any areas of flat colour.

    As far as a denoiser, I prefer the standalone versions; they don't add to the VRAM usage when rendering (a big concern on my low VRAM card), and they're also non-destructive; they leave the original, undenoised file, letting you layer them in Photoshop to taste. (I tend to mix back in a few percent of the noisy file to taste; it re-enhances a few details the denoiser might have overcorrected, and we're used to a little noise in images anyway - something too clean looks wrong).

    When it comes to darkness... well, when it's not feasible to 32bpc canvases, one of my favourite cinematographic techniques is "Day for Night". While it's a sometimes controversial technique because it's not realistic, my answer to that criticisms is that the audience actually being able to make out the image trumps realism and a lot of the problems that exist with live action "Day for Night" are completely under the artist's control in a CG render.

  • 3dOutlaw3dOutlaw Posts: 2,481
    Back in my day we didn't use a denoiser, we rendered for 32 hours then shrunk the image by 4x... and we liked it. Freaking kids these days...
  • SixDsSixDs Posts: 2,384

    smiley smiley smiley.

  • PadonePadone Posts: 4,081

    Also for post processing, especially for dark scenes, it is better to work with the exr beauty canvas. Another way would be to shoot "day for night" and make the "night" effect in gimp or photoshop, so to have enough gamut to work with. As for the iray denoiser, personally I'd avoid it until they do something better.

  • SnowSultanSnowSultan Posts: 3,803

    I'll re-render the scene tomorrow morning without the denoiser and compare. I did all the postwork on separate layers and left all the color corrections as adjustment layers, so I should be able to see an accurate comparison.

    Matt, when you say "Day for Night", you mean to render nighttime scenes with more daytime lighting? I can see that, most art and even cinema make nighttime or unlit dark indoor scenes much brighter than they would be in real life. For this particular scene, I used an moonlit HDRI with the brightness up a little bit and did some minor color correction. It's probably still brighter than it really would be, but it's fairly dark overall.

    LOL 3DOutlaw, I'm only waiting more than two hours for a render if I'm being paid to make it.   ;)

    Padone, that's a good idea too, I'll have to try a canvas sometime. I normally hate the things because I can never get predictable results with Photoshop's EXR conversion, but if it helps with postwork, it will be worth trying to figure something out.

     

  • Matt_CastleMatt_Castle Posts: 3,097

    Matt, when you say "Day for Night", you mean to render nighttime scenes with more daytime lighting? I can see that, most art and even cinema make nighttime or unlit dark indoor scenes much brighter than they would be in real life.

    Particularly with older camera technology, a common approach to shooting night time scenes is to shoot them during the day, but with high contrast and a blue tint to the colour temperature. The combination of the contrast and tone means that (if done well) the audience will naturally accept that the scene is supposed to be dark, even if what they're seeing is actually pretty bright.

    It also works well for certain digital image formats. The dynamic range of 8bpc isn't great even if you're using all of that range, but if you're trying to do a very dark image, you will often end up with this sort of colour banding (or just something that displays poorly if your monitor isn't calibrated the same as your audience's. I have sometimes seen people render images so dark they're barely perceivable).

  • nonesuch00nonesuch00 Posts: 18,830

    Thanks for the replies. Yeah like Richard said, letting the denoiser go a long time will cause more details to be retained. I might redo this render without the denoiser (when I have two hours to kill) and compare with all of the same adjustment layers as my current one.

    Richard, the scene is supposed to be at night, so the histogram is mostly tilted towards the blacks (left side), but not extremely so. I'm checking them in Photoshop CC 2020 and double-checked the color management settings, that hasn't been changed.

    Here are two cropped areas of the pic I'm working on that show the posterization pretty well. They might be really hard to see against a white background though.

    If using the denoiser is causing more details to be detailed as the render goes longer and your renders are not getting the details you expect then there is only one conclusion left; and that is the models and lighting combo you have set up don't have the details you think they have.

    An overly brightly lit scene will loose details as a render proceeds as the render colors in pixels than converge to the brightness of the excess bounced light in the scene, denoiser or not.

  • SnowSultanSnowSultan Posts: 3,803

    OK, 3000 iterations later as a Beauty canvas, placed in the same PSD as the first version, and...it's the same, only grainier.  These are the kind of days when I wish I could draw.   ;)

    Maybe it's just too dark, I don't know. Don't care at the moment either, I'll just spot render the bad areas and finish this thing for now. Thanks for the suggestions, I'll experiment later and hopefully be able to figure something out.

    shoulder2.png
    173 x 89 - 20K
  • Snow Sultan, use the Intel denoiser mcasual has a script. Or better yet, Blender beta 2.81 just got a new denoiser that has albedo, and everything render scene in Blender with Diffeomorphic plugin daz to blender 1.4. Works well you may have to tweak stuff but overall works pretty well.

  • mclaughmclaugh Posts: 221

    Matt, when you say "Day for Night", you mean to render nighttime scenes with more daytime lighting? I can see that, most art and even cinema make nighttime or unlit dark indoor scenes much brighter than they would be in real life. For this particular scene, I used an moonlit HDRI with the brightness up a little bit and did some minor color correction. It's probably still brighter than it really would be, but it's fairly dark overall.

    See this thread: tutorial-shooting-day-for-night

  • SnowSultanSnowSultan Posts: 3,803

    I may have narrowed down the problem, but to fix it, I'll need an answer to a question that I've asked before and never gotten a really accurate response to. No guesses please; if I have to write to Rob directly and ask him to get a straight answer, I will.  :)

     

    In the Advanced tab of the Render Settings panel, what values should be entered into the Medium and High Texture Threshold fields to disable ALL texture compression under all circumstances? I don't care if it slows the scene down or even crashes Studio, I need to test this with full quality 4K textures.

  • Both values would need to be larger, by at least one, than the longest side of any image used as a texture in the scene.

  • Matt_CastleMatt_Castle Posts: 3,097

    As long as the values are higher than the texture size, then the textures should not be compressed. (The compression works poorly on very small textures, hence why these are used to set a minimum threshold for compression).

    Although these values can't be set to more than 10000, it's rare to see anything try to use more than 8k textures, so in practice this effectively does disable compression.

  • SnowSultanSnowSultan Posts: 3,803

    So entering 4097 or higher in both boxes will effectively disable any compression of any texture 4K or less? I will try that and re-render, thank you.

    I still don't understand how to properly use those settings though. What value do you use if you *do* want 4K textures compressed to 2048 for example? And which box do you enter it in? Do the Medium and High settings refer to levels of compression (somewhat compressed or highly compressed?) or at what size a texture is compressed by a standard amount?

  • Matt_CastleMatt_Castle Posts: 3,097

    What value do you use if you *do* want 4K textures compressed to 2048 for example?

    For that, you'd need to use something like Scene Optimiser.

    The compression here refers to levels of compression, rather than resizing of textures. Medium compression is fairly low loss, High compression is a bit more lossy and can have problems with very densely patterned textures (Denim is often a problem). The values set the texture resolution at which the given level of compression is used.

    If you have it set to say, Medium 512, High 2048, then a 1000px texture would be compressed to Medium level, but a 2500px texture would be compressed to the High level.

  • SnowSultanSnowSultan Posts: 3,803

    Thank you, that makes a lot more sense.

    If you don't mind, could you let me know if these render settings could be improved? I did a spot render with the compression values set to 4097 (so it wouldn't activate), and while it looks better, it's still a bit blotchy. Basically, I have time and quality disabled, anywhere from 4000-6000 iterations, firefly filter is on, but the nominal luminance is 0 and noise degrain filtering is also disabled at 0, pixel filter is mitchell, radius is 1.00. I use bloom and the denoiser depending on the situation, denoiser kicks in at 8 iterations, bloom is set low, 10000 threshold I think. Thanks in advance for your time.

  • xyer0xyer0 Posts: 6,408
    edited November 2019
    3dOutlaw said:
    Back in my day we didn't use a denoiser, we rendered for 32 hours then shrunk the image by 4x... and we liked it. Freaking kids these days...

    LOL

    Post edited by xyer0 on
  • Matt_CastleMatt_Castle Posts: 3,097

    If you don't mind, could you let me know if these render settings could be improved?

    To be honest, I can't claim to be an expert on the best possible settings, particularly as I'm on a fairly low spec system that usually dictates what I can actually reasonably handle.

    I normally handle denoising and bloom as post-production, as it gives me a lot more control over them (without completely having to re-render if they don't work out).

     

  • DustRiderDustRider Posts: 2,904
    edited November 2019

    This is purely a WAG, and I'm not in a position to check that I'm even on the right track. But I think having the denoiser kick in after just 8 iterations might have something to do with the problem. Maybe try setting it to 24 or 32, or even 100 to see if it improves things. My reasoning here is that in a dark environment there isn't a huge spread in the pixel values in the samples shown, and the denoiser may be rounding the values enough to cause the "banding " you're seeing. 

    Sorry in advance if I'm way off base here.

    Post edited by DustRider on
  • Thank you, that makes a lot more sense.

    If you don't mind, could you let me know if these render settings could be improved? I did a spot render with the compression values set to 4097 (so it wouldn't activate), and while it looks better, it's still a bit blotchy. Basically, I have time and quality disabled, anywhere from 4000-6000 iterations, firefly filter is on, but the nominal luminance is 0 and noise degrain filtering is also disabled at 0, pixel filter is mitchell, radius is 1.00. I use bloom and the denoiser depending on the situation, denoiser kicks in at 8 iterations, bloom is set low, 10000 threshold I think. Thanks in advance for your time.

    I found abnormalities in my renders in any 4.12 flavor of DAZ when using Mitchell if I had it lower than 1.50. I noticed you have it on 1. Could try 1.5 and sharpen in post if you need. PiXimperfect has a method I think is phenomenal and use all the time (https://www.youtube.com/watch?v=Slkj-Xz4nHM). It seems I read on the forums here some months ago about 4.12 treating Mitchell different than in previous versions but may just have been a bad dream.

  • SnowSultanSnowSultan Posts: 3,803

    Thanks Matt, no problem.

    Dustrider: I tried that yesterday after you mentioned it and the denoiser never kicked in on the large image at all, not at 8, 50, or 100. It did work on a quick test picture, so I have no idea what's going on with that thing.

    Shane: That's interesting, I was just experimenting a little with that option too but didn't notice any difference. I'll try some test renders today with the different methods and compare. Thanks.

Sign In or Register to comment.