Daz Render keeps resetting?
sunnyjei
Posts: 497
Not always but many times now when I try to render something it will start to render to a certain seemingly random point and then it will restart the rendering iterations. The rendering window of the image resets back to an earlier/unclear version and then Daz starts to render again. Once it hits the same iteration number again it restarts. Anyone else had this problem or more importantly knows how to fix it? Thanks in advance!
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
is this 4.20.0.2 or an earlier version?
For me 4.20 but didn't want to limit it to that incase someone had it happen with an older version. I feel like I read about it happening to someone else with an earlier version but google and daz forums aren't giving me any matches
Do you use denoiser?
Yes, it keeps happening even when it's turned off (or on).
What does your log say (Help->Troubleshooting->View Log File), do not copy the text to the message area but attach the file to your post with "Attach a file" above the "Post Comment" button
I attached the log. All I could make from it is that denoiser failed a lot and running out of memory. In the past if an image/scene was too large/complex it would just crash but I never had problems with using the Spot render tool to render in small but now that also often gets to a certain number of iterations and resets
I am surpriced that you can render in Iray at all with a 4GB GTX card, but the change logs gave the impression that they have made some changes.
One change that jumped out, was releasing the memory when running out and recognizing that when it happens, and that may have something to do with your situation.
Starting rendering... Running out of VRAM... Dropping back to CPU and starting over... Releasing memory, Hey there's VRAM available, I better switch over, starting over again... Running out of VRAM... etc...
For the most part my gtx and Daz have worked well together- if at times slow. If I threw something too heavy/complex at it Daz would just crash. This getting so far and resetting by far is worse as sometimes it gets 20-30 minutes in and resets -at least if it immediately crashed I could try to reduce stuff before investing another 10-30 minutes into a render fail.
@sunnyjei here's the key line from your log file:
Iray's Scheduler is the master process that oversees how the actual processing cycles needed to complete a given render are split up between multiple render capable devices (either individual GPUs and/or individual CPU threads.) Although it is designed to handle individual rendering devices dropping out/coming back online during a render without losing overall progress, there is only so much disruption in the functioning of the rendering devices it manages (and their subordinate resources like GPU's VRAM) it can take before the entire rendering task goes sour and the scheduler itself needs to restart - necessitating a render startover at iteration 0 (what you are seeing.)
By the looks of your log file, your issue is with vram - specifcally that the scene attempting to be rendered here is dangerously close (but within the limits) of your GPU's vram capacity and that you are a multi-tasker. Since all running programs on a single-GPU PC necessarily share the same pool of vram, it is only a matter of time before some other program (web browsers are famous for this) also running while a render is going causes total vram usage to go past your system's capacity. Thereby triggering an Iray out of memory error and forcing Iray's Scheduler to drop the GPU and fallback to CPU (losing denosing in the process since that is currently GPU only.) As soon as overall vram usage drops enough to accomodate the scene again, Iray's Scheduler switches back to using GPU. Which, in your case, is quickly leading to another vram overload attempt and fallback. Which is leading to evenual Scheduler whiplash and a total restart.
My tips for possible solutions would be:
This last one won't help with your GPU falling in and out of service, but it might help against preventing the Scheduler from crapping out and having to start over at 0 iterations (since it means that the Scheduler should always see at least one device successfully working on the render, rather than there being gaps in time during which no devices are available and CPU fallback is being initialized/de-initialized as the GPU cycles in and out of service.)
If it helps, the same thing has been happening to me SINCE I UPGRADED TO 4.20.0.2.... but never before.
And I have a great PC and a very good NVidia graphic card.
I can'thelp but think that this is a 4.20.0.2 issue as the problem never happened before that.
Unfortunately, it isn't just you and I.....
From the digging that I've done of Google Search, there are similar threads going back to 2018, and each time there have been those who have offered their helpful advice on fixes. Maybe a VRAM problem, maybe a driver issue Maybe an IRay issue. Maybe this or maybe that.
Regardless, ever so often, the problem does pop up again for someone else. This time, it's you and I (and who knows else)
A question if Vram is full why do the Cuda cores stop processing?
Cards have a Graphics processor and Vram. Since Windows comes with an asynchronous queue for graphics memory. Even the Graphics Card device has a dedicated paging queue and requests are queued to the video memory manager thread pool for servicing various video memory manager requests. Every operation gets assigned a unique value that gets signaled when the video memory manager completes the operation. The paging is a regularly monitored fence object and can wait on these video memory manager services either on the CPU or on the GPU.
What am I missing?
I had this problem when I upgraded to the latest DAZ last weekend (11/03/2022 ish). I have a geforece RTX 3070 with 8GB of GDDR6. I'm running a liquid cooled i9 processor with 64GB of RAM. When running simple renders there was no problem, but on resource hungry stuff that I'd leave over night and/or whilst at work, I got the issue I think people are talking about.
Description - If shut all other applications down and run a juicy render at the 8 hour point it would restart, then after say 5h restart, then 2h restart ........30m restart and so on. And I'd have nothing to show for 24 hours of processing.
I fixed this by allowing my CPU to help out in the advanced settings.
I was worried that my renders would be slow, but the CPU clearly takes over at the point my render would be restarting, so it works quite well.
Whats happening, in my words. My graphic card runs out of memory at the 8h point. it restarts, but doesn't fully clear, so it then only process for 5h, the same thing happens again, but it can then only process for 2h and so on. Allowing my CPU to get involved, allows access to the 64GB of RAM and HD space, so the problem appears fixed.
Hope this helps
I've experienced this problem as well since upgrading to 4.20, which has a new version of Iray from NVIDIA. After reading a few threads about it, and checking my logs, I think I understand it. It's the Iray Scheduler causing the problem. When a render is running on CPU, the scheduler is deciding to try to run it on GPU instead, but if there's not enopugh vRAM, then it's switching back to CPU again, and over, and over again.
Work Around: I disabled CPU fallback, so if a scene doesn't fit on GPU it fails. Then for scenes that must be rendered on CPU, I disable GPU, and enable CPU. Scene renders like it should.
Clearly a bug that should be fixed.
And, again, one that I have NEVER experienced (and I've been using Studio since the "0.X" days) in all of the time that I've been using Studio and one that hasn't cropped up until 4.20
Based on some posts, the denoiser could also have something to do with this - Worth to try without denoising.
I have tried that.... problem is still there.
Now, I'm being advised by some friends that the best solution is to upgrade the GPU.... all for a "free" application.
G
The denoiser does not function at all when rendering on CPU, so that can't really be the problem. Also I never use it anyway. For people that do use the denoiser, I can see how the extra vRAM that it eats up could make the difference between CPU and GPU rendering.
It seems it's got something to do with changes to Iray, and how it chooses to use CPU or GPU.
....I wish there was another program I could use to render with it I might have to just figure out how to use blender because. I'm getting real sick of this...The application may be free, but we still have to buy the products to use in the application. I use the denoiser for test previews to get rid of grain faster.
I have this problem all the time after updating to 4.20. I also posted something similar to this in another thread. Basically my workaround is anytime I get into a state where the iray preview gets stuck in a loop or rendering stops at a black screen, relaunching Daz Studio and rendering the scene without making any edits will have no issues. If I start to move objects around (ex: Move a chair 2 feet to the left), then the rendering issues described in this thread can start to happen. Even though I did not add anything new to the scene that should impact memory. Just moving an object is all it takes. So my workflow is rebooting Daz Studio 10x a day each time I want to render my changes.
Fwiw this is a very different problem from the one described by the OP in this thread (who is talking about resets/failures in Iray final rendering - not Iray preview rendering.) It sounds to me like you need to tune your Iray preview settings (found in the Draw Settings tab when Iray is selected in the viewport DrawsStyle menu.) Particularly the value for "Response Threshold (msec)" found under Draw Settings > Editor > Drawing, where I have found that a setting which is too low can lead to pretty much exactly the behavior you are describing.
Thanks I will give that Draw settings suggestion a try. I should have clarified, I was having issues with final rendering the way it was mentinoed in this thread as well as issues with the iray preview. But perhaps the iray preview was a different issue. Hopefully the draw settings you suggested fixes that part of the problem.
Same problem here. Sometimes DAZ restarts the rendering process again and again, if i turn on the denoiser(default settings, kicks in at 8 Iterations). It renders exactly 15 iterations(it's always 15 when the bug appears, no matter what scene), then it falls back to the first iteration, moves on to 15 again and resets again, resulting in an infinite loop. Strange thing is, sometimes i just restart DAZ, load the exact same scene and it renders perfectly fine. Never seen this bug in a previous DAZ version, only occurs since the last DAZ update. Extremly annoying!
I would suggest you try to attach the log file (Help > Troubleshooting > Log file). That should list what is happening.
Well, i guess this might be the one:
2022-04-25 16:22:00.342 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Available GPU memory has increased since out-of-memory failure. Re-enabling CUDA device 0 (NVIDIA GeForce RTX 3060)
2022-04-25 16:22:00.407 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Initializing OptiX for CUDA device 0
2022-04-25 16:22:00.601 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering with 1 device(s):
2022-04-25 16:22:00.652 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : CUDA device 0 (NVIDIA GeForce RTX 3060)
2022-04-25 16:22:00.652 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering...
2022-04-25 16:22:00.652 Iray [INFO] - IRAY:RENDER :: 1.8 IRAY rend progr: CUDA device 0 (NVIDIA GeForce RTX 3060): Processing scene...
2022-04-25 16:22:06.168 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [ERROR] - IRAY:RENDER :: 1.6 IRAY rend error: CUDA error: the launch timed out and was terminated
2022-04-25 16:22:06.196 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [ERROR] - IRAY:RENDER :: 1.8 IRAY rend error: CUDA device 0 (NVIDIA GeForce RTX 3060): Scene setup failed
2022-04-25 16:22:06.196 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [ERROR] - IRAY:RENDER :: 1.8 IRAY rend error: CUDA device 0 (NVIDIA GeForce RTX 3060): Device failed while rendering
2022-04-25 16:22:06.295 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [ERROR] - IRAY:RENDER :: 1.8 IRAY rend error: CUDA device 0: cannot get memory statistics (the launch timed out and was terminated)
2022-04-25 16:22:06.311 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER :: 1.8 IRAY rend warn : CUDA device 0 (NVIDIA GeForce RTX 3060) is no longer available for rendering.
2022-04-25 16:22:06.312 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER :: 1.8 IRAY rend warn : All available GPUs failed.
2022-04-25 16:22:06.312 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER :: 1.8 IRAY rend warn : No devices activated. Enabling CPU fallback.
Also found these:
2022-04-25 17:51:37.388 [INFO] :: Unlocking viewport redraw...
2022-04-25 17:51:37.388 [INFO] :: Viewport redraw unlocked.
Otherwise there are just a million of these type of logs:
2022-04-25 17:54:46.460 Iray [INFO] - IMAGE:IO :: 1.0 IMAGE io info : Loading image "D:\...\Library\Runtime\textures\P3DA\HavenPark\P3DA_EndWall_N.JPG", no selector, pixel type "Rgb", 4096x4096x1 pixels, 1 miplevel.
So the gpu runs out of memory and falls back to cpu, which doesnt support the denoiser. But i can't believe that, because it happens with super small scenes as well. And even after some render loops, i still see the denoiser kick in at 8 iterations, so it's obiously still using the gpu. Plus i never had any issue with the denoiser before updating DAZ.
I have the same problem, but only very recently. My scene is fairly big (3x G8 + Environment) on a very old system (i7 3770K, GTX 1060, 32GB RAM). RAM usage is very high (already closed browser etc); seems DAZ uses even the HDD for a swap file. Dunno why, though; I had scenes with more G8 and they worked fine.
So I reckon it's something with the 4.20 version.
Differenet scenes will hae different footprints - the scene file is not the whole story, there are also the assets loaded from the Data and Textures folders, elements generated in-scene (such as strand-based hair), some of which may be affected by settings (HD mesh divisions).
Thanks for reminding me :D I'll try to reduce that kind of stuff. DoF will be my friend in this scene, I assume.