[SOLVED] Memory errors remain after buying brand new GPU (GTX 1080Ti-11GB)
PencilArt
Posts: 10
Hi,
Daz crashed all the time during rendering with memory errors in the log file. I thought it was because I reached the memory limit with my older GPU. I'm aware of possible memory leaks after a couple of renders. But even after a clean start, Daz crashed while rendering. So I decided to buy a complete new PC with 16GB of RAM and a GTX 1080Ti and 11GB of VRAM.
To my suprise the memory issues remained. Everything below 4 GiB works fine. Bigger scenes from 5 GiB and larger make Daz crash during rendering. There are "out of memory" messages in the log file. When I check the memory of my GPU with a nvidia diagnostic tool, there is only 45% of memory in use. I tried all kinds of benchmarks (3D mark, Octanebench, etc...) and they run fine on my GPU. I have no reasons to suspect any hardware issues whatsoever.
In the render settings I checked my GPU in the Photoreal tab and everything else (CPU, OptiX) is unchecked. I tried rendering 3 Iray cars (Roland, Astro and Ranger). All combinations of 2 cars are rendering fine. The three of them combined are making Daz crash.
Any advice how I could resolve this issue? Could this be a bug in Daz Studio? I'm trying to master Daz studio, but it's hard to keep the spirit up when it crashes all the time.
+++++++++++++++ DAZ Studio 4.10.0.123 starting +++++++++++++++++
Platform bits: 64
Running on Windows 10, Build 9200, No Service Pack Installed
CPU Information:
CPU String: GenuineIntel
CPU Brand String: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
CUDA rend info : Found 1 CUDA device.
IRAY rend info : NVIDIA display driver version: 398.36
IRAY rend info : Your NVIDIA driver supports CUDA version up to 9.2; iray requires CUDA version 8.0; all is good.
IRAY rend info : Using iray plugin version 4.6-beta, build 287000.7672 n, 23 May 2017, nt-x86-64-vc11.
IRAY rend info : CUDA device 0 (GeForce GTX 1080 Ti): compute capability 6.1, 11 GiB total, 9.14924 GiB available
IRAY rend info : Rendering with 1 device(s):
IRAY rend info : CUDA device 0 (GeForce GTX 1080 Ti)
IRAY rend info : Rendering...
IRAY rend progr: CUDA device 0 (GeForce GTX 1080 Ti): Processing scene...
IRAY rend stat : Geometry memory consumption: 69.3721 MiB (device 0), 0 B (host)
IRAY rend info : Initializing light hierarchy.
IRAY rend stat : Texture memory consumption: 5.40869 GiB (device 0)
IRAY rend info : Light hierarchy initialization took 0.36s
IRAY rend stat : Lights memory consumption: 9.68192 MiB (device 0)
IRAY rend stat : Material measurement memory consumption: 0 B (GPU)
IRAY rend stat : Materials memory consumption: 339.094 KiB (GPU)
IRAY rend info : CUDA device 0 (GeForce GTX 1080 Ti): Scene processed in 54.075s
IRAY rend info : CUDA device 0 (GeForce GTX 1080 Ti): Allocated 36.6213 MiB for frame buffer
IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): out of memory (while launching CUDA ior stack renderer in core_cuda_host.cpp:209)
IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): Failed to update ior stack
IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): Device failed while rendering
IRAY rend warn : All available GPUs failed.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
It's not a bug in Studio. Rendering those three cars in Iray is no more than 2GB.
Are you running the Nvidia GTX driver? Windows 10 typically replaces Nvidia's driver with it's own. Use a genuine Nvidia driver, however don't use the 391.35 version, it has been causing problems for people. Use and older Nvidia driver.
The Nvidia driver you are referring to was released in march 2018. There have been 6 updates since then. Do you have any reference stating that drivers released after march 2018 are not compatible with Daz Studio and should not be used?
I have installed the latest native nvidia drivers yesterday. As you can see in the log file I'm using the nvidia driver 398.36 which was released by Nvidia on june 26. Unfortunately it didn't resolve the issue.
I had a 1080ti that I used to use for renders and found that overclocking it resulted in instability often causing the render to experience this error. So if you have it overclocked drop it down some. I was able to maintain an overclock but had to find where it was stable for Daz iRay. This may not be the bottom line issue for you but it is what happened to me.
Forgot to also mention that I ended up using a previous version driver as the latest at the time, been a few months, was unstable for Daz iRay. So try using a previous version driver for the card as well if the OC is not helpful.
Thank you all for your response and advice.
My 1080Ti is factory overclocked. I tried downclocking it, but it doesn't seem to have an effect on the performance. I'm not sure it's adjustable.
I tried several older versions of Nvidia drivers, but unfortunately the issue remains.
It's sad that after investing 2500€ in a new system I hardly made any progress. I could render larger scenes with my old computer.
I do not mean to circle around the same premise but I understand how frustrated you are after investing money in more than capable hardware, much the same as I did, to not have it work so I want to provide other ideas for this that may lead to a solution.
These are the other things that come to mind (order I would check):
Special mention* I dont think this would be an issue or shouldnt be, but i noticed you mentioned having OptiX unchecked. Have you tried with it checked? Also, when you changed your GPU drivers each time did you do a complete uninstall, then reboot then reinstall? That has made a difference for me before when trying different drivers, not for daz specifically but other builds i've done and discovered flaky perf in some apps.
1) Have GPUz and task manager open to monitor making sure that the vRam is indeed released/free before hitting a render. For some reason when i rendered on my 1080ti i constantly had to close daz and even kill the process in task manager to release the held vram and then after I verified it was showing in gpuz as not used/free I would then relaunch Daz and open scene and render. (I feel this could be a quick simple thing to verify and even before hitting render you can guage somewhat how much of the vRam is being utilized already with the iRay preview view).
2) Try to load another large scene that doesnt use the same models/assets to see if maybe something from the saved file or those assets could be causing issue. Also make sure all your surfaces are using the uber base sahder so that iRay doesn't run into an issue, though I dont think this would cause the issue your seeing. Note* With geforce cards wddm for windows reserves card memory essentially stealing it from what you have available to use if on windows 10 its larger amount and based on the log you posted your scene was only 5.xxGiB for the largest portion consumed so you should be good there, and you may already know about that.
3) Especially if on Windows 10, verify that you have at least the creators update v1709 (some people have reported some errors with the very latest windows build but not sure that is related to the error your seeing), and if you dont already have that then update to that.
4) If your cpu and memory is OC'd set them back to defaults and test as anything in the pipeline could cause an error really.
5) Backup your library and then uninstall Daz and DIM and completely remove the current library from hdd/ssd and reinstall daz/dim and either move back over your library or download all fresh with DIM.
I really hope you are able to get this working.
And just to chime in with my silly question of the day, as it is/was over-clocked is it being adequaktey, and consistently, cooled?
Strange...
You had a problem on an old PC. You had the same problem on a brand new PC, with a new GPU and new hardware. Was ALL of the hardware new? If it was new, and all software was newly re-installed from scratch with no old corrupted files lying around from the old machine, then that's really strange.
Clearly, many of us have been using Studio for rendering big scenes on a 1080ti with no issues, including being able to access up to 10GB of the 11GB VRAM with big monster scenes. I'd focus on any hardware or software you moved over from the old machine to the new machine. Try to unplug/uninstall it and all associated software to take them out of the equation.
And I've become rather wary of the iray log file reports, since I think some of the "memory available" data is questionable to say the least. So I'd be skeptical that an "out of memory" report in the Iray log actually means it's out of memory. Maybe it's something else.
I doubt if there's a Studio bug, for the reasons I mentioned. Others are using it fine. Although I suppose it's possible that if you moved some Studio install (or other) files from your old machine to new machine, and they were corrputed somehow, that might explain something.
I'd try to start as much from scratch on the new machine to make sure you didn't bring any cooties from the old machine. And also take a look at the Security and Maintenance/Reliability History log to see if anything weird has been going on lately. Though if it happened on the old machine I wouldn't expect it to tell you much.
Thank you all for your comments. Thank you shaneseymourstudio for your checklist of things to try. Thank you ebergerly and SimonJM. It keeps my hopes up.
I did not reboot after each nvidia driver installation. You certainly have a point there. I will start over and try that again.
I use nvidia-smi for checking the available VRAM. I run it all the time to see if Daz "forgets" to release VRAM after rendering. At that point I restart Daz Studio and all is good again.
I have a complete new PC. No parts were re-used from my old computer.
Intel Core i7-8700 (3.2GHz)
16 GB DDR4
Inno 3D iCHILL GeForce GTX 1080 Ti X3 11GB GDDR5X
Power supply Seasonic 1000W
Is 16 GB RAM enough? I have 2 memory slots free. If it really matters I could increase that to 32 GB.
I also have an onboard display adapter Intel UHD Graphics 360. I tried connecting my monitor to the onboard display adapter. It frees up even more VRAM. This way there is only 227 MiB VRAM in use when Daz is running (according to the nvidia-smi tool). There is a screenshot in attachment. Unfortunately rendering this way didn't fix the issue.
I did try different combinations with Optix on or off. The issue remained.
The cooling looks fine. I don't suspect any issues there. I have been running 3DMark alot and the GPU performs as expected. I tried the Octane benchmark and also no issues there.
I have some more testing to do and will report back soon.
Thank you all so much for your help.
What's so strange is that it occurred on two completely different computers. So heating/cooling should be irrelevant, as should corrupt drivers, as should hardware issues, since it's two completely different machines, and highly unlikely they'd occur on both machines.
The only thing that makes some sense is that something (software or hardware) came over from the old PC to the new one and that is continuing to cause problems.
Again, I'd remove anything that is common between the two, and check Reliabiity History to see if it tells you something
I would definately try increasing to 32GB ram if you have the immediate option to do so just for testing. May eliminate more difficult steps if that helps with the issue. On some scenes I had seen as much as 22-28GB of ram being used before offloading it.
Well if you check your Performance Monitor and your system RAM isn't being nearly maxed out by Studio when rendering, then why suspect it's running out of RAM?
EDIT: BTW, I just loaded a scene that Studio took about 6GB of my system RAM, and 3.8GB of my GPU VRAM. So a ballpark estimate for using your entire, say, 10GB of VRAM would be that your system RAM in that situation might be around 10/0.6, or about 16GB. So if that's a reasonable estimate, 16GB should be right in the ballpark of what will fit on your 1080ti. And it sounds like your scene is nowhere near that, so I'm not sure system RAM is an issue.
Perhaps the title of my forum post was a bit confusing. The memory errors on my old computer were caused by normal behavior. I just didn't have enough VRAM on the GPU. I thought I would be rid of those errors when I bought myself a more powerful computer and GPU. Hence the title.
I think I tried everything, except reinstalling Daz. I already did a clean install when I bought my new PC a couple of weeks ago. I also got all the Win10 updates installed.
I tried different GPU driver versions. I tried different scenes and props. I tried all settings. But as soon as the used VRAM is between 3 and 4 GB, the "GPU out of memory" error kicks in and Daz falls back to CPU rendering and usually crashes. It almost makes me suspect that the memory on the GPU is fake or empty. -.- That would be something. But according to Daz Studio and Nvidia tools there is a beefy 11 GB on the GPU.
I still hope this is some kind of bug in Daz or Iray in combination with my new hardware. My old computer and GPU could handle bigger scenes. This turned out to be a terrible waste of money.
Perhaps I could file a support ticket and see what comes up. And in the end I can still give Octanerender a try.
Thank you all for your comments and advice.
In another recent thread a user was having the problem of memory issues rendering an image sequence, dropping back to CPU after the first image rendered on the GPU. The suggestion was to use a good nvidia driver that has worked for some, 388.59, and to not use Optix as it has been having issues according to some.
I bought a new computer back in 2016. It came with 16 GB ram. Just because it was inexpensive, I went ahead and ordered another 16 off Amazon. Easy to install and...not expensive.
Turned out to be an excellent decision. I'd watch scenes queue up for my 1080s and, for some reason, they'd eat up 20+ GB of system ram before loading onto the 8GB cards.
So, I agree with shane - increase to 32GB.
Disclaimer: I'm NOT an expert. This is just my opinion based on my experience watching how the system ram is used on my Windows desktop when rendering (ctrl-alt-del task manager).
Even if more ram didn't help, I still think it's worth it. If it were me, I'd then continue by installing GPUZ and doing stress tests (google: 'gpu memory tester') that use non-Daz programs to ensure the card is functioning properly.
While adding additional system memory isn't a bad idea, it has nothing whatsoever to do with the error being reported. In other words, its a solution in search of a problem.
What concerns me about the behaviour is not only does the GPU render fail before the cards memory is used, but that DAZ Studio "crashes". If insufficient memory on the video card were encountered, the normal behaviour would be to default to the CPU and carry on. Are you really saying that DAZ Studio actually crashes back to the desktop, or that the GPU rendering switches to CPU? That's an important difference in troubleshooting the issue.
Have you experienced any issues with other GPU-intensive applications on the new PC?
(BTW, the factory overclock should not be an issue as they tend to be conservative and well-tested)
I had a similar problem when rendering 3 characters. My ancient system was set in the bios to use primary on-board graphics(Intel 3000). I thought leaving my 2Gb Nvidia 950Ti to handle ONLY rendering kept maximum memory for that task but it seems that caused problems. Since changing the 950 to primary in the bios all has been working fine. Easy to check and costs nothing.
One other thing to check - try rendering the 3 cars after setting ALL surfaces to Weighted in Base Mixing and Glossy Weight to Zero - if it renders ok the problem might be your system(s) trying to calculate too many reflective surfaces.
Good Luck
The issue has been resolved!! Thank you Shane and Toonces for the golden tip! I'm so happy and excited. I included my test render of 3 Iray cars. Nothing fancy, no lights no nothing, just 3 cars. Isn't it wonderful
The solution was logical really and it all makes sense to me now. It was after buying an extra 16 GB of RAM that things became clear. The culprit was the "virtual memory pagefile" setting on my computer - it was set to fixed. Normally Windows 10 defaults to "automatically manage paging file size for all drives", but for some reason the shop where I bought my computer must have turned that option off.
The following happened while rendering:
Daz started filling up memory pagefiles. There is about 21.4 GB of pagefile memory needed to render my 3 cars. It takes a while and I could see the memory increase in task manager.
Because my pagefile memory was set to a fixed size (17GB), DAZ didn't have enough memory and it crashed. The "out of memory" messages in the log file were actually about system RAM and not about VRAM. That was the confusing part. It would also be nice if Daz could avoid the crashing part in a future update.
After increasing my RAM to 32 GB and setting the pagefile size to automatic, I can completely max out my 11 GB of VRAM without any issues. Optix settings included.
I'm still pretty shocked at the amount of system RAM Daz needs before it starts loading up VRAM on the GPU. I tested out "Chateau Blanche", a big castle. The pagefile memory went up to more than 40 GB(!).
I learned alot the last few days. Thank you so much for the massive response. This is a great community. I hope this thread might be helpful to others who experience the same issues.
Awesome! I am very happy you were able to resolve this and didn't waste money!!
This thread may help future new builders as well troubleshoot this issue.
I'm glad you got it fixed, though I'm kinda confused on what the actual issue was/is...
Sounds like you bought 16GB more system RAM, for a total of 32GB. And you set the pagefile to the Windows default of "auto" (aka "system managed size"). And now you can load a monster scene that generates a 40GB pagefile, which is more than the new installed RAM? And that's good, because that's what a pagefile does, it's a virtual RAM on disk in case you run out of system RAM. I posted previously in another thread that I suspected that the pagefile settings might be a factor in the long-running "myth" that W10 grabs a bunch of VRAM, and I manually set mine (on 64GB system RAM) to 16/32GB. Though I'm still not absolutely certain how or if that affects the VRAM usage. But with that 32GB manual setting my machine runs fine rendering scenes that take 40+GB of system RAM and fill my 2 GPU's (1070 and 1080ti) with no problem. So for me a manual pagefile setting works even if it's below the system RAM usage, but for you it didn't work that way on your first machine?
Confusing...
Anyway, what doesn't make sense for me is why your old computer had the same problem as your new one. Completely different hardware and software, and presumably the pagefile was set to "auto" on the old one. Yeah, your old PC had a lot less RAM apparently, but why wouldn't the old pagefile allow you to load a bigger scene like your new one does? You said the new one with 32GB system RAM loaded a 40GB pagefile? Why did your old machine run out of memory with an "auto" pagefile?
Also, you said you "maxed out" your 11GB VRAM on your 1080ti. What does "maxed out" mean? I get 10GB of my 11GB 1080ti utilized for my biggest scenes.
Also, when you say "pagefile memory went up to more than 40 GB(!)", is that the actual size of the pagefile.sys file on the hard drive? Because on my 64GB machine, if I set the pagefile to "system" (auto), the actual size of pagefile.sys is only just under 10GB, and that's before and while doing a render that fills up over 42GB of my system RAM. The pagefile size never changes from its system-determined 10GB, which is one reason I manually set a size range of 16-32GB.
I'll try to explain in some more detail.
In my experience, 3 types of memory are used during rendering:
VRAM (GPU), physical memory (system RAM) and pagefile size (virtual memory)
You will want to pay close attention to "Committed Memory" (= system RAM + pagefile size). You can monitor this with Task Manager (performance tab, memory).
When you hit the render button, Daz starts filling up Committed Memory. In task manager you can see it increasing. This takes from 30 seconds to a couple of minutes depending on the size of the scene. When the Committed Memory stops increasing, Daz starts filling up VRAM. This is really fast and takes less than a second. After these 2 steps have been successful the render process will start running on the GPU.
In step 1 when the available Committed Memory runs out Daz will close with an error message or just crash. Either way, there will be an out of memory message in the log file. This won't happen when the paging file size is managed by Windows automatically, as I already mentioned in my previous post.
In step 2 when the available VRAM runs out Daz will try to switch to CPU rendering and there will also be an out of memory message in the log file.
My scene with 3 cars used up 21GB Committed Memory and 5.5GB VRAM
Chateau Blanche (very big scene, had to delete some textures): +40GB Committed Memory, +9GB VRAM
I hope this helps a bit.
Yes, but you never explained why the same crash happened with your old computer, which presumably had a system-managed pagefile, which means that (assuming you have enough hard drive space), it shouldn't have crashed. But you said it did. It should have filled up system RAM, then the pagefile, and when it converted that scene data to Iray calcs and filled up the GPU VRAM, and found that to be insufficient, it still should have rendered on the old machine via CPU.
Unfortunately the old computer has been reset with a clean install. I cannot test it anymore. There was not much room left on the SSD hard drive. It might have been something else. I'm not sure.
But the new computer works like a charm now. Been rendering alot without any issues.