Switching to CPU rendering without maxing out GPU memory?
GreatGonzo
Posts: 58
Since changing my system I've hit an issue with from time to time Daz will switch to CPU rendering.
(The system change was a motherboard and CPU change and adding in an extra 1080ti)
It only happens when running a batch render, sometimes it will never switch to CPU, sometimes it will do so after maybe 5 images (same scene, characters and memoryt load). But if I restart the batch it's fine - which does not sound like a memory issue to me?
Checking GPU-Z the scene is pulling 3.5Gb on the 6GB 980ti cards and 4.7Gb on the 11Gb 1080ti cards so there seems like plenty of wiggle room on memory.
It doesn't appear to be thermal throttling as the render temps are:
CPU around 68c / 980ti's are at 68c / 1080ti cards are at 61c.
My only thought is that it could be because the system has a mix of GPUs? (2x 980ti and 1x 1080ti)
System: i7 7800x / MSi X-299 / 64Gb / 2x980ti / 1x 1080ti

Comments
That happens to me, too, when I restart a render a couple of times, or change camera position from one render of the scene to the next.
It seems that the memory of each rendered segment ist stacking up until there's no more space in the card's memory... But I can be wrong about that.
For one still open but finished or stopped renders keep draining GPU space away from what can be actively used. So where possible try to close render windows you don't need anymore. I never tried batch rendering but that might be the issue if those windows keep being open?
In the second case when you are still in the testing and do several renders with different lights or things like that, that seems to suck at GPU space as well. So whenever I am finished and have a rather complex scene I save, close DS start new and open the file fresh so it can start rendering on a clean space.
If you look at the DS log file (Help -> Troubleshooting -> View Log File), you'll probably see there's some kind of initialisation error. I've seen it happen on my single 1080ti card, sometimes with a scripted batch render, sometimes after I've restarted a render or switched camera a few times. Seems like something doesn't get reset or cleared properly between renders under some circumstances. At that point I find I need to restart DS to get GPU rendering back again.
It happens often enough that I don't walk away from the computer until I've verified that it's GPU rendering.
I'll check the log file next time it happens, due to my workflow it comes out at a daily 30-40 renders so I save the duf files for each panel seperately and then batch render the lot whenever I've finished / over night. So while they start off rendering fine something drops out after a time, but only sometimes - it seems to be completely at random and of course once it's happened it just keeps on rendering of CPU for everything else.
Happens to me as well. I often have to save and exit, re-load the scene to correct the issue. Like TangoAlpha, I also confirm Gpu rendering before letting the computer do its thing.
I need to learn your workflow! I'm good when I get two renders sometimes three when there is not much to change in the scene. 30 renders would be a dream...
I get this problem too. There's a scene I'm working on right now with a few dinosaurs in a Terrodome 3 landscape, and I can only do maybe 2 test renders after making adjustments to something in the scene, then on the 3rd, it'll drop to cpu rendering. It's pretty annoying to have to keep exiting and reloading. If only there was a way to flush the gpu memory without having to close DS.
I just want to jump in real quick and say that I believe that this has nothing to do with batch rendering. I cant do batch rendering, my video card wont take it. I do single renders, saving and exiting DS after each is complete because if I dont this, the issue will happen. This can happen to me after adding a new object, changing the lighting or a shader,etc. I always save and purge memory. I do use the Aux Viewport alot though, maybe that has something to do with it?
Also have the same issue, and also during batch renders. I have the same sort of work flow where I usually run renders overnight, and sometimes when I have five or more scenes it will still be running on CPU the next morning.
I posted awhile ago asking if there was some way to purge the memory without restarting DS, and I guess there isn't.
There is a way to purge memory in DS but I dont think it works as well as it should. I use it all the time and still have this issue. Its in your scripts > utilities folder.
Neat-o, I'll look into that, thanks. I wonder if a batch render script that also purged the memory in between loading files would help avoid things going to CPU.
What would be nice is a batch render programme that purged the memory each time it loads a scene... but I assume that does not exist because it would make sense. :)
I've noticed that when I'm rendering an animation (oh so slow but beside the point), checking with GPU-Z shows that the max VRAM stays pretty constant over the entire run. I render to an image series so it is like starting a new render for each frame except that the VRAM doesn't drop to the minimum but to an intermediate level between frames. That suggests there is something clever going on with memory management.
I've known renders drop to CPU even after they have been rendering on GPU for some cycles. Very annoying; fortuneately it is pretty rare though.
Actually I have seen that. But only when it was getting marginal on the vram usage. It woud get up to maybe a hundred or two cycles, and then drop to CPU. I took it as a sign that it was about time to run Scene Optimiser!
Gotta love Scene Optimizer. Daz should buy it from V3Digitimes and incorporate it into the DS Architecture.
For those who dont have it yet, https://www.daz3d.com/scene-optimizer , its worth having.
It's wishlisted; I keep forgetting about it.
...I remember something a while back about VRAM not clearing when performing subsequent renders in the same session. Usually it meant having to shut the Daz programme down and restart it to clear everything. There is a Purge Memory script, but I believe that only works with physical memory not VRAM.
Maybe we need a Purge VRAM script?
I use it all the time though I rarely (if ever) render extreme close-ups. I consider it a must-have.
In my experience one thing that helps a lot for memory and stability is to use a separate card for the viewport. Or use the cpu gfx if you only have one card. For example, in a desktop with a i7 and a 1070, you wire the mobo to the monitor so to use the i7 for the viewport. Then in the iray panel you set only the 1070 for rendering.
Then keeping the aux viewport closed and rendering directly to file (in render settings) also helps.
...or use a lower power GPU card just for the displays.
It works marvels, very recommended.
I use it for minimize textures at a minimum of 2048px, textures below that remains intact, and gettting rid of bump,normals or displacement on distant props.
Interesting development on this, I first hit this issue to the extent it became a problem when I installed a new GPU (Palit 1080ti). Eventually I got grumpy with it enough to do a clean install of Windows, Daz and Drivers and that seemed to fix the issue for about 95% of the time. A few days ago I installed anouther 1080ti (Gigabyte) and guess what... It's crashing to CPU render on every third or fourth render which makes my workflow unworkable again.
So, it seems to be (in my case anyway) something to do with when new cards are put on the system - Which would lead me to think it's 'probably' a driver issue.
I'll strip out the drivers that are there with Display Driver Uninstaller and put a completely fresh install and see if that makes a difference and let you know what happens.
Technically in batch rendering that could be solved possibly with something like Optane cachng the main scene files. Update, clean drievr install and and so far no crashes, or switching to CPU rendering
I'm curious, are you monitoring the GPU VRAM usage graph in GPU-Z or Task Manager to see what's going on between frames? Is the VRAM usage dropping after a frame is rendered, or does it stay somewhat fixed and then the next frame fills even more VRAM?
Using GPU-Z for monitoring RAM, not checked between frames from memory I'm pretty certain it does drop off, but will have to check on todays render batch.
But yesterdaty I noticed something potentially REALLY strange (I may possibly be reading this wrong or misunderstanding). According to the render log I have a scene that is loading in 9.76Gb of textures into the Vram. (see image). This is way above the two smaller cards but they all seem to be working, but drawing different amounts of Vram. The larger 1080ti cards have 11Gb and are pulling a couple of gigs more that the 980ti cards (which are both consistant at just over 4Gb). The Draw on card 0 will be slightly highter due to it being the main display card.
GPU-Z puts the draw on the cards at:
Card 0: 1080ti 11Gb - 5,976Mb (display)
Card 1: 1080ti 11Gb - 5,794Mb
Card 2: 980ti 6Gb - 4,175Mb
Card 3: 980ti 6Gb - 4,175Mb
Which appears to make no sense if the render log is saying there is somhow 9.76Gb being loaded onto a 6Gb card (Card 2) unless the 1080ti cards are somehow using the free memory to compensate for the smaller cards?
Any ideas?
First of all, the only reason I can see for the smaller cards to be using less VRAM than the larger cards is that the entire scene must fit on each card for it to participate in the render. Iray/CUDA will take the scene and start filling each card, and if one card runs out of VRAM it will stop filling and the larger cards will keep filling until the entire scene is loaded into VRAM. And when the render starts only the larger cards (which were successful in loading the entire scene) will participate in the render. So if there's a big discepancy in the VRAM utilizations among the cards you might want to check to see if some aren't participating because they ran out of VRAM.
As far as Iray reports in the log:
I tend to ignore them, since I have no clue what they REALLY mean, and I tend to believe Windows 10 since it is responsible for actually scheduling the VRAM, along with CUDA. Unless you have actually done detailed software development with Iray and know what the reports are actually referring to (which very few people do), I'd encourage everyone not to engage in the internet's #1 most popular sport: JUMPING to conclusions...
@gonzo_529d315939 "somhow 9.76Gb being loaded onto a 6Gb card"
There are default compression settings in Studio.
Nvidia control panel > PhysX > Processor: CPU, it took me months to find it.
Doing this, windows won't take anymore GPU ram and it will be all avaiable for DAZ, then it won't switch anymore to CPU. This solved all my problems.