Memory Consumption - Is this normal?
MadBirdCZ
Posts: 0
Hello everyone,
I started to work on a scene which is bigger than the usual ones I was creating in past. While adding details to the scene I hit a point where the work with the scene slowed down to a crawl. I realized that all of the 32GB of RAM are consumed by DAZ Studio (4.6.0.18 Pro 64bit) and that the system is swapping badly.
I checked the Scene Info tab and I found out that the whole geometry has 6.8M faces total (Scene Info). My first thought was that all the RAM gets eaten by preloaded textures, so I created a duplicate of the scene file, removed all textures from all characters and objects, saved the scene, quit the DAZ Studio and after launching it again I loaded this untextured scene. Unfortunately the memory footprint after the scene finished loading was again exceeding physical RAM and the system was swapping (Memory Utilization with textures compared to same scene without textures).
Then I quit the Studio again and created a sphere primitive with 4096 segments and 2048 sides. That produced a scene with higher face and quad count than the previous one, however the memory consumption was very tiny compared to the real scene (Sphere Scene Info and memory consumption)
Question is - is this normal, or am i facing some kind of a memory leak within DAZ Studio?
Also when I close the normal scene that consumes the 32+GB of RAM the application window closes almost immediately and about 8GB of RAM is released as free but in process list the DAZ Studio is still listed as running and consuming ~18GB RAM...
So my question as already shown in the topic is - Is this Normal?
--
Zdenek
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Out of physical memory is the worst form of bottleneck as it feeds back into all parts of the system, so you have my sympathies. I've noticed that DS4.6 seems to be consuming more memory than previous versions, but that may just me having had it catch my eye. I've constructed, and rendered, a couple of tolerably complex scenes in my 32GB of RAM. What HAS hammered the computer was a very poorly (IMHO) put together set of overlapping materials and custom shaders that seemed to gobble memory like it was going out of fashion during the render.
As it sounds like you've not even got to the render part yet I can only suggest deconstructing the 'problem scene' doing memory utilisation checks at key points along the way to see if you can spot an obvious culprit. I'd be curious to hear what you find out if you do this.
Hmm... 6.8 million total faces. I sort of think that is the issue. Its not the SIZE so much as DS dealing with all the mesh in 3D space. Never tried a file that large. I lag at 3.5 Million myself. Which is rather HUGE. And that does EAT Ram for the Calculations just so the Viewport can respond, when it does.
I can't even get mine to use more memory. I have 8GB installed and DS never uses more than 3.5+ (never even hits 4GB), but it'll eat my CPU (I have an overclocked i7-2600K to 4.7GHz- hyperthreading is enabled). This is while rendering.
How can you see the face count in DS?
Another funny bit is that after I did the collecting part and got the whole scene nicely packed for rendering on an external 3delight engine I downloaded the standalone Linux version of the 3Delight, installed it onto a box where i normally run Luxrender and let it crunch on the RIB file. During the render the Linux system (which has 16GB of RAM) consumed just about 5GB of it for the render.
So this makes me believe that the issue is somewhere in the Studio instead of the render process as also when I hit render in the studio the scene gets rendered and after the render is finished about 7GB of ram is then freed which sort of corresponds with the real size of all the resources in the scene.
---
Zdenek
Do you have any VERY old 3D mesh objects in the Scene file? I have loaded a few older items I collected around the web into DS4.6 and had issues. Never saw my Ram Ramp to full use but did notice one Freebe Tree I loaded jumped my ram use WAY up. I had planned to populate the scene with it but two would have killed all my ram. I even tried a Instance and that crashed DAZ Studio. I removed that item from my content folders. It is a DS4+ issue because I can use it fine in my DS3A, but I'm trying to move forward not backwards so DS3A is not a option in my eyes any longer unless I can not figure out how to do it in DS4+. Mostly I can so far.
I'm thinking that could be what you've run into. Or even just a BAD mesh, it might not need to be that old. Wish I could offer better help but So far I've not seen any mem leaks in 4.6. I do keep meters running on my System just because I'm a Nerd and like to know whats going on under the hood.
This sounds like you are using the 32bit version. The 64bit version can use more available memory.
---
Zdenek
Another thing to check for: how many objects have subdivision enabled? 4.6 changes the default SubD algo to Catmark, and that is more sensitive to mesh quality/construction than Catmull-Clark was. And who knows what bugs might be in it. So if there are a bunch of SubD objects, you could try switching them all to Catmull-Clark, save and reload and see if the problem persists.
Note that 4.6 also has a bug with switching SubD algo: if the figure has any confirmed items with smoothing & collision enabled, switching the SubD algo will hang the mesh smoother. The workaround is to switch the figure to Base resolution, change the algo, then switch it back to High resolution.
This sounds like you are using the 32bit version. The 64bit version can use more available memory.
Well, I got it after some testing. It WILL use more ram, but apparently what I've been doing hasn't pushed my PC to the point of needing it. I loaded some of my FB sets and props, used AdamR's Insane Quality render settings, went to a 4000x4000 image, and voila. Just under 5-6GB being used.
I just figured even with a good SSS mapped figure on insane and studio lighting that a slow render would get boosted a bit by using more ram, but my PC doesn't seem to agree. lol Unless I'm misunderstanding the relationship between CPU and Ram where rendering is concerned (I read somewhere iirc that cpu 'collects' the data and ram pushes the pixels out, or something- so maybe it's my CPU holding my ram back?).
Subdivision eats memory. Try putting all your mesh to base resolution to see how much the scene takes
OK... So the results are in...
I selected everything in the scene and reduced mesh resolution to Base. After that I switched the Subdivision Algorithm to Catmull-Clark (Legacy) and saved the scene. After quitting and re-opening the DAZ studio I loaded the scene and memory consumption dropped from 35GB (commit charge) to 28GB of physical RAM used out of 32.
Then I selected everything again and returned the base mesh resolution to High and repeated the save, restart and open process while leaving the SubDivision still set to Catmull-Clark. The memory utilization after the scene fully loaded and DAZ Studio became responsive is 28GB.
So it seems that mesh resolution does not really matter and the reason for the 7 more GB of RAM being eaten is the Catmark SubDivision algorithm.
Also the Base resolution of the figure meshes is not really useable as it generates a huge amount of poke-through issues.
Now however, the question still stands - is it normal or ist there some bug in the Catmark?
I've had WAY more than 6.8M faces in a scene.
This scene has 26M verts in it and oly uses about 22GB of RAM during render. About 9 on load iirc.
http://adamtls.deviantart.com/art/Urban-Centaur-372197828
What's happening to you is subdivision ballooning. When you subdivide, oh, say, Victoria 4... you're going from about 68K verts to something like 1.2M at full subdivision. This definitely uses quite a bit more RAM even when the "display" resolution is set to "base" or subdivision level is set to 0.
Nice scene.
In the one I'm working on I already have roughly 30 characters (V4 / Genesis) and SubDivision was for all characters left on default values (Mesh quality set to High and SubDiv level set to 1 with Catmark SubDiv algorithm). Then there are quite a few objects there.
Does the scene geometry info take the SubDivided surface poly count into account?
No, Scene Info only counts original mesh density. I've got a bug report on that somewhere.
To answer the OP's original question, yes, it's normal in the sense that ANY subdividing algorithm is going to increase the amount of RAM used for a scene. As you figured out, the difference is in which algorithm is being used.
As for the RAM not being released, that's normal for Windows. It can take a few minutes for a program to completely shut down and release the RAM, especially if it was actively processing data.
Just to tie up this thread - first of all thanks to all who replied and brought in pieces of insight to this issue.
I realized that I'm probably pushing DAZ Studio into a field it was never meant to go - e.g. creating and rendering big scenes in it as the studio seems to be quite inefficient when it comes to resource handling - or at least it feels like it.
Interesting is that when I just collect the scene into a RIB the folder has some 7GB of stuff in it and when I copy it to my Linux box and let the free 3delight standalone rendering engine crunch on the RIB it takes some 8GB of RAM. Where when I let the scene to be rendered in DAZ Studio the commit charge of the DAZ Studio alone jumps towards 40GB. That is quite a lot...
Problem is that the 3Delight standalone render engine that would use all cores costs over $1000 which is quite a lot for a non profit "just for fun" type of work and with the free one that is limited to two cores the scenes take ages to render.
So as I already maxed out the RAM in my computer I would have to upgrade mainboard and CPU, plus get another 32GB or RAM to have 64GB total. That would not come cheap either. So what I did was that I purchased an SSD drive that comes with 60 months warranty and moved swap and temps to it. So if it dies it won't be a big deal. It is still rather slow as there is nothing that can replace physical RAM but it is much faster than swapping 10+ GB to a physical spinning hard drive.
Not an ideal solution but probably the most and cheapest one.
It is a pity that the 3delight rendering engine that is part of DAZ Studio can't be used to render the RIB exported folders as it would not require to have the whole studio running and eating lots of system resources for no good (some sort of a command line or "thin" rendering client without GUI - just point it to the rib file/folder and out comes a rendered TIFF).
While there is a learning curve involved, and additional work to change material definitions, have you considered any of the other export plugin options? I render everything in LuxRender, for example, and it's very handy to have Lux run in the background as a separate process. Lux's built-in network rendering support is also a huge benefit when rendering large scenes (I always use at least two slaves in addition to my master machine, for example).
Yes I did experiment with Luxrender and I really love the results - light emitting surfaces for instance were something I fell in love with almost immediately. Problem is that scenes that took an hour or two to render in Studio's internal 3Delight took a day to render in Lux (in order to achieve an acceptable quality) even with two 4.4Ghz i7s crunching on it - and the scenes were never as big as the one I'm working on at the moment.
But then again, after learning the limits of my system and Studio, I think I will never try to create a scene as big as this one again.
DAZ Studio also has light emitting surfaces. I use them all the time.
Look in your main Genesis runtime for Light Presets\omnifreaker\UberAreaLight\!UberAreaLight Base.duf
It applies like a surface shader, so select the object on the scene tab, select the surface on the surface tab and apply. Then on the surface tab you can control the light, pretty much like a normal light.
This scene is primarily light with surface lighting:
http://adamtls.deviantart.com/art/Seditionist-306800160
This scene is *entirely* light by surface lighting, although admittedly in this one I used Shader Mixer to make the surface lights instead of using UberAreaLight:
http://adamtls.deviantart.com/art/Flame-in-the-Night-213073955