Polish Subway Huge VRAM demand

I bought the subway set from Polish, and I'm finding that it has huge, gigantic memory requirements.  I've tried set 1 and set 4 and both fallback to CPU rendering because they won't fit in my measly 6GB of VRAM (980ti).  The log indicates that set 1 needs 29GB of texture memory, and set 4 needs 24GB of texture memory.  Wow!  I've actually not run into this before, even with some of Stonemason's huge sets.

Is there a way to cut off parts of the scene that I don't want to minimize the memory requirement so that it can GPU render?

«134

Comments

  • BeeMKayBeeMKay Posts: 6,774

    In the render settings, have you tried "Optimize for memory" rather than "Speed"? Not sure if this helps, but it might.

    But you are right, this is extreme... I wouldn't even be able to render it in CPU mode.

  • LlynaraLlynara Posts: 4,737
    edited January 2017

    Is that the new Cyberpunk one? I just ran into that type of memory issue with the Antebellum Mansion, which is gorgeous, but requires 11 GB for textures on full preload.

    Strange, looking at the Cyberpunk Subway, it doesn't mention the size at all: 

    A large movie-like Cyberpunk Subway Station environment set with props and lights. Inside this set, you have 4 different preloaded Scenes ready to render, you can use also separate props and subsets included in this set to build your own scene as you like. Every models is well textured and optimized for Iray render. Hope you enjoy this one!

    Maybe there are just a lot of objects. Here's a couple of ideas, you've probably thought of these already: 

    • Don't do a full preload if you can help it. Load in chunks if there are presets for that. (Looks like there are, but I don't have the new set.)
    • Eliminate everything you don't need in the scene- ceilings that don't show up, floors, etc. If you're doing an inside pic, eliminate external things that don't show or vice versa
    • Check to see if there are textures on the insides of things that can be eliminated if objects are not open/seen- compartments, etc. 
    • Use Esemy's Reduced Texture script (free) to reduce it further. You may have to run it more than once. I played with it quite a bit on my art studio thread to fit things onto my 750Ti 2GB card. It works very well. I used it on many of the renders in that thread.
    Post edited by Llynara on
  • WendyLuvsCatzWendyLuvsCatz Posts: 29,285

    Octane Render

    out of core textures

    not that I own either set

     

  • Daz really needs to start posting requirements on the product pages.

  • AllenArtAllenArt Posts: 5,157
    edited January 2017

    The only solution I can come up with off the top of my head is to delete textures from the material zones that you don't see. That might be enough to knock the amount down to something that can fit on your card? I have a custom textureless single Iray material for just this purpose.

    Laurie

    Post edited by AllenArt on
  • Serene NightSerene Night Posts: 16,037

    Thanks for the information. I was going to buy it but no can do scenes with huge memory requirements

  • Oso3DOso3D Posts: 14,305

    If the material zones are broken up sensibly, you could do stuff with no map/procedural shaders, or plug in other stuff (like Mec4D shaders). Though that's a lot of work, and it rather requires that materials don't have crucial details baked in.

     

  • hphoenixhphoenix Posts: 1,331
    edited January 2017

    The product information lists 193 (!) texture/bump/normal maps, ALL of which are 2048x2048 or 4096x4096.  Just looking at the layouts and objects, I can guarantee that most items in any given render of that scenery would NOT need that kind of resolution.

    If the garbage can object is only 50 pixels tall in the final render, having a 2048x2048 texture on it is meaningless.  A 256x256 texture would look just as good, and take a tiny fraction of the VRAM compared to the bigger texture.

    29GB of textures alone?  That's just crazy.  Resize those puppies.  Esemwy and I (in another thread) are both working on image resizing scripts for just this purpose.  Based on this, I think I may have to make my scripts do XtraSmall resizes as well as Medium and Small......

    Again, I call out to PAs to make MULTIPLE sizes of your maps for your products.  And corresponding material presets.  So that this kind of stuff isn't as big of an issue......

    (now, if I can just work out that darn transparent PNG resizing issue, mine will be ready to go.....)

    edit:  Just checked....yep, there is NO nvidia card that can render this scene on CUDA.  P6000 Quadro has 24GB available, and at 29GB of textures, this scene can ONLY be rendered (at full texture resolutions) by CPU.  And then only if you have 64GB or more RAM, as with the OS/DS/etc., that won't all fit in RAM with only 32GB.  Or you'll be swapping to disk with virtual memory, which slows renders to a CRAWL.

     

    Post edited by hphoenix on
  • TooncesToonces Posts: 919

    I just looked at the Reduce Texture script everyone's been talking about:

    http://www.daz3d.com/forums/discussion/137161/reduce-texture-sizes-easily-with-this-script#latest

    It appears to be a great solution for Polish Subway and other large scenes. I really can't see any differences in the before/after comparisons of using the script. You can select all objects, deselect anything you want a closeup on, and reduce all textures at once! Assuming it works as well as everyone says, I'm glad PA's include the 4096 maps for closeups (though including multiple sizes woudl be fine too). Thanks to everyone who pointed out this cool freebie!

  • LlynaraLlynara Posts: 4,737
    edited January 2017

    I have used that Reduced Texture Script to reduce the files multiple times on the same scene and haven't seen any difference in quality. It would be hard to even spot the renders on my art thread where I did it. I know in a couple of the fairy pictures I did that. There's a way to save the reduced texture alongside the regular one, by holding the control button while you run the script (details on the script thread.) Occasionally I have trouble with a face texture loading and just reload that one from scratch. It's worth the effort and makes a lot of scenes more accessible to my small card. I'll be tackling the Antebellum Mansion with it next.

    Post edited by Llynara on
  • KindredArtsKindredArts Posts: 1,143

    Hey, there was a similar thread to this on the new mansion. I understand the concerns over resource usage, i really do. The numbers being thrown around are misleading however - The mansion shows around 13gb of uncompressed textures, however the actual GPU load is around 3.5gb. I know this isn't much help for anyone with sub 2gb cards, but these scenes are still GPU friendly with any mid-tier card. I'm really not trying to cause any drama, i just wanted to provide some sort of clarity. If anyone has this item, it would be useful to run a GPU monitor (like gpuz) to provide a more accurate load number.

  • LlynaraLlynara Posts: 4,737
    edited January 2017

    Sorry, I didn't realize that. I love the new info in verbose mode, but you're right, the numbers are misleading if they're being compressed. Sounds like a GPU monitor is worth running specifically for this reason. I don't have the subway set yet. Definitely on the wishlist though.

    Post edited by Llynara on
  • ZarconDeeGrissomZarconDeeGrissom Posts: 5,398
    edited January 2017

    That is a very nice looking set. Also, having hit the 32GB limit of my pathetic computers DIM slots, I'm not sure that IF it has 3DL mats, the subway would load in my willfully inadequate computer. Somewhere there has to be 16GB or 32GB DDR3 sticks (per stick, not total for the entire kit), just not anywhere that I can locate in non-ECC variety. I sort of wanted that two years ago Dram makers.

    Some where in the Beta thread I hinted at why it is such a problem with Iray, akin to the difficulties I had on a 32bit system with only 3GB of system ram back in Studio 3.5 days. With such limited memory, you are stuck with using environment spheres and ad-hock partial sets with limited number of figures for renders. You will never be able to load up a Stargate Atlantis city-ship model and render a fly-by with the usless skimping's of memory nvidia has been putting on there products.  If your lucky enough to be able to afford an i7 system with more then four memory slots (capable of more then 128GB of memory), then such incredible sets such as that subway are not unreasonable to work with in 3DL, and even have room for filling such a set with a believable number of figures for a full scene.

    Post edited by ZarconDeeGrissom on
  • hphoenixhphoenix Posts: 1,331

    Hey, there was a similar thread to this on the new mansion. I understand the concerns over resource usage, i really do. The numbers being thrown around are misleading however - The mansion shows around 13gb of uncompressed textures, however the actual GPU load is around 3.5gb. I know this isn't much help for anyone with sub 2gb cards, but these scenes are still GPU friendly with any mid-tier card. I'm really not trying to cause any drama, i just wanted to provide some sort of clarity. If anyone has this item, it would be useful to run a GPU monitor (like gpuz) to provide a more accurate load number.

    Thanks, @KindredArts , having someone with the specs to really test this (a TitanX) helps us understand just WHAT the stats Iray is putting out are.  So evidently those are the UNCOMPRESSED memory values, which are then compressed as it loads.  But even with similar compression, 29GB uncompressed textures is very likely to be very close to or over the 8GB limit for non-Titan-level VRAM.

    Still it would be nice if the 'verbose' iray details gave us actual USAGE in VRAM, rather than the uncompressed size to be transferred to the card.  What it is giving us isn't terribly helpful if it's uncompressed size.

     

  • FSMCDesignsFSMCDesigns Posts: 9,494

    Daz really needs to start posting requirements on the product pages.

    Also they need to monitor efficient modeling from their PAs as part of the quality control. There are many ways to model and with complex sets, there really needs to be some efficiency and thought put into not just the design, but the modeling and possible uses from the customer going forward. I have modded for games where the SDK gave you specific limits that you could not go over as not to impact performance in the game, something similiar should be in place with DS also

  • KindredArtsKindredArts Posts: 1,143
    hphoenix said:

    Hey, there was a similar thread to this on the new mansion. I understand the concerns over resource usage, i really do. The numbers being thrown around are misleading however - The mansion shows around 13gb of uncompressed textures, however the actual GPU load is around 3.5gb. I know this isn't much help for anyone with sub 2gb cards, but these scenes are still GPU friendly with any mid-tier card. I'm really not trying to cause any drama, i just wanted to provide some sort of clarity. If anyone has this item, it would be useful to run a GPU monitor (like gpuz) to provide a more accurate load number.

    Thanks, @KindredArts , having someone with the specs to really test this (a TitanX) helps us understand just WHAT the stats Iray is putting out are.  So evidently those are the UNCOMPRESSED memory values, which are then compressed as it loads.  But even with similar compression, 29GB uncompressed textures is very likely to be very close to or over the 8GB limit for non-Titan-level VRAM.

    Still it would be nice if the 'verbose' iray details gave us actual USAGE in VRAM, rather than the uncompressed size to be transferred to the card.  What it is giving us isn't terribly helpful if it's uncompressed size.

     

    No problem, i have voiced my concerns many times about resource usage. We are however trying to achieve professional quality rendering on consumer grade hardware, so there's a fairly fine line we have to tread. I'm hoping that high-capacity cards become more mainstream in the near future, to a point that compromises are no longer even a consideration. Fingers Crossed smiley

  • Serene NightSerene Night Posts: 16,037

    I'm not seeing anything in the promo renders that really loooks like it demands this degree of resolution to the textures to be honest. Especially given that it is a subway, and they are usually crowded with people, I would imagine adding memory intensive textures would be prohibitive to usefulness for such a large scene.

  • sapatsapat Posts: 1,735

    Guess I'll be returning this set.  I can't get it to render on my machine frown  So this has to end.  Most of us are hobbyists and not looking for commercial quality sets with those types of textures.  I'd be happy if they offered 2 versions for sale.  One for hi quality like the one out now, and one for those of us who can't have the latest and greatest computers/graphics cards and tons of other bling.  Guess I"ll be buying older stuff and converting iray like I've been doing when things go on sale.

  • Oso3DOso3D Posts: 14,305

    Again, if you are willing to work at it, I highly recommend using Mec4D's shaders or my free procedural shaders or something similar to rework the surfaces of products like this. If they are laid out sensibly, you can get texture load to VERY low levels without a huge amount of work (maybe).

     

  • DaWaterRatDaWaterRat Posts: 2,879
    sapat said:

    Guess I'll be returning this set.  I can't get it to render on my machine frown  So this has to end.  Most of us are hobbyists and not looking for commercial quality sets with those types of textures.  I'd be happy if they offered 2 versions for sale.  One for hi quality like the one out now, and one for those of us who can't have the latest and greatest computers/graphics cards and tons of other bling.  Guess I"ll be buying older stuff and converting iray like I've been doing when things go on sale.

    There's always the reduce textures script
    http://www.daz3d.com/forums/discussion/137161/reduce-texture-sizes-easily-with-this-script#latest

    I've had a few scenes that I couldn't get to render in Iray either, and this helped a lot.

  • LlynaraLlynara Posts: 4,737
    edited January 2017

    It's a beautiful set. I'll probably be getting it after I upgrade my card. If I tried it now, I think my computer would blow up. LOL. it's really straining on the Antebellum Mansion, which is not nearly as big. 

    Post edited by Llynara on
  • Serene NightSerene Night Posts: 16,037

    The set looks a bit sloppy to me. In several promos the trash is floating above the ground and the green aqua lighting looks a bit strange

  • Ghosty12Ghosty12 Posts: 1,349

    I think PA's need to realize that not everyone has superduper expensive video cards..  I think also Nvidia needs to take a page out of Otoy's book and have an implementation of dumping overflow onto the CPU and system ram..

    Really can not understand this need for super hi def textures 4096x4096 seems a bit of overkill even 2048x2048 seems a bit much..

  • frank0314frank0314 Posts: 10,821
    edited January 2017

    If you want detail then they have to be that big. Any smaller when you do close ups the texture gets fuzzy and pixilated. 4096x4096 is an industry standard so your going to keep seeing them. We are doing this work to produce quality work and we are going to do what's needed to get that quality. You can take the image into GIMP or Photoshop and reduce the size but you will lose quality.

    Post edited by frank0314 on
  • TooncesToonces Posts: 919

    I agree with frank, especially since there is a script and other mechanisms to easily reduce the textures.

    There is nothing for the reverse. If PA's started delivering only 2048x2048 and below, there is no script to magically increase the resolution for close ups.

  • xyer0xyer0 Posts: 3,909

    Maybe Daz can get him to do an update with an optimisation preload.

  • Oso3DOso3D Posts: 14,305

    The problem is that Iray really favors a material definition approach, rather than a texture map approach to design, and a lot of PAs just don't work that way.

     

  • Some great tips on here, thanks all.  The reduce texture script seems interesting, and I might try to play around with some textures to resize or delete them myself and see if I can make it go.  I have the Mecd set also.

    I did a CPU only render, my machine only has 16GB of system ram.  It does render, and it peaks at 11.7GB of system ram usage, so I guess the verbose mode is reporting uncompressed texture size as noted above.  I'm not sure if that's apples to apples or what, seems like maybe a card with 12GB might work then.

    Even with this "issue" I think this is an awesome set that I'll poke around and see if I can make use of.  I agree with some of the posters above - it's better to have "too big" of textures than to be stuck with pixelated renders for close-ups I guess.  Polish does do fantastic work for sure.

    Also, I tried memory optimized render, still doesn't fit in my card.

  • Oso3DOso3D Posts: 14,305

    I think memory optimized only affects instances.

     

  • kyoto kidkyoto kid Posts: 34,316
    hphoenix said:

    The product information lists 193 (!) texture/bump/normal maps, ALL of which are 2048x2048 or 4096x4096.  Just looking at the layouts and objects, I can guarantee that most items in any given render of that scenery would NOT need that kind of resolution.

    If the garbage can object is only 50 pixels tall in the final render, having a 2048x2048 texture on it is meaningless.  A 256x256 texture would look just as good, and take a tiny fraction of the VRAM compared to the bigger texture.

    29GB of textures alone?  That's just crazy.  Resize those puppies.  Esemwy and I (in another thread) are both working on image resizing scripts for just this purpose.  Based on this, I think I may have to make my scripts do XtraSmall resizes as well as Medium and Small......

    Again, I call out to PAs to make MULTIPLE sizes of your maps for your products.  And corresponding material presets.  So that this kind of stuff isn't as big of an issue......

    (now, if I can just work out that darn transparent PNG resizing issue, mine will be ready to go.....)

    edit:  Just checked....yep, there is NO nvidia card that can render this scene on CUDA.  P6000 Quadro has 24GB available, and at 29GB of textures, this scene can ONLY be rendered (at full texture resolutions) by CPU.  And then only if you have 64GB or more RAM, as with the OS/DS/etc., that won't all fit in RAM with only 32GB.  Or you'll be swapping to disk with virtual memory, which slows renders to a CRAWL.

     

    ...and people wonder why I am designing a duo Xeon render box with 128 GB of physical memory

    On my current system this would take almost a week to render without characters or any other effects as it would be in Swap Mode by day one. (I would probably even need to increase the size of the VM partition).

Sign In or Register to comment.