24 Cores Of Rendering Goodness

GarstorGarstor Posts: 1,411
edited December 1969 in Carrara Discussion

I finally stopped procrastinating and built my "big server." :coolsmirk:


Ultimately, this fire-breathing dragon is going to be running a simulated SQL Server 2012 failover cluster as the lab machine that I desperately need for work. But I wasn't about to let Carrara go without some loving!


I used 2 AMD Opteron 12-core CPUs and loaded up with 64 GB of RAM. Tomorrow I start building this machine's "little sister" (just one 12-core CPU and 32 GB RAM).


Here's the Render Room screaming along with (nearly) all the dials turned to 11...


Garstor is happy tonight!

«1

Comments

  • Virtual MediaVirtual Media Posts: 85
    edited December 1969

    Ya got me drooling :bug:

  • bighbigh Posts: 8,147
    edited December 1969

    nice to have big bucks

  • GarstorGarstor Posts: 1,411
    edited December 1969

    Actually, I cashed in the remainder of my Microsoft shares. Most of that is for my move to Seattle next year, but I do need these machines for work (and I do miss the Microsoft labs) to build a mini-network of virtual machines.

  • Virtual MediaVirtual Media Posts: 85
    edited July 2012

    I dropped about $6000 on a system last Christmas from http://www.digitalstormonline.com/ I've been more than pleased with the system and the service and actually thought I was rockin Daz Studio and Carrara pretty hard till I seen this!

    Post edited by Virtual Media on
  • GarstorGarstor Posts: 1,411
    edited December 1969

    I dropped about $6000 on a system last Christmas from http://www.digitalstormonline.com/ I've been more than pleased with the system and the service and actually thought I was rockin Daz Studio and Carrara pretty hard till I seen this!

    Ah the vagaries of Moore's Law!

    It was just over $6000 for both of my servers - nearly everything bought through New Egg; the hard drives I had to get through Amazon.

    For the record, the GI sample living room scene is providing quite the workout.

  • 3dView3dView Posts: 0
    edited December 1969

    Garstor---
    Wow ---sounds like a great machine-----if you could you post the specs ie. ram (company/ type) Mobo etc ---I am sure others might be interested as well. Always wondered about those opterons with all those cores. Its so helpful to have specs of a working machine to copy when making a new rig or buying components.

    And of course --the egg is just awesome.

    rich

  • tezentezen Posts: 0
    edited July 2012

    Hello Garstor!

    Nice to have so much buckets but its not nice to have a very big bug:
    Why are the shadows of "DAZ3D" incorrect ??? You can see it especially at the last "D"-letter...

    Greetings
    tezen

    Post edited by tezen on
  • GarstorGarstor Posts: 1,411
    edited December 1969

    3dView said:
    Garstor---
    Wow ---sounds like a great machine-----if you could you post the specs ie. ram (company/ type) Mobo etc ---I am sure others might be interested as well. Always wondered about those opterons with all those cores. Its so helpful to have specs of a working machine to copy when making a new rig or buying components.

    SPECS - msinfo32.exe
    OS Name Microsoft Windows Server 2008 R2 Enterprise
    Version 6.1.7601 Service Pack 1 Build 7601
    System Manufacturer ASUSTeK Computer INC.
    System Model KGP(M)E-D16
    System Type x64-based PC
    Processor AMD Opteron(TM) Processor 6238, 2600 Mhz, 6 Core(s), 12 Logical Processor(s)
    Processor AMD Opteron(TM) Processor 6238, 2600 Mhz, 6 Core(s), 12 Logical Processor(s)
    Total Physical Memory 64.0 GB


    PARTS
    ASUS KGPE-D16 SSI EEB 3.61 Server Motherboard Dual Socket G34 AMD SR5690 DDR3 1600/1333/1066/800

    AMD Opteron 6238 Interlagos 2.6GHz 16MB L3 Cache Socket G34 115W 12-Core Server Processor (x2)

    ASUS ENGTX550 Ti/DI/1GD5 GeForce GTX 550 Ti (Fermi) 1GB 192-bit GDDR5 PCI Express 2.0 x16
    * not a bleeding edge video card because the primary purpose of the machine is to run virtual servers - but I did need something beefier than an on-board video card

    Western Digitial Caviar Black 1TB SATA-3 hard drives (x5)

    Athena Power CA-SWH01B90 Black Steel Pedestal Server Case
    * this was the only case I could find to fit the over-sized motherboard - only caveat here is that it comes with a single fan, so I may need to add more in the future
    * also, I am not sure about blaming the case or the video card, but the when the card is seated in the mobo, it is still too tall to be secured to the case with screws


    BE AWARE: I am still investigating this and it is probably AMD's chip design fault...but I noticed in PerfMon that occasionally the total CPU processing would drop to near zero. It quickly shot back up to over 80% but would eventually repeat this. I suspect that this is caused by the on-chip L1 cache becoming invalid and needing to be flushed. Certainly a major performance if you are doing genetic research into cancer or something...but probably not serious for Carrara - just annoying for an uber-geek.

    Honestly; I would have preferred buying Intel but I wanted tons of cores and the price couldn't be beat. I was feeling better about the purchased when I learned that 3DAGE has used Carrara exclusively on AMD chips. :-) But for long-term, "real" server-grade work, Intel is probably the way to go.

  • GarstorGarstor Posts: 1,411
    edited December 1969

    tezen said:
    Nice to have so much buckets but its not nice to have a very big bug:
    Why are the shadows of "DAZ3D" incorrect ??? You can see it especially at the last "D"-letter...


    I hear you brah! Fortunately, I didn't make the scene! :coolhmm:

  • evilproducerevilproducer Posts: 9,040
    edited December 1969

    That's a nice system Garstor! Hopefully it'll be able to up your Carrara dosage and take the edge off your current environs. :cheese:

  • GarstorGarstor Posts: 1,411
    edited December 1969

    That's a nice system Garstor! Hopefully it'll be able to up your Carrara dosage and take the edge off your current environs. :cheese:

    Work??!!? What's that? SQL-what? SharePoint-huh? :-D

    Oh if I could only win a lottery...

    AbortThread(LAMENTATION);
    SetEvent(BACK_TO_THE_GRIND);
  • evilproducerevilproducer Posts: 9,040
    edited December 1969

    Garstor said:
    That's a nice system Garstor! Hopefully it'll be able to up your Carrara dosage and take the edge off your current environs. :cheese:

    Work??!!? What's that? SQL-what? SharePoint-huh? :-D

    Oh if I could only win a lottery...

    AbortThread(LAMENTATION);
    SetEvent(BACK_TO_THE_GRIND);


    That must be Windows code. Please don't port it to my system. ;-)

  • RoguePilotRoguePilot Posts: 239
    edited December 1969

    tezen said:
    Hello Garstor!

    Nice to have so much buckets but its not nice to have a very big bug:
    Why are the shadows of "DAZ3D" incorrect ??? You can see it especially at the last "D"-letter...

    Greetings
    tezen

    I don't think that's a bug so much as a consequence of the scene set-up and render settings.

    If you take a look, the light through transparency is disabled for the main setting but enabled for the indirect light.
    The caustics are also on. We are actually seeing shadows that are too dark rather than missing shadows and this shows on every letter.

    This may be a stylistic choice by the scene designer.

  • RoguePilotRoguePilot Posts: 239
    edited December 1969

    I've never seen a 24 core render before. Looks like fun.
    Why do the tiles only number to 8?

  • GarstorGarstor Posts: 1,411
    edited December 1969

    I've never seen a 24 core render before. Looks like fun.
    Why do the tiles only number to 8?

    That surprised me too. That might be considered a bug in Carrara display of the rendering tiles; however, I am more inclined to point at the MSINFO32 data...the AMD chips are actually 6-cores with Hyper-Threading / Hyper-Transport (or whatever) technology enabled.


    I bet that Carrara is actually counting cores via socket-to-logical processor ratios rather than summing up the grand total of all available logical processors.


    On that note, I'd love to see Carrara implement an affinity mask feature for CPUs. That way you could "give" X-number of cores to the Carrara process and reserve Y-number of cores for the rest of the system.

  • edited December 1969

    Garstor,

    Not really hyperthreading. They have 12 integer cores and share a floating point unit between each pair of integer cores. Much better design than Intels hokum with using wait states and claiming twice the power.

    Joe

  • RoguePilotRoguePilot Posts: 239
    edited July 2012

    Garstor said:


    On that note, I'd love to see Carrara implement an affinity mask feature for CPUs. That way you could "give" X-number of cores to the Carrara process and reserve Y-number of cores for the rest of the system.

    A setting in Carrara would be nice. I set affinity through the task manager.
    Apologies if you already know this but;
    right click on the Carrara process and select set affinity.
    Just uncheck which core(s) you want to release.
    You can do this mid render if you want and then re-check it when you want to give it full bore.
    A good thing is that Carrara will still use the cores when they are available but will release them when other processes want them.

    Personally I prefer to leave my render machine to render alone as much as possible but there are a few occasions when I need to use software that is not on my laptop or on my second string PC. If you are rendering on a server clearly that is not an option.

    Post edited by RoguePilot on
  • ds-mail_2e0cb9c256ds-mail_2e0cb9c256 Posts: 70
    edited December 1969

    The obvious question...how much speed improvement compared to a "normal" PC with 4 cores/8threads; say i7 2600 3.4GHz for duplicate render?

    My guess is 24 vs 8 cores is a little under 3x faster to account for overhead...and optimum size of render square tile size (assuming CPU speed is same and RAM sufficient)

    Do you have any data?

  • GarstorGarstor Posts: 1,411
    edited December 1969

    DougS said:
    The obvious question...how much speed improvement compared to a "normal" PC with 4 cores/8threads; say i7 2600 3.4GHz for duplicate render?


    I must confess my deepest shame for not having thought of this myself! :red:

    I'll get this remedied later today...find an older scene on my other machine and use the statistics tab from the Render Room to compare both boxes results.

    More to follow after the experiment!

  • GarstorGarstor Posts: 1,411
    edited December 1969

    A setting in Carrara would be nice. I set affinity through the task manager.
    Apologies if you already know this but;
    right click on the Carrara process and select set affinity.


    True! That is an easy option to forget about. It is also controlled by the OS rather than the process; so it doesn't persist between process lifetimes. I'll try to remember it though when I get a long-running physics animation calculating...

  • GarstorGarstor Posts: 1,411
    edited December 1969

    VERY surprising results in my "render race" between the two machines.


    I posted the MSInfo32 stats for the new AMD server earlier in this thread. Here are the stats for my older Intel machine:

    OS Name Microsoft Windows 7 Ultimate
    Version 6.1.7601 Service Pack 1 Build 7601
    System Type x64-based PC
    Processor Intel(R) Core(TM) i7 CPU 870 @ 2.93GHz, 2934 Mhz, 4 Core(s), 8 Logical Processor(s)
    Total Physical Memory 7.99 GB


    Here is the scene I was rendering: http://www.flickr.com/photos/garstor/6981911339/in/photostream


    Scene Render Settings
    Width 1680
    Height 1050
    AntiAliasing Best
    Object Accuracy 0.5
    Shadow Accuracy 0.5
    Sky Light True
    Indirect Light True
    Light Thru Transp. True
    Lighting Quality Excellent
    Accuracy 1
    Irradiance One for all frames
    Tile Size 16

    Caveat: on the new machine, Carrara could not find the Lana Elite texture or bump maps for Victoria (so she had a zombie/undead pall to her...poor thing!) or the texture for Michael's jeans. I am not sure if this affected the render times.


    Old Intel Machine
    Start Time: 8:15:49 AM
    End Time: 12:24:36 PM
    Elapsed: 4:08:38

    New AMD Server
    Start Time: 10:14:21 AM
    End Time: 1:49:40 PM
    Elapsed: 3:35:10


    Surprisingly close! I think that this shows Intel chips have better overall computational strength. However, bang-for-buck, AMD gets my vote...I simply wasn't willing to spend the $$$ needed for a 2-socket hex-core Intel machine.


    If anyone thinks I did something wrong here, I'm happy to re-test with further suggestions (at least until I pave over the Intel box...I'm thinking of getting rid of its SSDs as it keeps blue-screening on me).

  • bighbigh Posts: 8,147
    edited December 1969

    so you gain 30 mins. - boy I don't know - don't seem right to me

  • GarstorGarstor Posts: 1,411
    edited December 1969

    bigh said:
    so you gain 30 mins. - boy I don't know - don't seem right to me


    Yeah, not intuitive...I am guessing something is amiss. Perhaps some power setting (I did set Windows to High Performance though)...maybe something I missed in the BIOS. This is the first AMD machine I've ever owned.

  • RoguePilotRoguePilot Posts: 239
    edited December 1969

    Hi, I'm not sure about the total render time but try measuring from when Carrara actually starts dsplaying the image, it may be interesting.

    I think some things are still single threaded sometimes. (Could be wrong).

  • ds-mail_2e0cb9c256ds-mail_2e0cb9c256 Posts: 70
    edited December 1969

    And did you have any verification that all 24 cores were being used to render? I could be wrong but thought I read somewhere a couple years ago that Carrara was limited to perhaps 8 or 12 cores (on one machine, not across multiple PCs as a render farm). This may need some investigation. May be why no more than 8 tile squares of render show at a time?

  • Kendall SearsKendall Sears Posts: 2,995
    edited December 1969

    DougS said:
    And did you have any verification that all 24 cores were being used to render? I could be wrong but thought I read somewhere a couple years ago that Carrara was limited to perhaps 8 or 12 cores (on one machine, not across multiple PCs as a render farm). This may need some investigation. May be why no more than 8 tile squares of render show at a time?


    I seem to remember reading about an 8 thread limit somewhere as well, but I've been looking unsuccessfully for a reference before posting.


    Kendall

  • GarstorGarstor Posts: 1,411
    edited December 1969

    DougS said:
    And did you have any verification that all 24 cores were being used to render? I could be wrong but thought I read somewhere a couple years ago that Carrara was limited to perhaps 8 or 12 cores (on one machine, not across multiple PCs as a render farm). This may need some investigation. May be why no more than 8 tile squares of render show at a time?

    I was watching the render progress frequently. There was most certainly 24 distinct "render tiles" at work (remember I used a size of 16x16...and Carrara only puts numbers in them when they are 48x48)..

  • swordkensiaswordkensia Posts: 348
    edited December 1969

    Does task manager show all 24 cores running at 100%.??


    I have noticed on some scenes with IDL / GI enabled Carrara will crawl along and use only about 25-35% of your cores, yet if you disable IDL /GI on the same scene, it shoots back up to 100%.


    I have two Intel 3930K's networked and over clocked to 4.2ghz, and there is nothing more frutstrating than seeing Carrara not using all of my rendering cores.


    Peace,

    SK.

  • GarstorGarstor Posts: 1,411
    edited December 1969

    Does task manager show all 24 cores running at 100%.??


    I have noticed on some scenes with IDL / GI enabled Carrara will crawl along and use only about 25-35% of your cores, yet if you disable IDL /GI on the same scene, it shoots back up to 100%.


    I have two Intel 3930K's networked and over clocked to 4.2ghz, and there is nothing more frutstrating than seeing Carrara not using all of my rendering cores.


    Honestly, I didn't watch Task Manager. I am quite sure that I do not have IDL / GI in that scene though.

  • swordkensiaswordkensia Posts: 348
    edited December 1969

    In your scene settings you state.

    Sky light - true

    IDL - true

    Lighting Quality - excellent..


    Those settings would indicate that you DO have IDL / GI enabled for your scene..unless I am mis-undertsanding your nomenclature.


    Peace,


    SK.

Sign In or Register to comment.