Carrara not using all of my render nodes! Solved!

2456711

Comments

  • JonstarkJonstark Posts: 2,569

    I've been following this thread with half an eye, because I figured network renering is not really my thing, and I don't have the cash or space for a bunch of boxes to act as nodes. Then it twigged that I have this big i7-6700+GTX980ti box that just gets used for Iray and playing flight sim could also be a render node (the cranky old monitor hooked up to it is pretty much useless for working on. I'd hoped to hook up the DP output to the thunderbolt input on my iMac, but sadly Apple nixed that several OS updates ago. I can still remote desktop into it, but that's a real faff).

    So I installed the render node software and ran it up. My iMac needs to be the master even though it's only an i5, because it's my work system and has the really nice 27" screen (and isn't Windows! ;)) Anyhow it ran like a charm. Interestingly, my 4 little i5 cores didn't put in a showing until the network cores were halfway down the image. Nevertheless the test render (attached) which took 40+ minutes last night on my i5, was done in 10 minutes 48. I think I'll keep RenderNode running! laugh

    Sweet!  I didn't know all this time that you were using only an i5 to render with up til now, especially since you've got those big complicated scenes with lots of trees and leaves doing test renders as you were creating your scenes must have added a lot of render time you can now cut down substantially.  I also have a 6700, but on my main machine, and it usually gets well down the page before even the fastest of the render nodes starts kicking in  :)   That i7 6700 is a beast, it would be a crime not to use it as part of your rendering power.  BTW, are both machines Mac?  Or are you able to use a Mac and Wiindows together with no issues?  

  • DartanbeckDartanbeck Posts: 18,109

    Yeah, and his i5 blows away the little quad I have! LOL (AMD Quad-Core E2-6110 (1.50GHz))

  • DartanbeckDartanbeck Posts: 18,109

    Part of the beauty of Carrara... while we may have to ait a bit... we can render on pretty much any machine! ;)

  • TangoAlphaTangoAlpha Posts: 4,579

    Yep, the i7 is Windows and the i5 is Mac, so it's a mixed environment, and no issues on that regard. I've yet to do a big HQ render with this setup, but 12+ hours on a tree+water-rich environment was not unusual for the i5! (and an Iray render could take a week!)

    Full specs Mac: Core [email protected] 3.2GHz (don't know the model number), 24GB RAM. Win: [email protected], 32GB RAM

  • Jay VersluisJay Versluis Posts: 177
    edited September 2016

    Same here, I'm using a mixed setup as render nodes: three Macs (a quad-core i7 Mac Mini as my main system, plus two MacBooks, both i7s), a Surface Pro i5 and a Samsung whatever laptop featuring an i3. All 24 cores working well together.

    However there is one thing I've noticed about node rendering: artefacts the size of my render tiles.

    Here's one of my render challenge entries, it was a WIP render I did with my node setup:

    Notice the area around the number 9 (here's a closeup):

    It's as if every node returned a slightly different value for brightness here, some even with a different colour altogether. I rendered the same image without the nodes, and the problem did not appear. Perhaps it has to do with object or shadow accuracy, or the way the light is calculated (which happens locally on every node rather than centrally on localhost). 

    I don't recall the exact render settings right now (I think most were the defaults, plus Full Raytracig, Skylight and Caustics), but this is something to watch out for. 

    Screen Shot 2016-09-13 at 15.33.18.png
    751 x 566 - 827K
    Post edited by Jay Versluis on
  • DartanbeckDartanbeck Posts: 18,109

    Man, Jay... I love that render!

    Yeah, until TangoAlpha came in... I had no clue that we could do that! MacOS and Win together in a network?!!! Dogs and Cats living together in the streets... what next?

  • TangoAlphaTangoAlpha Posts: 4,579

    LOL, nobody said it couldn't be done, so I just did it! wink

    @ Jay, do they correspond with one particular machine? Might it have an old version of a shader or something? (maybe the version in the host machine wasn't saved?)

  • JonstarkJonstark Posts: 2,569

    You know, prior to actually using rendernodes I was afraid of something like that happening (what Jay shows with some render nodes rendering differently than the main machine, therefore giving a checkerboard result).  I'm pretty sure I had read comments on this problem 'back in the day' on prior versions of Carrara, and I was expecting it to be a problem, but haven't been able to replicate this problem at all myself.  I then thought maybe it was only for scenes with global illumination, but so far in my tests GI renders fine, no checkerboard effect.  Tried in animations, no checkerboards, so I just assumed maybe that was a problem for earlier versions of Carrara that had been fixed in 8.5, but perhaps not.

    Maybe it's something to do with ray depth for transparencies? Or refraction?  I'm just trying to guess what it could be since it looks like it only appears in the 'water' parts of the render.  Were you using caustics?  Maybe it's caustics that doesn't 'translate' across render nodes... ?  

  • DartanbeckDartanbeck Posts: 18,109

    That's right... evilproducer knows of at least one issue, and what causes it... but he's too busy to come play right now :(

  • Jay VersluisJay Versluis Posts: 177
    edited September 2016

    Well after plenty of node testing, I think I've found how the problem can be avoided. Like @Jonstark says, the troublesome spots only appear in the water, and I was indeed using Caustics. Solution 1: Switching Caustics off solves the problem.

    I've tried the same with seval other settings (Full Indirect Lighting, Sky Light) and all of them produced the problem.

    However, there's another way I could avoid the problem, while using Caustics just fine; Solution 2: disabling all Windows nodes also solves the problem, leaving my three Macs rendering in tandem. So perhaps there's something in the way each OS calculates floating point numbers differently (even though there shouldn't be of course). I would assume that on a Windows-only setup the problem also does not manifest. 

    I've attached the file, feel free to test this on your respective setups (Windows only, Mac only, Mac and Windows combined). Leave Caustics on to reproduce the problem (should be switched on in the render settings). Camera 1 is the full scene, while Camera 2 is zoomed in on the troublesome spot. Please report back what you find. Happy rendering!

    Oh, and if you want to compare render times: on my 24 core setup, Camera 1 renders in just over an hour at 1280x720, that's with all the render trimmings (Sky Light, Full Indirect Light, Caustics). 

    @TangoAlpha Good call, but I had saved all settings internally, so I assume Carrara would take all shaders from within the scene file.

    @Dartanbeck Thank you so much, I'm glad you like the scene! Inspired by all the talent in the render challenge - would have never happened without it :-)

    @Jonstark Hey thanks, glad you like the Tumblr gallery! 

    zip
    zip
    Carrara 9 Reef.car.zip
    8M
    Post edited by Jay Versluis on
  • JonstarkJonstark Posts: 2,569

    I loaded it up and gave it a try Jay, my network is all Windows btw.  At first I thought I was getting no artifacts/checkerboards, but then the area around the '9' I definitely got them (I guess that's where the light hits the water with the greatest caustic effect) which seems to lend credence to the theory that it's not a problem with Win/Mac render nodes not playing nice and simply that it's a broad render node bug that the caustics calculation/implementation does not carry across to the nodes and is enacted on the primary computer only.

    Looks like it's a limitation of network rendering, sadly.  I read in the Carrara 7 manual that Caustics isn't just an effect of light interacting with scene objects, but instead is a whole separate global illumination method/engine within Carrara, which may be why it doesn't translate to the nodes.  Honestly I never use caustics (except in rare experiments) which is probably why I never ran into the checkerboard effect before, so happily it's not much a limitation for my personal use.

  • TangoAlphaTangoAlpha Posts: 4,579

    Rendered in an hour 49 for me, using the default big tiles (could probably have shaved a good 15+ minutes off that with a smaller tile size). Still got the artifacts, but much less noticeable with the larger tiles. Most obvious were around the base of the top-left island and between the '9' and the bottom island.

  • Jay VersluisJay Versluis Posts: 177
    edited September 2016

    Thanks for checking it out guys! I've been rendering a larger version again only on my Macs, and the artefacts have also come back. So I guess we can eliminate a mix of OSes as the culprit, which is very nice. Caustics seems to be the main evildoer for render nodes then.

    Lucky for us, most render needs happen without Caustics, so we're good on that front and can still use our nodes.

    Hey @Jonstark, inspired by your Xeons, I've bid on one this morning on the cheap and won it, can't wait to start playing with it (16 more cores). 

    Post edited by Jay Versluis on
  • JonstarkJonstark Posts: 2,569

    Thanks for checking it out guys! I've been rendering a larger version again only on my Macs, and the artefacts have also come back. So I guess we can eliminate a mix of OSes as the culprit, which is very nice. Caustics seems to be the main evildoer for render nodes then.

    Lucky for us, most render needs happen without Caustics, so we're good on that front and can still use our nodes.

    Hey @Jonstark, inspired by your Xeons, I've bid on one this morning on the cheap and won it, can't wait to start playing with it (16 more cores). 

    Cool!  What did you get, if I may ask?  All these years ebay has been around, and I've never used it before these last couple of weeks, and I may be doing it wrong as I've never really bid, just looked for 'buy it now' and bought whatever the lowest price model that met the specs I was looking for was. Possibly I should have been bidding evern lower to see if I could get it at a steeper discount  :)   I just picked up a dual xeon (E5620's) for z600 for $225, which I think is a steal but maybe I could have got it for even less  :)   The best part is I can only imagine prices for 'old tech' (which is still very useful tech to us Carrarists!) only dropping lower as time progresses, and at these prices it really is crazy affordable; even if one of my 'new' $200 machines suddenly dies in a couple of years, no biggie as it didn't cost much to begin with and will probably cost even less to replace in the future.  Long rendertimes will be a thing of the forgotten past, lol   :)

  • Jay VersluisJay Versluis Posts: 177
    edited September 2016

    That's what I was thinking! I have no idea what the pricetag for a Z600 or a Z800 was when they were new, but just looking at what a current Xeon CPU alone can cost, I can only imagine it was unaffordable! 

    I got a Z600 with 2x Xeon 5560, 2.8GHz. Doesn't have a hard drive, only 4GB of RAM, an unknown graphics card (hopefully some NVIDIA flavour), and comes with a code for Windows Vista (har har har). But it was $55 when I discovered it last night, and this morning I bid on it, and finally got it for only $171 plus $50 shipping, and like you say, if it holds up for a couple of years, I'll get my money's worth. Can't wait to use it as a render node, and if the graphics card holds up, perhaps it makes for some speedier DAZ Studio renders too. Sadly on my Mac Mini, there's no way I can add a CUDA capable graphics card.

    Bidding can get you some great deals, but there's patience involved, and you may end up not getting what you want when it slips through your fingers at the last moment. Buy Now can be just as good, and you get what you want there and then.

    Post edited by Jay Versluis on
  • JonstarkJonstark Posts: 2,569
    edited September 2016

    That's what I was thinking! I have no idea what the pricetag for a Z600 or a Z800 was when they were new, but just looking at what a current Xeon CPU alone can cost, I can only imagine it was unaffordable! 

    I got a Z600 with 2x Xeon 5560, 2.8GHz. Doesn't have a hard drive, only 4GB of RAM, an unknown graphics card (hopefully some NVIDIA flavour), and comes with a code for Windows Vista (har har har). But it was $55 when I discovered it last night, and this morning I bid on it, and finally got it for only $171 plus $50 shipping, and like you say, if it holds up for a couple of years, I'll get my money's worth. Can't wait to use it as a render node, and if the graphics card holds up, perhaps it makes for some speedier DAZ Studio renders too. Sadly on my Mac Mini, there's no way I can add a CUDA capable graphics card.

    Bidding can get you some great deals, but there's patience involved, and you may end up not getting what you want when it slips through your fingers at the last moment. Buy Now can be just as good, and you get what you want there and then.

    I think I saw that one!  I know it's no big deal to install a hard drive, but I'm still new to all this taking apart and putting back together of computers (which previously in my mind were 'magic boxes' that shouldn't be opened) so I was too leery to go for one that didn't have the hard drive already installed lol  smiley  What a bargain, kudos!  I'm going to have to try my hand at actually bidding next time I pick something up from ebay (possibly in the future when I get brave enough to want to upgrade the Xeons to hex cores instead of the quad cores that I'm getting now, and I need to buy some hex cores).

    I also can't help but wonder what some of the systems would have sold for originally, when the tech was new.  I'm betting they were very very expensive indeed.  I saw this fairly recent article about 'building a monster dual Xeon 32 core render for less than the price of a Haswell-E i7' which mentioned the Xeon E5-2670 they were using originally sold for $1550 and only a few years later you are going for less than $100 on ebay.  (btw turns out the computer they built from scratch for less than what one single Haswell-E i7 cost at the time outperformed the newer tech computer in many side-by-sides).  Now the build they did in the article was still way too pricey for me and used mostly new equipment, except for the xeons, but it hit me how much this stuff gets discounted after only a few years, that's a tremendous price drop.

     

    Curiosity got the best of me and I looked it up, you know that X5560 you got?  It was originally priced at $1172!  For just one!  And you got 2 of them, and the case, power supply, motherboard, win Vista, 4GB ram, probably dvd drive, etc... all for only $171!  And while some might look down their noses at this 'old tech', this stuff is perfect and cutting edge to use as rendernodes for Carrara.  Man, it's a great time for a Carrarist to be alive   laugh

    Post edited by Jonstark on
  • GASP! So $3k just for the CPUs, plus $1-2k for the rest, we're probably looking at a new Z600 for $5k. According to Slashgear, the various models and configurations were sold for anywhere between $1800 and $7500 back in 2009. I'll let you know how I get on with it when it arrives.

  • TangoAlphaTangoAlpha Posts: 4,579

    Quick question: Is there a way to queue up renders from a bunch of different cameras on the same scene? I'm going to be out all afternoon, and I'd rather like Carrara to be productive in my absence, IYSWIM.

    I suspect I'm going to need to save a different version of the scene for every camera, but if there's a better way . . .?

  • render it as an anmated image series a new camera position each frame

  • TangoAlphaTangoAlpha Posts: 4,579
    edited September 2016

    Thanks ToeJam, that worked a treat.laugh Here's a one of the resulting wip renders, purely for the sake of it!

    tao-anim000.jpg
    1600 x 900 - 200K
    Post edited by TangoAlpha on
  • oooh apple orchard coming yes

  • @TangoAlpha Or you could just save the scene once with all your various cameras in it, and then load it into the render queue several times. Select each scene in the render queue one by one, and override the render settings for each slot - selecting a different camera, and giving it an appropriate file name. Then set off the queue, and your single scene should be rendered from every camera angle you've selected.

  • JonstarkJonstark Posts: 2,569

     

    ToeJam said:

    render it as an anmated image series a new camera position each frame

    Because my mind is slow I puzzled over this method you described for quite some time before I understood, simply a brilliant solution and one I intend to make use of in the future!  Thanks Wendy! (It is Wendy, right?  Apologies if I've got it wrong.)

     

    @TangoAlpha Or you could just save the scene once with all your various cameras in it, and then load it into the render queue several times. Select each scene in the render queue one by one, and override the render settings for each slot - selecting a different camera, and giving it an appropriate file name. Then set off the queue, and your single scene should be rendered from every camera angle you've selected.

    This is the method that immediately sprang to mind for me, and a perfectly workable method too.  The prior method Wendy described has the virtue of just being one scene to load up, and doing it as an animation means that your rendernodes can move on to the next 'frame' (picture/viewpoint) without having to wait for the first frame to complete.  Both ways would work fine though (at least I think so; in truth I haven't tried either method myself yet).

  • JonstarkJonstark Posts: 2,569
    edited September 2016

    Btw, my z600 server hasn't even arrived yet (probably not til this coming Thursday) and I'm already wearing out ebay searching for the next awesome server deal :)   I found a review of the z600 series when it came out, you were right on the money, these things went for over $5000 a pop.  It's amazing to think only 5 years later the same tech can be had for $200.  The next step up is the z620 series, apparently (or comparable servers Dell Precision T5600 or Lenovo D30 Thinkstation) with Sandy bridge era tech that can have dual 8-core xeons for 32 cores rendering in one box, but they are quite a step up in price.  The Lenovo's look the most expensive and the Dell's the least (not surprising to me, as I've had some bad experiences in the past with Dell) but even the least expensive is still nearly $600 for the used 32-core units.  But who knows, in another year or two they might fall into my price range as technology marches on    Soon I'll have 48 cores of rendering power, which is awesome, but my inner greedy pig wants ever more cores...  

    smiley

    Post edited by Jonstark on
  • Yes I am the user formerly known as Wendyluvscatz, and one of the first things I did when loading my first ever bought premde Carrara HowieFarkes scenes was chose each of the preset camera positions and render it out as a series.

  • TangoAlphaTangoAlpha Posts: 4,579

    It showed me that some of my camera angles were croc (got a lovely render of the inside of a tree!), and that some of my props and replicators still need work, but it was great to get 4 large size renders finished while I was out of the house! It might be the size of my screen, or the age of my eyes, but often postcard-size renders simply don't show me enough detail to spot what needs doing . . .

  • @TangoAlpha I couldn't agree more - small renders as previews aren't cutting it for me either :-)

    @Jonstark My Z600 is apparently arriving tomorrow - two days ahead of schedule! I've ordered a new GTX 970 graphics card and a bit more RAM too, in preparation for some DAZ Studio rendering. Will it fit, and will it work? Who knows... but what I haven't got yet is a simple USB keyboard! In this age of wireless tech, I no longer own one... can't install Windows without it. I've also been sneakily looking at the NEXT system I could order as a node... we're CRAZY you know that, right? Let's see what the Z600 brings (there are so many for around $250 with free shipping). I'm keeping my eye on a Z800 too, but they're twice as much (so are the Z620s), so perhaps sticking to the Z600 era is the cheaper option. Keep me posted on your progress!

  • 3drendero3drendero Posts: 1,911

    Maximum amount of cores per rig, below 500$, is probably and old AMD G34 4P folding monster, 48 cores for 400$ here:

    https://hardforum.com/threads/fs-48-core-supermicro-4p-amd-g34-workstation-servers-overclocking-ready.1908904/

    A few years ago they were 2000$, when shopping for used CPU's and modding regular heatsinks, OEM price was probably >5000$

    Today a 2p Xeon rig is faster, but that costs more 500.

  • MistaraMistara Posts: 38,675

    dreamin about a hexcore i7. 12 buckets.

    an ebay refurb machine mebbe i can afford one 

  • JonstarkJonstark Posts: 2,569

    @TangoAlpha I couldn't agree more - small renders as previews aren't cutting it for me either :-)

    @Jonstark My Z600 is apparently arriving tomorrow - two days ahead of schedule! I've ordered a new GTX 970 graphics card and a bit more RAM too, in preparation for some DAZ Studio rendering. Will it fit, and will it work? Who knows... but what I haven't got yet is a simple USB keyboard! In this age of wireless tech, I no longer own one... can't install Windows without it. I've also been sneakily looking at the NEXT system I could order as a node... we're CRAZY you know that, right? Let's see what the Z600 brings (there are so many for around $250 with free shipping). I'm keeping my eye on a Z800 too, but they're twice as much (so are the Z620s), so perhaps sticking to the Z600 era is the cheaper option. Keep me posted on your progress!

    Exciting that it's coming so soon! I checked my order again to be sure but looks like I'll have to wait til Thursday.  Ironically I have wired usb keyboards, so I have a spare I don't even need. I also use and prefer wireless keyboards (got 2 of them, which are more ergonomic to help me type faster and more comfortably).

    I was doing the math on what it would cost to buy two 6 core Xeons and swap them out with the 4 core Xeons that are coming with the Z600, when it struck me that it would only be slightly less than simply buying an identical 2nd z600 instead, and adding that into the network, so I'm re-thinking the idea of upgrading the Xeon cpus that are coming with the box.  I agree that the Z600 era is probably the sweet spot: tons of render cores that are decently powered at a ridiculously low price point.  But who knows, in a couple of years it would be fun if Z620 era stuff comes down to this same price level (of course, that would probably mean the z600 era stuff is going for even less...)    Can't wait to get my grubby hands on it and see how it performs with Carrara.

Sign In or Register to comment.