Iray completely resetting when dropping to CPU

Why in the name of SANITY is the Iray in 4.12.0.86 sometimes completely restarting when it drops to CPU?

I've just had a render (which would have been fairly acceptable if I'd been able to put the GPU rendering through a denoiser) completely start over when it went to CPU, totally trashing the process and wasting loads of time (and electricity, for that matter). And this isn't the first time it's happened, although fortunately it usually happens only a few iterations in and I can restart.

No, my system isn't spectacular - i7 4790, GTX 1050 Ti (4GB), 16GB RAM, Windows 8.1 - but this never happened back on 4.10 (I largely skipped over 4.11; rendering was noticeably slower and it didn't offer much of interest - I prefer standalone denoisers so I can mask/blend the results in PS) ; when it dropped to CPU before, it would at least preserve the existing progress.

Comments

  • TheKDTheKD Posts: 1,269

    I had the dropping to CPU in the middle of a render problem a lot when I "upgraded" to win10 for my newly built PC. After banging my head against the wall about it for a few weeks, I "downgraded" to win7, despite the fact my MOBO don't support it and I can't use the ethernet port on it now. In win7, the only time I have ever seen a render drop to CPU from GPU during a render is when the driver crashes, which is super rare. Ohterwise if the render is able to fit in VRAM to start, it renders until I tell it to stop, or it hits the iteration vcount I set it at. Something about the way later versions of win handle VRAM, does not seem to work so well with Iray rendering. In all other areas, like gaming etc, the GPU worked fine in win10.

  • Matt_CastleMatt_Castle Posts: 791

    The fact that the render ever drops to CPU mid-render is annoying, but I could handle that if I didn't have this new problem where if the render does drop to CPU, it sometimes starts over from iteration 1, completely trashing all the progress.

    If I come back to the computer and I find the render has dropped to CPU but kept its progress, that's something I have a chance of rescuing with a denoiser, a few spot renders and some postwork; If I come back to the computer and find that it got 80% through, then decided to completely trash everything and then start over at a snail's pace, then I'm forced to start over with a new game of Render Roulette and wonder if it's going to commit suicide again.

  • RayDAntRayDAnt Posts: 489
    edited November 8

    The fact that the render ever drops to CPU mid-render is annoying, but I could handle that if I didn't have this new problem where if the render does drop to CPU, it sometimes starts over from iteration 1, completely trashing all the progress.

    If I come back to the computer and I find the render has dropped to CPU but kept its progress, that's something I have a chance of rescuing with a denoiser, a few spot renders and some postwork; If I come back to the computer and find that it got 80% through, then decided to completely trash everything and then start over at a snail's pace, then I'm forced to start over with a new game of Render Roulette and wonder if it's going to commit suicide again.

    Do you still have the log file from when it happened? That's a very unusual phenomenon (iteration reset + cpu fallback in tandem.) Would be very curious to see what it says happened. Normally those two things are completely unrelated.

    Post edited by RayDAnt on
  • Matt_CastleMatt_Castle Posts: 791

    Unfortunately not. I was in a bit of a hurry, and while I took a look, I can't now remember what it said, and I completely forgot to save a copy - I'm not used to programs overwriting their log files quite as fast as Daz seems to.

    However, it has happened a few times now, so I'll try to remember to keep a copy if it happens again.

  • I just got this problem, and here is my log when it drops to CPU. It only happens to me if I'm using most of my VRAM. So I try to make my scenes light.

     

    2019-11-10 02:20:04.823 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00036 iterations after 201.966s.

    2019-11-10 02:20:24.917 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00040 iterations after 222.060s.

    2019-11-10 02:20:44.938 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00044 iterations after 242.081s.

    2019-11-10 02:21:04.791 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00048 iterations after 261.935s.

    2019-11-10 02:21:29.915 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00053 iterations after 287.058s.

    2019-11-10 02:21:54.320 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00058 iterations after 311.464s.

    2019-11-10 02:22:28.136 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00064 iterations after 345.279s.

    2019-11-10 02:22:59.194 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00071 iterations after 376.337s.

    2019-11-10 02:23:34.294 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00078 iterations after 411.437s.

    2019-11-10 02:24:14.149 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00086 iterations after 451.292s.

    2019-11-10 02:25:06.673 Iray [VERBOSE] - IRAY:RENDER ::   1.0   IRAY   rend progr: 3.37% of image converged

    2019-11-10 02:25:06.944 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00095 iterations after 504.088s.

    2019-11-10 02:25:48.458 Iray [INFO] - POST:RENDER ::   1.0   POST   rend info : using cuda device 0, "GeForce GTX 1080 Ti" (6.1), buffers: fp16, cuDNN v7500, rt v10010

    2019-11-10 02:25:48.680 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.690 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.690 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.701 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.846 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.857 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.865 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.866 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.873 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.890 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.895 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:48.895 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.023 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.025 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.027 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.038 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.043 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.113 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.122 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.217 Iray [INFO] - POST:RENDER ::   1.0   POST   rend info : layers created for resolution 5376 640, inp 6, outp 3, cuDNN memory 8.1 MB, total 563.3 MB

    2019-11-10 02:25:49.514 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.529 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.533 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.538 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.565 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.565 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.569 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.578 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.585 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: out of memory

    2019-11-10 02:25:49.643 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.664 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.668 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.675 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:49.849 Iray [INFO] - POST:RENDER ::   1.0   POST   rend info : layers created for resolution 5376 384, inp 6, outp 3, cuDNN memory 8.1 MB, total 341.2 MB

    2019-11-10 02:25:49.934 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: Autoexposure failed (invalid argument), now using default hdr scale: 1.000000

    2019-11-10 02:25:49.940 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: invalid argument

    2019-11-10 02:25:52.082 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.6   IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while launching CUDA renderer)

    2019-11-10 02:25:52.082 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_BAD_PARAM

    2019-11-10 02:25:52.082 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_BAD_PARAM

    2019-11-10 02:25:52.088 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.6   IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): Failed to launch renderer

    2019-11-10 02:25:52.088 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_BAD_PARAM

    2019-11-10 02:25:52.088 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.093 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.093 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): Device failed while rendering

    2019-11-10 02:25:52.093 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.14  IRAY   rend warn : All available GPUs failed.

    2019-11-10 02:25:52.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER ::   1.14  IRAY   rend warn : No devices activated. Enabling CPU fallback.

    2019-11-10 02:25:52.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while initializing memory buffer)

    2019-11-10 02:25:52.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: All workers failed: aborting render

    2019-11-10 02:25:52.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.109 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.109 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:52.109 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.114 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:52.114 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.114 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:52.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.14  IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:52.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.125 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

    2019-11-10 02:25:52.125 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: CUDNN_STATUS_EXECUTION_FAILED

     

    (above line repeats several hundred lines) 

     

    2019-11-10 02:25:52.709 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER ::   1.0   POST   rend error: A CUDA error occurred: an illegal memory access was encountered

    2019-11-10 02:25:52.715 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER ::   1.0   POST   rend warn : Denoising unsuccessful, probably out of device memory. aborting.

    2019-11-10 02:25:52.716 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER ::   1.0   POST   rend warn : Failed to allocate device memory, falling back to CPU

    2019-11-10 02:25:52.716 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER ::   1.0   POST   rend warn : Image denoising is not available on the CPU and will be disabled.

    2019-11-10 02:25:52.743 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER ::   1.0   POST   rend warn : Postprocessing is running on the CPU, but Denoiser is requested. Denoising is not available on CPU.

    2019-11-10 02:25:53.309 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Received update to 00104 iterations after 549.247s.

    2019-11-10 02:25:53.466 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.0   IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:53.471 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.0   IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:53.471 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.0   IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:53.476 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.0   IRAY   rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)

    2019-11-10 02:25:53.536 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER ::   1.0   IRAY   rend error: Scheduler was aborted for restart and needs to be restarted

    2019-11-10 02:25:53.585 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CPU: using 6 cores for rendering

    2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering with 1 device(s):

    2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : CPU

    2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER ::   1.0   IRAY   rend info : Rendering...

    2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER ::   1.5   IRAY   rend info : Initializing Embree

    2019-11-10 02:25:54.521 Iray [VERBOSE] - IRAY:RENDER ::   1.14  IRAY   rend progr: CPU: Processing scene...

    2019-11-10 02:25:55.358 Iray [INFO] - IRAY:RENDER ::   1.14  IRAY   rend info : CPU: Scene processed in 1.768s

    2019-11-10 02:25:55.364 Iray [VERBOSE] - IRAY:RENDER ::   1.5   IRAY   rend stat : Native CPU code generated in 0.069s

    2019-11-10 02:25:55.815 Iray [INFO] - IRAY:RENDER ::   1.14  IRAY   rend info : CPU: Allocated 992.25 MiB for frame buffer

  • RayDAntRayDAnt Posts: 489

    Ah, You didn't mention you were using the ai denoiser when this happenens. Because your scene is near the threshold of VRAM, and Nvidia's ai denoiser uses a significant amount of VRAM as well (for this scene +500MB) that is causing both the Iray rendering process to fail (resulting in CPU fallback) AND the Iray scheduling process to crash (resulting in an iterations restart.)

    The solution is to not use the built-in denoiser when rendering scenes close to the limit of GPU VRAM (there's at least one standalone alternative to it from Intel you could try using instead.) It might also be worth submitting a bug report about it to Nvidia somehow (that scheduler crash seems pretty suspect - you'd think the denoiser would fail and the scheduler would catch the failure and just continue without it.)

  • Matt_CastleMatt_Castle Posts: 791
    edited November 10
    RayDAnt said:

    Ah, You didn't mention you were using the ai denoiser when this happenens.

    I didn't mention it, because it's not my log file (or my post!). (I am however happy for others to share their information; it's more practical to keep all the information on this problem in the same place).

    Personally, I always denoise afterwards because I a) prefer the results from Intel's AI denoiser to Nvidia's and b) prefer to be able to keep the original and denoised files so I can mix them in Photoshop, masking different areas at different strengths depending on exactly how noisy they are. (Although I will usually fade in a few percent of the original across most of the scene; a little noise usually looks better than something too clean.)

    However, on that note, I do generally run with Optix Prime Acceleration enabled, which might be something to explore, because I know that can also use a reasonable chunk of VRAM.

    Post edited by Matt_Castle on
  • RayDAntRayDAnt Posts: 489

    I didn't mention it, because it's not my log file (or my post!).

    Oops - was in the midst of commuting and on mobile when I saw that post and assumed it was also from you without seeing the name.

     

    However, on that note, I do generally run with Optix Prime Acceleration enabled, which might be something to explore, because I know that can also use a reasonable chunk of VRAM.

    Could very well be. Any technically extra feature (OptiX Prime, denoising, etc) that involves a memory usage penalty on top of what the scene itself needs to be rendered is suspect any time you're working with scenes close to the physical memory limit of your GPU.

Sign In or Register to comment.