Iray completely resetting when dropping to CPU
Matt_Castle
Posts: 2,346
Why in the name of SANITY is the Iray in 4.12.0.86 sometimes completely restarting when it drops to CPU?
I've just had a render (which would have been fairly acceptable if I'd been able to put the GPU rendering through a denoiser) completely start over when it went to CPU, totally trashing the process and wasting loads of time (and electricity, for that matter). And this isn't the first time it's happened, although fortunately it usually happens only a few iterations in and I can restart.
No, my system isn't spectacular - i7 4790, GTX 1050 Ti (4GB), 16GB RAM, Windows 8.1 - but this never happened back on 4.10 (I largely skipped over 4.11; rendering was noticeably slower and it didn't offer much of interest - I prefer standalone denoisers so I can mask/blend the results in PS) ; when it dropped to CPU before, it would at least preserve the existing progress.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
I had the dropping to CPU in the middle of a render problem a lot when I "upgraded" to win10 for my newly built PC. After banging my head against the wall about it for a few weeks, I "downgraded" to win7, despite the fact my MOBO don't support it and I can't use the ethernet port on it now. In win7, the only time I have ever seen a render drop to CPU from GPU during a render is when the driver crashes, which is super rare. Ohterwise if the render is able to fit in VRAM to start, it renders until I tell it to stop, or it hits the iteration vcount I set it at. Something about the way later versions of win handle VRAM, does not seem to work so well with Iray rendering. In all other areas, like gaming etc, the GPU worked fine in win10.
The fact that the render ever drops to CPU mid-render is annoying, but I could handle that if I didn't have this new problem where if the render does drop to CPU, it sometimes starts over from iteration 1, completely trashing all the progress.
If I come back to the computer and I find the render has dropped to CPU but kept its progress, that's something I have a chance of rescuing with a denoiser, a few spot renders and some postwork; If I come back to the computer and find that it got 80% through, then decided to completely trash everything and then start over at a snail's pace, then I'm forced to start over with a new game of Render Roulette and wonder if it's going to commit suicide again.
Do you still have the log file from when it happened? That's a very unusual phenomenon (iteration reset + cpu fallback in tandem.) Would be very curious to see what it says happened. Normally those two things are completely unrelated.
Unfortunately not. I was in a bit of a hurry, and while I took a look, I can't now remember what it said, and I completely forgot to save a copy - I'm not used to programs overwriting their log files quite as fast as Daz seems to.
However, it has happened a few times now, so I'll try to remember to keep a copy if it happens again.
I just got this problem, and here is my log when it drops to CPU. It only happens to me if I'm using most of my VRAM. So I try to make my scenes light.
2019-11-10 02:20:04.823 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00036 iterations after 201.966s.
2019-11-10 02:20:24.917 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00040 iterations after 222.060s.
2019-11-10 02:20:44.938 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00044 iterations after 242.081s.
2019-11-10 02:21:04.791 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00048 iterations after 261.935s.
2019-11-10 02:21:29.915 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00053 iterations after 287.058s.
2019-11-10 02:21:54.320 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00058 iterations after 311.464s.
2019-11-10 02:22:28.136 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00064 iterations after 345.279s.
2019-11-10 02:22:59.194 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00071 iterations after 376.337s.
2019-11-10 02:23:34.294 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00078 iterations after 411.437s.
2019-11-10 02:24:14.149 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00086 iterations after 451.292s.
2019-11-10 02:25:06.673 Iray [VERBOSE] - IRAY:RENDER :: 1.0 IRAY rend progr: 3.37% of image converged
2019-11-10 02:25:06.944 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00095 iterations after 504.088s.
2019-11-10 02:25:48.458 Iray [INFO] - POST:RENDER :: 1.0 POST rend info : using cuda device 0, "GeForce GTX 1080 Ti" (6.1), buffers: fp16, cuDNN v7500, rt v10010
2019-11-10 02:25:48.680 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.690 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.690 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.701 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.846 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.857 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.865 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.866 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.873 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.890 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.895 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:48.895 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.023 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.025 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.027 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.038 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.043 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.113 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.122 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.217 Iray [INFO] - POST:RENDER :: 1.0 POST rend info : layers created for resolution 5376 640, inp 6, outp 3, cuDNN memory 8.1 MB, total 563.3 MB
2019-11-10 02:25:49.514 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.529 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.533 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.538 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.565 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.565 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.569 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.578 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.585 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: out of memory
2019-11-10 02:25:49.643 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.664 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.668 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.675 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:49.849 Iray [INFO] - POST:RENDER :: 1.0 POST rend info : layers created for resolution 5376 384, inp 6, outp 3, cuDNN memory 8.1 MB, total 341.2 MB
2019-11-10 02:25:49.934 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: Autoexposure failed (invalid argument), now using default hdr scale: 1.000000
2019-11-10 02:25:49.940 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: invalid argument
2019-11-10 02:25:52.082 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.6 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while launching CUDA renderer)
2019-11-10 02:25:52.082 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_BAD_PARAM
2019-11-10 02:25:52.082 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_BAD_PARAM
2019-11-10 02:25:52.088 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.6 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): Failed to launch renderer
2019-11-10 02:25:52.088 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_BAD_PARAM
2019-11-10 02:25:52.088 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.093 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.093 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): Device failed while rendering
2019-11-10 02:25:52.093 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER :: 1.14 IRAY rend warn : All available GPUs failed.
2019-11-10 02:25:52.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - IRAY:RENDER :: 1.14 IRAY rend warn : No devices activated. Enabling CPU fallback.
2019-11-10 02:25:52.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.099 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while initializing memory buffer)
2019-11-10 02:25:52.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: All workers failed: aborting render
2019-11-10 02:25:52.104 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.109 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.109 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:52.109 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.114 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:52.114 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.114 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:52.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.14 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:52.119 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.125 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
2019-11-10 02:25:52.125 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: CUDNN_STATUS_EXECUTION_FAILED
(above line repeats several hundred lines)
2019-11-10 02:25:52.709 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - POST:RENDER :: 1.0 POST rend error: A CUDA error occurred: an illegal memory access was encountered
2019-11-10 02:25:52.715 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER :: 1.0 POST rend warn : Denoising unsuccessful, probably out of device memory. aborting.
2019-11-10 02:25:52.716 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER :: 1.0 POST rend warn : Failed to allocate device memory, falling back to CPU
2019-11-10 02:25:52.716 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER :: 1.0 POST rend warn : Image denoising is not available on the CPU and will be disabled.
2019-11-10 02:25:52.743 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [WARNING] - POST:RENDER :: 1.0 POST rend warn : Postprocessing is running on the CPU, but Denoiser is requested. Denoising is not available on CPU.
2019-11-10 02:25:53.309 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Received update to 00104 iterations after 549.247s.
2019-11-10 02:25:53.466 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:53.471 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:53.471 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:53.476 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: CUDA device 0 (GeForce GTX 1080 Ti): an illegal memory access was encountered (while de-allocating memory)
2019-11-10 02:25:53.536 WARNING: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(305): Iray [ERROR] - IRAY:RENDER :: 1.0 IRAY rend error: Scheduler was aborted for restart and needs to be restarted
2019-11-10 02:25:53.585 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : CPU: using 6 cores for rendering
2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering with 1 device(s):
2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : CPU
2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER :: 1.0 IRAY rend info : Rendering...
2019-11-10 02:25:53.591 Iray [INFO] - IRAY:RENDER :: 1.5 IRAY rend info : Initializing Embree
2019-11-10 02:25:54.521 Iray [VERBOSE] - IRAY:RENDER :: 1.14 IRAY rend progr: CPU: Processing scene...
2019-11-10 02:25:55.358 Iray [INFO] - IRAY:RENDER :: 1.14 IRAY rend info : CPU: Scene processed in 1.768s
2019-11-10 02:25:55.364 Iray [VERBOSE] - IRAY:RENDER :: 1.5 IRAY rend stat : Native CPU code generated in 0.069s
2019-11-10 02:25:55.815 Iray [INFO] - IRAY:RENDER :: 1.14 IRAY rend info : CPU: Allocated 992.25 MiB for frame buffer
Ah, You didn't mention you were using the ai denoiser when this happenens. Because your scene is near the threshold of VRAM, and Nvidia's ai denoiser uses a significant amount of VRAM as well (for this scene +500MB) that is causing both the Iray rendering process to fail (resulting in CPU fallback) AND the Iray scheduling process to crash (resulting in an iterations restart.)
The solution is to not use the built-in denoiser when rendering scenes close to the limit of GPU VRAM (there's at least one standalone alternative to it from Intel you could try using instead.) It might also be worth submitting a bug report about it to Nvidia somehow (that scheduler crash seems pretty suspect - you'd think the denoiser would fail and the scheduler would catch the failure and just continue without it.)
I didn't mention it, because it's not my log file (or my post!). (I am however happy for others to share their information; it's more practical to keep all the information on this problem in the same place).
Personally, I always denoise afterwards because I a) prefer the results from Intel's AI denoiser to Nvidia's and b) prefer to be able to keep the original and denoised files so I can mix them in Photoshop, masking different areas at different strengths depending on exactly how noisy they are. (Although I will usually fade in a few percent of the original across most of the scene; a little noise usually looks better than something too clean.)
However, on that note, I do generally run with Optix Prime Acceleration enabled, which might be something to explore, because I know that can also use a reasonable chunk of VRAM.
Oops - was in the midst of commuting and on mobile when I saw that post and assumed it was also from you without seeing the name.
Could very well be. Any technically extra feature (OptiX Prime, denoising, etc) that involves a memory usage penalty on top of what the scene itself needs to be rendered is suspect any time you're working with scenes close to the physical memory limit of your GPU.