AI as render engine
colcurve
Posts: 171
in The Commons
Why cant we use diffusion as render engine/controlnet? It would give so many possibilities, even for old content.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2025 Daz Productions Inc. All Rights Reserved.
Comments
nobody is stopping you
I often run Filament or OpenGL renders through Stable Diffusion Image2Image
What do you mean? Rendering and generative AI are doing vey diffrent things, even if both produce an image. Are you wanting a way to use the current state of the Viewport for an image-to-image generation without having to save and import the image into the AI tool?
@richard: no, I do not think so. what you design in a scene is input to a render engine and/or to a controlnet. basically a 3d scene is the controlnet for a "render" engine, be it iray or "stable diffusion, flux", or any other magic
when I currently browse the store, I nowadays have a specific look of vintage items, because I think they have a good idea but night have latest shaders or whatever. maybe i think the textures are not modern but as depth map/outline render it could be excellent... as controlnet input
I think if daz would offer tools to use old content as controlnet... it would raise the value of large contents of the store by 100x..
A thing to keep in mind is that, if things continue as they seem to be going, that art created by AI programs will be be far less protected by copyright, if at all. Beyond that, I've tried a number of the various programs, most of them still have a lot of issues with basic anatomy. That said, I recently bought Gigapixel to enlarge some old images and I've been experimenting with using the AI function it on some of my newer images with the enhancement/extrapolation levels set very low, then using the reuslting image to augment parts of the original render. Where it really excels is on cleaning up digital hair and creating realistic... if often way over the top... wrinkles on cloth or on the skin at joints like the underarms. Unfortunately, where it's still incredibly weak is on things like hands, feet, and smaller skin details like nipples , navels, small details like text tends to become gibberish, and while the AI will refine some elements of a facial structure, it also tends to make the characters look significantly different from the original model. In other words, it's still a long way away from being something that I'd be ready to trust as the central point of control for creating anything I was going to put my name on.
There are nothing alike, text to image (or image to image style change) vs 3D to iray render are fundamentally different field, while AI generated images offers some charm, the lack of fine control, unpredictable generating process, and moral limition severly limited what creators can/can not do.
No, the processes are completely different - when rendering you place elements in the scene with properties and those completely determine what the render engine produces. With Generative systems it produces a cloud of noise and refines it using statisical associations from the prompts and influenced by any control net image etc. - the process is still deterministic, but one of the determining factors is a pseudo-random seed value used to generate the initial noise cloud.
Unsure what OP really wants with Daz. The act of inputting your prompt becomes the rendering. Daz Studio isn't for those who really like to dabble in AI. Yes, there's an AI version, but it doesn't hold a candle compared to the base product and what can be done with human artistic skills. And yes, AI can also be used to enhance images, but that's a different beast than doing entire image generation. At that point, you already spent time on your rendering in Daz Studio. And we have Photoshop for that kind of thing, also with Daz support.
I much prefer Daz Studio as the ability to fully control what I'm doing is paramount to the finished product. That control, human eye, experience and drive also saves an immense amount of time and frustration when I'm tweaking things to exactly how I envision them. But these things come with working on improving oneself's abilities at something, and sadly, many don't like that and look for quick fixes, instead of spending time to develop said skill (not saying you OP, just in general, it's way too common today).
They want an AI render engine to piggyback off of Iray and 3DL. The workflow would be something like: Set up scene > Render with Iray or 3DL > Enhance with AI. All from within Daz. No 3rd-party apps or AI models needed.
colcurve, read this thread
https://www.daz3d.com/forums/discussion/591121/remixing-your-art-with-ai#latest
are suggestions and examples
That's not how I read the OP. I got the impression that they wanted to set up their scene inside DS, then output it directly to an AI (which, as Richard pointed out, isn't really how this works).
yeah, iray is ray pathtracing, bouncing rays off 3D co-ordinates
a mathematical geometry based process
Diffusers and Ai is something quite different, a dataset trained on images
you can use depth maps and images rendered in DAZ in ControNet which I do and what that thread I linked is all about
someone could make a plugin but Photoshop bridge already exists if you want to go down that path
no need for a plugin though as I often run Fooocus and DAZ together myself just rendering the image then loading it into Fooocus
using openGL to do my prompt image in DAZ keeps the VRAM available, is sufficient for canny guidence
Ah, well, in that case, I stand corrected.
This is not a contradiction. Render produces an image and this can be input to any step in the overall pipeline, daz offers an AI denoise postfilter to iray as well already.
How much render and how much AI postprocess should be left to the user.
AI has problem that it does not have good control with the prompt, thats a spin but it is not precision of any sort.
DAZ has infinite control but as such can take forever to tweak things or set them up.
Using a Daz scene as depth map can bring interesting outcome. Clicking it to configure canvas however and convert the exr to an image was much work so I bought the depth map maker from the store, but this tool often crashes for me and it does not support limiting the scene to a certain depth, so you have can invisible/far away things in your scene that ruin the depth map because the value range gets overstreched.
But you can also use e.g. partial finished render as controlnet, some controlnets have an "anything" approach accepting just any image. Or you can denoise (as you said) with low/medium intensity.
Doing so, it does not matter (so much) whether the things in your scene are iray or 3dlight shaded.
I understand diffusion as a post-render step.
Ideally, we would have a DAZ node for comfyui. It would read image data from DAZ application (render, depth pass, specific material as mask) and make it accessible to comfy, there you could further consume it. E.g. if DAZ could provide 1.) a render and 2.) a mask image with the same camera as the render, one could use comfy to modify the masked part in the image with something the render engine has a difficult job at (e.g. "add a background crowd", "three creepy zombies looking at "xyz"" or anything else.
I think using 3d scene with AI made specific elements/postprocess is just the best way to use these things.
The good thing with old DAZ products is that they are pretty lightweight. Less faces, highly responsive viewport, ...
DAZ has a unique position as simplified enabler of 3d, in contrast to blender where you need a PhD to use it. I think ignoring the mixed render/AI use cases is bad decision, as it misses a rather large business opportunity which does not take vast resources to create, and the blender/... community will develop the pipeline anyway and it will be competing the DAZ ecosystem which would be the greatest waste.
If someone PA /DAZ developer here reads this: Welcome to make contact...
...like an AI powered post render realism filter?
Use a script to render and save (or save the last draw) and then launch the stand-alone AI with the saved iamge as input and whatever other parameters were required.
See http://docs.daz3d.com/doku.php/public/software/dazstudio/4/referenceguide/scripting/api_reference/samples/rendering/render_post_process/start for a starting point using a render
You can just download the C++ SDK and create one yourself.
Ai image generation tech, such as stable diffusion, does not simulate light bounces like Iray, or have a 3d reference of the scene. So it's more like post processing on an already-rendered image.
You can give stable diffusion extra information about 3d space using ControlNets. That can help reduce the divergence from your original input image as the ai reimagines it.
For example, you can use a normal map to indicate surface directions, and a depth map to show each object's distance from the camera, canny or lineart to preserve detail areas, and tile to maintain overall composition. Other tools can do object recognition or body parts recognition, like automatically detecting and detailing eyes, or style transfer to completely change the look and feel.
Daz Studio can already render some of the maps useful as input for ControlNet. (Depth and Normal in particular.) So somebody could definitely make a plugin to help smooth the workflows. Some future iteration of Daz Studio could also push a scene into the cloud, render it in Iray or Filament, and help apply some ai post processing without the user needing a powerful PC or fiddling with a complicated stable diffusion setup. ie There's some opportunity to make tools like stable diffusion more accessible to 3d artists.
I got DepthMap Maker for the same reasons - but it's worth noting that depth ControlNets are not trained on actual depth images (there just aren't the massive quantities required), but synthesized ones. So in many cases you may be better off having the ControlNet process your regular viewport image/render and have it synthesize depth data similar to what it was trained on, rather than give it actual depth data that may contain artifacts that it was not trained on.
It does seem way past time, though, for a utility that would output an OpenPose model, like posemy-dot-art.