Has Anyone here experimented with Gaussian Splatting?

So, super quick primer, Gaussian Splatting is a relatively very new 3d rendering technique where instead of polygons and textures and all of that, you can have a 3d scene depicted in real time as millions of semi-transparent dots in a 3d space and that can then be just rendered in a 3d engine and consequently sidestep issues of having very high resolution 3d models attempting to be rendered in real time, amongst other advantages (such as avoiding needing to have interactive distributions rights for every 3d asset in the scene because you are not actually distributing 3d assets).

Right now, I don't have a pipeline for going from a DAZ Studio scene to a gaussian splatting output, has anyone here done anything like that?

I'm gonna be investigating this and I can come back and repost.

«1

Comments

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    not successfully devil

    took all the sequences of photographs 

    tried all the programs like 3DZephyr, Meshroom

    ended up with bizarre mishapen objects that looked even worse when I joined the dots cheeky

    now I just use AI instead, on my PC Hunyuan3D

    4 photographs front, back, left, right.

    though online ones like Meshy etc are better, but I am not paying when I can do it for free

     

  • DustRiderDustRider Posts: 2,911
    edited August 30

    So, super quick primer, Gaussian Splatting is a relatively very new 3d rendering technique where instead of polygons and textures and all of that, you can have a 3d scene depicted in real time as millions of semi-transparent dots in a 3d space and that can then be just rendered in a 3d engine and consequently sidestep issues of having very high resolution 3d models attempting to be rendered in real time, amongst other advantages (such as avoiding needing to have interactive distributions rights for every 3d asset in the scene because you are not actually distributing 3d assets).

    Right now, I don't have a pipeline for going from a DAZ Studio scene to a gaussian splatting output, has anyone here done anything like that?

    I'm gonna be investigating this and I can come back and repost.

    I've made a few just evaluating them vs SFM 3D photogrammetry (i.e. making a 3D model from the photo "scan"), but haven't done anything serious with them. They definitely are less resource intensive than 3D models and that can be an advantage for real-time display. But they do have some downsides like areas of poor resolution, stray splats, and weird blurry areas. Just Like SFM photogrammetry, if you can't see something (or see it clearly) in the original photographs/images, it will also be missing/blurred in the splat (it must also be in more than one picture). I saw something the other day about using AI to make a gaussian splat from a single photograph. I didn't investigate that as my needs are for accurate data, not made up data. I'm sure an AI created gaussian splat would suffer from the same issues that AI image/video generation has (hallucinations and making stuff up that is just plain wrong).

    I highlighted a portion of your post that might be your a misunderstanding of the tech, or my misunderstanding of what you mean. From your post, I get the impression that you are thinking about taking DAZ assets and making gaussian splats from images of them? If so, I would think that this does indeed need at least an interactive license to use in a game, but possibly not for personal use (you should probably send a ticket to tech support if you plan on doing this to make sure it is OK). Gaussian splats are indeed 3D data, points in 3D space with a sort of "texture map" (plane) placed on each point. Gaussian splats can also be turned into 3D models, therefore I would seem that distribution of a gaussian splat based on DAZ assets would be distribution of a derivative 3D product.

    Post edited by DustRider on
  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498
    edited August 30

    yeah, what I did was Photogrammetry but it uses Gaussian splats to do it

    you have the scenes made of point clouds

    Twinmotion which I use for rendering a lot, loads them too but never found any for free to test it

    Post edited by WendyLuvsCatz on
  • DustRiderDustRider Posts: 2,911
    edited August 30

    There are some free downloadable GS models on Sketchfab if you want something to test in Twinmotion.

    Yep, GS is just another form of photogrammetry. To make your own GS models you want your photos to have at least a 70% overlap from photo to photo. It also helps if there is a slow steady movement of the camera so the software can easily go from photo to photo. when turning corners you also need to move a bit slower so the software can "follow" the movement. Super wide angle cameras can help a lot.

    My dabbling with GS is using a SLAM Lidar scanner with dual fisheye cameras. The cameras are needed for coloring the lidar point cloud, but you can also generate a GS from the photos. The upside to this method is the GS will have accurate geometries because it uses the lidar point cloud to help align the photos (The GS will be as accurate as your lidar scan, if it drifts, so will the GS). The downside is the scanner I'm using (low end .... "low" cost) doesn't have a top notch camera system, so the splats aren't awesome (I have a Insta360 X5 with an adapter, but I haven't had time to process any of that data yet). As you probably know, GS processing take a long time (unless you do it on line, but that costs$$$).

    Edit: Thanks for the tip on Twinmotion and GS, I might have to try that sometime.

    Post edited by DustRider on
  • DiasporaDiaspora Posts: 552
    edited August 30

    Thank you both for your comments
    @Wendy,

    I don't quite understand, "PC Hunyuan3D", so you have used that successfully for GS from photographs but not from DAZ Studio renders? Or could you use that technique for Daz Studio renders but have only tried it with photography?

    @DustRider,

    "From your post, I get the impression that you are thinking about taking DAZ assets and making gaussian splats from images of them?"

    Yes, if that's the best workflow I can get out of DAZ. From what you're telling me of your experiments, the ideal solution would be if there existed a way to get a gaussian splat directly from the render engine but it sounds like I'm asking for the moon here so I'll just do the best I can with what I have available.

    Theoretically though, if there existed a way to get a gaussian splat directly out of DAZ Studio, where the GS generator could have access to the 3d environment, it would be possible to make extremely accurate and extremely high resolution Gaussian Splats.

    "(you should probably send a ticket to tech support if you plan on doing this to make sure it is OK). "

    Ok, I will do that if I get to the point that I intend to redistribute. I would argue though that although it sounds like a Gaussian splat has enough information that a generative AI model could construct a somewhat accurate "broad strokes" 3d model from it, a gaussian splat is more comparable to a render (especially since apparently it can only be made from renders, the GS generator has no direct access to the 3d space itself) that a lot of the information you need to have a truly useful 3d model, the topology, the UVs, the actual high resolution texture maps, the morphs, the rigging (in cases of characters), all of that data doesn't make it into the Gaussian Splat same as how it doesn't make it into the 2d renders that we're allowed to copyright and redistribute.

    But, I'll talk to DAZ before I make anything public.

    "SLAM Lidar"

    To improve accuracy, can't I do something similar in 3d software/Daz Studio? This sounds really similar to a z-depth pass.

    Post edited by Diaspora on
  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    I meant I find it easier to create 3D meshes from photos with AI than using Gaussian splatting techniques 

    that's all

  • lilweeplilweep Posts: 2,864

    Diaspora said:

     it can only be made from renders

    You mean 2d images (rather than renders) i assume. 

    I cant imagine people are going to go hunt down some person's random Gaussian Splat and then use some other AI model to extract degraded 3d models from it when there are already other more convenient ways to generate 3d models from images, or just ouright buy them from the store.
     

    This sounds really similar to a z-depth pass.

    There are gaussian splat approaches which allow crosstalk from different inputs and models, including depth information. 

    To me, it sounds like interactive license wouldnt even be required for Gaussian Splat from Daz renders for the reasons you mentioned earlier about the assets themselves not being embedded in the output.

    Daz render as a precursor to GS seems a bit unnecessary anyway since you can get AI to generate a GS from scratch.

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    DustRider said:

    There are some free downloadable GS models on Sketchfab if you want something to test in Twinmotion.

    Yep, GS is just another form of photogrammetry. To make your own GS models you want your photos to have at least a 70% overlap from photo to photo. It also helps if there is a slow steady movement of the camera so the software can easily go from photo to photo. when turning corners you also need to move a bit slower so the software can "follow" the movement. Super wide angle cameras can help a lot.

    My dabbling with GS is using a SLAM Lidar scanner with dual fisheye cameras. The cameras are needed for coloring the lidar point cloud, but you can also generate a GS from the photos. The upside to this method is the GS will have accurate geometries because it uses the lidar point cloud to help align the photos (The GS will be as accurate as your lidar scan, if it drifts, so will the GS). The downside is the scanner I'm using (low end .... "low" cost) doesn't have a top notch camera system, so the splats aren't awesome (I have a Insta360 X5 with an adapter, but I haven't had time to process any of that data yet). As you probably know, GS processing take a long time (unless you do it on line, but that costs$$$).

    Edit: Thanks for the tip on Twinmotion and GS, I might have to try that sometime.

    damn I find that site confusing 

    that said, I can download from Sketchfab inside Twinmotion using my Epic Games account (and Quixel Megascans too) 

    not sure about GS though will look

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    I have no idea if this is a Gaussian Splat

    it likely was at some point though

    captured using Windows game capture but of course I can render it in Lumen or Path Tracing if so inclined

  • DiasporaDiaspora Posts: 552

    WendyLuvsCatz said:

    I meant I find it easier to create 3D meshes from photos with AI than using Gaussian splatting techniques 

    that's all

    OOH, ok, my end goal is not to create meshes from images, it's to create 3d content rendered in real time that I and potentially others can look at.  

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    well actually Twinmotion can do that as a presentation

    a cooked Unreal Engine format 

    but you would need the gaming licenses for content used

  • DiasporaDiaspora Posts: 552
    edited August 30

    ""Daz render as a precursor to GS seems a bit unnecessary anyway since you can get AI to generate a GS from scratch.""

    This is a really odd sentiment to express in a forum for a particular 3d animation software. 

    Are you saying I should make gaussian splats from just a prompt? That doesn't appeal to me, if that's what I wanted to do, why would I be in the Daz Studio Discussion forum to ask about that?

    I'm not trying to use Daz Studio as an arbitrary 'precursor', I have specific scenes that I have made in Daz Studio that I am exploring redistributing as Gaussian Splats.

    Post edited by Diaspora on
  • DiasporaDiaspora Posts: 552

    WendyLuvsCatz said:

    well actually Twinmotion can do that as a presentation

    a cooked Unreal Engine format 

    but you would need the gaming licenses for content used

    "Cooked unreal engine format?"

    I'm not against getting interactive licenses but my main concern is that it would possibly be very slow. But I do appreciate the lead very much and I will look into that.

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    it plays like a game with WASD controls (and footstep sound) or flying camera

    basically like a game made with Unreal Engine but much faster and easier 

    I am trying to export that scene in my video now to demonstrate as it's CC by Atribution

  • DiasporaDiaspora Posts: 552

    I really need to look into that cause gaussian splats are really interesting but maybe I get the best bang for my buck, so to speak, by just having the models themselves in the scene.

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498
    edited August 30

    I can't walk around this one, it's too small blush

    and footsteps are not sounding (this is a later version of TM than I used before).

    anyway. it has all the menu options including connecting a VR headset which I don't have

    just compressing a zip for you to put on Google Drive

    is 5.32GB so slow to upload for me, looking like 2 hours!

    will link when done, will delete after a week because that's more than a third of my free space!

    Capture.PNG
    1920 x 1040 - 3M
    Post edited by WendyLuvsCatz on
  • DiasporaDiaspora Posts: 552
    edited August 30

    "anyway. it has all the menu options including connecting a VR headset which I don't have"

    Oh, I definitely do have a VR headset (three actually) and the fact that Twinmotion allows for VR without any hoop jumping apparently, that really interests me cause one of the main reasons I'm looking into gaussian splatting and such is because a few years ago I went pretty far down the rabbit hole of stereo 360 degree renders that could then be watched in a VR video player and it is a pretty interesting approach but the problem with it is that it's a very narrow range of the video that actually feels properly 3d, it's not nearly the same experience as when you're in a VR game and everything is being rendered in real time and the perspective is correct regardless of where you look or what angle you look from. 

    Anyway, with all that said, I really do appreciate you going to such lengths to help me have a taste of what is possible, it's super late here but I will check it out first thing tomorrow!

    Post edited by Diaspora on
  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498
    edited August 30

    you can export EXR panoramic renders too for the DAZ environment dome

    I do that a lot

    it has sculptable terrains, particles, plants with wind, volumetric clouds, all the good stuff

    Post edited by WendyLuvsCatz on
  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    https://drive.google.com/file/d/1LRw7qFobvGddPFj0sA7M9qOaETQtKuJW/view?usp=sharing

    link for Presenter example

    Name
    The Smoking Room
    Author
    The Hallwyl Museum (Hallwylska museet)
    License
    CC Atribution
    License Timestamp
    Aug 30, 2026, 6:54:40 PM

  • umblefuglyumblefugly Posts: 172
    edited August 30

    I used Postshot from Jawset during the beta and even created a splat importer for Cinema 4D, results are really good once you get shaders set up properly (which the plugin does now). Cool stuff, but in my everyday usage I don't have a need for it.
    Rendered and vertice/point view in cinema 4D, and the last image is my result with a 3DGS Ply import with Iray material conversion in DS.

     

    To do what you are requesting requires significantly more work IMO and then there are going to be speed issues exporting good enough data and images for training. It can be done though.

    Grogu_Splat.PNG
    1920 x 1058 - 1M
    Grogu_Splat_Vertices.PNG
    1920 x 1059 - 1M
    DS_Splat_Grogu.PNG
    1920 x 1080 - 3M
    Post edited by umblefugly on
  • DiasporaDiaspora Posts: 552

    @Wendy:

    Just watched your presenter and WOW! That was amazing, had to turn the speed way down so that I wouldn't fly through the room but that is really cool and I'm really eager to start cooking up my own stuff in Twinmotion. (Still gonna look at Gaussian Splatting but it's good to have options!)

    I obviously have a lot of my own homework to do but if you don't mind a few questions that I hope have relatively simple answers, I've already started dabbling, I did an FBX Export like I saw you write about elsewhere and got one of my own animations to play in Twinmotion! Yay

    1: The clothes and hair of my character were just in a T-Pose and not being animated, is there just some box I forgot to check somewhere?

    2: On the flipside, the clothes did have textures from Daz Studio but the character didn't have their skin texture come over? Is that another checkbox or is it a much more involved operation?

    3: Is there any way you know of to get D-Former deformations to make it into Twinmotion or alternately the D-formers could be baked as per frame morphs?

    Thanks again for helping to guide me to this point. 

  • DiasporaDiaspora Posts: 552

    @Wendy Seems I've at least partially if not entirely answered my second question, in Daz Studio FBX Export window, I checked 'Collect Textures' and now my characters have skins!

  • DiasporaDiaspora Posts: 552

    @Wendy Seems I've at least partially if not entirely answered my second question, in Daz Studio FBX Export window, I checked 'Collect Textures' and now my characters have textured skins!

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    merge skeletons in DAZ export for your hair and clothing to follow

    I myself avoid all the map exports by using a much older version of TM that uses original paths but that may not be available to you

    it does mean everything is locked to this computer and drive setup though but there is always the option to collect textures afterwards in Twinmotion 

    one can sort materials alphabeticaly and drag the first identical one onto the next ones to consolidate and reduce the number of materials but make sure they can be, because going back is impossible after saving a scene and means reimporting

    also annoyingly especially for buildings, none are 2 sided (Unreal does this too) so you need to check that under miscellaneous on every one

    don't use animated for static objects either only rigged animated figures 

    bring animated objects in as one mesh, static sets keep hierarchy 

    probably much more to ask but I thought of the ones that annoy me devil

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498

    I also started a thread yonks ago

    https://www.daz3d.com/forums/discussion/326986/twinmotion-renders

    lots is likely out of date though because they add heaps of stuff to each new version 

    the first version 2019 for example still had speedtrees which while less realistic than the current trees had species I liked and matched my iClone ones which was useful and Adobe Fuse people I also could replicate and rig with Mixamo so combining scenes in different software was easier 

    now it has AnimaXYZ people who still look derpy but at least you can load lots of them and they follow paths enmass, 

    the cars are nice but the version I use now (2025#) they won't do left hand drive so everything I render is not in Australia wink probably fixed in 2026 but that has too many other bugs, the Twinmotion Facebook group is full of people complaining about so I won't touch it yet.

  • DiasporaDiaspora Posts: 552

    These are all great insights but I have to ask about the biggest showstopper I've run into...

    How can I get Twinmotion to import animated morphs from FBX?

    I tried looking this up but the article from 2024 I found made it sound like there used to be a lot more import options in Twinmotion when right now there seems to just be three checkboxes and 1 numeral field.

    I also reimported my fbx back into daz studio and imported it into blender and can confirm that the fbx absolutely has the animated morph data in it, for whatever reason twinmotion 2026.1 just isn't bringing it in. Have you had this issue? The epic games launcher appears like it will allows me to painlessly choose an earlier version of Twinmotion that goes all the way back to 2023. Which version are you currently using?

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498
    edited August 31

    I did say only bone animations are supported 

    so any clothes should be rigged to follow the figures or have childbones that move it in DAZ like the old days before Dforce

    facial expressions need to use bones, some do

    also FBX is usually base resolution, some subdivision possible but HDmorphs wreck it because they use a mysterious encrypted way of applying them not compatible with FBX export, the only way to use HDmorphs is a separate obj export applied as a morph to a subdivided figure but not in Twinmotion or even Unreal AFAIK

    it could be done I believe in Blender, also there actually is a Blendshapes to bone modifier for Blender

    I tried it once, you get hundreds of tiny animated bones especially with dforce, messy looking rig wise

    I use the last 2025 version because the latest apparently is full of bugs but no difference in FBX limitations 

    Blendshapes (morphs) could still come, Unreal updates their stuff often

    or more likely Alembic ,abc a mesh cache format increasingly popular and grooms for hair

    if it does, Sagan is the DAZ plugin you need

    Post edited by WendyLuvsCatz on
  • DiasporaDiaspora Posts: 552

    "I did say only bone animations are supported "

    Oh... that's quite a substantial limitation.

    I will look into the Blendshapes to bones thing but shoot, this would actually represent quite a wall of how far I can expect to go. Aaaah, this is why I was kinda hoping this Gaussian Splats thing would work out cause in those cases, I don't need to worry about how any of this rigging stuff can make it between different software platforms.

    https://dev.epicgames.com/documentation/twinmotion/animation-overview?lang=en-US

    "Twinmotion does not support"
    "Any kind of point-level animation, including but not limited to morph targets and blend shapes."

    Yep, explicit confirmation. It would be incredibly convenient if they added in support in the next few months but I don't see a twinmotion roadmap anywhere so I don't think I should bank on that. 

    Ok, well, I guess I'll sleep on it and think on what's possible in the future.

    "

    or more likely Alembic ,abc a mesh cache format increasingly popular and grooms for hair

    if it does, Sagan is the DAZ plugin you need"

    Thank you for the lead! Really hope Epic adds this cause it sounds like this would address most of my concerns. 

  • WendyLuvsCatzWendyLuvsCatz Posts: 41,498
    edited August 31
    Post edited by WendyLuvsCatz on
  • DiasporaDiaspora Posts: 552

    "there is always Unreal Engine enlightened"

    True, I guess I just need to figure out how I turn an FBX I pull out of Daz Studio into something like what twinmotion does with presentations that I can then distribute to others, I have no idea how to do that with Unreal but I'm gonna try to figure out how.

    Sidenote, I looked into it and Twinmotion documentation doesn't make any mention of animated point clouds anywhere, from what I'm reading I should just assume that animated Gaussian Splats in Twinmotion are a dead end as well. And yeah, from what I'm reading, architecture is Twinmotion's main focus so that will guide their priorities and be the reason why Vertex Animation support has been only under consideration for almost 3 years now. 

    Yeah, I guess Unreal Engine directly is my best bet cause it would be one thing if Vertex Animation was in progress already but it might be multiple years if ever this feature gets applied.

    So at this point if I could ask one more thing, can you point me to any documentation on Blendshapes to Bones? I haven't been able to find any mention of it about Blender.

Sign In or Register to comment.