Showcase: Bringing Daz figures To Life in UE5

dave_0aa47f5a80dave_0aa47f5a80 Posts: 7
edited September 17 in Unreal Discussion

Starting with an FBX character exported from Daz3D, we produce a brand-new, custom set of Unreal compliant JCMs by effectively sampling mesh deformations over a wide range of body positions.
We then rebuild the skeleton from scratch to produce a character optimised for animation in Unreal Engine. This is a completely automated process performed by a Python script run in Maya in conjunction with a custom Daz script. These two scripts represent many months of work!

Custom JCMs applied to the shoulders of a converted Daz3D Character

 

Demo Movie

 

Technical Overview Movie

 

Despite some minor differences in the skeleton structure, the process works seamlessly with all Daz Genesis 8 and 8.1 figures, both male and female, with or without anatomical elements. The system also works perfectly with the higher resolution FBX characters exported by the DazToUnreal plugin.

Once the converted character has been manually imported into the Unreal Editor, applying JCMs using posedrivers, setting up physics, converting standard UE control rigs and other animation tasks all become relatively straightforward. Compelling, realistic mesh deformations across a wide range of body positions that are not possible with the standard Daz JCMs become achievable.

 

Detail

There are two parts to the conversion process. The first is a Maya Python script that takes the FBX file exported from Daz3D and rebuilds the skeleton for optimal UE compatibility:

 ~ Converts the joint rotation order to XYZ
 ~ Reorients all the joints, pointing X down the joint and making Z the primary swing angle
 ~ Reorients the root node
 ~ Converts all inline twist bones to leaf twist bones
 ~ Adds the standard Unreal IK joints (optional)
 ~ Adds muscle joints for 'jiggle' effects (attach physics bodies to drive morphs, optional)
 ~ Adds extension joints to finger and toe tips for VR grabbing and precise IK solutions (optional)

The second part of the system is a Daz script that, when used in conjunction with the Maya script, produces a definitive set of JCMs by effectively sampling mesh deformations for a wide range of body positions. The resulting JCMs are optimised for the UE's linear skinning method - Daz uses the dual quaternion skinning method, so this is necessary for good results.

The character mesh is also combined with the clothing meshes during the process, so all exported JCMs (and other morphs) are applied to all meshes at once. This virtually eliminates poke-through issues and makes the application of JCMs at runtime a far more efficient process, especially when used in conjunction with Unreal's posedriver technology.

We also produced a cut-down version of the skin materials optimised to minimise draw calls, enhanced them with micro-normal and micro-roughness maps taken from the Metahuman project and added configurable 'Selective AO' masks for problem areas like nostrils and ears.

An IK retargetting rig, standard backwards / forwards control rig, ground following rig, eye tracking rig and semi-automated method for characterising the figure in MotionBuilder have also been produced.

 

Background

The system was originally developed so that we could include Daz characters in the VR meeting application we are developing, as we wanted to make the extensive Daz3D catalogue more accessible for avatar creation. It has since turned into a project in its own right. We are considering offering either a Daz figure conversion service or selling the system. There would be some work to get the system to a marketable state, so we thought we'd showcase it to try and gauge the level of interest. We'd very much welcome your comments and feedback!

Post edited by dave_0aa47f5a80 on

Comments

  • VividImaginatorVividImaginator Posts: 18
    edited September 17

    Interesting, one of the most tedious things about using Daz characters on Unreal Engine is dealing with the skeletons.

     

    After long hours of trial-and-error, I was able to find workarounds for the main issues with the skeletons but it is a tedious, time consuming, process.

    Having a conversion tool would be awesome; Maybe Daz 3D could be interested; they could integrate the conversion to their Daz to Unreal bridge; that would be awesome.

     

    I would be reluctant to use a conversion service thought; I use custom models, I desing and morph-in my characters' look, they are part of my copyrighted works.

    I don't use "out of the box" Daz characters and I'm not comfortable with uploading my private designs anywhere. Not until I release a complete UE5 game.

     

    I don't know what your upload policy would be but ensuring that custom, private, designs will not be leaked or "shared" would be a requirement for me... and for many.

    Just something to consider.

    Post edited by VividImaginator on
  • Make it like some Daz studio plugins or independent utilities being sold in Daz store, and it's better to be independent of third party software or bridges. Because users may prefer different 3D software, Maya has the official support from Epic with Livelink, Blender is free and popular with very useful plugins, also users can use different bridges like DazToUnreal or Diffeomorphic or some experimental pipelines. This is awesome and would be the best seller if it's user friendly with documentation and video demonstrations.

  • dave_0aa47f5a80dave_0aa47f5a80 Posts: 7
    edited September 18

    Thank you for your feedback!

    @VividImaginator:

    I hear you on the trial and error comments - it was after spending a good deal of time trying to animate my fingers in VR using Valve Index Knuckles that I decided to look at reorienting the joints. I wasn't planning to spend so much time on the project back then, but it's done now.

    I completely understand your concerns about uploading your files for conversion. Whilst we are considering all options to monetise the system, I'm also not sure I'd really want to start offering that as a service either, as now that we have Daz based avatars we can use I'm very keen to get back to developing the open source, decentralised VR meeting app we have had on hold for a while now.

     

    @ VividImaginator, @acatmaster

    It would be great to get the system integrated into the DTU bridge, but since most of the heavy lifting is done by the Maya Python script the system is only really available as a tool for Maya users, at least in it's current form. In theory it would be possible to port it to a C++ plugin, but I taking on such a big project at the moment is not really a priority.

    I'd be happy to consult on such a project is someone else wanted to do so though.

     

    Bigger Picture

    As hardware gets faster and software optimisations like Nanite and Lumens continue to evolve, I see huge potential for Daz3D characters being used as avatars in the near future. I believe the early success of platforms like Second Life are due in no small part to their focus on fostering an ecosystem of user generated content - unlike many of the limited, in-app avatar creation options offered by some of the currently proposed next generation metaverse solutions.
    There is already a huge range of Daz characters / apparel available and Daz Studio offers many creative possibilities beyond that. I think it would be great if both regular users and professional avatar creators could use Daz Studio to create avatars and seamlessly upload them to metaverse platforms - this is what we're aiming for with our app and this is the main reason I have developed this conversion system.

    Post edited by dave_0aa47f5a80 on
  • EllessarrEllessarr Posts: 1,327

    a little later to the party but still here, it's really awesome specially if it could be used as a sort of "addon" for the daztounreal plugin and could work without the maya extra step, if your team would talk with david and daz team to see how to make then work together it will be really perfect, we really need the jcms improviment and fix for the rotation/orientation stuffs for daz characters it would be awesome.

  • @Ellessarr 

    Thank you for the feedback :-)

    I one hundred percent agree with what you say - all the features I've built into the system were born out of necessity for achieving compelling results when animating Daz characters in Unreal.

    I'd be very open to talking to David (am guessing Vodhanel?) and the Daz team about a possible collaboration / ways to implement my process outside of Maya.

  • EllessarrEllessarr Posts: 1,327
    edited September 19

    dave_0aa47f5a80 said:

    @Ellessarr 

    Thank you for the feedback :-)

    I one hundred percent agree with what you say - all the features I've built into the system were born out of necessity for achieving compelling results when animating Daz characters in Unreal.

    I'd be very open to talking to David (am guessing Vodhanel?) and the Daz team about a possible collaboration / ways to implement my process outside of Maya.

    awesome!!!!!, that is something i would really love to see, a colab would make both have a awesome work and we finally get a perfect daz to unreal transition for the characters, with all jcsm, bones corrections, also maybe awesome physics and mats, making the characters goes to unreal in the most perfect optmized way. 

     

    specially without the need of a third part program like maya or blender as it was supposed to be, you just export the character using the plugin and it go direct to unreal with everything working.

    Post edited by Ellessarr on
  • Hey Dave

    As a mocap specialist and studio that uses Daz characters a lot, this is very, very interesting to us. How can we get in touch?

    thanks

  • @Bryan Steagall - I have sent you a PM via Daz3D forum's messaging.

  • EllessarrEllessarr Posts: 1,327

    being curious, the talk with david or daniel are having any adavances??

  • Thank you to everyone who has provided feedback. Based on the encouraging comments I've received here, on youtube and in private messages / emails, I have decided to start looking at ways in which I can make this available for Daz users.

    There are quite a few considerations to work through and discussions to be had, but I intend to get this out there as soon as possible. I'll post updates here as things progress.

     

Sign In or Register to comment.