Loading...
> >

Animating character using ARKit face tracking

daniel_5b3edb82daniel_5b3edb82 Posts: 0
edited October 2019 in Daz Studio Discussion

Hi,

I'm trying to figure out how to use ARKit face tracking data in order to animate a character in DAZ.

My current approach is to make a pose injection file using that data. 

Anybody tuckled this sort of problem before?

Post edited by daniel_5b3edb82 on

Comments

  • Catherine3678abCatherine3678ab Posts: 8,010

    ? current approach? We don't inject poses, we either move bones [G3/8] or make morphs.

  • TheMysteryIsThePointTheMysteryIsThePoint Posts: 2,887

    I've thought about this. I think the ARKit data is just FACS Action Units. It should be easy to write a script to read a file and dial in the intensities. The work is in creating all the morphs to approximate each of the 52 FACS AUs that ArKit recognizes.

    Just another one of many things that I haven't gotten around to playing around with, but I can't see why it wouldn't work.

    Also, do yourself a favor and read "Stop Staring" by Jason Osipa.

  • daniel_5b3edb82daniel_5b3edb82 Posts: 0
    donalddade said:

    I've thought about this. I think the ARKit data is just FACS Action Units. It should be easy to write a script to read a file and dial in the intensities. The work is in creating all the morphs to approximate each of the 52 FACS AUs that ArKit recognizes.

    Just another one of many things that I haven't gotten around to playing around with, but I can't see why it wouldn't work.

    Also, do yourself a favor and read "Stop Staring" by Jason Osipa.

    Thanks!

  • Catherine3678abCatherine3678ab Posts: 8,010
    edited October 2019
    donalddade said:

    I've thought about this. I think the ARKit data is just FACS Action Units. It should be easy to write a script to read a file and dial in the intensities. The work is in creating all the morphs to approximate each of the 52 FACS AUs that ArKit recognizes.

    Just another one of many things that I haven't gotten around to playing around with, but I can't see why it wouldn't work.

    Also, do yourself a favor and read "Stop Staring" by Jason Osipa.

    Dialing in intensities is not the same as making injections. If you could write the script for such a project that would be wonderful.

    In the meantime there is this product which has HD Visemes which helps with the realism for the mouth of G8F.

    While the tutorial was written at the time G3 was new, bones are bones and can be used to make posing expressions possible. Now if one were to dial in an expression, export that out, reset the main figure, bring back in the expression - one could make a morph from that.

    Using the Face Bones of G3 post link

    Stop Staring is available online if anybody is interested, pdf too. Google search page.

    Another useful item for script writing:  https://www.daz3d.com/daz-studio-4-5-sdk

    Post edited by Catherine3678ab on
  • TheMysteryIsThePointTheMysteryIsThePoint Posts: 2,887
    Catherine3678ab said:
    Dialing in intensities is not the same as making injections. If you could write the script for such a project that would be wonderful.

    I didn't assert that they were. I actually don't know what "injections" refers to. I just observed that one of the iPad apps that captures ARKit data can produce a text file that is just a list of coefficients for each basis, versus time, and that the strengths of morphs are modeled the same way. So I have a certain level of confidence that ARKit data can drive facial morphs, if a morph-per-FACS-AU were available.

    Can you talk about "injections"?

    Catherine3678ab said:

    In the meantime there is this product which has HD Visemes which helps with the realism for the mouth of G8F.

    There are lots of products that could serve as starting points. But I am not sure that visimemes map 1:1 to AUs. In any case, there are reference photos of all the FACS AUs, so one could probably hand craft them quite well with whatever morph products one wanted.

    Catherine3678ab said:

    Another useful item for script writing:  https://www.daz3d.com/daz-studio-4-5-sdk

    That's the C++ SDK. It's not required for script writing.

  • Catherine3678abCatherine3678ab Posts: 8,010
    edited October 2019
    donalddade said:
    Catherine3678ab said:
    Dialing in intensities is not the same as making injections. If you could write the script for such a project that would be wonderful.

    I didn't assert that they were. I actually don't know what "injections" refers to. I just observed that one of the iPad apps that captures ARKit data can produce a text file that is just a list of coefficients for each basis, versus time, and that the strengths of morphs are modeled the same way. So I have a certain level of confidence that ARKit data can drive facial morphs, if a morph-per-FACS-AU were available.

    Can you talk about "injections"?

    Catherine3678ab said:

    In the meantime there is this product which has HD Visemes which helps with the realism for the mouth of G8F.

    There are lots of products that could serve as starting points. But I am not sure that visimemes map 1:1 to AUs. In any case, there are reference photos of all the FACS AUs, so one could probably hand craft them quite well with whatever morph products one wanted.

    Catherine3678ab said:

    Another useful item for script writing:  https://www.daz3d.com/daz-studio-4-5-sdk

    That's the C++ SDK. It's not required for script writing.

    "Can you talk about "injections"?" For the legacy characters in D/S something called "injections" were used to make it possible to add morphs to the now legacy figures. It is no longer supported per sa. With the Genesis clans [my term for all the Genesis characters] we do not add or remove any mesh data, simply re-arrange it. Morphing with these figures is a lot easier :-)

    "But I am not sure that visimemes map 1:1 to AUs" -- highly unlikely! The provided visimemes and add-ons tend to facilitate for the speech capabilities of another plugin so that the lips [more or less] go along with the spoken text. [Mimic Live]

    If the end result is for one's own work only, then yes one can use all kinds of morphs to assist in the project however for redistribution it tends to be best to make 'em all from scratch. [or purchase licenses etc. to use the other morphs, varies per vendor].

    Obviously I'm not a script writer ;-) Okay ... well that's the whatever AFAIK for making plugins that can/might/hopefully should work in D/S.

    Recently another plugin was released for Blender which enables one to morph GF8 ... I'm not a Blender expert however it appears to have a delightful feature to morph symmetrically which could cut down the work time. It's in the freebies section too.

    52 morphs, so curious here, if one had 52 morphs, what next? Like is this what you're after, just to have 52 morphs of various eye movements.

    Post edited by Catherine3678ab on
  • Catherine3678abCatherine3678ab Posts: 8,010

    Found this in the store that might be of interest: https://www.daz3d.com/real-facial-animation-for-genesis-8-maless-and-females

    "affordable mocap expressions"

     

     

Sign In or Register to comment.