Loading...
> >

mcjHIDKit - live puppetry in daz studio using a microphone and a gamepad (WIN-PC)

mCasualmCasual Posts: 4,604
edited February 2013 in Freebies

────────────────────────────────────────────

download and manual -----> https://sites.google.com/site/mcasualsdazscripts2/mcjhidkit

────────────────────────────────────────────
History
────────────────────────────────────────────
Feb 7 2013 10:00 pm Handles higher data rates like 30 Hz
Feb 6 2013 3:00 am initial release, DS1, DS2 versions non-operational
Feb 6 2013 3:20 am version for DS1 DS2 fixed
Feb 6 2013 4pm de-selection of target nodes while the script is running - practical for Live Shows and screen-captures
────────────────────────────────────────────

you will be able to control a character using a gamepad ( ... a gamepad with an HID driver )

and any sound present at the microphone ( or soundcard in general ) will be usable to drive a morph channel like "mouthOpen"

all this can be done live and can be recorded on Daz studio's timeline

in the emotiguy example shown below, i recorded movements generated using the microphone,
in 3 passes, one for the mouth-open morph, one for the wink morph and one for the smile morph

but i could've done it in 1 pass or live, using the mic + 2 joystick buttons

the system should work with most modern (HID) gamepads like my logitech usb-rumblepad

if will work only on a windows PC since it relies on an exe program called the "server". On Studio's side the 'client' is a daz script which should be usable on DS 1, 2, 3, 4, 4.5

Post edited by mCasual on

Comments

  • TugpsxTugpsx Posts: 732
    edited December 1969

    This is another very interesting project. I like where you are going in the development of interactive animation for Daz Studio.
    I wonder if the use of an Xbox Kinect would serve as an input device. Whereas the information generated by the onscreen figure could be segmented and used to drive the character movement.

    I know it would take a decent resolution for facial movement, but for starters, body movement such as raising an arm, bending hands, lift a shoulder or even fingers may be within the relm of the project as it gets refined.

    Again thanks for sharing your projects with us and we look forward to the results of your quest to enhance Daz Studio and more.

  • mCasualmCasual Posts: 4,604
    edited October 2012

    Tugpsx said:
    This is another very interesting project. I like where you are going in the development of interactive animation for Daz Studio.
    I wonder if the use of an Xbox Kinect would serve as an input device. Whereas the information generated by the onscreen figure could be segmented and used to drive the character movement.

    I know it would take a decent resolution for facial movement, but for starters, body movement such as raising an arm, bending hands, lift a shoulder or even fingers may be within the relm of the project as it gets refined.

    Again thanks for sharing your projects with us and we look forward to the results of your quest to enhance Daz Studio and more.

    eventually i may try something with the kinect, there's OpenKinect OpenNI and brekel which i gather have all that's needed to get skeletal poses from a kinect connected to a PC . Last year James Walsh http://vimeo.com/19570277 had a working kinect plugin for daz studio - there may be posts about it elsewhere in the forum. Longer ago he was able to implement GLSL shaders like chrome, in Studio's 3dviewport

    a few weeks ago i started making tests with OpenCV (computer vision) , i bought an HD webcam, and made a baseball cap with 2 colored markers on it, and wrote a program that reports the position of the markers and forwards them to daz studio, 10 or 30 times per second. i'll eventually get back to that project. With more complex processing i could also detect and follow eyes. with 3 markers i could get head position and orientation.

    there's a system that exists with 3 IR beacons on a baseball cap + a webcam, and can be used for head tracking, it's compatible with some PC Games.

    with my WiiKit it wouldnt take much work to so things like this http://www.youtube.com/watch?v=Jd3-eiid-Uw

    Post edited by mCasual on
  • TugpsxTugpsx Posts: 732
    edited December 1969

    Awesome. Keep up the great work we all look forward to your projects.
    .

  • mCasualmCasual Posts: 4,604
    edited October 2012

    the buttons will be able, to work as momentary switches,

    meaning, you press a button, the animated object jumps to its full-on position

    or

    they will function as dampened switches,

    meaning, you press the button, the animated object ramps up until it reaches the full-on position,

    stays there as long as you press the button,

    and when you release the button, it ramps down until it reaches its full-off position

    ---

    the finite state machine ( http://en.wikipedia.org/wiki/Finite-state_machine )
    for the dampened buttons

    motorized_button.png
    800 x 800 - 98K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited December 1969

    here you can see an animation done live using 2 gamepad buttons
    they are "dampened", with an "attack" and "release" time of half a second

    dampen.gif
    480 x 640 - 384K
  • TugpsxTugpsx Posts: 732
    edited December 1969

    Wow you work fast. Great idea.

  • mCasualmCasual Posts: 4,604
    edited October 2012

    adding push-on-push-off outputs for each button

    i guess there must be other types i need to add, like dampened push-on-push-off

    trying to think where the complete list of button types is posted , i would think, on a site for PLCs

    popo.jpg
    1000 x 1000 - 61K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited October 2012

    in red you can see the undampened output of Button #1 of the game pad was used to animate the YTranslate parameter of a Null node

    and in red you can see the dampened output of the same button #1 was used to animate the YTranslate of a another null node

    then mcjTransport was used to make the paths of those 2 nulls visible ( amd setInterpolation script was user to linearize the speed of the nulls )


    in the second picture, you can see in green the unfiltered output for the gamepad button #1

    and in red the push-on-push-off output for the gamepad button #1

    keen.jpg
    1280 x 160 - 34K
    damppp.jpg
    1280 x 240 - 86K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited December 1969

    the dampened push-on-push-off buttons

    press and release, the output of the switch will ramp from 0 to 1
    then stays at the 1 level
    press again the output of the switch will ramp from 1 to 0
    and stay at the 0 level

    etc ...

    dampenedpopo.png
    1280 x 800 - 230K
  • yungexecyungexec Posts: 6
    edited December 1969

    awesome purchased key rig and daz had the nerve 2 tell me 2 downgrade 2 daz 3.x in order 2 use this....make this work and i'll be the first to pay $20 bucks 4 it...u already a legend...me thinks this will make u a god!!

  • yungexecyungexec Posts: 6
    edited December 1969

    honestly i think the best use of the game-pad is controlling the camera...just play mw3...

  • mCasualmCasual Posts: 4,604
    edited December 2012

    yungexec said:
    awesome purchased key rig and daz had the nerve 2 tell me 2 downgrade 2 daz 3.x in order 2 use this....make this work and i'll be the first to pay $20 bucks 4 it...u already a legend...me thinks this will make u a god!!

    actually i'm almost certain this will work on all versions of Daz Studio

    it's the same system as the wii-kit,
    ( https://sites.google.com/site/mcasualsdazscripts/mcjwiikit-for-daz-studio-3-pc-version-zero-beta-not-stable-yet )

    on Daz Studio's side of things, , there's a script, which is compatible with Daz Studio 1, 2, 3, 4 ( not sure if i tested it on DS4.0 or 4.5)
    it launches a program (for windows only) that constantly watches what the gamepad is doing

    good thing you reminded me i was forgetting this project !

    so i'll finish the dress for Aiko3 and get back to it before xmas

    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited January 2013

    been working on it a bit today

    i linked the buttons of the gamepad (logitech dual action rumble) to the YTranslate of the button objects

    the 2 analog joysticks ( 4 potentiometers) were linked to the XTranslate and ZTranslate coordinates of 2 button objects

    here's the content of the links table

    
link joy.btn.1.val button_1 Y Translate -50 0
link joy.btn.10.val button_10 Y Translate -50 0
link joy.btn.11.val button_11 Y Translate -50 0
link joy.btn.12.val button_12 Y Translate -50 0
link joy.btn.2.val button_2 Y Translate -50 0
link joy.btn.3.val button_3 Y Translate -50 0
link joy.btn.4.val button_4 Y Translate -50 0
link joy.btn.5.val button_5 Y Translate -50 0
link joy.btn.6.val button_6 Y Translate -50 0
link joy.btn.7.val button_7 Y Translate -50 0
link joy.btn.8.val button_8 Y Translate -50 0
link joy.btn.9.val button_9 Y Translate -50 0
link joy.or.1 button_13 Z Translate 1 -178
link joy.or.2 button_13 X Translate 1 306
link joy.or.3 button_14 Z Translate 1 -178
link joy.or.4 button_14 X Translate 1 128
    blinka.gif
    400 x 400 - 136K
    nubuttonz.gif
    400 x 400 - 308K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited February 2013

    ~ news update ~~ news update ~~ news update ~~ news update ~~ news update ~

    just a quick news bulletin to say, i worked on the gamepad server side of things

    and now have a steady stream of data coming out of it

    the release web page will be here :

    https://sites.google.com/site/mcasualsdazscripts2/mcjhidkit

    i'll possibly tie all loose ends tomorrow and release it !

    the image below shows the 4 ouputs for button 1 of my gamepad

    raw value, dampened, push-on-oush-off-dampened and push-on-push-off

    the status of the audio and the gamepad were sent from the server to the client in daz studio 10 times per second.

    the final version will probably include the ability to increase the sampling rate to 30 times per second

    mcjClient.dsa_.png
    273 x 273 - 14K
    mcjClientDS1DS2.ds_.png
    273 x 273 - 14K
    workin.jpg
    737 x 823 - 117K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited February 2013

    the latest version of the HIDServer will display the states

    you will also have the opportunity to change the report rate, the ramp up and ramp down times for the "dampened" buttons

    in the third image you can see the report rate was 10 Hertz, the ramp up and ramp down times were 5000 ms

    dpopop.gif
    320 x 400 - 191K
    hid2.jpg
    347 x 205 - 14K
    id.jpg
    320 x 400 - 34K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited December 1969

    ────────────────────────────────────────────────────────────────
    ~ Release News ~ Release News ~ Release News ~ Release News ~ Release News ~ Release News ~
    ────────────────────────────────────────────────────────────────

    https://sites.google.com/site/mcasualsdazscripts2/mcjhidkit

    a first release

    if you use it at the default data rate of 10 per seconds, it seems to work quite well

    ────────────────────────────────────────────────────────────────

  • mCasualmCasual Posts: 4,604
    edited December 1969

    here's a demo made today

    https://www.facebook.com/video/embed?video_id=10151703443064746"

    i'll probably add a button to mcjClient, to make it de-select the nodes

    this way if you screen-capture a session, you wont have the wireframe boxes (bounding boxes) around the selected objects

  • mCasualmCasual Posts: 4,604
    edited February 2013

    - update - update - update - update - update - update - update - update -

    de-selection of target nodes while the script is running

    this should be practical for live shows and screen-captures

    since it removes the selection boxes around the target objects

    dsel.png
    1280 x 720 - 169K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited February 2013

    good news - i figured why the data rate was limited to 10 Hz


    communication between the server and the client is done over stdin / stdout

    apparently buffering and polling happens around 10 times per second ( in Daz or in Windows )

    if the report rate from the server is over 10 Hz, sometimes

    the client (script) will receive more than 1 status report at a time.

    in the previous version of the script, those reports were thrown out for fear of being corrupt

    the next version of the script will instead unpack them and feed them to the part of the script that processes them

    --

    just now i tested 30 Hz data rate

    at 60 Hz i still get some "misses", say, 1 miss per second


    actually the image below isnt a valid test, by mistake i re-launched the server after changing the data rate to 30 !

    but it gives you a glimpse of an upcoming script that will use the HIDServer to pilot a camera or a car

    clip0007noco.gif
    616 x 351 - 595K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited February 2013

    - update - - update - - update - - update - - update - - update - - update - - update - - update - - update - - update -

    new version handles higher data rates like 30 Hz

    also fixed Load/Save of links lists issues

    bass.gif
    600 x 337 - 63K
    Post edited by mCasual on
  • TugpsxTugpsx Posts: 732
    edited December 1969

    Great job will give this a try. Thnks for sharing your work.

  • mCasualmCasual Posts: 4,604
    edited December 1969

    Tugpsx said:
    Great job will give this a try. Thnks for sharing your work.

    thanks

    eventually i think i'll make an OSC ( http://opensoundcontrol.org/ ) plugin
    which will let people interface a lot of things including the kinect and even musical keyboards

  • TugpsxTugpsx Posts: 732
    edited February 2013

    Actually you are almost there. Check the MS Kinect dev kit

    Post edited by Tugpsx on
  • mCasualmCasual Posts: 4,604
    edited March 2022

    oh my 2013 we're talkin 2013 :)

    my gamepad was so clean and shiny

    here's all the math needed to write a renderer

     

    aa.jpg
    580 x 800 - 102K
    Post edited by mCasual on
  • mCasualmCasual Posts: 4,604
    edited March 2022

    mcjHIDServer.exe will not work unless you have the proper version of

    msvcr100.dll

    installed on your PC

    so you may have to find a copy of msvcr100.dll and place it in the same folder as mcjHIDServer

    mine is about 755KB in size and seems to be version 10.0... from 2015

    i may add it in the mcjHIDKit zip/rar package in the near future, now you may say "how near" and that's not a saying that's a question so it's different.

     

    warnin.jpg
    1000 x 401 - 102K
    Post edited by mCasual on
  • TugpsxTugpsx Posts: 732

    Good find. With that in place we can then map keyboard to gamepad and more.

Sign In or Register to comment.