Daz Speech Salad - ANIBLOCKS Talking LipSync System / Conversation generator - Enjoy :)
Norbz
Posts: 14
Took this on as a challenge for myself - to solve my own "I hate animating conversations in DAZ, and the tools available..." problem, I'm rather happy with the results of my 'solution'. Though it's a rather 'simple' approach to it all, it's very effective and fast. I hope it's going to be a great problem solver for a lot of other users too (it's free :)), and even help you pros make your animations even faster. I won't type much as it's all in the video and graphic below, but I AM eager to see what you all think of this.
Download: https://www.patreon.com/posts/41001292
Youtube:
Post edited by Norbz on
Daz 3D is part of 
Connect
DAZ Productions, Inc.
224 S 200 W, Suite #250
Salt Lake City, UT 84101
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Wow, that looks really useful. Thanks
Norbz, Wowwwwwww. Wowwwwwww and WOWWWWW!!! Amazing my friend. if you have seen any of my posts you know that I am a huge fan of animation in DAZ, particularly MMD conversions into aniblocks. Sometimes the motions are ripped and do not have any facial motions available. This will be a huge innovation for getting facial data into motions that do not have lip sync available. Thank you so much. Just Awsome!!!
Looks cool
Yoiu're really talented and good at that. Great skills and organizing.
Cheers, hope you get a lot of use out of it :).
Awesome, I'm curious to see where and how else this is going to help people like yourself. I hadn't thought of how this could help existing processes like yours, happy it's a good addition to your toolbox :).
Thanks Wendy :)
Thank you I appreciate the comment, despite my talent I am actually surprised nobody else has done something like this to date :).
I'm going to do another demo video shortly, just showcasing all the aniblocks in a row (should have done that on the original video but was too excited to release it..). Looking forward to seeing the first person actually use it and publish with it. Eager to know if it is as easy for others to use as it is for me :).
I had such an ideal like yours and posted about it in the forums 4 or so years ago for something that the DAZ 3D business could do themselves for us customers so I'm thrilled with your work! I wanted them to design a scripting language that would be in the form of a script for a stage play like a playwrite could use to write a play with. The words and animations would be driven in a 'playwrite' written script in a 'playwrite markup language' that DAZ Studio would use to run the 'play'
So I'm very happy and impressed with the work you've done. It's super nice of you. And it looks to me from your tutorial video that it could be used as the the 1st set of actions and words for such a playwrite markup language like I asked DAZ 3D to create a few years ago. Now I'm a programmer by college degree but also unfortunately I've suffered a TBI which makes my ability to sit, concentrate, design, program, and work nearly non-existant unfortunately (eg to give you an ideal of the frustration TBI can cause: I've been working 6 years to finish a 6 month DIY project to get my house in comfortable living condition) otherwise I'd promise to work on such a playright markup language (think HTML or a DUF file that completely drives an animation and speech in DAZ Studio) for the work you have done myself. Best I can do in my condition is privately hope I can work on it in a couple of months during this winter after I finally finish the last room in my house.
Do you know what I mean by a playwrite scripting language for DAZ Studio? I got the impression from the lack of response to my original posts a few years back no one understood what I was talking about or they just weren't interested. Would you be interested in such a scripting language; it would I think need to be a superset of DAZ Studio's DSON DUF scripting API?
Also, if you were interested you should know the Unity and UE4 are capable of such things already although they could be made easier for the 100% non-programmers out there. Adventure Creator is a $$$ cost product with similar goals in the Unity Asset Store for example (but doesn't look anything like a playwrite's script would look) and that's the whole ideal behind UE4 'Blueprints' and 'Visual Scripting Languages' although they are more intended for non-programmers desiring to make a game than a playwrite desiring to animate a play so in that respect the groundwork you have done is somewhat unique.
In any case thanks so much!
Thanks for the feedback! And wow at your 13K posts here, you're a like a Triple O.G. :).
Your project idea sounds both awesome and very deep. After studying all the aspects I could re how the current scripts and audio file to visemes translation works, I kind of understand why the tools are incapable of perfect translation and why all the end results I get are totally mismatched from the audio, and the text file, and the variations never really matter (I've been producing music/audio engineering for 25yrs, so well versed with the ways I can try to manipulate the translators through EQ changes, frequency focus, levels, voice types, speeds, gating vs non, mouth smacks and lip/tongue/moisture sound interruption, etc.).
I say all that to say - we all have different voices, pronounciation, speed of speech, pitch, tone, even saliva distro.. there is no script that could catch all that efficiently in my opinion. Couple that with all the things necessary to make speech look realistic outside the head, and although not impossible, I think such scripts could cap out quick or just give warbled results like the current ones do.
I can envision a timeline editor similar to animate2 where you control each important component for speech for the head/face on sliders so that you can morph quick tweens while viewing .wav data file under it (I really wish DAZ implemented a simple way to see the wav. I currently take my logicX and position it under daz with the audio lined up to the timeline for quick reference sometimes - mind you now I won't really have to). But yeah so far this randomizing of ANIblocks to match speech is great - and more importantly imo = Giving users full control of speed, individual shapes/words/etc and maximizing animate2 use efficiently is a good catch all without having to go heavy into a full scripting language.
My mind was racing as I deved this out on how/what to do for a 2.0 version and I have some cool 'add-ons' I'll probably make for this to enable more interaction between characters (fight scene generator, aniblock a jump, choose from 10 kicks while in the air, combine with some spins, voila.. ). So I guess in a way I'm doing a manual version of the concept, automating/scripting it more does sound fun.
It's ambitious, I encourage you to keep exploring ways to try a trial run through it - I have been through a few long term project cycles and currently love to create, hate to maintain/long-term-build :).
Forgive me, but if you downloaded already, I ask that you download again as I just fixed about 20 of the aniblocks (noticed lot's of mistakes like hands going through legs and random glitchy glitch frames), and I've also added 5 new ones for singing/rapping where the body bops and the character is on a mic bopping and one slow opera like, and some simple 'waving' ones, annnd a few additions like finger flutters and stuff that is probably unnoticable unless you're looking for it..) :). Going to be making a new video shortly, the new upload is available now.
I haven't unzipped yet so that's fine
I use puppeteer often myself, it's a great tool but I don't usually create aniblocks so certainly appreciate the effort
I have a few Mimic babble files I use a lot too, just me babbling random stuff saved as animated poses as often my videos do fine with just music because adding sound becomes a whole foley stage nightmare.
^ Nice. I literally just updated the .zip file on patreon a few min before posting so I don't think you have the new one unless you literally just grabbed it. If so - cools :).
Yep, when making this, I started with a simple normal talking speed and saying random paragraphs for lipsync, which from what I understand is the same thing (maybe tweaked), then once I started variating them very slow to very fast and trying to fully understand how the translator works and how to manipulate it, I learned to absolutely hate lipsync with a passion now. Especially compared to puppeteer or even right in timeline manually with powerpose/etc. Trying to hold a simple 'hi' with a long 'iiiiiii' or 'eyyyyy', anything more than half a second and nope, random mouth shapes/visemes, trying with a generated straight tone - nope, the thing has a mind of its own and I'm starting to think it's just a 'on/off' volume trigger or three inside it that randomizes face/mouth lol.
Thank you, much appreciated :-)
AWESOME!!!
I have been playing with the freebie version of a DeepFace Fake Face AI (it's a freebie on Android & iPhones called ReFace) and it's pretty impressive much of the time so I disagree although ultimately it's the script that would be feeding a series of movement clips and sound clips using DAZ compatible models in a DUF file and a DeepFace AI DAZ Studio plugin would do the alignment and syncing you talk about not being possible by computer.
Can that AI do it now as is? No. Is it feasible with increasing HW & SW sophistication to make it a goal to realistically shoot for in the future with some alterations? Most definately, although I don't know personally how deep I'll get into messing with such software ports to DAZ Studio (after all if I'm able to to do a port that sophisticated then I need to retire the federal government's classification of me as permanently disabled not that I'd care in the least.
).
Thanks again for the update.
Cool I'll have to check it out. Re being classed disabled, I can relate heavy, since you shared I'll share: Mom only has one hand, only her thumb on the left hand. Her man only has one leg, now kinda bedridden, it's like a ying-yang of love, whatever needs strong hands/jars/etc he does, whatever needs stairs/legs, she does :). Strong inspirational people, and my bio dad had an accident xx years ago and was also listed disabled, however covid took him a few months ago so he's RIP'ing no longer suffering. Health first brother, projects come and go, money comes and goes, you only got one you. This year has taken a lot from me, death comes in threes so my pops best friend also passed like a month after him, and I'd rather not speculate who the third is but I kinda see it.. anyway, sorry if TMI, you touched a soft spot for me re being disabled.
Blah, on a brighter note - today is my B-day, my big ass came into this world at 3:15pm euro time, so I'm 1.5hrs + 42yrs old now, weeeeee :).
Happy Birthday!
@Norbz Congratulations on a job well done . I tested each aniblock you created, everything worked great. very seamless . I have not rendered anything out with them yet. I tend to lack voice actors in my part of the world. so I have not had much need for duologue or speech . but this is a really good set up when i do use speech. I properly would purchased a set like this, So thank you very much for sharing them.
You can clone someone's voice and generate speech with this text-to-speech software. The speech generated sounds a lot more natural and less robotic than most other TTS programs I have found.
Happy birthday!
& on another good side, technology is helping the disabled, no matter how they got that way, manage their disabilities more, I guess, pleasantly, would be the right word.
Where do i download this
https://www.patreon.com/posts/41001292
You gotta scroll down a bit, you will see a link NORBZ-DAZ-SPEECH-SALAD.zip
thanks for fast responce i did not see it in 1st post
TY~!
Eyyy that's love thanks IVY, there are still a few I'm going to work on as I don't like some of the hands but overall I'm happy all my years in polytechnical art schools/life-drawing classes is paying off re having an eye for this and a natural understanding of the human body :). I have many sketch-books filled with like 20 eyes on a page for x pages, then 20 noses, then arms, then full body.. I used to kinda hate doing it, model in front of the class, sometimes changing poses every 15-20 seconds and we have charcoal and newsprint, then those damn sketch-books haha, years of this.. later as my studio and work developed I was greatful of course, and now this stuff comes kinda easy but I'm still rough as heck in certain areas.. Anyway, happy to read this is good enough to pay for and you see it as a good addition to your arsenol. Bang bang!
TY, I will say, disabled people are very strong and inspirational. Watching someone have a healthy life then get sick is awful re their change, seeing someone sick or disabled their whole life yet still be happy because that's all they know is bitter sweet. My mom say's "When I wake up with the pain, I know I'm alive, I'm ok. It's when I wake up with no pain that I think something is wrong, and I may not be ok".. :).
^ Glad you got it sorted, I updated the OP as well to include the link. Also new video is ready, I tried adding the SpeechSalad to my music (I'm an artist too :)), and it turned out not bad, I am now thinking of doing an official 'Singing/Rapping/etc' vocal SongSalad?! version of this to focus on individual single mouth shapes to fully match each syllable of slow singing. The Rapping you can almost get away with being totally random, just start/stop talking points need to be on point and the rest is hit/miss random, but usually hit.
I collected it, and dropped it into aniblocks, works great, thank you!!!
I'm still figuring things out though, because I'm relatively new to DAZ, so e.g.:
We all have a chance to buy some items at 70% off today for animation, so:
=> I'm trying to compare free Puppeteer vs. aniLip & aniHead vs. Mimic (I have mimic pro).
=>Aniblock converters from v4 to Gen, Gen2, and Gen3
WICKED! Norbz thank you for sharing bro xx
_ and these are for which version of Genesis ?? (G8 only or others) ._ thanx
^ Cheers guys glad you are downloading ok and testing ok, if you do anything cool and it ends up online please show/tell in here so we can see your action :).
For now they are for G8 only, looking into how to port/convert them to legacy versions too.
+ alrite then _ Thanx very much ~
I tried this. Took some time to figure out how to set everything up but not too hard to do. The results are not bad, although the cloned train-clean-100 dataset voices consistently sounds much better than a clone of a voice from my own mp3 audio file that I load into the toolbox.
It's a great solution. Unfortunately, under Genesis 8.1, only the posings work, but not the expressions. Maybe there will be a current version for it soon ????