Sagan: A DAZ Studio to Blender Alembic Exporter

mrpdean_7efbae9610

TheMysteryIsThePoint said:

If I remember correctly, the verts/polys are the same but normals still get smoothed. Or maybe I'm thinking of the pre-opensubdiv days. You've created sufficient doubt that I'll have to check again. :)

Edit: I remember now... even at subd 0 there is some smoothing, with opensubdiv.

Thank you, that is genuinely useful to know for my current workflow.

clandescent1

mrpdean_7efbae9610 said:

clandescent1 said:

@mrpdean_7efbae9610 Yes every 3d application can count vertecies, but the count of vertecies is not the same for base mesh and subd1 mesh. Idea is to use base mesh count as identifier and then if you know the subd level just look up the appropriate subdivided mesh to match the alembic mesh. I think that is better then to keep track of all the vertex counts for each subd level of each object and treat every one of them as unique objects.

To elaborate further how this can be used to streamline alembic to for example blender workflow. You have genesis 8 figure, hair and an outfit. You export that in alembic then in blender you have the figure, the hair and an outfit prepared as assets. It is better to prepare subdivided assets of the same objects separately instead of trying to dynamically build an object because porting daz assets to blender often requires adjustmenent that are not ideal to do programmatically so I think it is best to just manually prepare your assets. Then you have an intermediary addon that either keeps track of the vertex count and subd level of assets or somehow reads them, then if vertex count of the base mesh and subd level you can dynamically decide which asset to instance into the scene all from just reading the alembic file. Alembic is great for porting meshes, its the best, but even with the best case scenario for "Lord of the Plugins", it will still not be ideal. I think the most ideal solution is to just manually prepare assets and have them dynamically update through an addon, this I think is the closest we can get to a "live link".

 

Thanks @clandescent1

I think I understand it now.

Just as a quick thought, you might be able to get to the same point mathmatically.

For example, using Python you could do something like this:

EDIT:  Looks like code blocks don't display well on the Daz Forum so I'll put the code here as a quote instead:

# The vertex count of the prepared asset
# How you query the vert count will depend on the application being used
prepared_asset_vert_count = 16245

# An array of the vertex counts of the objects in the alembic cache
# How you get this list of vert counts will depend on the application being used
vert_counts_of_candidate_objects_in_alembic_cache = [24563, 64980, 36050]

# Loop through each of the objects in the alembic cache
for index, vert_count in enumerate(vert_counts_of_candidate_objects_in_alembic_cache):
    # If the object in the alembic cache is divisible by the prepared asset's vertex count
    if vert_count % prepared_asset_vert_count == 0:
      # Calculate it's subd level    
      subd = (vert_count / prepared_asset_vert_count) / 4
      print("Alembic object %s is a likely match to the prepared asset at a subd level of %i" % (index, subd))

 

Would output this, noting that the array is zero based:

Alembic object 1 is a likely match to the prepared asset at a subd level of 1

This would only work if the original geometry in Daz was modelled cleanly, entirely using quads.  It would fall apart if there were any triangles or n-gon's in the original geometry. However, I think this would also be true even if the vert counts and subd levels were stored as an attribute in the alembic cache.

Thanks for your input, however i think it is not very practical as manually counting vertecies can bring things to a halt especially if you consider multiple high-res objects.

clandescent1

TheMysteryIsThePoint said:

The idea of "design purity" is not an academic point. It comes from the idea of Separating Policy from Mechanism, as well as One Function, both which are long standing maxims since the early 70s. I don't remember the issue but that is probably at least partly the reason, if Padone objected. Software engineers have long understood that violating these concepts leads to overly complex software that is difficult to understand, reason about, update, and get to work with other software. (Read "The Art of UNIX Programming" by Eric S. Raymond) That means the good things that people ask for are slower in the coming.

I think naming objects is a core function of Alembic export and I'm going to figure out a general way to satisfy your requirements. But materials, on the other hand, is not. In fact, materials are expressly not addressed by Alembic at all, and that is the chief thing that made the complexity in Sagan v1/2 get out of hand. I'll have to address materials in a separate app, and give it a more powerful mechanism to deal with geoshells and whatnot. And of course there is the fact that I haven't figured out geoshells yet, anyway...

We'll get there... I ask for your patience.

Ok but can we still get subd and base mesh vert count option to append either in a path or as attribute? This has to do with just meshes, not materials..

mrpdean_7efbae9610

clandescent1 said:

mrpdean_7efbae9610 said:

clandescent1 said:

@mrpdean_7efbae9610 Yes every 3d application can count vertecies, but the count of vertecies is not the same for base mesh and subd1 mesh. Idea is to use base mesh count as identifier and then if you know the subd level just look up the appropriate subdivided mesh to match the alembic mesh. I think that is better then to keep track of all the vertex counts for each subd level of each object and treat every one of them as unique objects.

To elaborate further how this can be used to streamline alembic to for example blender workflow. You have genesis 8 figure, hair and an outfit. You export that in alembic then in blender you have the figure, the hair and an outfit prepared as assets. It is better to prepare subdivided assets of the same objects separately instead of trying to dynamically build an object because porting daz assets to blender often requires adjustmenent that are not ideal to do programmatically so I think it is best to just manually prepare your assets. Then you have an intermediary addon that either keeps track of the vertex count and subd level of assets or somehow reads them, then if vertex count of the base mesh and subd level you can dynamically decide which asset to instance into the scene all from just reading the alembic file. Alembic is great for porting meshes, its the best, but even with the best case scenario for "Lord of the Plugins", it will still not be ideal. I think the most ideal solution is to just manually prepare assets and have them dynamically update through an addon, this I think is the closest we can get to a "live link".

 

Thanks @clandescent1

I think I understand it now.

Just as a quick thought, you might be able to get to the same point mathmatically.

For example, using Python you could do something like this:

EDIT:  Looks like code blocks don't display well on the Daz Forum so I'll put the code here as a quote instead:

# The vertex count of the prepared asset
# How you query the vert count will depend on the application being used
prepared_asset_vert_count = 16245

# An array of the vertex counts of the objects in the alembic cache
# How you get this list of vert counts will depend on the application being used
vert_counts_of_candidate_objects_in_alembic_cache = [24563, 64980, 36050]

# Loop through each of the objects in the alembic cache
for index, vert_count in enumerate(vert_counts_of_candidate_objects_in_alembic_cache):
    # If the object in the alembic cache is divisible by the prepared asset's vertex count
    if vert_count % prepared_asset_vert_count == 0:
      # Calculate it's subd level    
      subd = (vert_count / prepared_asset_vert_count) / 4
      print("Alembic object %s is a likely match to the prepared asset at a subd level of %i" % (index, subd))

 

Would output this, noting that the array is zero based:

Alembic object 1 is a likely match to the prepared asset at a subd level of 1

This would only work if the original geometry in Daz was modelled cleanly, entirely using quads.  It would fall apart if there were any triangles or n-gon's in the original geometry. However, I think this would also be true even if the vert counts and subd levels were stored as an attribute in the alembic cache.

Thanks for your input, however i think it is not very practical as manually counting vertecies can bring things to a halt especially if you consider multiple high-res objects.

Really? That must be a Blender thing (I'm not familiar with it) as it's pretty much instantaneous in Houdini and 3ds Max regardless of the poly count.

TheMysteryIsThePoint

clandescent1 said:

TheMysteryIsThePoint said:

The idea of "design purity" is not an academic point. It comes from the idea of Separating Policy from Mechanism, as well as One Function, both which are long standing maxims since the early 70s. I don't remember the issue but that is probably at least partly the reason, if Padone objected. Software engineers have long understood that violating these concepts leads to overly complex software that is difficult to understand, reason about, update, and get to work with other software. (Read "The Art of UNIX Programming" by Eric S. Raymond) That means the good things that people ask for are slower in the coming.

I think naming objects is a core function of Alembic export and I'm going to figure out a general way to satisfy your requirements. But materials, on the other hand, is not. In fact, materials are expressly not addressed by Alembic at all, and that is the chief thing that made the complexity in Sagan v1/2 get out of hand. I'll have to address materials in a separate app, and give it a more powerful mechanism to deal with geoshells and whatnot. And of course there is the fact that I haven't figured out geoshells yet, anyway...

We'll get there... I ask for your patience.

Ok but can we still get subd and base mesh vert count option to append either in a path or as attribute? This has to do with just meshes, not materials..

Yes, I'm going to employ the general, user-defined naming scheme everywhere it is applicable.

clandescent1

@mrpdean_7efbae9610 You are right, I just assumed that blender doesnt store vertex count of its objects in blend data because its slow to count them also considering painfully slow mesh editor in blender so that was my logic. However it isnt slow to count vertecies, subd level 4 of genesis 8 which is 4192586 vertecies took less then a millisecond. But I'd still prefer a stored count due to potential issues you mentioned.

clandescent1

@TheMysteryIsThePoint If you load a .sagan config that was created in a different scene file, it will keep getting reset like before even if you click to save it. Like if i load a config that is made in a different scene and click done and then re open sagan its gone again. Also I suggest "Save As" button for config, because if you have loaded config you cant unload it and there is only save option which overwrites.

mrpdean_7efbae9610

TheMysteryIsThePoint said:

Flowed the container xform fix into Version 3.3.0.0 Please test.

Sorry. I forgot to get back to you on this. Been using v3.3 for a while now and it's working great. Thanks!

TheMysteryIsThePoint

mrpdean_7efbae9610 said:

TheMysteryIsThePoint said:

Flowed the container xform fix into Version 3.3.0.0 Please test.

Sorry. I forgot to get back to you on this. Been using v3.3 for a while now and it's working great. Thanks!

Glad to hear that. I'm particularly interested in your UE5 stuff (after watching the State of Unreal, that tech is hard to ignore), so please never hesitate to tell me how I can support you.

I'm almost done with the general naming scheme, and I'll look at the config loading/saving that clandescent1 talked about.

 

TheMysteryIsThePoint

Hi everyone,

Version 3.4 is up in the usual place.

It includes:

• A more flexible naming scheme that includes labels, names, subd levels, and more.
• Alembic does not support armatures, but now Sagan can export empties representing the transform of any joint. This is useful for parenting a crown to a head, etc... Right now, the empties are unoriented, so the Y axis is not directed along the length of the bone, but it can be parented with "keep offset" and I really haven't found aligning to be that useful as the orientation of the bone never seems to align well with the geometry anyway. Maybe I'll orient the empties if there is zero DSBS associated with doing so, but there're actually other things I'm more interested in, e.g. USD support.

As usual, test, and let me know...

 

mrpdean_7efbae9610

TheMysteryIsThePoint said:

Hi everyone,

Version 3.4 is up in the usual place.

It includes:

• A more flexible naming scheme that includes labels, names, subd levels, and more.
• Alembic does not support armatures, but now Sagan can export empties representing the transform of any joint. This is useful for parenting a crown to a head, etc... Right now, the empties are unoriented, so the Y axis is not directed along the length of the bone, but it can be parented with "keep offset" and I really haven't found aligning to be that useful as the orientation of the bone never seems to align well with the geometry anyway. Maybe I'll orient the empties if there is zero DSBS associated with doing so, but there're actually other things I'm more interested in, e.g. USD support.

As usual, test, and let me know...

Working well for me so far, although I've only been using one of the naming schemes. Haven't looked at the others yet.

Thanks for the update!

Krys Kryngle

Great add-on.  Very nice work and really appreciate your efforts on this. 

I've spent the last two days working on exporting DAZ to Unreal using the DAZtoUnreal pluigin.  The intention is to use the new ML Deformer in Unreal so we don't have to use shapekey jcm's anymore.  I ran into one small issue when exporting and was wondering if anyone had any suggestions.  The DAZtoUnreal plugin exports an fbx, animation, and alembic.  When adding a geograft to the figure, the alembic file somehow generates some loose vertices which causes a vertex count mismatch between the fbx and the alembic exports.  For example, when adding the base genesis 8.1 genital geograft, we get an extra 100 loose verts in the alembic file.  I get the same results (extra verts) when using DAZ2UE or Sagan.  When I export with the obj sequence the geometry is perfect.

I know geografts are constantly a pain, but I can't figure this one out other than to export the obj series and convert to an alembic cache (which I will probably try).  Any help would be much appreciated.  Tks.

TheMysteryIsThePoint

Krys Kryngle said:

Great add-on.  Very nice work and really appreciate your efforts on this. 

I've spent the last two days working on exporting DAZ to Unreal using the DAZtoUnreal pluigin.  The intention is to use the new ML Deformer in Unreal so we don't have to use shapekey jcm's anymore.  I ran into one small issue when exporting and was wondering if anyone had any suggestions.  The DAZtoUnreal plugin exports an fbx, animation, and alembic.  When adding a geograft to the figure, the alembic file somehow generates some loose vertices which causes a vertex count mismatch between the fbx and the alembic exports.  For example, when adding the base genesis 8.1 genital geograft, we get an extra 100 loose verts in the alembic file.  I get the same results (extra verts) when using DAZ2UE or Sagan.  When I export with the obj sequence the geometry is perfect.

I know geografts are constantly a pain, but I can't figure this one out other than to export the obj series and convert to an alembic cache (which I will probably try).  Any help would be much appreciated.  Tks.

Geografts. I just couldn't reverse engineer it with the given documentation. Nothing I've read online worked. I guess I can take another stab, with a fresh look.

But when you say that obj sequence works, are you referring to Sagan obj sequence export? That would be strange because the underlying calls to get the geometry is independent of how the geometry is actually exported, i.e. either both or neither should cause a vertex count mismatch...

And I assume you are using Sagan v3?

 

Krys Kryngle

Sagan 3.4

Yes, the obj sequence export from Sagan.  I have not fully tested this yet.  I had some issues converting the obj sequence to abc but I'll get it.  Just takes time.

If you've already looked at it, please don't feel obligated to try again if you don't think it can be fixed.  It's a wierd issue for sure.  I just checked again and the fbx, obj, and abc file all have the same vertex count and vertex order when I delete the extra vertices in Blender.  Basically I select the character model in edit mode, select by trait, and select loose geometry.  I tried it with 2 different geografts and they both had extra verts.  Some of the verts are along the seam of the geograft while a few others seem to be floating in space.  I tried deleting any loose geometry in DAZ but it didn't find anything.  The obj exports match up fine (as far as I can tell, I'll have to verify to be 100% sure). 

I did manage to export a character with Epic Eyes and got the ML Deformer working, which was the whole point.  I'll post up some examples when I can get them looking a little more presentable.  So thank you to you and all the other smart people out there making the tools for us to use. 

TheMysteryIsThePoint

That's wierd, but at least something I can investigate without your context... the Wavefront export and Alembic export should be virtually the same.

The geografts issue is nagging me... I just don't understand very well some of the concepts involved, like Shapes. What the heck is a Shape? I really thought that the Hidden Face Bit was the solution, but it wasn't set for any faces, even ones that the geograft hid. I gave up at that point in 3.3, I think. Maybe a fresh look again is just what I need. It doesn't seem like this should be hard to figure out, but I'm just missing it. I think I'll ask Thomas or David... one of them has simply got to know :)

I'm glad you got what you needed to work, though. And when you're ready, I'm (and probably a lot of other people) very interested in seeing what you did with the ML Deformer, which keeps inching me closer and closer to UE...

 

lilweep

Krys Kryngle said:

Sagan 3.4

Yes, the obj sequence export from Sagan.  I have not fully tested this yet.  I had some issues converting the obj sequence to abc but I'll get it.  Just takes time.

If you've already looked at it, please don't feel obligated to try again if you don't think it can be fixed.  It's a wierd issue for sure.  I just checked again and the fbx, obj, and abc file all have the same vertex count and vertex order when I delete the extra vertices in Blender.  Basically I select the character model in edit mode, select by trait, and select loose geometry.  I tried it with 2 different geografts and they both had extra verts.  Some of the verts are along the seam of the geograft while a few others seem to be floating in space.  I tried deleting any loose geometry in DAZ but it didn't find anything.  The obj exports match up fine (as far as I can tell, I'll have to verify to be 100% sure). 

I did manage to export a character with Epic Eyes and got the ML Deformer working, which was the whole point.  I'll post up some examples when I can get them looking a little more presentable.  So thank you to you and all the other smart people out there making the tools for us to use. 

The ML Deformer looks really good.  I willl be curious to see the results of your experiment and workflow. 

mrpdean_7efbae9610

TheMysteryIsThePoint said:

That's wierd, but at least something I can investigate without your context... the Wavefront export and Alembic export should be virtually the same.

The geografts issue is nagging me... I just don't understand very well some of the concepts involved, like Shapes. What the heck is a Shape? I really thought that the Hidden Face Bit was the solution, but it wasn't set for any faces, even ones that the geograft hid. I gave up at that point in 3.3, I think. Maybe a fresh look again is just what I need. It doesn't seem like this should be hard to figure out, but I'm just missing it. I think I'll ask Thomas or David... one of them has simply got to know :)

I'm glad you got what you needed to work, though. And when you're ready, I'm (and probably a lot of other people) very interested in seeing what you did with the ML Deformer, which keeps inching me closer and closer to UE...

 

I don't know if this helps but a "Shape" as far as Daz Studio is concerned appears to just be any type of geometry node. Figures, props, etc

In this example, there are 3 "shapes" in the scene. Michael 8.1 (figure), the male genitalia (geograft) and the special round glasses (prop):

When exported as an FBX and inspected inside Houdini, I get a name attribute for all 3 "shapes":

Like I say, I've no idea if that's helpful or not but figured it couldn't hurt to mention it :)

It looks like every polygon in the FBX belongs to one, and one one, "shape".

With geografts, you get the duplicate geometry problem. So with the male genitalia for example, as well as being exported as it's own separate "shape", a copy of it's polygons are also merged into it's parent figure (in this case the Michael 8.1 figure).

With the Alembic exporter, I don't know if it's possible to tell if a "shape" is a geograft or not, but if it is possible, you might just be able to ignore it altogether as presumably it's polygons will also be part of the parent figures geometry.

TheMysteryIsThePoint

mrpdean_7efbae9610 said:

TheMysteryIsThePoint said:

That's wierd, but at least something I can investigate without your context... the Wavefront export and Alembic export should be virtually the same.

The geografts issue is nagging me... I just don't understand very well some of the concepts involved, like Shapes. What the heck is a Shape? I really thought that the Hidden Face Bit was the solution, but it wasn't set for any faces, even ones that the geograft hid. I gave up at that point in 3.3, I think. Maybe a fresh look again is just what I need. It doesn't seem like this should be hard to figure out, but I'm just missing it. I think I'll ask Thomas or David... one of them has simply got to know :)

I'm glad you got what you needed to work, though. And when you're ready, I'm (and probably a lot of other people) very interested in seeing what you did with the ML Deformer, which keeps inching me closer and closer to UE...

 

I don't know if this helps but a "Shape" as far as Daz Studio is concerned appears to just be any type of geometry node. Figures, props, etc

In this example, there are 3 "shapes" in the scene. Michael 8.1 (figure), the male genitalia (geograft) and the special round glasses (prop):

When exported as an FBX and inspected inside Houdini, I get a name attribute for all 3 "shapes":

Like I say, I've no idea if that's helpful or not but figured it couldn't hurt to mention it :)

It looks like every polygon in the FBX belongs to one, and one one, "shape".

With geografts, you get the duplicate geometry problem. So with the male genitalia for example, as well as being exported as it's own separate "shape", a copy of it's polygons are also merged into it's parent figure (in this case the Michael 8.1 figure).

With the Alembic exporter, I don't know if it's possible to tell if a "shape" is a geograft or not, but if it is possible, you might just be able to ignore it altogether as presumably it's polygons will also be part of the parent figures geometry.

@mrpdean_7efbae9610

Thanks for that analysis... it's always helpful to see Houdini's more detailed view.

I actually said "Shape" when I should have said "Assembly"... Shapes are exactly what you investigated and figured out, but an Assembly escapes me. I can't find an explanation of the concept, the problem they solve, nor how. They seem to lie to you about subdivision levels, and the hidden face calls don't work as I expect them to. It's one of the few things left that I just don't get, and for that reason, the geografted node looks to Sagan just like any other node and all it can do is to export it just like any other. It bugs the heck out of me but I have not been able to understand it.

What you said about the geograft being part of another node is what I thought an "Assembly" might be (it's called an assembly, afterall), but it doesn't appear to be.

 

mrpdean_7efbae9610

TheMysteryIsThePoint said:

mrpdean_7efbae9610 said:

TheMysteryIsThePoint said:

That's wierd, but at least something I can investigate without your context... the Wavefront export and Alembic export should be virtually the same.

The geografts issue is nagging me... I just don't understand very well some of the concepts involved, like Shapes. What the heck is a Shape? I really thought that the Hidden Face Bit was the solution, but it wasn't set for any faces, even ones that the geograft hid. I gave up at that point in 3.3, I think. Maybe a fresh look again is just what I need. It doesn't seem like this should be hard to figure out, but I'm just missing it. I think I'll ask Thomas or David... one of them has simply got to know :)

I'm glad you got what you needed to work, though. And when you're ready, I'm (and probably a lot of other people) very interested in seeing what you did with the ML Deformer, which keeps inching me closer and closer to UE...

 

I don't know if this helps but a "Shape" as far as Daz Studio is concerned appears to just be any type of geometry node. Figures, props, etc

In this example, there are 3 "shapes" in the scene. Michael 8.1 (figure), the male genitalia (geograft) and the special round glasses (prop):

When exported as an FBX and inspected inside Houdini, I get a name attribute for all 3 "shapes":

Like I say, I've no idea if that's helpful or not but figured it couldn't hurt to mention it :)

It looks like every polygon in the FBX belongs to one, and one one, "shape".

With geografts, you get the duplicate geometry problem. So with the male genitalia for example, as well as being exported as it's own separate "shape", a copy of it's polygons are also merged into it's parent figure (in this case the Michael 8.1 figure).

With the Alembic exporter, I don't know if it's possible to tell if a "shape" is a geograft or not, but if it is possible, you might just be able to ignore it altogether as presumably it's polygons will also be part of the parent figures geometry.

@mrpdean_7efbae9610

Thanks for that analysis... it's always helpful to see Houdini's more detailed view.

I actually said "Shape" when I should have said "Assembly"... Shapes are exactly what you investigated and figured out, but an Assembly escapes me. I can't find an explanation of the concept, the problem they solve, nor how. They seem to lie to you about subdivision levels, and the hidden face calls don't work as I expect them to. It's one of the few things left that I just don't get, and for that reason, the geografted node looks to Sagan just like any other node and all it can do is to export it just like any other. It bugs the heck out of me but I have not been able to understand it.

What you said about the geograft being part of another node is what I thought an "Assembly" might be (it's called an assembly, afterall), but it doesn't appear to be.

Out of curiousity, if you could tell that the node was a geograft, I wonder what would happen if you just excluded the entire node from export?  Would the parent figure still contained the merged polygons?

mrpdean_7efbae9610

I'm sure you've seen this already, but just in case:  https://www.daz3d.com/forums/discussion/109271/is-a-child-a-geograft

Possibly these aren't available in the SDK, I've no idea.

TheMysteryIsThePoint

Those functions are all in the DzFigure class and described in dzfigure.h, but this is not enough to know how the functions are supposed to be used.

If I recall correctly, the geograft shows up as just another node. If I export it, there is duplicate geometry. If I don't, there is geometry missing.