Help understanding Metadata

WildlyfeWildlyfe Posts: 36

I need help understanding Metadata, how it works and why it isn't working for me. In order for it to work, what needs to happen, where do the files need to be. For example, by default, I know the metadata files reside in the Support folder. Where does the file need to be? Must it reside only in the Studio\My Library\Runtime\Support folder? Must the files associated with it reside in the same Runtime folder? Will it only work if the associated product files are not moved from the default libraries? I usually install my files to a temporary directory adjust the files to my own structure before placing them in the final directory (I prefer to put all the files for an item in one folder for example - lights, camera, mats, poses... in the same folder as the base item). I also have multiple runtimes to work with.

Comments

  • cridgitcridgit Posts: 823
    edited December 1969

    Wildlyfe said:
    I need help understanding Metadata, how it works and why it isn't working for me. In order for it to work, what needs to happen, where do the files need to be. For example, by default, I know the metadata files reside in the Support folder. Where does the file need to be? Must it reside only in the Studio\My Library\Runtime\Support folder? Must the files associated with it reside in the same Runtime folder? Will it only work if the associated product files are not moved from the default libraries? I usually install my files to a temporary directory adjust the files to my own structure before placing them in the final directory (I prefer to put all the files for an item in one folder for example - lights, camera, mats, poses... in the same folder as the base item). I also have multiple runtimes to work with.


    Hi Wildlyfe

    Sure is an interesting topic. I recommend you follow the link in my sig to the metadata toolkit. There are three detailed tutorials with step by step guides explaining everything about metadata and showing you how to make it, use it and share it.

    You might also want to check out this thread on using Smart Content http://www.daz3d.com/forums/discussion/8724/ and this one ont he discussion around metadata guidelines: http://www.daz3d.com/forums/discussion/7542/

  • Richard HaseltineRichard Haseltine Posts: 19,413
    edited December 1969

    When you install content with metadata a script is placed in the Run Once folder (in the DAZ 3d\Studio 4\ folder in your application data folder) to place the metadata file in the main \Runtime\Support folder - so it doesn't matter whether the file is installed to the main folder or not. You are then prompted to add the item to the CMS then next time you start DS or process the queue, at which point the data is processed and added to the CMS database and the file in the Support folder is not needed again 9unless you reset the database). Moving files around will break the metadat unless you do it after the content has been processed and use the Content Library pane to do the moving.

  • cridgitcridgit Posts: 823
    edited December 1969

    When you install content with metadata a script is placed in the Run Once folder (in the DAZ 3d\Studio 4\ folder in your application data folder) to place the metadata file in the main \Runtime\Support folder - so it doesn't matter whether the file is installed to the main folder or not. You are then prompted to add the item to the CMS then next time you start DS or process the queue, at which point the data is processed and added to the CMS database and the file in the Support folder is not needed again 9unless you reset the database). Moving files around will break the metadat unless you do it after the content has been processed and use the Content Library pane to do the moving.


    I'll add one comment to Sir Richard's. For homebrew metadata or manual installation (i.e. without the installer) you can copy the three metadata files to the Runtime/Support folder. DAZ Studio won't know about when it starts up (unless you place a copy of the DSA file in the RunOnce folder), but if you choose Content Library/Content DB Maintenance/Reimport Metadata it will be in the list and you can import it manually.

  • GeddGedd Posts: 2,444
    edited November 2012

    So if I understand this correctly...

    If someone edits the metadata file in the support folder, then 'reimports' the metadata.. it will adjust the metadata in the CMS and any future reimporting of the metadata will follow the imported version, the original metadata having been wiped from the CMS, untill ofc the support file gets overwritten by an update to the item from DAZ. Furthermore, if one were to back up the edited metadata file, merge it with any revised one from an update from DAZ, then place the updated file in the support folder, this updated version would also maintain the structure the user wished in the CMS rather then constantly blowing out the data with some default data that didn't follow the end users personal preferences. Is this essentially correct?

    All of this is ofc with the provision that DAZ does not support modifying the metadata files and anyone doing so would be doing it at their own risk.

    (A train and a car both have their purposes, but a train stops at places I don't care about, and doesn't go everywhere I do.)

    Post edited by Gedd on
  • Richard HaseltineRichard Haseltine Posts: 19,413
    edited December 1969

    That should be the case, though I've had very limited success in getting it to work.

  • GeddGedd Posts: 2,444
    edited December 1969

    Hmm, well I had hopes of being able to possibly create an interface around modifying/customizing the files.. but if manually editing isn't functional, automating it would be a non starter.

  • WildlyfeWildlyfe Posts: 36
    edited December 1969

    Thanks cridgit for the tutorials and forum links. Lost of great info there. I guess from what I am reading, that those of us who have restructured our runtimes are a bit out of luck at this time for using supplied metadata? Right now, I can re-import metadata, but while I can see it in the list for import, the info doesn't seem to actually install. Nothing shows up. I assume this is because it can't find the files where it expects them to be for association. I had hoped I could install them anyway and edit the paths to link to where I located the files, but I guess not. I really like the idea of using metadata, but not sure I want to re-install the huge amount of content I have in order to use it.

  • cridgitcridgit Posts: 823
    edited December 1969

    Wildlyfe said:
    Thanks cridgit for the tutorials and forum links. Lost of great info there. I guess from what I am reading, that those of us who have restructured our runtimes are a bit out of luck at this time for using supplied metadata? Right now, I can re-import metadata, but while I can see it in the list for import, the info doesn't seem to actually install. Nothing shows up. I assume this is because it can't find the files where it expects them to be for association. I had hoped I could install them anyway and edit the paths to link to where I located the files, but I guess not. I really like the idea of using metadata, but not sure I want to re-install the huge amount of content I have in order to use it.


    You're welcome.

    If you look inside one of the metadata DSX files, you'll see the assets have relative path references (relative to My Library) e.g. runtime/poses/pose1.pz2). That means the content directory doesn't matter. But if you've renamed files or moved them within the My Library folder, those relative paths won't work.

    As far as I know you can edit base paths in Studio, but I haven't tried doing this to get metadata to work.

    There is a batch install DOS script you can use if you're on Windows - let me know if you need the link. I predict it'll only be a matter of time before you take the plunge and reinstall ;-) A number of people I know of have already done this. Its just easier that way.

  • GeddGedd Posts: 2,444
    edited December 1969

    Actually, I rename all of my items also so that they have natural names rather then bg_btn_smzt or whatever unintelligible thing they get named at times. You know, the name one sees in the interface. I'm guessing this also breaks metadata?

  • cridgitcridgit Posts: 823
    edited December 1969

    Gedd said:
    Actually, I rename all of my items also so that they have natural names rather then bg_btn_smzt or whatever unintelligible thing they get named at times. You know, the name one sees in the interface. I'm guessing this also breaks metadata?


    It should break it if you do that BEFORE installing the metadata, but if you do it AFTER in Studio (right-click rename) I would expect Studio is smart enough to know to update the metadata references as well. Haven't tried though, so you might want to give it a shot.

  • GeddGedd Posts: 2,444
    edited November 2012

    Ok, there is no way that would work. Renaming 1000's (10's of 1000's even) of files one at a time by right clicking etc... is ... not workable to put it mildly. I batch rename the files many at a time, and we can't do that within the DS interface. This points out exactly what the problem is when a program requires everything to go through it's interface to maintain integrity. And before anyone says it, media players etc.. all allow external applications to modify metadata and they update their internal database to match. It appears to me what is really needed is a way to modify the support files externally and have the DS program reimport the metadata to sync it internally, and to be able to save out changes made internally to the support files. This is the way better modern programs work.

    Post edited by Gedd on
  • cridgitcridgit Posts: 823
    edited December 1969

    Gedd said:
    Ok, there is no way that would work. Renaming 1000's (10's of 1000's even) of files one at a time by right clicking etc... is ... not workable to put it mildly. I batch rename the files many at a time, and we can't do that within the DS interface. This points out exactly what the problem is when a program requires everything to go through it's interface to maintain integrity. And before anyone says it, media players etc.. all allow external applications to modify metadata and they update their internal database to match. It appears to me what is really needed is a way to modify the support files externally and have the DS program reimport the metadata to sync it internally, and to be able to save out changes made internally to the support files. This is the way better modern programs work.


    Nice concept in theory :-)

    Practically though, how do you propose any program (DAZ Studio, Poser, Maya, iTunes, Media Player, Powerpoint, Picasa etc.) identifies an asset (a file) if it has been renamed and/or moved BEFORE metadata has been added? I would very much like to here your proposed solution, because with my 30 years of computing experience (yeah since 1983 baby) all I know is the filename/path is the ONLY way to uniquely identify an asset WITHOUT metadata (some kind of supplementary signature added to the file).

    The media players etc. you are referring to have the following characteristics:
    1) they are bundled with the operating system which enables the operating system to provide media-editing features to provide for a seamless experience (I'm referring to both Windows and Mac).
    2) they identify the file using either the filename or its metadata in the first place, without which they simply don't know what song/album it is.

    Personally, I simply don't understand why you feel compelled to rename the files and move them around when you have higher-level tools at your disposal for organizing things. However, that's your right and choice, but please provide a feasible proposal and perhaps it could be done. I really can't think of how this is possible without using either filename/path OR metadata.

  • GeddGedd Posts: 2,444
    edited November 2012

    First of all, neither Mediamonkey, XBMC, nor Media Companion come with any OS. They handle metadata very well as does many programs. The ones that come bundled with the OS, MS Media Player and ITunes *do not* handle metadata properly as they *only* keep the metadata within an internal indexing system. This means the metadata doesn't get properly shared to other applications that look to the (music etc...) They are an example of how not to handle metadata. Metadata can be handled by placing it in the header of the file, if the file structure is defined in such a way as to accommodate that (this is the preferred way when available) or in a 'sidecare' file. This sidecar file can be xml, proprietary, etc... This in effect is what DAZ is doing with the support files. So far so good. They are actually following best practices up to this point. Where they fall away from that is when they don't handle the sidecar files in a way that sets a standard that other programs can access and modify these sidecar files, and further, provide a way in the program (DAZ Studio) to syncronize these sidecar files with the database (CMS.)

    Edit: technically they do provide a way to syncronize the database as one can 're-import' and this should do exactly that. However, this doesn't seem to be working properly.

    Second Edit: There are many programs that handle metadata using the methods I have discribed and this *is* considered current best practice. I simply pointed out the three above as they are readily accessible for looking at. Mediamonkey and Media Companion probably being best two of the three for simply understanding functionality.

    Post edited by Gedd on
  • cridgitcridgit Posts: 823
    edited November 2012

    We've got 2 issues here. Firstly, does DAZ Studio handle metadata correctly and allow external modification of metadata. Secondly, should we be allowed to rename and reorganize our files and folders as we see fit and still expect our metadata to work inside DAZ Studio.

    Let me tackle issue #1 first. Your statement that DAZ Studio does not support this is simply not correct. DAZ Studio handles external modification of DSX metadata perfectly. See the screenshots below for an example. The point is clear - anybody or any program can modify the DSX files and after reimporting the metadata DAZ Studio absorbs those changes.

    You might ask why its necessary to reimport the metadata rather than read it "live" from the DSX files, or why not automatically import updated metadata when DAZ Studio starts up. The former case won't provide the performance of a database, and the latter will slow down startup times for ALL users.

    Second issue: "why can't I rename my files and folders as I see fit outside of DAZ Studio"? The problem here is that when DAZ Studio first identifies a new asset, the only information it has about what that asset is, is its file name and path. Without that information THERE IS NO WAY IN GODS KINGDOM to know what the asset is. So if you insist on changing the files and paths DAZ Studio won't be able to connect the asset's metadata to it, and you're on your own. If you leave the file names and paths intact and import the metadata, DAZ Studio knows what the asset is and attaches the metadata to it. If you THEN change the file name and path (through DAZ Studio), it should keep track for you (I haven't tested this). Suggesting the program is deficient because it doesn't allow you to arbitrarily change file names and paths outside of its control, doesn't make sense.

    Hopefully I have now more clearly stated my position and shared enough information with you so that you can see the point. I'm not diagreeing with your right to choose the workflow that suits you. I'm explaining why there are limitations as a result of your choice.

    Picture5.jpg
    1509 x 848 - 143K
    Picture4.jpg
    1502 x 848 - 159K
    Picture3.jpg
    1502 x 846 - 137K
    Picture2.jpg
    1502 x 846 - 122K
    Picture1.jpg
    1506 x 848 - 133K
    Post edited by cridgit on
  • GeddGedd Posts: 2,444
    edited November 2012

    If any program tried to read the data *live* it would cause it to 'break' too often (file locking issues etc.. when multiple apps tried to access the same file.)

    Actually, I apologize. The issue that I believe is tripping you up is actually a good point and one I didn't want to get into necessarily because I didn't want to get that deep into it. I'll be brief because I don't want to bore everyone. Basically the added complexity DAZ has is that unlike a video or music file, a 3d object is multiple objects, and therefore more complicated to deal with. The issue with paths and filenames have to be either avoided being changed or done so in a way that is robust enough to handle it properly and efficiently. Going into it further as to how to deal with that is too boring for most so I'll spare everyone from that. The one thing I will point out is that separating 'tags' from files and paths, ie... keeping them to separate and well defined areas of the sidecar file, is a step towards making this feasible.

    Understand, I haven't played with trying to modify the files. I was actually responding to Richard's post saying he hadn't had much luck manually modifying the files. Since I trust in his competence, I figured there are probably gotchas involved. That was the impetus of all of this on my part really. I know that all of this is in flux to some extent as DAZ formalizes some of this so I'm not in a rush to try delving too deeply into this atm.

    That last part, the fact that import doesn't zero out old metadata, nor provide a simple interface where all of the metadata is visible in a single screen for a given object/set of objects... that's an example of a gotcha.

    Post edited by Gedd on
  • GeddGedd Posts: 2,444
    edited December 1969

    I should also say, you took the time to try to explain your point and do screen captures to make it clear. Thank you for that.. I just hope it can be kept on a level where people don't get emotionally vested in something that should be a technical issue, myself included ofc. There isn't any reason anyone should get upset about any of this.

  • cridgitcridgit Posts: 823
    edited November 2012

    No need to apologize. The discussion is good because it gets us on the same page and hopefully moves DAZ Studio and the community forward. Agree there is no need for any of us to get emotional - we're just trying to understand each other coming at an issue from different angles - that's why I took the time to demonstrate what I was saying with screenshots. I guess the reason I'm taking quite a strong position on this is because you are coming across as taking a strong position, but you may not have invested the same level of time in understanding the subject (see my last note below for an example).

    Unfortunately I don't understand what issue is tripping me up, I hadn't intended to respond on RH's issue, but to the issue you raised on batch renaming files. I don't see how you can expect to do that outside of Studio and have the default metadata work, so I showed you how modifying the DSX will allow you to do what you want to do - automate renaming files outside of Studio while keeping metadata intact. I was trying to help *you* address your issue.

    Gedd said:
    That last part, the fact that import doesn't zero out old metadata, nor provide a simple interface where all of the metadata is visible in a single screen for a given object/set of objects... that's an example of a gotcha.


    I agree that Studio's metadata interface and Smart Content are far from perfect (I've filed dozens of feature requests to further improve the interface). However, Studio *does* zero out old metadata and replace it with the new. I was referring to categories specifically. Obviously it shouldn't zero out your old categories because then you lose any additional categorization you've done every time you update the product. That decision makes perfect sense to me.

    Post edited by cridgit on
  • GeddGedd Posts: 2,444
    edited November 2012

    cridgit said:
    ... However, Studio *does* zero out old metadata and replace it with the new. I was referring to categories specifically. Obviously it shouldn't zero out your old categories because then you lose any additional categorization you've done every time you update the product. That decision makes perfect sense to me.

    On the surface, yes this would seem to make sense. However, there are better ways to handle this that would accommodate merging the various streams without conflicting but again, this starts to get a bit into the deeper end, and without assigning resources that aren't necessarily available at this time for this task it might be the best option.

    As for mass renaming files messing with the 'default' metadata.. yes you are correct. Without some way to efficiently update the support files it would break the system.

    This basically gets at the center of the situation. Many people have voiced their situation that they prefer renaming and restructuring the files to the point of giving up metadata if necessary. That is, if forced to choose between them, they do not choose metadata but rather choose rename and restructure. I think it is only respectful to appreciate their right to feel that if they can only have one or the other, they will choose restructure and rename. One shouldn't have to choose, eventually.

    The final point is that if the metadata is too hard to change, if there isn't an easy to manage interface for doing mass changes on the metadata, it will be useless for many. One size does not fit all in this case. Is there really any good structure of metadata that will work for most people? It seems the focus should be on creating flexible ways of creating and managing various metadata structures, with methods to save/share/modify metadata templates of objects. That way, people could create their own 'groups' to create metadata in ways that make sense to them.

    Categories actually fill this role for many people, which is why they say they don't understand why they should care about metadata. While there is a functional difference between metadata and categories, I don't think the differences are ones that some people care about in the case of metadata in it's current state.

    Post edited by Gedd on
  • GeddGedd Posts: 2,444
    edited November 2012

    Another quick note on handling metadata. I have found that the programs that 'consume' metadata are often not the best ones for modifying it. bundling both in one package often makes the program complicated to use. Mediamonkey is great for ripping cd's and modifying the mp3 tags, and for some it is great for playback. For many though, it is more than they need for playback. One can use Mediamonkey to manage an mp3 library, while using many other apps to actually play back (consume..) including an apple ipod. Microsoft's Media Player and Apple's ITunes both read the tags Mediamonkey puts 'in the mp3 file' and so need nothing other then the modified mp3 to have all of the metadata intact. If the same song is ripped in MS or Apple's product, they put the metadata in an internal structure only and trying to move the mp3 results in a file with a database key for a filename and no metadata, ie useless to any other program other then the one it was ripped in. Also, if something happens to the index of the ripping program if it was either MS or Apple's product... well you can guess.. *poof.* All one has to do to back up music ripped in Mediamonkey is save the mp3 files. To restore, wipe the index in the program or reload the program and point it to the parent folder for the music and say 'index.' In MS/Apple (if the song/album was ripped in either,) One would have to go through a complicated process of backing up the music, the index, and restoring both 'properly.' Not nearly as easy for someone not technical, and more of a bother for anyone who is. Note, anyone can move their whole music library to another computer and point MS/Apple player at the folder and they will index the folders ripped by Mediamonkey and all of that data will instantly work in either/both.

    Media Companion looks up the metadata for movies, updates and manages the xml sidecar files, and technically can play them but isn't the best player by far. XBMC is a very nice media player, but not so great at managing the tags. XBMC does use the same xml format that Media Companion does though, so someone can use one for tagging/managing, one for consuming. With these programs, since the sidecar xml files are stored in the same folder as the movie, all one has to do to back them up is back up the folder with the movie in it. To restore, simply wipe the index or reinstall the base app and point it to the movies parent folder and say 'index,' simple and robust.

    In either case, each song, album, movie, etc.. is an autonomous unit with it's metadata contained within it's structure or in a folder with it's metadata. One can back up 5 items, 50 items, 500 items to one location, some other number of items to another location... reassemble them however they want... it all works.

    If one can wrap their mind around how these programs work with the metadata, they will see the future imo.

    Post edited by Gedd on
  • GeddGedd Posts: 2,444
    edited December 1969

    Ok, sorry but I have to throw in one more point.

    Metadata is a changing field right now. There is experimentation on the best ways to implement and consume it. Take 'Tag Clouds' on websites. In some cases they are simply an amusing sidebar item that people glance at and otherwise ignore. In other cases they fit very well and get used. The point is, people are playing with the whole idea of metadata and various ways of utilizing it.

  • WildlyfeWildlyfe Posts: 36
    edited December 1969

    OK, I think I have a direction now. After studying the tutorials and playing a bit, I have been able to mess with the metadata and make my own work for some content that came without. I like the possibilities that the metadata opens up for better workflow. It is really nice now that I see how it all actually works. Not simple, but not nearly as bad as I had thought and better than I thought. I didn't really grasp the relationship of the tabs to the categories or metadata before. I am going to bite the bullet and start re-installing my content. I can see how in the long run, it will be for the best.

    ....So if you insist on changing the files and paths DAZ Studio won’t be able to connect the asset’s metadata to it, and you’re on your own. If you leave the file names and paths intact and import the metadata, DAZ Studio knows what the asset is and attaches the metadata to it. If you THEN change the file name and path (through DAZ Studio), it should keep track for you (I haven’t tested this).

    It does. I have been trying to figure out how to change the file names and paths inside DAZ Studio, and I don't think it it works like it used to. I am pretty sure we used to be able to drag and drop between directories, but not now. You can however, select files from a category and from there have the option to edit the file pathways. You can select all files with that base pathway and change them to another base pathway. Seems to work, so I plan on re-installing to a NEW directory and then moving it all over to the main one whenever I am ready.

    In making my own metadata, I have found a few questions regarding the default categories. Many of the categories seem to be duplicated under "Presets". What is the reasoning for choosing, say "Preset/morphs" versus the "Shaping" category? and What is the difference between "apply" and "inj". I have several products that seem to place everything in "presets" and other similar products that don't use the presets at all. I am finding it a bit confusing.

  • cridgitcridgit Posts: 823
    edited December 1969

    Wildlyfe said:
    In making my own metadata, I have found a few questions regarding the default categories. Many of the categories seem to be duplicated under "Presets". What is the reasoning for choosing, say "Preset/morphs" versus the "Shaping" category? and What is the difference between "apply" and "inj". I have several products that seem to place everything in "presets" and other similar products that don't use the presets at all. I am finding it a bit confusing.


    Super to hear you're getting your teeth stuck into this. We have a discussion on metadata guidelines / consistency going on here: http://www.daz3d.com/forums/discussion/7542/

    Presets has been dropped and the categories that used to be under Presets are now directly in the Default category. So Default/Presets/Morphs should be dropped and Default/Shaping should be used instead. The new DAZ products are using the new default categories, but many of the older products still need to be updated.

    Inject is for injecting a morph and typically leaving the slider on 0, whereas Apply moves the slider to 1 (or 100%). Some injects also apply the morph, and some applies also inject the morph, so its a grey area. I tend to put anything that injects a morph (whether it applies the morph or not) in the Inject category, and the few poses that apply a morph go into Apply. Currently I have probably around 10 Injects for every 1 Apply.

  • WildlyfeWildlyfe Posts: 36
    edited December 1969

    Thanks Cridgit, for the update. Exactly the info I needed. I have been trying to follow the guideline discussion, but got a bit lost. A lot to digest. Finally I can see it all coming together.

  • MythmakerMythmaker Posts: 255
    edited December 1969

    cridgit said:


    Presets has been dropped and the categories that used to be under Presets are now directly in the Default category. So Default/Presets/Morphs should be dropped and Default/Shaping should be used instead. The new DAZ products are using the new default categories, but many of the older products still need to be updated.

    Inject is for injecting a morph and typically leaving the slider on 0, whereas Apply moves the slider to 1 (or 100%). Some injects also apply the morph, and some applies also inject the morph, so its a grey area. I tend to put anything that injects a morph (whether it applies the morph or not) in the Inject category, and the few poses that apply a morph go into Apply. Currently I have probably around 10 Injects for every 1 Apply.


    Exactly the clarification Daz noobs like me was looking for! Thanks a zillion.

    There are so many ways to do the same thing in Daz, and so much historical baggage to sieve through. But I'll get there...

Sign In or Register to comment.
Rocket Fuel