Digital Art Zone

It is going to be a very long read again. But to all metadata creator knowing those differences might be very important.
Not knowing them might be the very reason why we see so many products released without all metadata included.

Please help filling in the gaps and correct me when I make wrong assumptions.

The following questions seem still to be rather unanswered all through the forum:

Which exact data is stored as “User Data”?
When is data stored as “User Data”?

The process of exporting metadata .dsa / .dsx files:

Go to the “Content Library” tab to the “Product Library” open up the Content DB Editor from the tab options and export the .dsa and .dsx files to Documents\DAZ 3D\Studio\My Library\Runtime\Support.

To make sure that those files are not overwritten when you reinstall product updateds or make changes that you do not want to save it is important to make a backup of them immediatley and save them in an other location.
A practical way seems to be the backup location of the original product installers.

- - -

The drawback of this is that if you do not make that backup installing products again will just overwrite all the categories and changes you made in the Content DB Editor. To remedy this it seems DAZ has introduced some new functions with DS4.5:

But do they really work like intended? Either I do not understand the process properly or those have quite some pitfalls.

Export “User Data” from Content DB Maintenance

With DS4.5 one has the option to export “User Data” from the “Content Library” tab options “Content DB Maintenance”

When one uses this function a file called UserData_1.dsx is placed in Documents\DAZ 3D\Studio\My Library\Runtime\Support

But which information is exactly stored in that file?

When you are importing new metadata you can see the option to load the “User Data” as well.
When you are importing new metadata there is also the option to check “User Data overwrites Product Data”.

The intention seems to be that when you install an update of the same product only the product content will be updated but not the metadata category changes.

It seems that the “User Data” is stored under

C:\ProgramData\DAZ 3D\Content Management Service\databases

I do not know if that is true and which file exactly contains the user data.

- - -

At first glance this seems to be rather useful. But then this usefulness is degraded by the fact that I do not have a very deep understanding what exactly all counts as user data. Is it just the categories created by the user or more?

Possible things that could go wrong when working with “User Data”:

User created metadata that is imported again is not treated as “User Data”

I did notice that in the “Select Categories” window that you can open by right clicking any file in about any window you can see some Categories marked with an “U”.
Those seem to be the categories that are created by the user.
I nevertheless noticed that by reimporting metadata files I created myself by exporting .dsa and .dsx files those categories previously marked with “U” are now seem not to be recognized as user created categories. The “U” is not showing after reimporting.

Are “file type”, “compatibility bases” “Tag” changes also considered as user data changes?

- Metadata customization is not just creating categories. The user might make some chances in the Content DB Editor.
Changing file types. Setting up compatibility bases. Adding tags.
Are those changes saved as “User Data” as well?

The difference of the “Content DB Editor” of the “DAZ Studio” the “Poser Formats” and the “Product Library “

The content DB Editor is mainly opened from the “Content Library” tab options.
Nevertheless is seems the exact library you open the “Content DB Editor” has a huge impact what kind of options you can see.

When you are opening the “Content DB Editor” to create your own metadata you need to open it from the “Content Library” tab “Product Library. Only then the user has acess to both the “Products” and the “Assets” subtabs.
It seems as long as you the the option “Auto Export” checked all changes you made will be saved to the .dsa and .dsx files
Documents\DAZ 3D\Studio\My Library\Runtime\Support

But when one opens the “Content DB Editor” from the “DAZ Studio or the “Poser Formats” libraries two it seems two things are different:

- The user can only access the “Assets” tab.
- There is no “Auto Export” function

This means by making changes in the Content DB Editor from the DAZ libraries no updates are made to the .dsa and .dsx files located in /Support.

But it seems those changes are also saved somewhere.

My guess would be under

C:\ProgramData\DAZ 3D\Content Management Service\databases

This leads to the questions:

Are all changes made in “Content DB Editor” accessed from the “DAZ Studio” or the “Poser Formats” libary treated as “User Data”?

Are all changes made in the “Content DB Editor” accessed from the “Product Library” treated as “User Data”?

Or and that might have dire complications is any Data that is exported as .dsx and .dsa files not treated as “User Data” anymore?
If that would be true then those changes I made would not be exported anymore by the “Export Userdata” function. And to make things worse when one forgets to make a backup of the .dsa and .dsx files installing updated products would just overwrite those files.

This could be one of the major reasons why we often see official DAZ metadata released that does not seem to contain any metadata:

- The “Product” metadata was created from the “Product Library” and the “Assets” metadata was created from the “DAZ Studio” or “Poser Libraries”. If after that the metadata creator just copies the .dsa / .dsx files and zips them up for distribution all the changes to the “Assets” metadata will not yet be included.

Therefore it is very important to always go to the “Product Library” and export the .dsa and .dsx files after all changes are made!!!
Only after that final step has be done one can go and collect the .dsa and .dsx files for distribution.

Does the “Synch Product and Local Database” checkbox in the Content DB Editor have any impact on User Data?
In the Content DB Editor you can find a checkbox called “Synch Product and Local Database”

I still only have guesses what the Product and what the Local Database are.

C:\ProgramData\DAZ 3D\Content Management Service\databases

But is the “Local Database” the same as “User Data”?

And what are the consequences if this checkbox is checked or not? Is it related to user Data at all?

What does this all mean for the “Export User Data” functions?

To me it seems the “Export User Data” and “Userdata overwrittes Product Data” function is absolutely useless if it does not include all the changes to categories and the Content DB Editor.

Also if you exported .dsa and .dsx files it is still necessary to make a backup of them in another location.
If you do not make a backup they will just be overwritten when you reinstall updated DAZ products.

Without properly understanding the “User Data” and how it works this function has absolutely no value to me.
To me it seems the only reasonable way of saving ones metadata changes is to use the “Export” option from the “Product Library”.

Feature request 1): Batch Export of all individual “Product Library .dsa and .dsx files:

To me currently what seems to be missing in Content DB Maintenance is a function that really exports ALL metadata of all products.

At the moment when I want to make sure that all changes I made to the metadata of a product can be imported again for a new installation are to manually export the .dsa and .dsx files to Documents\DAZ 3D\Studio\My Library\Runtime\Support .

But to go to each Product open up the Content DB Editor and trigger the “Export” or “Auto Export” function for each individual product would take a very long time.

Instead it would be great to have the option to batch export the whole “Product” library as individual .dsa and .dsx files to Documents\DAZ 3D\Studio\My Library\Runtime\Support .

- - -

Well thank you for reading through this. I hope you found some information valuable to you. I would be very glad if someone could take the time to fill in the missing gaps and correct me if I put some information there that is not correct.

I’m no expert on metadata but I will try and help out.

The .DSX files in Runtime\Support\ are plain text. 
That means they can be opened in a text editor to see exactly what data they store.  This includes products & the exported UserData_1.dsx file/s from Content DB Maintenance. Make sure your text editor is refreshed if it’s open when you make changes to the files.

From my tests most user metadata changes seem to be saving, exporting and importing properly.

Changes made from “Content Library: Products: Content DB Editor” saves User Data for:
Asset Value, Content Type, Categories, Compatibilities, Compatibility Base & User/Keywords.  But I could NOT get it to save user data for Description, Audience or Tags (so I would use userwords/keywords instead of tags). 

Changes made from “Content Library: DAZ / Poser Formats: Content DB Editor”:
As far as User Data, this seems to work the same as doing it from products (see above).  It just does not save the product files & does not access the top level product information. Although if you open the product later and save it, it will update the product files with the changes.  This even works after closing & re-starting DS.

Other changes made from the Content Library:
Tags (keywords & notes) & Right clicking: Categorize all seem to be working.  Categories might need to have at least 1 item assigned to them to work. 

In the Content DB Editor make sure “Sync Product & Local Database” is checked.  Otherwise some User Data values I listed above may not be saved on export and I ran into issues updating Compatibilities when it was off.

CMS:
I can’t access CMS but I assume it’s the same data being stored when we export User Data.  The main differences being that it’s easier to backup, edit & restore since we can work with text files and don’t have to start/stop the CMS. 

Here is how to test out if your changes are being saved/exported & imported properly:
1. Make changes to a test asset’s metadata.
2. Export your user data in Content DB Maintance.
3. Open the runtime/support/UserData_1.dsx file in a text editor & search (ctrl+F) for your changes to confirm.
4. Remove or edit the changes you made in step 1 but don’t export, to see if the 1st changes can be restored.
5. Go to Content DB Maintance & select “Re-Import Metadata”:
6. Uncheck “All Products”, make sure user data files & override are checked, Accept.
7. Check your test asset to see if the changes from step 1 have been restored.

Hope some of this was helpful.  I recommend updating the original post to make it more clear and concise as you discover answers to make it easier for people to learn from and add to. Also might help to cross link to this thread: http://www.daz3d.com/forums/viewthread/6123/

Here are some links that might also be helpful:
http://docs.daz3d.com/doku.php/public/software/dazstudio/4/userguide/creating_content/packaging/tutorials/adding_metadata/start
https://helpdaz.zendesk.com/entries/22523891-trouble-shooting-the-smart-content-pane

Swawa, thank you a lot for sharing your findings!

Especially a great idea to have a look at the .dsx files with a text editor. I seem always to be a little to carefull in fear of unintentionally deleting some lines. I was able to have a look at some .dsx files with Notepadd ++ and it was interesting to see especially what was saved in my userdata.

Allthough the whole interconnection between the user data and manually exporting a .dsx files still seems unclear.

For example:
If I export a .dsx files to /runtime/support will the data of that product be removed from the userdata?

To me it seems not. At the moment I have 105312 lines in the UserData_1.dsx file.

- - -

In the last weeks I continued to reinstall content and set up metadata sometimes from scratch and sometimes from what was there.

My conlusion at this moment is that the “Userdata” is of no help at all for my needs.
I had the whole CMS crashing on me with all data lost. From that experience I am not willing to trust that all the changes made will be restored with the user data.

What I really want is that all changes I make are saved automatically on a product level.

If I make changes to the metadata I do not want those changes saved in the User Data I want those changes to be automatically saved in the .dsa or .dsx files in /runtime/support.

Unfortunately an option that all changes are automatically saved to the .dsa and .dsx files are not availabe at the moment.
The only way to trigger an autosafe function is to go to the Product Library and open up the Content DB Editor from there.

So that is what I do:

Go to the Product Library / Content DB Editor and export the .dsa and .dsx after every change I make.
Then in the evening I make a backup of the /support folder

This seems the only way to make sure to have them ready if I reinstall products at a later point.
There would be no need to uninstall and reinstall products if DS could handle an infinite amount of products at the same time.
But even with the .duf systems it seems that DS slows down when 100s to 1000s icons are loaded at the same time in an open tab.
So for now I only want those products installed that I really need on a regular basis.

Workflow Example1 :
If I make a change in the categories I want that change immediatley saved in the .dsa and .dsx files of the product.

Workflow Example 2:
If I make a change in the Content DB Editor from the DAZ or Poser Library I want that change immediatley saved in the .dsa and .dsx files of the product.

It is especially unfortunate that by right clicking smart content icons there is a quick link to the DAZ and Poser Librarys but no quick link to the Content Library Product Library.

And no the Product Library is not the same thing as the “Product View” in smart content.

If there would be such a quick link it would take less time to search around for the content in the Product Library and makeing changes and saving them would be faster.

Remember only by accessing the Content DB Editor from the Product Library are changes saved to the .dsa and .dsx files at the moment.

Workflow Example 3:
Under the old metadata system shoes were categorized as “Shoes”. Now they are categorized as “Footwear”.
It would be a very quick process to recategorize all “Shoes” at once to “Footwear”. But unfortunately like that I loose track which products I recategorized. So now instead of batchconverting all “Shoes” to “Footwear” I have to make that changes product by product from the “Product Library” to make sure that those changes are exported to the .dsa and .dsx files and not just to the user data.

- - -

Well reality is there do not seem to be many people in the forum who really care about making metadata for content themselves.
So the only people concerned with those problems are the DAZ metadata team and they do not seem to frequent those forums.

I am glad that you posted the links to the other threads.

But it seems most questions aksed in those threads were never answered anywhere as well.

I guess all we can do is to try and figure out this things ourselves as you suggested by trial and error.

To me it looks like support/user data stores any changes from the “original” product state (excluding: tags, description & audience). Saving the support/product files do not seem to remove any information from support/user data, unless you remove things that were added to the original state.  I do not know if there is work around for this.

It is annoying that some changes must be exported through DB Maintenance. And also that when using DB editor from Poser/DAZ Formats it also has to be opened again from products to export. Do you know if the CMS database directory stores all of our metadata changes and we could just back that up?  It is annoying to have to stop, restart it, etc though. I feel safest backing up everything.

I quick link to Product Library would be nice… but can 1 item be in more than 1 product?  I’ve been able to paste references into multiple products but I’m not sure if your not supposed to.  And they could even have a link to a list of products I guess.

It is a shame that metadata and many other features of DAZ have very poor, out of date or no documentation.  So many people have to waste so much time just because they did not do what should have been a requirement from the start. I wonder how many hours have been lost.  Honestly the time commitment is a bigger factor than $ when I consider buying most products at this point. 

I thought the entire point of metadata was that it was supposed to save time?  I feel like all the trouble of managing it may take up more time then it saves.  I seem to do OK making my own directory structures with old content but the auto material matching and stuff is nice. 

I would love a install/update system like steam. 

Swawa - 12 December 2012 09:21 PM

Do you know if the CMS database directory stores all of our metadata changes and we could just back that up?  It is annoying to have to stop, restart it, etc though. I feel safest backing up everything.

I asume that in the CMS databases folder all metadata and the userdata is saved. I agree that stopping the CMS and restarting is somewhat more complicated that it should be to make a backup. But then again I only have to do it once a day in the evening to make sure the work done will be saved.

Swawa - 12 December 2012 09:21 PM

I quick link to Product Library would be nice… but can 1 item be in more than 1 product?  I’ve been able to paste references into multiple products but I’m not sure if your not supposed to.  And they could even have a link to a list of products I guess.

 

Having the same items in more than one product should be no problem at all. In fact the way I see it that would be exactly the benefit of having metadata. While in the content library you can only have one folder structure to look at things with the metadata categories you can set up as many ways to order the content as you like.

Examples:
- You could put together custom products of any kind of items you used for a project to quickly set it up again.
- You could add all content of one artist into a product to have quick access to all items he created.
- You could create products with your favorite shaders of different shader bundles.
- You could create products with your favorite props for different themes to quickly populate scenes
- You could create products with your favorite poses to quickly set up people standing in the background

One needs to keep in mind that for some tasks it might also be helpful to use additional categories or a more detailed category structure.
Nevertheless the plus side of setting up custom products at the moment seems to be that those are not influenced by other content that you install. The category system very quickly becomes less useful when content is installed that is not up to date with the “new” metadata guidelines.
Products and Categories could also be used in combination. For example you could set up the category “My Favorites” and put all your custom products there.

Swawa - 12 December 2012 09:21 PM

It is a shame that metadata and many other features of DAZ have very poor, out of date or no documentation.  So many people have to waste so much time just because they did not do what should have been a requirement from the start. I wonder how many hours have been lost.  Honestly the time commitment is a bigger factor than $ when I consider buying most products at this point. 

I thought the entire point of metadata was that it was supposed to save time?  I feel like all the trouble of managing it may take up more time then it saves.  I seem to do OK making my own directory structures with old content but the auto material matching and stuff is nice.

This one of my biggest concernes as well. When I buy products I unfortunately feel that I cannot trust that the metadata is set up properly.
Each time I have to doublecheck everything. At this very moment there are still “new” items released in the store that are set up with the old metadata category system.

So after every purchase I have to go and clean it all up. Moving everything from those old “Presets/Materials, Preset/Poses folder to the new Materials and Poses categories. Checking all the wardrobe and props categories. Make sure to move the former sets and scenes to environments.

I did try backing up CMS, removing it’s database files & re-importing my support/product & user data files. Didn’t check it all but it seemed like almost everything was restored how I customized it.  I was missing a few custom products I made but I might not have set them up correctly.  So I restored my backed up CMS. 

Yeah, Metadata could be great if it was more consistent, reliable, well documented, easy/fast to manage.  In it’s current state it has been a nightmare. It doesn’t help that I bought a ton of stuff in a short time with all the sales.  So for now I’m really cutting back.

Swawa - 13 December 2012 01:01 AM

Yeah, Metadata could be great if it was more consistent, reliable, well documented, easy/fast to manage.  In it’s current state it has been a nightmare. It doesn’t help that I bought a ton of stuff in a short time with all the sales.  So for now I’m really cutting back.

Exatly as you desribed one has not only to consider the price one pays for products but also the cost of the time one needs to set them up with correct metadata.

It causes to much effort to mass buy Gen 4 content without metadata and old installers that cannot be processed with a batch installer.
It seems from a time perspective risky to buy any Gen5 content without DSON files because it seems a waste of time to set them up and then have to reinstall a DSON update a few weeks later.

Well the good news is that most products that are now released in the DSON format are in a good shape.

There is a lot of content I did not yet reinstall because I am waiting for a DSON update.
Especially waiting for a DSON Super Suit update, and updates for figure bundles like Hitomi, Jason, Amy.

Only time can teil if I will be waiting forever. I tried to get information which content they plan to update with DSON files but no luck there.

Anyway, for the moment I rather have only a few products installed in my runtime but those I all can find in the right “Categories” and use.