Digital Art Zone

 
     
A complete reinstallation of DS4.5 with 1299 products - The need for an automated installation and update system
Posted: 19 November 2012 01:47 AM   [ Ignore ]
Active Member
Avatar
RankRank
Total Posts:  314
Joined  2011-03-26

I am currently facing the task to reinstall DS4.5 with 1299 products.

Why the need to reinstall?

1) The metadata and Content Management System got corrupted during a fatal crash while metadata was queued to process:

After several fatal errors what seems to be the main engine behind DS4.5 the metadata has stopped working.

See this thread:
http://www.daz3d.com/forums/viewthread/11759/


2) The speed of DS4.5 still seems to get slower the more content is installed.

The implementation of .duf files increased the working speed with DS4.5 a lot.
Nevertheless I still notice quite a lag when browsing smart content or the normal content database.
I have installed 1299 products that is 120GB of data in my DAZ documents folder.

3) Gen4 content without metadata takes too much time to browse and bloats the library structure.

All generation 4 content that is spread out over several folder locations like “files” “poses” and “props” is just not reasonable to navigate when one has 1299 products installed. The questions that rise while working with that old folder structure are:

Where is the figure?
Some figures are saved under a product name folder. Some figures are saved under the Published Artist folder. Some figures are saved in special categories like DAZ figures or are treated as add ons to V4, M4. In short it is really difficult to find anything without metadata when one has 1299 products installed.

Where are the texture sets to the product?
- Many names of the texture sets are not the same as the orginal products. Some are files under the texture artist name, some have the letters MAT in front of them or other initials.

Are there props?
- Some figures have props. Browsing back and forth between figures and props

With the time it takes to navigate that old contents file structure it seems not worth anymore to reinstall it.


3) New DSON installers

With 4.5 some content has been updated to feature the new .duf file type and the DSON format

4) Duplicate ID error
Many files have been updated to fix the duplicate ID error that was now triggered with the new DS 4.5 version


5) New metadata categorization system

The metadata team has after a year of trial and error now set fixed guidelines how they want to categorize all content.
Now tags for files are also improved to make the search function more useful.


What would help me in making this reinstallation of all content easier?

I am aware of this thread:
http://www.daz3d.com/forums/viewthread/3885/

and I am also aware that there seems to be an unofficial history manager that shows updates of DSON files and makes files downloading a little easier. But can a third party application be trusted? Is it not DAZ job to provide solutions for their customers?

Yes it is great that some community members took initiative but what is needed are offical reports by DAZ staff not somewhere scatter on the forum but in one easy to browse database

I) A browsable database of all my orders and files that shows the following information:

- a fast way to see all products that were updated since I last downloaded them

In the old store one could at least manually compare the dates by looking at the entries in the column “last updated” and “last downloaded”

- an indication if the product still has the .dsf or the new DSON .duf file format

- an indication if the product has “no metadata”, “old metadata” or “metadata that follows the new guidelines”


II)Feature request: DAZ Studio as an automated content management system

To me the main purpose of DAZ Studio is to find, browse and setup content to export to other applications.

- It seems that there is a limit to how much content can be handled. It seems 120GB is too much.
- It seems also obvious that having the latest files and updates for any software is crucial

Manually updating and downloading 1299 files is just insane. Every week new updates are released.
It is just not reasonable time management for creative people to spend their time downloading and installaing all day long.
To work efficiently it also seems reasonalbe to only have those files installed which are crucial for the current project that is being worked on.

What is needed is an automated system that has the following features:

- DS is used with an user account that has access to a personalized database on the DAZ server
- The user can browse the Database either directly in DS or online.
- The Database makes use of all existing metadata to search purchased products.
- The user can select which products are automatically downloaded and updated and which not.
- The user can choose if newly purchased product is downloaded and installed automatically right after the purchase or at the next scheduled installation and updating session.
- The user can select for each product if updated metadata is downloaded or the “User Data” metadata is kept
- The user can specify key products that are always installed on his system
- The user can assign products to custom temporary installation groups that download, install and uninstall quickly in the background when needed.

- - -

All wishfull thinking? Well you cannot say that I did not tell you what I need. Now the question is if someone out there can make this happen.

I look at everyting from a global time management perspective. One company invest its money and the working hours of its employes to come up with a solution. Afterwards hundred to thousands of people all over the world can benefit because their time is not wasted anymore with unnecessary download, installation and update tasks that conflict with any creative mood.

Games on pc and console now download, update and install patches and subscribed addon content in the background. Homes are equiped with network cloud solutions where mobile phones, tv boxes, pads and pc are synchronized.
It should not be an unsolvable problem to come up with an automated system for DS.

- - -

Does this all help to solve my current situation?
No. At the moment I am before the impossible task to reinstall the most import of my products again.
My goal is to start with products that were updated with DSON and those newer products that are following the new metadata guidlines.

If anyone can provide me with more lists which files were updated I would be greatfull.

I mean seriously, does DAZ not have an internal database where they keep track of all their products and their current update state?
Can you not make at least that information public allready?

In addtion there must be some timeline which products will be updated with DSON and which products if any will be updated with new metadata following the guidelines.

It would help the decision making process when to update and download wich products so much easier if that information was public.

Throw us a bone here. I am facing the task to install the most important of 1299 products again. I really do not want to reset them each individually to figure out which were updated. Really not.

 

 

 

 

 Signature 

My facebook page is always updated first with Computer Graphics, Photo and Video projects.
- - -
On deviantArt I post Computer Graphics and Photo projects.
- - -
On youtube you will find my Video projects.
- - -

Profile
 
 
Posted: 19 November 2012 06:06 AM   [ Ignore ]   [ # 1 ]
Administrator
Avatar
RankRankRankRank
Total Posts:  7272
Joined  2003-10-09

By what has beens aid the forthcoming Content Install Manager should address many of your wishes.

 Signature 

DAZ Studio Frequently Asked Questions

Index of free DAZ Studio scripts and plugins list

Profile
 
 
Posted: 19 November 2012 08:11 AM   [ Ignore ]   [ # 2 ]
Active Member
Avatar
RankRank
Total Posts:  314
Joined  2011-03-26
Richard Haseltine - 19 November 2012 06:06 AM

By what has beens aid the forthcoming Content Install Manager should address many of your wishes.

That would be awesome. Allthough the name just says Content “Install” Manager so I assume there will not be an update functionality?

Is there an offical thread about that “Content Install Manager” where I can find more information?

And when is it scheduled to be released?

For me the decision is really should I invest several days to install now or is the Content Install Manager “just around the corner”?

 

 Signature 

My facebook page is always updated first with Computer Graphics, Photo and Video projects.
- - -
On deviantArt I post Computer Graphics and Photo projects.
- - -
On youtube you will find my Video projects.
- - -

Profile
 
 
Posted: 19 November 2012 01:21 PM   [ Ignore ]   [ # 3 ]
Administrator
Avatar
RankRankRankRank
Total Posts:  7272
Joined  2003-10-09

There isn’t an official thread, so I would assume the release isn’t imminent. But it was said, when it was first discussed, that it would have the ability to handle product updates if the user wished.

 Signature 

DAZ Studio Frequently Asked Questions

Index of free DAZ Studio scripts and plugins list

Profile
 
 
Posted: 19 November 2012 01:45 PM   [ Ignore ]   [ # 4 ]
Active Member
Avatar
RankRank
Total Posts:  314
Joined  2011-03-26
Richard Haseltine - 19 November 2012 01:21 PM

There isn’t an official thread, so I would assume the release isn’t imminent. But it was said, when it was first discussed, that it would have the ability to handle product updates if the user wished.

Thanks for getting back with this great information. This calmed my nerves for now so I can focus on reinstalling the most important content. Letrs hope the best a working solution can be found in time.

 Signature 

My facebook page is always updated first with Computer Graphics, Photo and Video projects.
- - -
On deviantArt I post Computer Graphics and Photo projects.
- - -
On youtube you will find my Video projects.
- - -

Profile
 
 
Posted: 20 November 2012 12:53 AM   [ Ignore ]   [ # 5 ]
Active Member
RankRank
Total Posts:  889
Joined  2009-12-07

While you’re waiting, here’s a batch installation script http://www.daz3d.com/forums/viewthread/4603/ which contains a link to the original batch installation script written by Tugspx. I modified it to process different types of installers in successive waves (by uncommenting the different parameters lines). Legionair added two other scripts but I haven’t looked at what he’s done with those.

 Signature 

PoseMaster for DAZ Studio and PoseMaster People Pack now sold at Rendo.

Profile
 
 
Posted: 20 November 2012 04:58 AM   [ Ignore ]   [ # 6 ]
Active Member
Avatar
RankRank
Total Posts:  314
Joined  2011-03-26
cridgit - 20 November 2012 12:53 AM

While you’re waiting, here’s a batch installation script http://www.daz3d.com/forums/viewthread/4603/ which contains a link to the original batch installation script written by Tugspx. I modified it to process different types of installers in successive waves (by uncommenting the different parameters lines). Legionair added two other scripts but I haven’t looked at what he’s done with those.

Cridgit!

Thx a lot for sharing this!

I followed the link, read through everything and followed the link further to a post in the forumarchive where the original unchanged script by Tugspx is to be found:

http://forumarchive.daz3d.com/viewtopic.php?t=180348&postdays=0&postorder=asc&start=0

I quickly tried with 4 products I purchased last sunday and it worked.

- - -

Now the question will be how many installers I can safely process at the same time without running the risk that some metadata is being skiped when added to the queue.

My cautious approach was so far to unly install 10 products at a time and the restart DS4.5 to process the metadata queue.
Maybe that is to cautious? But then again it was also last sunday that DS4.5 triggered a fatal error while trying to process metadata of 15 queued products. On the other hand I seem to have managed to install whole “pro bundles” without any loss. Seems all a bit trial and error so far.

 

 Signature 

My facebook page is always updated first with Computer Graphics, Photo and Video projects.
- - -
On deviantArt I post Computer Graphics and Photo projects.
- - -
On youtube you will find my Video projects.
- - -

Profile
 
 
Posted: 20 November 2012 05:06 AM   [ Ignore ]   [ # 7 ]
Active Member
RankRank
Total Posts:  889
Joined  2009-12-07

I documented my experience in the post on the old forum. I ran it on about 250 products, but had to do it in several cycles (about 3 or 4 cycles I think). Installers that worked moved into a subfolder, so the next time I ran the script it picked up the remaining ones and tried them again. Right at the end there were maybe a dozen I needed to install by hand.

I didn’t have any problem with metadata crashing - it just takes a loooong time to import grin

 Signature 

PoseMaster for DAZ Studio and PoseMaster People Pack now sold at Rendo.

Profile
 
 
Posted: 20 November 2012 05:46 AM   [ Ignore ]   [ # 8 ]
Administrator
Avatar
RankRankRankRank
Total Posts:  9023
Joined  2003-10-09
linvanchene - 20 November 2012 04:58 AM
cridgit - 20 November 2012 12:53 AM

While you’re waiting, here’s a batch installation script http://www.daz3d.com/forums/viewthread/4603/ which contains a link to the original batch installation script written by Tugspx. I modified it to process different types of installers in successive waves (by uncommenting the different parameters lines). Legionair added two other scripts but I haven’t looked at what he’s done with those.

Cridgit!

Thx a lot for sharing this!

I followed the link, read through everything and followed the link further to a post in the forumarchive where the original unchanged script by Tugspx is to be found:

http://forumarchive.daz3d.com/viewtopic.php?t=180348&postdays=0&postorder=asc&start=0

I quickly tried with 4 products I purchased last sunday and it worked.

- - -

Now the question will be how many installers I can safely process at the same time without running the risk that some metadata is being skiped when added to the queue.

My cautious approach was so far to unly install 10 products at a time and the restart DS4.5 to process the metadata queue.
Maybe that is to cautious? But then again it was also last sunday that DS4.5 triggered a fatal error while trying to process metadata of 15 queued products. On the other hand I seem to have managed to install whole “pro bundles” without any loss. Seems all a bit trial and error so far.

 


Coded section removed, as it appeared to be merely a copy and paste of the information in the old forum, and it was breaking the forum formatting.

 Signature 

Chohole’s Space        Neil’Vs Freebies and stuff        Bryce Rendering Challenge        Freebie Challenge
11915 was the number

Profile
 
 
Posted: 20 November 2012 11:12 AM   [ Ignore ]   [ # 9 ]
Active Member
Avatar
RankRank
Total Posts:  314
Joined  2011-03-26
chohole - 20 November 2012 05:46 AM


Coded section removed, as it appeared to be merely a copy and paste of the information in the old forum, and it was breaking the forum formatting.

1) My intention was to provide for myself and some interested people a backup of the code in case the archive gets removed in a few years.

2) Posting the code here would make it able to be found by using the forum search function of the current forum.

3) For me it would have been a quick way to access the code from other places than home. The threads I am subscribed to are some kind of a “favorite” / bookmark system that work from any pc as long as I can log into the forum.

- - -

I apologize for breaking the forum formating.

 Signature 

My facebook page is always updated first with Computer Graphics, Photo and Video projects.
- - -
On deviantArt I post Computer Graphics and Photo projects.
- - -
On youtube you will find my Video projects.
- - -

Profile
 
 
Posted: 20 November 2012 01:31 PM   [ Ignore ]   [ # 10 ]
Administrator
Avatar
RankRankRankRank
Total Posts:  9023
Joined  2003-10-09

If you can find a way to post it without breaking the forum formatting, then please do repost it.

However by using the “Code” html format, it was showing the full strings without a break, across the width of the page and beyond, if you see what I mean.

 Signature 

Chohole’s Space        Neil’Vs Freebies and stuff        Bryce Rendering Challenge        Freebie Challenge
11915 was the number

Profile