How long does your DIM take to populate?
Wonderland
Posts: 7,133
in The Commons
It might be because I have a ridiculous amount of content or the fact that my Daz library is on an external hard drive, but DIM takes about 20 minutes for anything to even show up... I was wondering how long it takes other people...

Comments
See here for a solution: https://www.daz3d.com/forums/discussion/347621/dim-startup-time-solved#latest
It shouldn't have anything to do with where your DAZ Library is as it doesn't seem to access it before you start installing/uninstalling and whatever you can do with your files. But if your downloaded files are on the removable drive and it's slow (USB 2?) it may have to do with that.
Mine takes about 2 minutes to update online from when I click the Start button. That's with all DIM data (downloaded files, manifest files, thumbnails) placed on an older fairly slow internal HDD (no SSDs). I have 15580 products and in DIM it looks like this:
11964 Ready to download
2441 Ready to install
6279 Installed
BTW, I can't rule out that caching (I have DIM open most of the time) may affect the speed. I'll try later after a reboot (note that a restart may not clear the disk caches, you need to do a cold boot, preferably with the power plug removed for a couple of minutes to ensure that all is cleared. Pressing the PCs Start button after it's been unplugged is also a good idea as some PSUs may hold the power for a while, you'll usually see the power LED light up a for a moment if you do that which means there is still some power left.).
OK, tried again after a cold boot with power off. Took 3:45 minutes this time with no other things running, so caching seems to be a factor.
I see about six and a half minutes from filling in my password to getting the list of available downloads, ready to install, and installed. My download directory is on an external USB 3 drive (2 TB) and my DIM control and installation manifest files are on my C: drive in the default locations. I have a bit over 13,000 products showing in DIM.
Items that may influence time:
1) DIM updates the date/time stamp of every manifest (dsx) file in the download directory
2) DIM walks the installation manifest files in your appdata directory
3) DIM scans your Daz product library
From these three sources DIM fills in the three tabs - available downloads with product updates split out, ready to install (with, again product updates split out) and installed products. (1) is dependent on the size of the download directory and the speed of the disk and the disk interface. My download directory is 1.31 TB, with 26,300+ files on, as I said, a USB 3 drive connected to a USB 3 port. (2) is dependent on the number of installed files and the speed od the disk the manifest resides on; on my laptop, this is an internal 7200 RPM 1 TB drive and I only have 1,000 products installed. (3) is dependent on your network connection and I'm running on Starbucks' Google gigabit wifi.
I'll time the main system when I get home tonight, but it will be an apples to turnips comparison - the home system has the 'download' directory copied to an internal 2 TB SATA drive and the install manifest (all 13,000+ products installed) is on an SSD - and there's no network connectivity. So I expect startup to be faster.
To put the different times in perspective: with the downloaded files being kept in their original location: startup was over 30 minutes. With the downloaded files moved to a different folder, startup time is now 35 seconds.
It doesn't seem to do that (as I wrote in my first post). I've tried to rename the DAZ Library as well as the Connect library and then update DIM online, it didn't complain but updated as normal. If it does and doesn't warn you if the DAZ Library is missing it's badly coded, but I doubt the DAZ programmers are that careless (that would normally also cause some serious malfunction).
For me, it's about 2 minutes after DIM starts up to sort through the downloads and figure out which ones aren't installed yet. I've been doing manual downloads for about 7-8 months now, as my rendering computer is now isolated from the interweb thingie, to help reduce the risk of nasty bugs.
Anyways, my setup has the Daz files in the 'Default' DIM location, on a 2TB Samsung 970 EVO NVME SSD. So the drive access times are pretty fast. As others have noted, if you are accessing an external drive, etc. this can impact sort times. Nothing wrong with that approach, just felt the need to mention that access times for external drives over USB is often slower than internal NVME based drives, or SATA drives for that matter...
My guess from what I read here is that the number of downloaded files matters a great deal, when it comes to update speed. It seems to update the .dsx files for each downloaded product plus it's probably doing some compare between this and that every time it updates which is probably taking a while.
Not your local 'My Library' - the Daz Product Library under your account - and I'm assuming the process is semi-optimized to look at updates since last connect, but I may be wrong. This is how it finds out about new purchases and updates.
ETA - Startup time on the home system is just under 11 seconds. There is no network connection, so no download of data from Daz AND no update of the dsx manifests in the download directory. 24 ready to install and 13,182 installed.
From this, the bulk of the startup time is network access and updating the dsx manifests in the download directory.
I do have my downloaded files in another folder. I have 17,266 files installed...
What is the path to that "Daz Product Library"?
There is a DIM folder called "ManifestFiles" which contains modified versions of the .dsx files that accompanies the install zips in the "Downloads" folder, these ManifestFiles contain information about the installed products and DIM uses them to keep track of what is installed and where. I've moved these and all other product related files that DIM is using to another drive:
This is the .dsx file that accompanies the product file(s), and below the modfied version which contains local install data:
<?xml version="1.0" encoding="UTF-8"?>
<ProductSupplement VERSION="0.1">
<ProductName VALUE="Victoria 2.0 Character Pack 1"/>
<ProductStoreIDX VALUE="8-1"/>
<UserOrderId VALUE="6717907"/>
<UserOrderDate VALUE="2011-01-10T19:39:07Z"/>
<InstallerDate VALUE="2013-02-08T00:05:00Z"/>
<ProductFileGuid VALUE="5d4b40a5-b6e0-da8e-9a4d-f46b438e95b6"/>
<InstallTypes VALUE="Content"/>
<ProductTags VALUE="DAZStudio4_5,DAZStudio4,DAZStudio3,DAZStudioLegacy,Poser9,PoserLegacy,CloudAvailable"/>
</ProductSupplement>
-----------------------------------------------------------------------
<?xml version="1.0" encoding="UTF-8"?>
<DAZInstallManifest VERSION="0.1">
<GlobalID VALUE="3835eed4-edb1-04ca-54be-0f94c44b0b0e"/>
<MetadataGlobalID VALUE="3835eed4-edb1-04ca-54be-0f94c44b0b0e"/>
<SmartContent VALUE="False"/>
<ProductName VALUE="Victoria 2.0 Character Pack 1"/>
<ProductStoreIDX VALUE="8-1"/>
<ProductFileGuid VALUE="5d4b40a5-b6e0-da8e-9a4d-f46b438e95b6"/>
<InstallTypes VALUE="Content"/>
<ProductTags VALUE="DAZStudio4_5,DAZStudio4,DAZStudio3,DAZStudioLegacy,Poser9,PoserLegacy,CloudAvailable"/>
<ArchiveDate VALUE="2013-02-08T00:05:00Z"/>
<InstallerDate VALUE="2013-02-08T00:05:00Z"/>
<UserInstallAccount VALUE="Account"/>
<UserOrderId VALUE="6717907"/>
<UserOrderDate VALUE="2011-01-10T19:39:07Z"/>
<UserInstallDate VALUE="2017-05-29T04:47:42Z"/>
<InstalledSize VALUE="543613"/>
<UserInstallPath VALUE="F:/My DAZ 3D Library"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Alien.fc2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Alien.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Elderly.fc2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Elderly.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Heavy Torso.fc2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Heavy Torso.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Oriental.fc2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Oriental.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Super Model 2.fc2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Face/V2 Character Head/Super Model 2.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Alien Body.pz2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Alien Body.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Elderly Body.pz2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Elderly Body.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Heavy Torso Body.pz2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Heavy Torso Body.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Oriental Body.pz2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Oriental Body.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Super Model 2 Body.pz2"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Libraries/Pose/V2 Character Body/Super Model 2 Body.rsr"/>
<File TARGET="Content" ACTION="Install" VALUE="Content/Runtime/Support/DAZ_3D_8_Victoria_2_0_Character_Pack_1.dsx"/>
</DAZInstallManifest>
I think he means at the DAZ3D end, your Product Library.
@Ati, you mean "different folder," right? Not "different drive?"
I read the other thread you linked. But I was not certain then either. Are you simply moving all the DIM downloaded zips into a different folder on the same drive?
C:\Users\[user]\Documents\DAZ 3D\Studio\My Library\ ?
I've moved that to F: too. And can't find anything there that looks like being DIM related. Neither in the original C: folder which is still there. In the folder moved to F: there is a data\auto_adapted folder with many subfolders, but they seem to contain .dsf files only. Maybe Connect related?
I find it all quite confusing, DAZ is placing files all over the place on C:, both for DS, DIM and the CMS.
Correct. Step 3 is to parse your Daz store product library - not something on your system. Then, when you download the files with DIM the existing store dsx file gets modified to add your order ID and order date before it is saved in your download directory.
Not on your computer, the DAZ3D servers, your Product Library on the web site. They have records of all your purchases, so they check what you have downloaded vs. what is available (new or updates). If you move the accompanying .dsx files in the Package Archive (or change the folder without moving them over), DIM will show them all as "Available for Download" because it thinks you don't have anything.
Ah jeez, I used to move the download files to a second drive pretty regularly, once I got a new drive that had a ton of space I hadn't done it in a while. Made a huge difference, so thanks for pointing this out!
Different folder, same drive. It used to be in "F:\Program Files\DAZ 3D" (for me), and I moved them over to "F:\Program Files\DAZ 3D backup". Now I just need to move the new ones over once in a while, and it should remain fast, while I still have the original zips for backup purposes. (This location is not listed in DIM.)
Ah, OK. I've never thought about the Product Library in connection with DIM. I've written software that pulls user and product data from the Product Library in the user accounts and there's no DIM data there so that confused me. DIM connects to its own "private" server AFAIR.
So then DIM is parsing and updating every file in your product list if you save them locally. I can be impatient but it's not like 3 1/2 minutes is an unreasonable amount of time to wait for DIM to parse all 4000+ of those saved download files xml data when DIM's main purpose anyway is to download files that are often gigabytes in size and take hours in doing so (in my case with a 3.5mb/s download speed). DIM took 3 minutes to startup? Excellent. Now be prepared to wait another 40 minutes as it downloads the new 630MB DS PB version.
Thanks so much @Ati. I'll do that. I had gotten accustomed to waiting a half hour or less for DIM to come on.
Yeah, that half an hour waiting time was starting to bother me. I only downloaded my purchases every couple of weeks because of it. Now I can go back to downloading them immediately. And it was a pretty quick "fix," or workaround, or whatever you want to call it.
This is great! I only turn on the internet connection of my Daz computer when I have to download files; so, now I can go from an hour down to a few minutes.