Winamp & SHOUTcast Forums Dealing with extremely large file count question?

 4th September 2017, 12:22 #1 MyPC8MyBrain Member   Join Date: Nov 2015 Posts: 73 Dealing with extremely large file count question? Hi Guys, a question when dealing with a very large amount of file being installed... (200k count atm later on ill need to do the same for 1.5mil!) using the new code bit from earlier post i am able to run the setup itself fairly quickly, now comes the tricky part; the uninstall, in this step i have to specifically revert the install procedure; which is returning each file to its original location, with a large file count and no option to use wildcard the uninstall bit is well over 20mb, this is due to specific instruction given to each file, the process of building the installer due to the long uninstall list is taking forever (up to 8 hours) to build, (this installer only moves existing files from multiple dir's to a single dir, the uninstaller reverts that step; but i have to be specific in that step,) i think i need to build an external exe that does these specific revert back steps (they don't change) so when i rebuild i don't have to wait 8 hours for each build to complete, is there another way to speed this up? TIA Chris
 5th September 2017, 00:52 #2 JasonFriday13 Major Dude     Join Date: May 2005 Location: New Zealand Posts: 858 Post a short snippet of code that shows what you are doing for the uninstall files, there might be a bottleneck that could be widened. One way to speed it up is during the install make a list of all the files that were moved and save it in the install directory somewhere as a file. Then on uninstall, just loop through the same file and do your specific work on each one. "Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me) NSIS 3 POSIX Ninja Wiki Profile
 5th September 2017, 07:34 #3 MyPC8MyBrain Member   Join Date: Nov 2015 Posts: 73 Thank you Jason here is a small snippet from my uninstall script, its more of a patch installer in this case where we don't install files just move existing files from one dir to another during install; in install step i can use a wild card when moving from multiple folders into one; the uninstall is tricky without the use of wildcard, i also know the file names and location in advance so this list doesn't need to be "generated" during setup, PHP Code:  Section Uninstall SetDetailsPrint none         SetOverwrite on           rename "$INSTDIR\Patch\00000118.BMP" "$INSTDIR\Dir0\00000118.BMP"           rename "$INSTDIR\Patch\00010119.BMP" "$INSTDIR\Dir1\00010119.BMP"           rename "$INSTDIR\Patch\00010120.BMP" "$INSTDIR\Dir2\00010120.BMP"           rename "$INSTDIR\Patch\00020121.BMP" "$INSTDIR\Dir3\00020121.BMP"           rename "$INSTDIR\Patch\00020122.BMP" "$INSTDIR\Dir4\00020122.BMP"           .....            (the list continues with about 200k similar rename entries)         SetAutoClose true SectionEnd   the uninstall bit just moves files from one dir to another, no new files added or removed, TIA Chris Last edited by MyPC8MyBrain; 5th September 2017 at 10:05.
 5th September 2017, 12:58 #4 JasonFriday13 Major Dude     Join Date: May 2005 Location: New Zealand Posts: 858 You shouldn't be using Rename to move files around, use CopyFiles and Delete. You know that you can keep the directory structure the same when you move the files over, this allows you to use the same wildcard loop for the installer and uninstaller and all you change are the directories it points to. ProcessTextFiles will have to be modified to take two directories instead of one. On the other hand, if you know the files to be moved at compile time, you can use compile time commands to generate two nsis script files, one for the installer and one for the uninstaller. Then you just !include that file in the section instead of doing it at install time. "Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me) NSIS 3 POSIX Ninja Wiki Profile
 5th September 2017, 13:02 #5 MyPC8MyBrain Member   Join Date: Nov 2015 Posts: 73 Thank you Jason wouldn't copy then delete take longer? ATB Chris
 5th September 2017, 13:07 #6 JasonFriday13 Major Dude     Join Date: May 2005 Location: New Zealand Posts: 858 I don't know if it will take longer, I haven't timed it. "Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me) NSIS 3 POSIX Ninja Wiki Profile
 5th September 2017, 13:59 #7 JasonFriday13 Major Dude     Join Date: May 2005 Location: New Zealand Posts: 858 Another option is using the system plugin to call MoveFileEx(), you will have to read the documentation on how to use it. "Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me) NSIS 3 POSIX Ninja Wiki Profile
 5th September 2017, 17:59 #8 MyPC8MyBrain Member   Join Date: Nov 2015 Posts: 73 Thank you Jason i will look into your suggestion, since this is a "patch installer" we are not installing or removing files at any point, we just move them within the same top folder from multiple dir to single dir (installer part) and vice versa (uninstaller part) from single dir back to multi spesific dir location, uninstall is a spesific revert off a list embedded in the uninstaller, and thats where the issue of this long stall is happening, there are no files to compress/decompress in the process, its just move instructions on install with wildcard; and than uninstaller spesific restore, i thought renaming files makes the most sense in this situation since t s just a ptr record change, TIA Chris
 7th September 2017, 19:10 #9 MyPC8MyBrain Member   Join Date: Nov 2015 Posts: 73 here is what i came up with so far, working off external list like so should speed things up, since rename wont take one line operation i had to use two files; one with a line containing the copy "from" path, and the other the copy "to" path, this works fine in my standalone test with small number of files, once in the full script it will stop looping after one loop, what am i doing wrong here? PHP Code:  section ReadRegStr $INSTDIR HKCU "Software\..." Path Open: FileOpen$1 "$INSTDIR\from.txt" r FileOpen$2 "$INSTDIR\to.txt" r IfErrors exit loop: FileRead$1 $3 FileRead$2 $4 IfErrors closelist Rename$INSTDIR\$3$INSTDIR\$4 Push$3 Push $4 Goto loop closelist: FileClose$1 FileClose $2 IntOp$5 $5 + 1 Goto exit exit: sectionend  TIA Chris  7th September 2017, 19:27 #10 MyPC8MyBrain Member Join Date: Nov 2015 Posts: 73 as usual as soon as i post for advice i find the answer a minute later, i just had to clear errors first ATB Chris  8th September 2017, 06:25 #11 JasonFriday13 Major Dude Join Date: May 2005 Location: New Zealand Posts: 858 Interesting, you don't need 'push$3' and 'push $4', because you aren't popping them off anywhere. Same with IntOp$5 $5 + 1 ($5 isn't used anywhere). But I assume they are being used in your actual script? "Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me) NSIS 3 POSIX Ninja Wiki Profile
 8th September 2017, 19:43 #12 MyPC8MyBrain Member   Join Date: Nov 2015 Posts: 73 Thank you Jason $3 and$4 are the lines passed from reading each of the lists, $5 is leftovers from my test; i was sure it increments my next line Ive actually made few more adjustments but i seem to be stuck atm, the approach i used above works with one condition; that all files be present, if one file is missing the whole script exits in the middle of uninstall instruction with external list (not what we want); i need to keep parsing through the list to the end regardless of file present in origin or not, my latest attempt is to pass a blank line ""/NULL as the returning value to exit the list by using a blank line in from.txt but that not working for me atm from.txt content file looks like this PHP Code:  CB\test1\image0.JPG CB\test1\image1.JPG CB\test1\image2.JPG CB\test1\image3.JPG  to.txt content looks like this PHP Code:  CB\test2\image0.JPG CB\test3\image1.JPG CB\test4\image2.JPG CB\test5\image3.JPG  PHP Code:  section ReadRegStr$INSTDIR HKCU "Software\..." Path Open: FileOpen $1 "$INSTDIR\from.txt" r FileOpen $2 "$INSTDIR\to.txt" r IfErrors exit         loop:         FileRead $1$3         FileRead $2$4     StrCmp $3 "" closelist Rename$INSTDIR\$3$INSTDIR\$4 Push$3         Push $4 Goto loop closelist: FileClose$1         FileClose $2 IntOp$5 $5 + 1 Goto exit exit:  ATB Chris  8th September 2017, 22:07 #13 MyPC8MyBrain Member Join Date: Nov 2015 Posts: 73 i settled on this for now in from.txt last line is "end" PHP Code:  ReadRegStr$INSTDIR HKCU "Software\..." Path         ClearErrors         Open:         FileOpen $1 "$INSTDIR\from.txt" r         FileOpen $2 "$INSTDIR\to.txt" r         IfErrors exit         loop:         FileRead $1$3         FileRead $2$4         StrCmp $3 "end" closelist Rename$INSTDIR\$3$INSTDIR\$4 Goto loop closelist: FileClose$1         FileClose \$2         Goto exit         exit:   ATB Chris
 Winamp & SHOUTcast Forums Dealing with extremely large file count question?