Old 4th September 2017, 12:22   #1
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Question Dealing with extremely large file count question?

Hi Guys,
a question when dealing with a very large amount of file being installed...
(200k count atm later on ill need to do the same for 1.5mil!)

using the new code bit from earlier post i am able to run the setup itself fairly quickly,
now comes the tricky part; the uninstall,

in this step i have to specifically revert the install procedure;
which is returning each file to its original location,
with a large file count and no option to use wildcard the uninstall bit is well over 20mb,
this is due to specific instruction given to each file,
the process of building the installer due to the long uninstall list is taking forever (up to 8 hours) to build,

(this installer only moves existing files from multiple dir's to a single dir,
the uninstaller reverts that step; but i have to be specific in that step,)

i think i need to build an external exe that does these specific revert back steps (they don't change) so when i rebuild i don't have to wait 8 hours for each build to complete,

is there another way to speed this up?

TIA
Chris
MyPC8MyBrain is offline   Reply With Quote
Old 5th September 2017, 00:52   #2
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 854
Post a short snippet of code that shows what you are doing for the uninstall files, there might be a bottleneck that could be widened.

One way to speed it up is during the install make a list of all the files that were moved and save it in the install directory somewhere as a file. Then on uninstall, just loop through the same file and do your specific work on each one.

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 5th September 2017, 07:34   #3
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Thumbs up

Thank you Jason
here is a small snippet from my uninstall script,
its more of a patch installer in this case where we don't install files just move existing files from one dir to another during install; in install step i can use a wild card when moving from multiple folders into one;
the uninstall is tricky without the use of wildcard,

i also know the file names and location in advance so this list doesn't need to be "generated" during setup,

PHP Code:
Section Uninstall
SetDetailsPrint none
        SetOverwrite on
          rename 
"$INSTDIR\Patch\00000118.BMP" "$INSTDIR\Dir0\00000118.BMP"
          
rename "$INSTDIR\Patch\00010119.BMP" "$INSTDIR\Dir1\00010119.BMP"
          
rename "$INSTDIR\Patch\00010120.BMP" "$INSTDIR\Dir2\00010120.BMP"
          
rename "$INSTDIR\Patch\00020121.BMP" "$INSTDIR\Dir3\00020121.BMP"
          
rename "$INSTDIR\Patch\00020122.BMP" "$INSTDIR\Dir4\00020122.BMP"
          
..... 
          (
the list continues with about 200k similar rename entries)
        
SetAutoClose true
SectionEnd 
the uninstall bit just moves files from one dir to another, no new files added or removed,

TIA
Chris

Last edited by MyPC8MyBrain; 5th September 2017 at 10:05.
MyPC8MyBrain is offline   Reply With Quote
Old 5th September 2017, 12:58   #4
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 854
You shouldn't be using Rename to move files around, use CopyFiles and Delete.

You know that you can keep the directory structure the same when you move the files over, this allows you to use the same wildcard loop for the installer and uninstaller and all you change are the directories it points to. ProcessTextFiles will have to be modified to take two directories instead of one.

On the other hand, if you know the files to be moved at compile time, you can use compile time commands to generate two nsis script files, one for the installer and one for the uninstaller. Then you just !include that file in the section instead of doing it at install time.

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 5th September 2017, 13:02   #5
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Thank you Jason
wouldn't copy then delete take longer?

ATB
Chris
MyPC8MyBrain is offline   Reply With Quote
Old 5th September 2017, 13:07   #6
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 854
I don't know if it will take longer, I haven't timed it.

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 5th September 2017, 13:59   #7
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 854
Another option is using the system plugin to call MoveFileEx(), you will have to read the documentation on how to use it.

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 5th September 2017, 17:59   #8
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Thumbs up

Thank you Jason
i will look into your suggestion,

since this is a "patch installer" we are not installing or removing files at any point,
we just move them within the same top folder from multiple dir to single dir (installer part)
and vice versa (uninstaller part) from single dir back to multi spesific dir location,

uninstall is a spesific revert off a list embedded in the uninstaller,
and thats where the issue of this long stall is happening,

there are no files to compress/decompress in the process,
its just move instructions on install with wildcard;
and than uninstaller spesific restore,

i thought renaming files makes the most sense in this situation since t s just a ptr record change,

TIA
Chris
MyPC8MyBrain is offline   Reply With Quote
Old 7th September 2017, 19:10   #9
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Question

here is what i came up with so far,
working off external list like so should speed things up,
since rename wont take one line operation i had to use two files;
one with a line containing the copy "from" path, and the other the copy "to" path,

this works fine in my standalone test with small number of files,
once in the full script it will stop looping after one loop,

what am i doing wrong here?

PHP Code:
section
ReadRegStr $INSTDIR HKCU 
"Software\..." Path
Open
:
FileOpen $"$INSTDIR\from.txt" r
FileOpen 
$"$INSTDIR\to.txt" r
IfErrors 
exit

loop:
FileRead $$3
FileRead 
$$4
IfErrors closelist
Rename $INSTDIR
\$3 $INSTDIR\$4
Push 
$3
Push 
$4
Goto loop

closelist
:
FileClose $1
FileClose 
$2

IntOp 
$$1
Goto exit

exit:
sectionend 
TIA
Chris
MyPC8MyBrain is offline   Reply With Quote
Old 7th September 2017, 19:27   #10
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Thumbs up

as usual as soon as i post for advice i find the answer a minute later,
i just had to clear errors first

ATB
Chris
MyPC8MyBrain is offline   Reply With Quote
Old 8th September 2017, 06:25   #11
JasonFriday13
Major Dude
 
JasonFriday13's Avatar
 
Join Date: May 2005
Location: New Zealand
Posts: 854
Interesting, you don't need 'push $3' and 'push $4', because you aren't popping them off anywhere. Same with IntOp $5 $5 + 1 ($5 isn't used anywhere). But I assume they are being used in your actual script?

"Only a MouseHelmet will save you from a MouseTrap" -Jason Ross (Me)
NSIS 3 POSIX Ninja
Wiki Profile
JasonFriday13 is offline   Reply With Quote
Old 8th September 2017, 19:43   #12
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Thumbs up

Thank you Jason
$3 and $4 are the lines passed from reading each of the lists,
$5 is leftovers from my test; i was sure it increments my next line

Ive actually made few more adjustments but i seem to be stuck atm,
the approach i used above works with one condition; that all files be present,
if one file is missing the whole script exits in the middle of uninstall instruction with external list (not what we want); i need to keep parsing through the list to the end regardless of file present in origin or not,

my latest attempt is to pass a blank line ""/NULL as the returning value to exit the list by using a blank line in from.txt

but that not working for me atm

from.txt content file looks like this
PHP Code:
CB\test1\image0.JPG
CB
\test1\image1.JPG
CB
\test1\image2.JPG
CB
\test1\image3.JPG 
to.txt content looks like this
PHP Code:
CB\test2\image0.JPG
CB
\test3\image1.JPG
CB
\test4\image2.JPG
CB
\test5\image3.JPG 
PHP Code:
section
ReadRegStr $INSTDIR HKCU 
"Software\..." Path
Open
:
FileOpen $"$INSTDIR\from.txt" r
FileOpen 
$"$INSTDIR\to.txt" r
IfErrors 
exit

        
loop:
        
FileRead $$3
        FileRead 
$$4
    StrCmp 
$"" closelist
        Rename $INSTDIR
\$3 $INSTDIR\$4
        Push 
$3
        Push 
$4
        
Goto loop
        
        closelist
:
        
FileClose $1
        FileClose 
$2
        
        IntOp 
$$1
        
Goto exit
        
        exit: 
ATB
Chris
MyPC8MyBrain is offline   Reply With Quote
Old 8th September 2017, 22:07   #13
MyPC8MyBrain
Member
 
Join Date: Nov 2015
Posts: 73
Smile

i settled on this for now
in from.txt last line is "end"

PHP Code:
        ReadRegStr $INSTDIR HKCU "Software\..." Path
        ClearErrors

        Open
:
        
FileOpen $"$INSTDIR\from.txt" r
        FileOpen 
$"$INSTDIR\to.txt" r
        IfErrors 
exit

        
loop:
        
FileRead $$3
        FileRead 
$$4
        StrCmp 
$"end" closelist
        Rename $INSTDIR
\$3 $INSTDIR\$4
        
Goto loop

        closelist
:
        
FileClose $1
        FileClose 
$2
        
Goto exit
        exit: 
ATB
Chris
MyPC8MyBrain is offline   Reply With Quote
Reply
Go Back   Winamp & SHOUTcast Forums > Developer Center > NSIS Discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump