File Size limit when using File Read?

99999999 · 18th February 2008, 18:22

So I have a problem when using FileRead. The problem appears to be a file size limit when using FileRead. For each of the targets I am building, I have a command file, which is read through and parsed, and that command file tells the installer how to extract files from external zip files.

Keep in mind that these targets can have anywhere from 20 to 200,000 files, and be as large as 20gb. The problem I've encountered, is that when the target has a large number of files, the command file wil grow accordingly, and when the installer then attempts to call FileOpen, and then FileRead, FileRead fails and returns me 0 data, on the first read of the file. This work perfectly with targets that have much smaller sets of files. An example of a failing target would be a target that has 10k of files, and it creates a command file of 500kb.

So does NSIS have a a buffer it reads the entire file into? If so what is the length of the buffer? I can certainly split the command file into multiple files. By the looks of the command I was expecting maybe a windowed buffer, or straight reads from the file itself.

Anders · 18th February 2008, 18:40

FileRead reads one line at the time, so unless you have lines longer than 1024 bytes (buffer size limit for normal nsis builds), I don't see how you can have problems with files smaller than 2GB (or 4GB, not sure what the limit is)

Afrow UK · 18th February 2008, 19:27

If the 1024 limit isn't enough, you can try the special build (8192): http://nsis.sf.net/Special_Builds

Stu

99999999 · 18th February 2008, 19:32

Yeah, its not the length of the lines, I already do use less than 1024 characters per line, ( I checked,) I can't see any control characters either.

I am using version 2.24 of NSIS, with the big strings patch.
So I've checked and the file handle is valid, on both calls to FileRead.

18th February 2008, 18:22	#1
99999999 Junior Member Join Date: Nov 2006 Posts: 43	File Size limit when using File Read? So I have a problem when using FileRead. The problem appears to be a file size limit when using FileRead. For each of the targets I am building, I have a command file, which is read through and parsed, and that command file tells the installer how to extract files from external zip files. Keep in mind that these targets can have anywhere from 20 to 200,000 files, and be as large as 20gb. The problem I've encountered, is that when the target has a large number of files, the command file wil grow accordingly, and when the installer then attempts to call FileOpen, and then FileRead, FileRead fails and returns me 0 data, on the first read of the file. This work perfectly with targets that have much smaller sets of files. An example of a failing target would be a target that has 10k of files, and it creates a command file of 500kb. So does NSIS have a a buffer it reads the entire file into? If so what is the length of the buffer? I can certainly split the command file into multiple files. By the looks of the command I was expecting maybe a windowed buffer, or straight reads from the file itself.

18th February 2008, 18:40	#2
Anders Moderator Join Date: Jun 2002 Location: ${NSISDIR} Posts: 2,779	FileRead reads one line at the time, so unless you have lines longer than 1024 bytes (buffer size limit for normal nsis builds), I don't see how you can have problems with files smaller than 2GB (or 4GB, not sure what the limit is) IntOp $PostCount $PostCount + 1

18th February 2008, 19:27	#3
Afrow UK Moderator Join Date: Nov 2002 Location: Birmingham, England Posts: 8,202	If the 1024 limit isn't enough, you can try the special build (8192): http://nsis.sf.net/Special_Builds Stu Need an installer? http://www.afrowsoft.co.uk

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

18th February 2008, 19:32	#4
99999999 Junior Member Join Date: Nov 2006 Posts: 43	Yeah, its not the length of the lines, I already do use less than 1024 characters per line, ( I checked,) I can't see any control characters either. I am using version 2.24 of NSIS, with the big strings patch. So I've checked and the file handle is valid, on both calls to FileRead.