Old 29th March 2014, 00:33   #1
arwa
Junior Member
 
Join Date: Mar 2014
Posts: 5
Out of range #12345 with mmap, at only 1.4 GB

Hi,

We're using NSIS for creating the installer for a software which includes two relatively large files, around 700 MB each. I love this software, btw.

Until now everything has worked perfectly. But now the software is updated and requires a new installer, and the files have grown slightly (about 710 MB per file) and then I get the NSIS build error:

"Internal compiler error #12345: error mmapping file (1447194380, 33554432) is out of range."

The installer obviously isn't hitting the theoretical 2 GB .exe limit here but it's something else at work.

I get the exact same error on Windows XP, Windows 7 and Windows 8, and I've tried it on multiple computers, file systems and %temp% folder locations. There's plenty of space on all drives and RAM usage is negligible. The same error occurs also using the native 64-bit prerelease of NSIS.

We cannot use compression, for technical reasons (file watermarking) so there are not many options to try. We don't want to use multiple files, because it's a downloadable software, and since we can't compress the file we can't use two files and put them into a single .zip archive either.

I was going to have a go at producing a fix for this myself, but I was unable to make the NSIS source code build... Normally I only work with Visual Studio and XCode so I'm a bit impaired when it comes to command line building and anything Python or make/scons. I managed to build a 3rd party NSIS "large installer" mod found here:

https://github.com/ywjheart/Nsis64

as it provided a Visual Studio solution. Curiously, this mod has this problem solved and correctly produced a 1.4 GB installer. But unfortunately we had other problems with that version, and we had to take out parts of our script to make it build (but size-wise it worked fine).

So... sorry to bother you with this, and I guess I'm asking here sort of as a last resort, since I'm not really sure how to proceed now. This is quite maddening, since we use some very specific features in NSIS which we cannot find in other programs. What would you recommend a safe setup to get the NSIS project to build? Would you recommend setting up a separate system with Visual C++ Express 2005 or some other version? Is there a preference for a certain Windows version?

On the other hand, I wonder if this is an easy fix. I only briefly looked at the mmap.cpp file, and the first thing that struck me was that SIGNED integers was being used for indexing and file sizes, although Windows file mapping functions uses unsigned integers (DWORD) and should be able to, theoretically, map files up to 4 GB even in a 32-bit process. And the error above is triggered by the condition:

if (!m_iSize || offset + size > m_iSize)

which would produce a false error if m_iSize is larger than 2GB (or 2,147,483,648) because m_iSize becomes negative. And I don't think this would trigger any error earlier when mapping the file, because the Windows mapping functions casts the size parameter as a DWORD, unless I'm mistaken?
arwa is offline   Reply With Quote
Old 29th March 2014, 01:26   #2
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,169
Makensis can not map more than 2GB because it does not set the large address aware flag. If that PE flag is set you could get 3GB on a 32bit OS and 4GB on WOW64. Even then you will have to deal with fragmentation and might not be able to map a > 1GB chunk in the real world. I believe NSIS maps small chunks so maybe we can work around the signed issue. I don't remember what m_iSize is but I'll take a look tomorrow.

Even if everything is changed to unsigned the final combined limit would still be 2GB because the top bit is used to store some compression information.

If you need help building you might need to start a separate thread. There is no Windows version requirement but getting scons to pickup the compiler and SDK paths can be tricky sometimes.

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Old 29th March 2014, 09:52   #3
arwa
Junior Member
 
Join Date: Mar 2014
Posts: 5
Quote:
Originally Posted by Anders View Post
Makensis can not map more than 2GB because it does not set the large address aware flag. If that PE flag is set you could get 3GB on a 32bit OS and 4GB on WOW64. Even then you will have to deal with fragmentation and might not be able to map a > 1GB chunk in the real world. I believe NSIS maps small chunks so maybe we can work around the signed issue. I don't remember what m_iSize is but I'll take a look tomorrow.

Even if everything is changed to unsigned the final combined limit would still be 2GB because the top bit is used to store some compression information.

If you need help building you might need to start a separate thread. There is no Windows version requirement but getting scons to pickup the compiler and SDK paths can be tricky sometimes.
Thanks for getting back to me so quickly!

That sounds very reasonable, requiring the large addr. flag. And the final installer wouldn't use more than 1.5GB at most, so it's only a question of being able to make it build.

I don't think memory management is an issue as you say, chunk processing seems to work, as the makensis.exe process doesn't appear to use any more than 50-100 Mb at a single time. And since the problem occurs at the same byte on three independens systems its probably not a real world/defragmentation issue either in this particular case.

We've setup a new system for building NSIS, and its working better this time. Although for the moment we're stuck with a "zlib (win32) is missing" error, although everything appears to be setup correctly with environment variables (trying three different zlib versions). But we should follow your advice and make that a separate thread if we can't resolve it.

Thanks again!
arwa is offline   Reply With Quote
Old 29th March 2014, 20:53   #4
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,169
I can generate a setup with two 999mb files or one 1.34gb + one 660mb and these installers end up as 1.99gb exe files. Once I go past that I hit your problem.

Quote:
setcompress off
setdatablockoptimize off # I'm using sparse (zero filled) files
CRCCheck off

Section "Test"
Goto end
File $%temp%\sparse1.tmp
File $%temp%\sparse2.tmp
end:
Quote:
SetCompress: off
SetDatablockOptimize: off
CRCCheck: off
Section: "Test"
Goto: end
File: "sparse1.tmp" 1447194381 bytes
File: "sparse2.tmp" 692060160 bytes
SectionEnd


Processed 1 file, writing output (x86-ansi):
Processing pages... Done!
Removing unused resources... Done!
Generating language tables... Done!

Output: "C:\Users\Anders\AppData\Local\Temp\Test.exe"
Install: 1 page (64 bytes), 1 section (2072 bytes), 4 instructions (112 bytes), 1 string (563 bytes), 1 language table (198 bytes).

Using zlib compression.

EXE header size: 34304 / 35840 bytes
Install code: 2381 / 2377 bytes
Install data: 2139254549 / 2139254549 bytes

Total size: 2139291234 / 2139292766 bytes (99.9%)
Switching to unsigned types is painful since you have to modify IGrowBuf and IMMap and if you insist on using SetCompress off then this is probably not going to help you get under the 2gb final limit...

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Old 30th March 2014, 00:32   #5
arwa
Junior Member
 
Join Date: Mar 2014
Posts: 5
After some experimentation I've found out that the problem happens only if you call "File.." twice on one of the files, even with SetDatablockOptimize on (which we use). We include the same file multiple time, on purpose, because the file is a plug-in which the user may actually want to install to multiple locations at the same time, on the same system.

So it seems the 2GB limit is counted against the sum of all files included using "File..." in the script, including possible duplicate file includes that use the same address space in the produced .exe. And this is why we cannot build our 1.4 GB installer in this particular case.
arwa is offline   Reply With Quote
Old 30th March 2014, 04:29   #6
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,169
I added some code to trunk that should fix this for "setcompress off" and "setcompressor /solid ...", non-solid has to rely on compression to get you under the limit...

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Old 30th March 2014, 10:57   #7
arwa
Junior Member
 
Join Date: Mar 2014
Posts: 5
That fast, really? That's fantastic. Thank you so much! I'm going to make sure we can get the source to build, and let you know when I'm able to try it out with our installer script!
arwa is offline   Reply With Quote
Old 31st March 2014, 11:13   #8
arwa
Junior Member
 
Join Date: Mar 2014
Posts: 5
Thumbs up

Ok, great news. I managed to get the build working on a W7 machine instead of the XP machine where I couldn't make zlib work (probably user error on my behalf, but now it works anyway) and I managed to produce an NSIS version which builds our ~1.5 GB installer correctly, it seems.

The "bad" news (not really that bad) is the fix didn't work initially due to a syntax error. The overflow detecting code:

if (this->build_optimize_datablock && st + sizeof(int) + length < 0)

is always false, because of the sizeof() operator. In C++ the sizeof() operator is unsigned, which in turn will automatically cast those kind of expressions (st + sizeof(int) + length) < 0 as unsigned comparisons. So they always return false for less-than-zero comparisons.

There appear to be similar situations in NSIS on some other places. For example in build.cpp, row 868. Although I haven't tried it out in that case I strongly suspect also that code will never kick in because it also uses < 0 as a way to detect a possible int overflow, and it includes a sizeof() value in the comparison. It could be compiler dependent though. But I suspect it to be a bug which could result in inexplicable "#12345 deflateInit() failed" errors when creating large compressed installers.

The easiest way to fix this in the NSIS source is to always cast the left hand operator as (int) when comparing to < 0. That works as long as you don't try to add files > 2 GB to an installer (I know this isn't supported, but anyway) because then the integer will flip around and become positive again, and possibly produce a crash. For future/large file compatibility I would suggest to use a 64-bit long long int, and do a comparison with a lot of headroom instead, for example:

if (this->build_optimize_datablock && (st + (long long)sizeof(int) + length) > INT_MAX)

For the record, a simple find/replace of all integers with unsigned integers, in mmap and growbuf, also produced an NSIS version which dealt fine with our larger installer. There where a few obvious things that had to be corrected (such as not using < 0 for detecting overflows, and using %u instead of %d in output) but other than that NSIS seemed to work happily with considerably more than 2 GB of data, temporarily, at least as long as the optimized .exe was less than 2 GB in the end, as in our case. But I'm sticking to your solution as it works great for us already, and doesn't feel as intrusive or error-prone.

The only "quirk" is that the NSIS output stats no longer shows correctly how much data the datablock optimizer saved. It says ~0.0%.

Thanks again for all the help! We really appreciate it, and made a little donation to you guys as well and hope it helps keeping the project running or pay for some beers.
arwa is offline   Reply With Quote
Old 31st March 2014, 16:18   #9
Anders
Moderator
 
Anders's Avatar
 
Join Date: Jun 2002
Location: ${NSISDIR}
Posts: 5,169
Yeah I was just following the rest of the code with the sizeof but we can just remove it and lose 4 bytes in the process.

The optimizer stat can easily overflow for big files, I'll see if I can change it to 64 bit...

IntOp $PostCount $PostCount + 1
Anders is offline   Reply With Quote
Reply
Go Back   Winamp & Shoutcast Forums > Developer Center > NSIS Discussion

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump