NSIS Installer for big files (more 4 GB and zip)

[Jekano]

Hello,

I want to make a installer with one big file. The size of data more then 4GB. But NSIS can do file only <2GB now.

I try installer based on zip file. But its size more 4GB. When I try open file .zip I get "Error openning zip file". What kind error - I don't know.

Can I create one file installer with size more then 4GB? How do this?

Thanks!

[Afrow UK]

No you can't.

Stu

[atomjack]

I looked at this a few months back. Basically there is no good solution currently for installing large files. I ended up distributing a small installer together with a set of .tbz2 files and using the 'untgz' plugin to decompress the files: the 'untgz' plugin was the only one that I found that could handle files that were larger than 4GB.

If your individual files are smaller than 2GB then you can compress them into individual zip files (so long as no zip file holds more than 2GB of data you're OK). Then there are various unzip plugins which will work. However, there's no solution that will let you distribute a single >4GB installer.

[digthegoat]

Now that it is 8 years later, any way to do this? I have a file that is encrypted with a 10-15 character password and lost the password. Total file is something like 8 GB.

[Anders]

Now that it is 8 years later, any way to do this? I have a file that is encrypted with a 10-15 character password and lost the password. Total file is something like 8 GB.

You can use https://sourceforge.net/projects/nsisbi/ but I don't see how that is related to you losing a password.

[extremecarver]

Does the NSISBI work together with lzma2 for multicore support?
(without it creating installers will be very slow - as lzma supports only single core - it would be great to have this working in Nsis 3 together with a modern version of 7zip supporting lzma2 so creating the installer will be much quicker).

[JasonFriday13]

No, my NSISBI fork for external files does not add compressors, if I did I'll be adding lz4 for speedy compression because the output file can be huge.

The LZMA compression module has a special exemption from the 7-Zip developer for use in NSIS, I doubt lzma2 will be added any time soon.

[extremecarver]

okay - thanks. Would have been nice. I know most users don't mind. It's just that I create about 600GB of NSIS installers per week - and just running lzma for packing adds up lots of time (well I serialized it - but better would be to have it running on 8 cores instead of 1 core and running several compressors in parallel because that slows down the hdd more).

[Anders]

The LZMA compression module has a special exemption from the 7-Zip developer for use in NSIS, I doubt lzma2 will be added any time soon.

The lzma license has changed and you don't need the exemption now (IANAL)...

[extremecarver]

Just to pop this up once again. Any chance of including lzma2 into Nsisbi (well it would be nice in normal nsis too). Also I think the lzma module should be much quicker even using singlecore - guess it has not been updated long long time either.


For my testcase I get:
Compressor off 548MB
LZ4 489MB (just a tiny bit slower then compress off)
LZ4 465MB with SetCompressorDictSize 1: Very Slow - Comparable to BZIP2 speed. Really Useless Mode.
Zlib 420MB (a lot slower - but still way faster than lzma, a bit slower than simply compressing the whole data lzma2 with 7zip on a 4 Core processor with Hyperthreading - but yeah it's also single core use only)
BZIP 2: 420MB -- very slow
LZMA 377MB. very very slow.

So lz4 without Dictionary is kinda nice if compression/uncompression speed is really important - and as it's nearly as fast as no compression at all - it is attractive. On my dataset ZLIB is not doing too bad - considering it is still pretty fast - and gets the size down again a lot from LZ4.
LZMA is king however about compression ratio - and if NSIS/NSISBI could move to lzma2 that would at least on the compression side make the speed pretty attractive.


Note - on NSISBI there is still a problem in the log with choosing lz4 as Compressor:
SetCompressor: lz4
warning 8026: SetCompressorDictSize: compressor is not set to LZMA. Effectively ignored.




Another problem is - I cannot unextract/open installers created by Nsisbi with 7zip. That's really not good as my installers also serve as container for linux users to download and just unpack the data. It does not matter which compressor I choose - the problem exists as soon as I use NSISBI - if I go back to NSIS I can again unpack all installers.

[JasonFriday13]

That's because I have changed the first header in the exehead to support external files, so no you can't unpack them unless 7zip changes their code.

Also, SetCompressorDictSize is just two lines of code, one is the warning and the other just sets the value. All compressors get passed this value, only lzma uses it and I decided to use it for a different purpose (lz4 high compression).

[extremecarver]

I still think lzma2 (or xz with multithreading or any other lzma based compiler with multithreading support) would be great. Much faster packing and unpacking.
Yes lz4 is super fast - but for many people downloading files over 2GB is still slow - so lzma2 would be the best compromise for speed vs size (also unpacking of lzma2 is much faster than lzma). Maybe even replace lzma with lzma2 in nsisbi?

[extremecarver]

Just popping this up again. Any chance of moving to lzma2 (both Nsis and Nsisbi). Even most notebooks are now quadcore, 8 threads, and Nsis installers are painfully slow.

On my notebook the speed of unpacking via nsis7z for external files is about 10x the speed of nsis unpacking lzma. Packing is about 5x slower. I really cannot understand why NSIS and NSISBI cannot move to lzma2. Multi Processor support is soo soo needed. Also as it saves battery on notebooks. Notebooks are way more efficient using multicore over singlecore..

(I create every week about 400GB of nsis installers. and external files are just much more cumbersome compared to offering all in one file). For an average download speed of 50Mbit lzma2 is overall way faster than lz4 (time downloading and time unpacking combined). Yes if you have a 1000mbit line then lz4 is the answer. But very few people do (and it creates much more load on the server).

The exception to use lzma2 /lzma is not needed anymore AFAIK - so it would be great to upgrade from lzma to lzma2.

[Anders]

As most things in a open source project, lack of developer time and interest would be a big reason.

Inno Setup supports lzma2 AFAIK.

[JasonFriday13]

Yep. Anyone is welcome to have a crack at it and implement it, it doesn't have to be me or the nsis devs alone.

On a side note, there are multi-threaded versions of all 3 codecs used in nsis. But the hard bit is abstracting the threading out and putting it into a generic threading class for all three compressors to use (I have briefly looked at it in the past, as I'm running out of things to do on nsisbi).

But yes, time is my factor. Currently working 60 hours a week, and all I want to do on my days off is rest (I currently work in a hot factory, not an air-conditioned office like alot of programmers do).