NSIS 3.0 Breaks Dynamically Linked Plugin DLL?

[suwalski]

Hello,

I'm trying to upgrade from NSIS 3.0b2 to NSIS 3.0 and it seems that the changes to load DLLs more securely breaks my custom plugins.

My software and NSIS plugins are built with VS2013 Express. I install msvc[pr]120.dll to the program directory (running as ExecutionLevel user). In the past, the plugins extracted in the NSIS temp directory were able to load the DLL from the CWD (program install location), but now that seems to fail.

I don't have any way of backing this statement up, other than the installer fails on Windows 7 and 8, and works on Windows 10, which has msvc*120.dll in the Windows directory.

Any ideas?

Thanks,
--Pat

[suwalski]

I've just tried adding copies of msvcp120.dll and msvcr120.dll to $PLUGINSDIR using the "File" directive, and it now executes on Windows 7 and 8. It also, unfortunately, leaves a copy of msvcr120.dll in the temporary directory after the installer is complete.

Clearly, my hypothesis is correct.

The question now remains, is this a known change, or a bug? I see it as a pretty major regression that breaks binary compatibility with what's always worked in NSIS.

I can use my workaround for now, but I don't think it's very pretty.

[Anders]

Is your plugin .dll left in $pluginsdir as well? If so, the plugin is not unloading correctly.

Are you installing msvc*.dll in $instdir as well? If so you can use System::Call 'KERNEL32::AddDllDirectory(w "$instdir")' to allow .dll files to load from there.

If all else fails, statically link your plugins or use a old version of Visual Studio.

[suwalski]

Originally Posted by Anders

Is your plugin .dll left in $pluginsdir as well? If so, the plugin is not unloading correctly.

I've been looking for some documentation about unloading plugins. I don't do anything special, but I get left with nsResize.dll, UserInfo.dll, and one of my own plugins. The plugin written by me that is left is by far the simplest one, it's about 40 lines and just parses a string.

Looking for plugins with unload functionality, I see an empty unload function in NSISList, but not much more info than that. Any tips?

Originally Posted by Anders

Are you installing msvc*.dll in $instdir as well? If so you can use System::Call 'KERNEL32::AddDllDirectory(w "$instdir")' to allow .dll files to load from there.

Works like a charm! I saw some mention of it in the SF issue that tracked the modernized DLL loader, but I didn't think it could be called right from the script.

Thanks for the tips.

[suwalski]

Originally Posted by suwalski

I get left with nsResize.dll, UserInfo.dll, and one of my own plugins.

Fixing my own plugin involved adding a DllMain() function that saves the HANDLE hInstance, and then hooking up an empty PluginCallback() that returns 0. I guess this is necessary for successful unload.

I do still wonder why nsResize and UserInfo (used via MultiUser.nsh) doesn't get unloaded properly.

[suwalski]

Originally Posted by suwalski

I do still wonder why nsResize and UserInfo (used via MultiUser.nsh) doesn't get unloaded properly.

So, looks like UserInfo doesn't have a callback at all, and so it stays around.

nsResize only calls its RegisterCallback() function in the setRTL() function that isn't called by anything in its NSH file. Adding "nsResize::setRTL 0" in the installer script makes it unload cleanly and get deleted, proving this theory.

If it's a requirement that RegisterCallback() needs to be called reliably, then that information should probably be put out there to get these things fixed.

[Anders]

The old plug-in model works like this:

NSIS calls GetModuleHandle with LoadLibrary as a fallback, the plug-in function and FreeLibrary. If SetPluginUnload is used or /NOUNLOAD is specified, FreeLibrary is not called.

Some plugins provide a Unload or another "no operation" function to help with unloading if you are supposed to call them with /NOUNLOAD in some instances.

New model:
Plugin calls RegisterPluginCallback, this forces the plugin to stay loaded and NSIS will unload it after .onGuiUnload.



UserInfo should not be called with /NOUNLOAD IIRC. nsResize is a 3rd-party plug-in and I don't know anything about it.

[suwalski]

I'm simply stating that whether it's the old loader or new loader, plugins that don't register a callback apparently do not get unloaded cleanly, meaning that the files do not get deleted. This is easy to demonstrate.

So either:
- It is a bug that plugins without callbacks do not unload
- It is a bug that a plugin does not call RegisterPluginCallback

[Anders]

Originally Posted by suwalski

I'm simply stating that whether it's the old loader or new loader, plugins that don't register a callback apparently do not get unloaded cleanly, meaning that the files do not get deleted. This is easy to demonstrate.

So either:
- It is a bug that plugins without callbacks do not unload
- It is a bug that a plugin does not call RegisterPluginCallback

Works fine for me:

Section
UserInfo::GetAccountType
Pop $0
GetDllVersion "$SysDir\ntdll.dll" $8 $9
IntOp $1 $8 >> 16
IntOp $2 $8 & 0xffff
IntOp $3 $9 >> 16
MessageBox mb_ok "$0$\n${NSIS_PACKEDVERSION}$\n$1.$2.$3"
SectionEnd

gives me

User
0x03000090
6.2.9200

and $pluginsdir is deleted correctly.

Other reasons for things not unloading might include Anti-Virus, indexing and/or backup software holding locks on things.

[suwalski]

Okay, I won't argue. Thanks for looking.

For anyone finding this because of the dependency library problem I was having, I should add that AddDllDirectory needs to be called after files are installed. Since I'm using the Microsoft runtimes from $INSTDIR, when I call AddDllDirectory is called, $INSTDIR has to be there already or it will not take effect. For me, that meant the call happened right before the plugin was first used.

[Anders]

Originally Posted by suwalski

Okay, I won't argue. Thanks for looking.

If you don't get the same result as me, copy my code and post the output if it is different and give us some information about your Windows version and 3rd-party security software. You could also see if Process Monitor provides any clues. If you believe there is a bug then I'd like to get to the bottom of it...

And what happens if you use NSIS 2.46?

[suwalski]

FWIW, the unload issue was directly related to the "SetPluginUnload" flag being set to "alwaysoff". Of course, this was very difficult to notice when someone buried it years ago in an included file and not camel-cased.

Hopefully this thread proves useful for those getting stuck on the DLL load path.