There are two separate problems here.
1) The file you are upgrading does not have to be registered, it does not contain the dllunregisterserver entry point. Regsvr32 reports this too and does not register the file (read the regsvr32 message carefully, it says that it can load the dll, but there is no entry point). You should define UPGRADEDLL_NOREGISTER.
2) There are interface problems with the Modern UI Welcome page. It looks like NSIS failed to create a temporary plug-ins folder. But if there would be a problem with the temporary folder, NSIS should report that. Are you sure you are using the latest development version (see http://nsis.sf.net