http + ftp download plug-in

[Takhir]

Originally Posted by bsmith1

I know this was done on purpose since it is supposed to be "silent", however is there an option or a way to implement that the progress windows shows up, like a /NOSILENT option?

Yes, this is possible to implement, I even tested 'small' changes and found that for installer with /S dialog appears in the center of screen. Looks a bit ugly for silent install, IMHO would be better to move it to the right bottom corner. Or to left top. May be /NOSILENT /POSITION X:Y (percents, pixels?). So we need to understant a full set of options for this feature. Code update for this may take time (and I need to find this time in my schedule ).

[Necku]

/RESUME seems buggy for me.

If I enable it without the extra text parameter then the download errors with "connecting". If I add the extra text, then I properly get a pop up, however clicking retry seems to do nothing and the pop up just comes up over and again and the download never resumes, when clicking cancel I get "reget error". Any idea on what could be going on ?

-Using win7 64bit
-compiled non unicode script
-file is in direct download and support resuming (through firefox or download manager)
-no other options used for inetc
-tried disabling device or unplugging cable gives same result

Help appreciated!

[Takhir]

/RESUME RETRY_QUESTION - parameter is mandatory. On error during download plug-in asks you what to do - continue download or fail with rising some error (reget error in this case). If network cable still disconnected, Resume popup will appears again. Normal behavior is to restore connection (where is my VPN! ) and press Resume button after this only.

[Necku]

Thanks for the reply Takhir!

For the parameter: yes I was confused by the existence of a default value. Seeking for example I figured it was used if you use an empty string "" as parameter, problem solved.

However for the pop up, this is exactly my issue, the plugin seems to fail to detect my connection is available again. i.e. after plugging the cable back in and even if internet works perfectly, the pop up with retry/cancel error will keep showing up, refusing to restore the download.

[Necku]

Looks like I owe you an apology. Your files work perfectly. As the history page show (http://nsis.sourceforge.net/File:Inetc.zip) someone else applied modifications on your plugin which caused the bug, using the last version uploaded by yourself (16:04, 13 July 2010) works perfectly. I don't know why/who modified it but you might want to revert it.

Thanks for your awesome work

[Takhir]

Sorry, after reading code : Reget error for http means that InternetSetFilePointer() failed with (REST for ftp). So connection itself was already established, but server could not continue download from this position. Dynamic content names, unknown file size in reply, server functionality limitations - what else? Could you reproduce this on another server? On static content/file?

[Takhir]

Thanks, Necku, presisely, JohnTHaller added 2 flags "for cleanliness", but "This handle must not have been created with the INTERNET_FLAG_DONT_CACHE or INTERNET_FLAG_NO_CACHE_WRITE value set." Main idea from the very beginning was to add reget support. But I'll think a little is this a bug or feature . May be well need to rollback inetc version.

[Necku]

You're welcome, I'm happy as long as it works

On a side note I'll pm you the URL I used for tests so you can see by yourself

[Takhir]

Unfortunatelly I cannot login to sourceforge now (server internal error), so I am attaching patched version here for the moment. New flags will be used if resume option not set only. Necku, if you will have time, please test this

[Necku]

Originally Posted by Takhir

Unfortunatelly I cannot login to sourceforge now (server internal error), so I am attaching patched version here for the moment. New flags will be used if resume option not set only. Necku, if you will have time, please test this

Works like a charm

[cole2k]

First I'd like to say thanks for the plugin, it looks to be exactly what I've been looking for.

I am having a small issue with inetc::get it works for the most part, but on my server logs it's showing as POST instead of GET.

The version I'm using is the one you posted above on the 14th.

Example:

Section "Dummy Section" SecDummy
inetc::get /USERAGENT "Some Browser" \
"http://somesite.com/somepage.html" "$EXEDIR\somepage.html"
Pop $0
MessageBox MB_OK "Download Status: $0"
SectionEnd

[Takhir]

I re-tested 2 last versions and in apache logs I see:
127.0.0.1 - - [25/Feb/2011:19:45:35 +0300] "GET /aut_1986.htm HTTP/1.1" 200 17952
127.0.0.1 - - [25/Feb/2011:19:55:11 +0300] "GET /aut_1986.htm HTTP/1.1" 200 17952
for the following script:
inetc::get /useragent "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)" "http://localhost/aut_1986.htm" "$EXEDIR\aut.html" /end
In code I see 3 situations where POST may have place: 1) inetc:ost(); 2) /file ; 3) /NOUNLOAD in previous POST request. Any other ideas?

Latest release is here http://nsis.sourceforge.net/File:Inetc.zip

[cole2k]

Thanks Takhir,
I've tried your example with the latest version and it works fine. My problem appears to be with the agent string I've been trying to use.

Works fine:
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)

Causes POST:
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET CLR 1.1.4322; InfoPath.2)

[Takhir]

You are right, cole2k, string size limit 128 chars in current version for user agent. Most of strings in inetc allocated dynamically with size 1k or 8k depending on NSIS build, but not user agent. Most of servers support at least 8kB fields. I'll fix this in next release.

[yyv]

Dear sirs,
I’m trying to use your plugin with parameter/RESUME
Everything seems to work normally.
We started downloading, then disconnected the network cable, downloading broke and a message requesting to resume/cancel the downloading appeared. We connected the cable, clicked “resume” and downloading continued.
But actually, there is no real downloading. The plugin downloads everything from the start till the moment of break and then continues downloading.
For example, I’m downloading a 600 mb file.
When 200 mb are downloaded, I disconnect the network and then connect it.
The downloading process seems to continue but the progress bar doesn’t move ahead.
Once the plugin downloads those 200 mb once again he continues the further download.
Is this the way the plugin operates or is it an error?
What can we do to resume the downloading from the point of downloading break?
Thank you.

[klopfdreh]

Hello,

first of all thanks a lot for that great plugin!

I saw that there is a new release with this resume flag and I tried it by myself. I've the same issue as yyv.

In addition to that there are sometimes errors while downloading files that are bigger in size while there is kaspersky running on that client. The antivirus scanner trys to check the complete file after download and then the inetc.dll is running into a timeout which is not regarding due to an issue of the connection. This problem appears on several systems - windows xp / vista / 7.

Any ideas?

Thanks a lot again and hopefully we will se a lot of updates to this plugin in feature!

[Takhir]

I prepared small patch for reget feature, but we still have this problem on nginx server (apache is OK), so please wait few days.
No ideas about virus protection....

[klopfdreh]

@Takhir - what I also found out so far about the "kaspersky issue" is that downloads of the same files in browsers also take for about ~30 seconds till they are finished. During this time the progress bar and the percentage info stucks on 99%,too.

The delay until the Internet Explorer throws an error is much later then of inetc.dll. Maybe it is regarding how the WinInet lib is used of both (please correct me if I'm wrong).

What I have tried so far is to reset the browser settings. Reference to microsoft answer: Download stucks at 99% Solution - still the same issue.

Is there a delay not regarding an connection timeout that can be configured till the download will be aborted?

Thanks a lot again for your fast answer!

[klopfdreh]

I looked into your source code and there are a lot of other TIMEOUT settings that can be applied to "hinternet" - and in the past, some of them, weren't implemented.

InternetSetOption(hSession, INTERNET_OPTION_SEND_TIMEOUT, &timeout, sizeof(DWORD));
InternetSetOption(hSession, INTERNET_OPTION_DATA_SEND_TIMEOUT, &timeout,
sizeof(DWORD));
InternetSetOption(hSession, INTERNET_OPTION_RECEIVE_TIMEOUT, &timeout,
sizeof(DWORD));
InternetSetOption(hSession, INTERNET_OPTION_DATA_RECEIVE_TIMEOUT, &timeout,
sizeof(DWORD));
InternetSetOption(hSession, INTERNET_OPTION_CONTROL_RECEIVE_TIMEOUT,
&timeout, sizeof(DWORD));
InternetSetOption(hSession, INTERNET_OPTION_CONTROL_SEND_TIMEOUT, &timeout,
sizeof(DWORD));

The reason why I did this is, because I tried to increase the timeout and shut down my http server during the download. After that the error message appeared in exaclty the same time.

Maybe it is a bug?!

[Takhir]

May be. But MSDN says that DATA_SEND/RECEIVE_TIMEOUT related to ftp only. CONTROL_RECEIVE is identical to RECEIVE. So the only flag that can help with big files download via HTTP is INTERNET_OPTION_RECEIVE_TIMEOUT (probably no separation of HTTP header and data).
Currently plug-in can use not default INTERNET_OPTION_CONNECT_TIMEOUT only (if set in parameters - /timeout). I do not remember why it was added in 2005 , but IMHO this option is good. This way we can change IE default setting for connection t/o. If you can test RECEIVE_TIMEOUT flag and confirm that this resolves situation with antivirus (this may be true if Kaspersky catches data before Inetc), I'll add this new option.
Thanks, Takhir

[klopfdreh]

I tried various settings now in
HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings

ReceiveTimeout (REG_DWORD) = 300000
KeepAliveTimeout (REG_DWORD) = 300000
ServerInfoTimeout (REG_DWORD) = 300000

But I had no success, because I received the error message every ~3000 Milliseconds and not as expected ~5 minutes.

So I had a deeper look into the source code.

Please correct me if I'm wrong, but you are using a thread which is waiting if there is a transfer and if not it will abort after 3000 milliseconds, right?

Line: WaitForSingleObject(hThread, 3000) == WAIT_TIMEOUT

Could it be that the timeout which is going to be set via RECEIVE_TIMEOUT and CONNECT_TIMEOUT should influence this timeout, because the thread would exit to early?

In my opinion the changes of the RECEIVE_TIMEOUT would be only a little (~5 lines -> cmd line parsing and call in the function itself), so I would add it to the next version, to set it as needed.

P.S.: A few options are not mentioned in the wiki (for example: nocookies).

[Yathosho]

could it be the plugin ignores the default font rendering settings when used in /BANNER mode? it seems to ignore the fact i have cleartype enabled.

[klopfdreh]

Hello again,

@Takhir : Anything new to Yathosho and my objections?

[Takhir]

Originally Posted by klopfdreh

Please correct me if I'm wrong, but you are using a thread which is waiting if there is a transfer and if not it will abort after 3000 milliseconds, right?

Line: WaitForSingleObject(hThread, 3000) == WAIT_TIMEOUT

No, in this line plug-in waits transfer thread termination after Cancel button was pressed. On server termination DATA_RECEIVE_TIMEOUT probably defines time. If you tested InternetSetOption(hSession, INTERNET_OPTION_DATA_RECEIVE_TIMEOUT, &timeout,
sizeof(DWORD)); and this work as expected please let me know (or I'll test this when I'll have time - a lot of job now ). Both options - connection and receive timeouts have sence, so let's test and add new feature .

> P.S.: A few options are not mentioned in the wiki (for example: nocookies).
Thanks!

> could it be the plugin ignores the default font rendering settings when used in /BANNER mode? it seems to ignore the fact i have cleartype enabled.

Yathosho, please send me minimum size sample script and I'll test this. Only english/russian/italian systems are available now for me .

[klopfdreh]

@Takhir: Thank a lot for the fast answer again. Well I don't have the environment to compile the source . Would you please so kind and test it?

The way I did was simply to limit the transfer rate of the http server and shut it down, before the download was finished. The response always appears after ~3 Seconds, thats why I was supposing it could be the line I worte. And in the case of the virus scanner it also appears in ~3 Seconds, because the file is beeing scanned.

It would be really really nice, because we are using your lib to update clients automatically in an environment with ~1500 users.

I think it should be only a little change, just those lines that get the parameter out of the call and if the parameter contains the receive timeout the call of the internetSetOption.

[klopfdreh]

Now I tried to compile it with cygwin by myself and I got some errors during the compilation (picture)

@Takhir: Because I can't compile it easily - would you be so kind and add/change the following lines to inetc.cpp and recompile it? I think it should work that way:

line 208:

code:
timeout,

receivetimeout = 0;

line 799:

code:
if(receivetimeout > 0)

	InternetSetOption(hSes, INTERNET_OPTION_DATA_RECEIVE_TIMEOUT, &receivetimeout, sizeof(receivetimeout));

line 1374:

code:
else if(lstrcmpi(url, _T("/receivetimeout")) == 0)

{

	popstring(url);

	receivetimeout = _tcstol(url, NULL, 10);

}

That would be very very nice of you. If you could do this and post a new build in here I would check if that will solve our problem!

Edit: I was able to solve the error with the exdll by using the exdll.h and the api.h of the current source of nsis but the error in inetc.cpp on line 648 still appears.

[Afrow UK]

You don't need to use strtol (you don't want a dependency on CRT). NSIS exposes popint() and myatoi() that you can use.

Stu

[klopfdreh]

Ok, I solved it for my environment. I used dev-c (g++) within cygwin:

g++ -c inetc.cpp -I/usr/include/mingw -lwininet -lcomctl32
g++ -shared -o inetc.dll inetc.o -I/usr/include/mingw -lwininet -lcomctl32

and I used the exdll.h and api.h attached to the post.

@ Afrow UK: It is not my code. I only want to recompile and test the changes posted above.

Edit: Well my dll doesn't work! I hope I will get it running today.

Edit2: Mmhhh I don't know why it is not working for me, the result if I compile the dll is, that the return value is not "OK". Example:

inetc::get /NOCOOKIES /TIMEOUT 300000 /NOCANCEL /CAPTION "$(^Name)" /RESUME "..."

Pop $R0

MessageBox MB_OK $R0

$R0 is always the first paramter I give into the dll call....

[Takhir]

I am sorry... and thank for your efforts!
I am attaching build with your changes, please send me info if you need something else and I'll try to do this tonight.

[klopfdreh]

@Takhir: Thank you very very much! Maybe you are able to tell me what I did wrong with the compilation?

I don't know why it doesn't worked for me - even NSIS was compiling fine...

Edit: I tried the new version with the internetreceive timeout and if I have the kaspersky virus scanner installed, it trys to check the download after the tranfer and then aborts. I've tested it on a vmware box already...

Edit2: BUUUUUTTTT!!! I tested the ReceiveTimeout in the registry again and then the download worked for me. Seems that the internetSetOption with INTERNET_OPTION_DATA_RECEIVE_TIMEOUT doesn't apply it to the current connection.

Brrrr.... this smells like an ugly sh* but I'm going to set the dword previously:

WriteRegDWORD HKCU "Software\Microsoft\Windows\CurrentVersion\Internet Settings\" 'ReceiveTimeout' 0x001b7740

[Takhir]

Thanks, klopfdreh

Attached plug-in worked as expected (for me ).
Finally: INTERNET_OPTION_RECEIVE_TIMEOUT sets what we need. Interesting information from MSDN 2005: INTERNET_OPTION_DATA_RECEIVE_TIMEOUT 8 Not implemented.
Now Inetc options are /connecttimeout and /receivetimeout, both in seconds (not milliseconds!). I.e. 60 sec timeout will be /receivetimeout 60

And ... this not took many time after I understood how to test timeout - simple php script with sleep() . Included to zip. please let me know if big files problem resolved.

[klopfdreh]

Great! Thanks a lot! I'm going to test it without the reg key and with this new version, now!

Edit: It is working like a charm, even without the registry entry - thank you very very much!

I think we should add those changes to the wiki, too. It is a great change and really usefull!

[Takhir]

Thank you very much, klopfdreh! I'll add this patch shortly to archive.

[Yathosho]

Originally Posted by Takhir

Yathosho, please send me minimum size sample script and I'll test this. Only english/russian/italian systems are available now for me .

(screenshot based on your cameron diaz example)

is the banner using the system font? it should be antialiased according to my windows settings, but it's not.

[Yathosho]

btw, i have tried to use InetLoad with three switches (/TIMEOUT 5000 /RESUME /BANNER) but the download only works when using just one of these. the documentation and examples don't deal with multiple switches, is it even possible to chain these?

[zenpoy]

Hi all - I've noticed that when using embedded progress bar the text above the main progress bar is always printf("Downloading %s",filename), ignoring the caption switch. so I've changed the code a bit to make the text printf("%s",caption) if the caption switch exists.

If you think the this is really a bug you can merge it. Anyway, this is what the I thought the expected behavior would be.

Thanks,

J

[ManoKum80]

Feature Add Request -
- If download file already exists, check hash, if OK do not re-download
- Resume download even if user Cancels it, something like /NODELETE option on nsisdl v1.4
- on resume prompt, window title is 'Inetc plugin', it should be either removed completely or changeable via some option

Currently both above are available nsisdl v1.4 but it does not have authentication/proxy handling features that inetc can do.

PS - let me know if above features are already available and i am missing something.
Great thanks to all!

[MSG]

Originally Posted by ManoKum80

- If download file already exists, check hash, if OK do not re-download

This is actually trivial to implement in NSIS code. Use the md5dll plugin.

[zenpoy]

I think I found a bug in Inetc but I'm not sure - for some reason it clears the stack after cancelling. Does anyone see this behavior also?

EDIT: It clears the stack when clicking cancel and then it pops up an are you sure dialog - while it continues to download - if it finishes to download while the dialog is still open and then I press "Yes" (which means - yes, I'm sure I want to cancel) It clears the stack. If everyone see this behavior - I think it's pretty critical...

[Takhir]

Originally Posted by zenpoy

I think I found a bug in Inetc but I'm not sure - for some reason it clears the stack after cancelling. Does anyone see this behavior also?

inetc can download few files, so stack can include few pairs of URL/file, on termination plug-in should remove tail from stack. Best of all is to use /end paramater in your command line.