nsJSON plug-in

[daveb]

Just checking in again on these two problems in nsJSON....
this is a great plugin but very difficult to use as we keep hitting the limit and getting truncated/invalid json packets as responses.
These are the two problems detailed in my March 2022 post...

#1: 256 char limit to a POST response
#2: support for "RawOutput": false ...so that the "Output" node is proper json, not a string like this "{"key1":"value1", etc... }"

Would be great if these two problems in nsJSON could be fixed... anyone? Or recommend a different json plugin if this one is deprecated?

[Anders]

I tried contacting the author but no response so far.

[daveb]

Some progress to report and a workaround in my case.

1. the truncation problem remains, and appears to be a limit of 256 characters in the POST response that appears in the "Output" node.

2. Still getting empty ouput {} unless I set "RawOutput" to true. This gives the \" escaped quotes, but *provided* the string is not truncated (ie <256 characters) then "Output" is a wellformed string with a closing quote and can be loaded in then json nodes extracted from the response. Here's a summary of what I did... this can possibly be simplified but is definitely working within these limitations (mainly the 256 character response limit):


nsJSON::Set /tree HttpWebRequest /value `{ "Url": "${HttpWebRequestURL}", "Verb": "POST", "DataType": "JSON", "RawOutput": true }`
nsJSON::Set /tree HttpWebRequest "Headers" "Authorization" /value "Basic QQQQQetc"
;....etc for other headers...
nsJSON::Set /tree HttpWebRequest "Data" "MyPayloadKey1" /value `"$R0"` ; $R0 has some payload data
;....etc for other payload keys, then do the POST....
nsJSON::Set /tree HttpWebResponse /http HttpWebRequest

; check the response code
nsJSON::Get /tree HttpWebResponse "StatusCode" /end
Pop $R1
StrCmp $R1 "200" +1 badstatus

; extract the "Output" node as POST response
nsJSON::Get /tree HttpWebResponse "Output" /end
Pop $R1
nsJSON::Set /tree TheResponse /value `$R1`

; now we can look at the values of response keys like this
nsJSON::Get /tree TheResponse "Message" /end ; this is the "Message" node in TheResponse
Pop $R5
DetailPrint `Here is TheResponse.Message node: [$R5]`

nsJSON::Get /tree TheResponse "AnotherResponseKey" /end
Pop $R6
DetailPrint `Here is TheResponse.AnotherResponseKey node: [$R6]`

;....etc for other response nodes


= = = = = =
Note: I would have thought that providing a [Nodepath] as in the documentation would work, instead of first loading the "Output" node into another tree.
Maybe that's due to dealing with the "RawOutput"? but this doesn't work...

nsJSON::Get /tree HttpWebResponse "Output" "Message" /end
Pop $R1

...giving an empty string in $R1. I'm guessing that the problem is that the actual "Output" node is a quoted and escaped string, not a json node even though it converts nicely to json when handed in to this line of code...

nsJSON::Set /tree TheResponse /value `$R1`

So that's a summary of my workaround for the moment.
Would be great if these two problems in nsJSON could be fixed...
#1: 256 char limit to a POST response
#2: support for "RawOutput": false ...so that the "Output" node is proper json, not a string like this "{\"key1\":\"value1\", etc... }"

[daveb]

I also have been trying to hit a simple api with a json POST and get a json response.
I experimented for quite a while and was always getting "Output":{} as reported by Minother.
Testing was based on using Postman with the same json POST to the same server, and it was giving a proper json response.

At last I tried setting "RawOutput": true
This now gives me something closer... at least the Output node is now a string that resembles the correct/desired json response.
Two catches...
1. All the quotes (") in the Output node are now escaped as \"
2. Even when I try to remove these \ characters, it isn't quite right.
It appears to be truncating the first of what should be two subnodes, and the {} are now mismatched.

Here's the (edited) sample output with the \" turned into " and some linefeeds added to make reading easier....

{
"Output": "{
"License":{
"SyncTime":"2022-03-25T00:43:34",
"Status":"Active",
"Expiry":"2022-05-02T00:00:00",
"UserID":"900000",
"Level":"-16641",
"MaxVer":"3072",
"AccNum":"XXXXX",
"AccName":"YYYYYYYYYY YYYY",
"AccEmail":"[email protected]",
"PCFinger":"4380FB1YYYYYYY",
"StatusCode": 200
}

Notice "Output": "{ etc... there is a spurious " before the opening {, and
notice also, after "PCFinger":etc, it is just truncated and missing the rest of this node plus the whole of the next node.

I really would like to read the response as json - it is valid json when I get it in a test using Postman. Certainly would be better to get the response as json, and be able to get subnodes etc. But as reported by Minother above, if I turn off RawOutput, I get an empty Output node with a correct "StatusCode": 200 in the response

Any help appreciated.. I am new to using this nsJSON plugin which looks to be perfect for the job needed, if I could tame it...

[Anders]

I don't think storing a small JSON file on a SSD is going to hurt it much. Windows writes a lot more to the drive just in daily use.

[Minothor]

From a quick look at the code, it looks like you can set "Agent" to whatever you want. Perhaps Github wants a "Mozilla" in there? From a quick test in another tool it looks like Github does not care much about the UA.

Do you have access to a web server when you can put a json file for testing? I have never used this plug-in to read directly from the web so I don't know much about that part. As a last resort, use INetC to download to $pluginsdir and then parse that file.

Sadly I do not have a webserver handy myself, I could set up a test on localhost in a pinch if needed.

I'm leaning towards the INetC plugin too, since we already use it to download the binaries, but I'm unsure how to parse the body of a GET response as a string rather than persisting it on disk.
(I'm refactoring the installer away from hardcoded values that have to be manually updated and trying to minimise needless disk writes since a lot of our users are likely to be running SSDs as their primary drive.)

Thanks for bearing with me on this, it's been an interesting but long overdue first foray into NSIS.

looks like it may be a similar issue to the ones Ahmett linked in Post 64.
Swapping to that version gave me partial output, but it looks like the response is getting cut off part way through the "assets_url" node of the reply.

I'll keep digging.

[Anders]

From a quick look at the code, it looks like you can set "Agent" to whatever you want. Perhaps Github wants a "Mozilla" in there? From a quick test in another tool it looks like Github does not care much about the UA.

Do you have access to a web server when you can put a json file for testing? I have never used this plug-in to read directly from the web so I don't know much about that part. As a last resort, use INetC to download to $pluginsdir and then parse that file.

[Minothor]

This seems wrong to me. What is `Output`? I think you need to break this down into smaller steps and confirm each little step is working. Focus on getting a correct value into $arraySize before moving on.

Hi, thanks for responding so quickly, the reason for "Output" being part of the path was due to that first segment of code I linked*, it's the JSON tree associated with the Async response.
It appeared as though the tree had the body of the response under a node named "Output", so I prepended that to the path and tested again.

As it currently stands, both Async or regular HTTP GET requests to that URL seem to produce a 200 (OK) response but an empty body.
(even when serializing the tree immediately upon completion of the request)

So far, I haven't been able to replicate that behaviour in a browser or using the Rested Firefox plugin to replicate the User-Agent and headers that nsJSON is sending - both the browser and the plugin return the expected information from GitHub's API.

At this point I can only assume it's either some bug or I've overlooked something particularly silly, hence coming here to ask persons more knowledgeable and familiar with both NSIS and this plugin.

*A big thanks to whoever edited my post to fix the code blocks by the way, I couldn't find an Edit option available to me.

[Anders]

nsJSON::Get /tree asyncHttpWebResponse /count `Output` `assets` /end

This seems wrong to me. What is `Output`? I think you need to break this down into smaller steps and confirm each little step is working. Focus on getting a correct value into $arraySize before moving on.

[Minothor]

Could someone point me in the right direction on this?

I'm trying to load JSON from GitHub's API, but the response I'm getting in the output file is:

{
    "Output": {
        
    },
    "StatusCode": 200
} 

Comparing the Empty output to the expected result of an unauthenticated GET request: https://api.github.com/repos/HoboVR-...eleases/latest

Leaves me confused as to where exactly I'm going wrong.

I've read and re-read the ReadMe and HttpWebRequest.nsi example file, I've checked my var, tree names and paths and like the Response body - I'm drawing a blank.


Test Script:

!include nsDialogs.nsh
!include MUI2.nsh
!include "winmessages.nsh"
!include LogicLib.nsh
!include "Locate.nsh"
!include registry.nsh
!include x64.nsh

;!define LATEST_RUN "https://api.github.com/repos/HoboVR-Labs/hobo_vr/actions/runs?per_page=1&branch=master&event=push&status=success"
;"hobovr-build-windows-latest"

Name "HoboVR"

!define httpWebRequestURL "https://api.github.com/repos/HoboVR-Labs/hobo_vr/releases/latest"

!insertmacro MUI_LANGUAGE English

Var windowsBinaryURL
Var currentIndex
Var arraySize
Var nodeContentType

Section
; [url]https://api.github.com/repos/HoboVR-Labs/hobo_vr/releases/latest[/url]
; -> assets -> [COUNT] -> iterate:
; "application/x-ms-dos-executable"

  ;Get the latest successful Run

    nsJSON::Set /tree httpWebRequest /value `{ "Url": "${httpWebRequestURL}", "Verb": "GET", "Async": true , "Decoding": true , "AccessType": "PreConfig" }`
    DetailPrint `Getting latest build from: ${httpWebRequestURL}`
  ;  DetailPrint `Download: ${HttpWebRequestURL}`
    nsJSON::Set /tree asyncHttpWebResponse /http httpWebRequest

    ; Wait until done.
    ${Do}
      Sleep 1000
      nsJSON::Wait httpWebRequest /timeout 0
      Pop $R0
      ${If} $R0 != wait
        DetailPrint `Request complete!`
        ${Break}
      ${EndIf}
      DetailPrint `Waiting...`
    ${Loop}

  ; Get the Artifacts array
  nsJSON::Get /tree asyncHttpWebResponse /count `Output` `assets` /end
  Pop $arraySize

  StrCpy $currentIndex 0

  IntCmp $arraySize 0 artifactNotFound artifactNotFound

  DetailPrint "$arraySize compiled binaries found..."

  checkNextArtifact:
  DetailPrint `Checking Artifact at Index: $currentIndex...`
  nsJSON::Get /tree asyncHttpWebResponse `Output` `assets` /index $currentIndex `content_type` /end
  Pop $nodeContentType

  DetailPrint "Artifact ContentType: $nodeContentType"
  StrCmp $nodeContentType "application/x-ms-dos-executable" artifactFound

  IntOp $currentIndex $currentIndex + 1
  IntCmp $currentIndex $arraySize artifactNotFound checkNextArtifact

  artifactFound:
  nsJSON::Get /tree asyncHttpWebResponse `Output` `assets` /index $currentIndex `browser_download_url` /end
  Pop $windowsBinaryURL
  DetailPrint "Found binary: $windowsBinaryURL"
  Goto end

  artifactNotFound:
  DetailPrint `No Binary could be found.`
  CreateDirectory $EXEDIR\Output
  nsJSON::Serialize /tree asyncHttpWebResponse /format /file $EXEDIR\Output\asyncHttpWebResponse.json
  DetailPrint `Response saved to: $EXEDIR\Output\AsyncHttpWebResponse.json`

  end:
  DetailPrint `Test Finished.`

SectionEnd 

[T.Slappy]

@Afrow: In the most recent version there is a small bug:

If there is an empty array like this:

{
"ChunkDbs": [],
"FormatVersion": 0
}

The Json (read from file in my case) is neved read correctly and the Error flag is set, however empty arrays are perfectly correct in Json.


Edit:

Quick fix is to update the function with another if condition:

static struct JSON_NODE* EatNode(PTCHAR pszBuffer, int* piPos, BOOL bIsValue)

...
        else if (EatChar(pszBuffer, piPos, TEXT('[')))
        {
            pNode->eType = JNT_ARRAY;
            if (EatChar(pszBuffer, piPos, TEXT(']'))) // <<<< Start: This fixes empty arrays
            {
                pNode->eType = JNT_VALUE;
                pNode->pszValue = TEXT("null");
            }                             // <<<< End fix
            else
            {
                pNode->pValue = EatNodeArray(pszBuffer, piPos);
                if (!pNode->pValue || !EatChar(pszBuffer, piPos, TEXT(']')))
                {
                    JSON_Delete(&pNode, NULL);
                        return NULL;
                }
            }
        }
... 

[ahmett]

some more detail for an ANSI setup;

NsJSON v1.1.0.6 is right version for these;
nsJSON::Set /file /Unicode (Don't work in v1.1.1.0)
nsJSON::Serialize /format /file /Unicode (Don't work both in v1.1.1.0 and 1.1.0.9)

[ahmett]

It fails to read Unicode JSON files in ANSI setup! (/Unicode tag only works in Unicode setup builds)

https://nsis.sourceforge.io/File:NsJSON.zip

I've now checked the previous version and it does not have this problem!

[Anders]

Can't handle empty objects?

${IfNotThen} ${FileExists} "$%TEMP%\test.txt" ${|} ExecWait `"$sysdir\cmd.exe" /C echo.{"node1":{node122:{x:0}}} > "$%TEMP%\test.txt"` ${|}
ClearErrors
nsJSON::Set /file "$%TEMP%\test.txt"
${IfThen} ${Errors} ${|} MessageBox mb_ok Err ${|}
nsJSON:elete 'node1' 'node122' /end
nsJSON::Set 'node1' 'node122' /value '{deleteme:0}'
nsJSON:elete 'node1' 'node122' 'deleteme' /end
nsJSON::Set node3 node31 node313 node3131 a31313 /value "Z1Z1Z1"
nsJSON::Serialize /format /file "$%TEMP%\test.txt"

This works the first time, displays error if you run it again because node122 is {}?

[r2du-soft]

nsJSON get value

hi
i read a json value from variable but i have some problems:

this code can't show result value address:

StrCpy $0 '{"status":"ok","username":"rose","password":"software","expire":"2020","used_time":[],"address":"http:\/\/google.com\/apps"}'



nsJSON::Set /value $0

nsJSON::Get "address" /end
Pop $R0
DetailPrint $R0

but this code show result value address:

StrCpy $0 '{"status":"ok","username":"rose","password":"software","expire":"2020","used_time":"[]","address":"http:\/\/google.com\/apps"}'



nsJSON::Set /value $0

nsJSON::Get "address" /end
Pop $R0
DetailPrint $R0

The difference in codes is [] and is "[]"

my problem is the [] , if the value is entered in double quotation "[]" the result successfully displayed
but if in variable not insert [] to quotation then result can't successfully displayed