Working with Newline delimited JSON using can-ndjson-stream in TypeScript
What is Newline delimited JSON?
ndJSON is basically a collection of JSON objects separated by `\n` (new line character — hence the name). Each line is a valid JSON object.
ndJSON is primarily used for streaming or accessing large JSON data where we need the flexibility to access the data without having to parse the whole object first. For example when JSON data is send from the server we have access to the information with each line that is being send and can already respond to it since each line is an independent JSON object.
Recently I had to deal with ndJSON in a React app when making use of the Lichess API — which streams all the responses in the ndJSON format.
To read the ndJSON stream I used the can-ndjson-stream package from canjs. You can find it here. The only downside is that I was using TypeScript and the library is written in plain JS but it’s quite easy to create a type for it since all it does it returns a reader with a read() function that returns a Promise which holds the data and a done variable.
All you need to do then is pass whatever response you get into the ndjson reader which will be able to parse every line until the done variable is false which means it reached the end of the stream.
In my case, the API I was working with streamed the responses as ndJSON for each event that was send from the API. All I had to do was pass the data to the reader and then keep looping through new lines that were send until it reached the end.
Here’s an example of a function that makes use of the can-ndjson-stream reader to parse ndJSON data :
More information about the ndJSON format at http://ndjson.org
