Combine Multiple JSON Files Json.NET -
i have api receives json calls push files (800kb-1mb) (1 each call), , have hourly task takes of json files in last hour , combines them single file make better daily/monthly analytics on.
each file consists of collection of data, in format of [ object {property: value, ... ]. due this, cannot simple concatenation it'll no longer valid json (nor add comma file collection of collections). keep memory foot-print low possible, looking @ following example , pushing each file stream (deserializing file using jsonconvert.deserializeobject(filecontent); however, doing this, end collection of collection well. have tried using jarray instead of jsonconvert, pushing list outside of foreach provides same result. if move serialize call outside foreach, work; however, worried holding 4-6gb worth of items in memory.
in summary, i'm ending [ [ object {property: value, ... ],... [ object {property: value, ... ]] desired output [ object {property: value (file1), ... object {property: value (filen) ].
using (filestream fs = file.open(@"c:\users\public\documents\combined.json", filemode.createnew)) { using (streamwriter sw = new streamwriter(fs)) { using (jsonwriter jw = new jsontextwriter(sw)) { jw.formatting = formatting.none; jarray list = new jarray(); jsonserializer serializer = new jsonserializer(); foreach (ilistblobitem blob in blobcontainer.listblobs(prefix: "sharepointblobs/")) { if (blob.gettype() == typeof(cloudblockblob)) { var blockblob = (cloudblockblob)blob; var content = blockblob.downloadtext(); var deserialized = jarray.parse(content); //deserialized = jsonconvert.deserializeobject(content); list.merge(deserialized); serializer.serialize(jw, list); } else { console.writeline("non-block-blob: " + blob.storageuri); } } } } } 
