Read big JSON files?

[thesoftwarephilosopher]

Bug Report

🔎 Search Terms

"large json files"

Problem

None of the predefined sections are relevant here.

I'm trying to get types from a JSON file using resolveJsonModule. I'm using VS Code to do everything here. My JSON file is 4.9 MB and 200k lines. When I try to inspect the result of require('./big-file.json), it says {}. But it works great for another smaller file, albeit a bit slow.

Not sure if this is relevant or not, but it also fails for this big file in quicktype and works for the slightly smaller file. I assumed they're using TypeScript's parser/compiler under the hood, which would explain it. If they're not, then maybe it's a Node.js limitation that it's coming across.

To be clear, the require call itself works fine and returns a valid JSON object. So it might not be a Node.js memory limitation issue. But it might be one if TypeScript is more than doubling the memory by all its type analysis, or by duplicating substrings, etc.

Just some thoughts. I have no idea what's going on. All I know is that I was hoping VS Code would analyze my JSON files for me, but it's only working on some files.

[RyanCavanaugh]

Not sure how quicktype works.

Our JSON support involves using the normal type system mechanics and is going to not do well on huge huge huge files. I'd really recommend generating a .d.ts file from a smaller subset of it or writing a manual definition if your file is that big; there's not much we can do to improve performance in this particular case.

[thesoftwarephilosopher]

That's pretty much what I did, manually figured out the types at runtime and wrote them down in a .d.ts file. So never mind I suppose. Thanks.

[jzxchiang1]

Same error here.

[alanscandrett]

So I have a large GeoJSON object as a file and it runs into this same issue. All the worlds nations outlines are represented in coordinate pairs.

I have however included a context.d.ts file which is read successfully, but the GeoJSON object is not and it seems to only appear as an empty object (error below):
Property 'type' is missing in type '{}' but required in type 'CountryJson'.ts(2741)

the context.d.ts file:

declare namespace SomeNamespace {
    export interface CountryJson {
        features?: Array<T>;
        type: string;
    }
}

After deleting countries from the array, starting by removing half and doing so until it goes, the error eventually disappears. I got it down to the difference of one additional country. I tried deleting the same country and random countries and it has the same effect.

It seems like TS taps out with larger objects, it would be good if the error message was clear about this as it caused quite a bit of time to be wasted figuring this out. I also am not sure how to proceed as it makes no sense to split the file when it is within the JSON and GeoJSON spec, but the error is never going to go away..

[kdthomas2121]

@alanscandrett did you get anywhere with this? I'm trying to do the same with a geojson file

[alanscandrett]

@alanscandrett did you get anywhere with this? I'm trying to do the same with a geojson file

Unfortunately not, I just split the dataset :/

[MrScriptX]

Same problem, that is an issue. My file isn't that big (33k lines) and it's just a bunch of arrays. No complex nesting or anything.