JSON column with all empty arrays is dropped

[tkarabela]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
import io

data = io.BytesIO(b'''\
{"id": 1, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
{"id": 2, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
{"id": 3, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
{"id": 4, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
''')

df = pl.read_ndjson(data)
print(df)

# shape: (4, 3)
# ┌─────┬─────────────┬─────────────────────┐
# │ id  ┆ zero_column ┆ empty_object_column │
# │ --- ┆ ---         ┆ ---                 │
# │ i64 ┆ i64         ┆ struct[1]           │
# ╞═════╪═════════════╪═════════════════════╡
# │ 1   ┆ 0           ┆ {null}              │
# │ 2   ┆ 0           ┆ {null}              │
# │ 3   ┆ 0           ┆ {null}              │
# │ 4   ┆ 0           ┆ {null}              │
# └─────┴─────────────┴─────────────────────┘

Log output

No response

Issue description

JSON parsing seems to drop null values (reported in #7858 and #11860) and empty arrays (reported here).

I encountered this when reading several files with pl.read_ndjson(...) and then trying to join the resulting dataframes using pl.concat(...). Some files have only null or [] in a given column, which causes errors like ShapeError: unable to append to a DataFrame of width 7 with a DataFrame of width 8, despite all the input files having the same structure.

Related PR: #11880

Expected behavior

import polars as pl
import io

data = io.BytesIO(b'''\
{"id": 1, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
{"id": 2, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
{"id": 3, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
{"id": 4, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": null}
''')

df = pl.read_ndjson(data)
print(df)

# shape: (4, 5)
# ┌─────┬─────────────┬────────────────────┬─────────────────────┬─────────────┐
# │ id  ┆ zero_column ┆ empty_array_column ┆ empty_object_column ┆ null_column │
# │ --- ┆ ---         ┆ ---                ┆ ---                 ┆ ---         │
# │ i64 ┆ i64         ┆ list[null]         ┆ struct[1]           ┆ null        │
# ╞═════╪═════════════╪════════════════════╪═════════════════════╪═════════════╡
# │ 1   ┆ 0           ┆ []                 ┆ {null}              ┆ null        │
# │ 2   ┆ 0           ┆ []                 ┆ {null}              ┆ null        │
# │ 3   ┆ 0           ┆ []                 ┆ {null}              ┆ null        │
# │ 4   ┆ 0           ┆ []                 ┆ {null}              ┆ null        │
# └─────┴─────────────┴────────────────────┴─────────────────────┴─────────────┘

In other words, I would expect df to be the same as:

df = pl.DataFrame([
    {"id": 1, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": None},
    {"id": 2, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": None},
    {"id": 3, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": None},
    {"id": 4, "zero_column": 0, "empty_array_column": [], "empty_object_column": {}, "null_column": None},
])

which produces the output show above.

Installed versions

--------Version info---------
Polars:              0.19.15
Index type:          UInt32
Platform:            Windows-10-10.0.22621-SP0
Python:              3.11.4 (tags/v3.11.4:d2340ef, Jun  7 2023, 05:45:37) [MSC v.1934 64 bit (AMD64)]

----Optional dependencies----
adbc_driver_sqlite:  <not installed>
cloudpickle:         <not installed>
connectorx:          <not installed>
deltalake:           <not installed>
fsspec:              <not installed>
gevent:              <not installed>
matplotlib:          3.8.2
numpy:               1.26.2
openpyxl:            <not installed>
pandas:              2.1.3
pyarrow:             <not installed>
pydantic:            <not installed>
pyiceberg:           <not installed>
pyxlsb:              <not installed>
sqlalchemy:          <not installed>
xlsx2csv:            <not installed>
xlsxwriter:          <not installed>