un-tracked columns are giving json error when pyarrow schema have feild with nullable=False and create_checkpoint is trigged

[sherlockbeard]

Environment

Delta-rs version:
0.18.2

Binding:
python

Environment:

• Cloud provider:
• OS: mac m1
• Other:

---

Bug

What happened:
un-tracked columns are giving Json error when pyarrow schema have feild with nullable=False and create_checkpoint is trigged

What you expected to happen:
create_checkpoint command is successful

How to reproduce it:

import deltalake
import pyarrow as pa
import polars as pl
from importlib.metadata import version
print(version("deltalake"))

pylist = [{'year': 2023, 'n_party': 0}, {'year': 2024, 'n_party': 1}]

my_schema = pa.schema([

    pa.field('year', pa.int64(), nullable=False),

    pa.field('n_party', pa.int64(), nullable=False),])

data = pa.Table.from_pylist(pylist, schema=my_schema)

print(data)
deltalake.write_deltalake("temp5", data, configuration={"delta.dataSkippingNumIndexedCols": "1"},)


print(deltalake.DeltaTable("temp5").to_pyarrow_table())

deltalake.DeltaTable("temp5").create_checkpoint()

print(deltalake.DeltaTable("temp5").to_pyarrow_table())

error

    self._table.create_checkpoint()
Exception: Json error: whilst decoding field 'add': whilst decoding field 'stats_parsed': whilst decoding field 'minValues': Encountered unmasked nulls in non-nullable StructArray child: Field { name: "n_party", data_type: Int64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }

More details:
see https://delta-users.slack.com/archives/C013LCAEB98/p1721098853604139 thread