Extract add.stats_parsed with wrong type

[yjshen]

Bug

During delta log reading, extracting add.stats_parsed with the wrong StringArray type (on line 82) results in a double stats_parsed column in the result batch.

delta-rs/crates/core/src/kernel/snapshot/replay.rs

Lines 76 to 89 in eb8c19c

	fn map_batch(
	batch: RecordBatch,
	stats_schema: ArrowSchemaRef,
	config: &DeltaTableConfig,
	) -> DeltaResult<RecordBatch> {
	let stats_col = ex::extract_and_cast_opt::<StringArray>(&batch, "add.stats");
	let stats_parsed_col = ex::extract_and_cast_opt::<StringArray>(&batch, "add.stats_parsed");
	if stats_parsed_col.is_some() {
	return Ok(batch);
	}
	if let Some(stats) = stats_col {
	let stats: Arc<StructArray> =
	Arc::new(json::parse_json(stats, stats_schema.clone(), config)?.into());
	let schema = batch.schema();

Also, the newly generated stats_parsed from stats has a different array length than other columns.

source: Arrow { source: InvalidArgumentError("Incorrect array length for StructArray field \"stats_parsed\", expected 10 got 14")

What you expected to happen:

1. extract add.stats_parsed with StructArray type
2. generated stats_parsed should have a same number of rows as other cols.

How to reproduce it:

More details:

[ion-elgreco]

@yjshen can you add some reproducible code?

[sjohnston]

I've seen this error when trying to read an old table with 10s of thousands of partitions. I started removing old partitions and vacuum/compact and the error went away. Maybe the schema changed at some point or something wrong with the old data. I'm not sure.