GH-37630: [C++][Python][Dataset] Allow disabling fragment metadata caching

[pitrou]

Rationale for this change

Parquet file fragments currently cache their (Parquet) metadata for later accesses when scanning has finished.
This can produce surprisingly high memory consumption in cases where:

1. the dataset is only scanned once, rather than repeatedly (this is very common)
2. there is a high metadata-to-data ratio; this can happen when the schemas on disk are very wide, with few rows per file and/or a low number of columns selected for reading

What changes are included in this PR?

Add an option to disable metadata caching on Parquet file fragments.

Are these changes tested?

Yes, by new unit tests. Also, reading a wide dataset locally has been confirmed to consume much less memory when the new option is toggled.

Are there any user-facing changes?

No.

• GitHub Issue: [C++] Potential memory leak in Parquet reading with Dataset #37630

[github-actions]

⚠️ GitHub issue #37630 has been automatically assigned in GitHub to PR creator.

[pitrou]

In #45287 (comment) it was mentioned that clearing physical_schema_ further improved the memory profile, but it's not obvious that it would be correct.

[icexelloss]

I wonder if we can have a mode to release the fragment once the data for that fragment has been read in a "scan once" usage pattern. But also I don't know how hard it is to change that.

Per physical_schema_ - I am not familiar with the details but it seems fine to me if this is "scan once" and the fragment is not used again?

[timothydijamco]

I posted some thoughts about clearing physical_schema_ in #45287 (comment)

[pitrou]

I added a change that clears the cached physical schema, but keeps the original schema when it was passed via the constructor.