Expose configuring input CSV file formatting options

[sstroemer]

As discussed internally on 17.01.2025.

The most common two use cases might be:

# "German" with comments
config:
  files:
    _csv_config:
      comment: "#"
      delim: ";"
      decimal: ","

# "English" without comments
config:
  files:
    _csv_config:
      delim: ","
      decimal: "."

Auto-summary below.

---

This pull request introduces several changes related to CSV file handling and configuration in the project. The key updates include the addition of a new example configuration file, modifications to the file parsing logic, and updates to the CSV reading function to support custom delimiters and decimal separators.

CSV Configuration Enhancements:

• Added a new example configuration file 49_csv_formats.iesopt.yaml to demonstrate different CSV formats and configurations.
• Updated _prepare_config_files! function in src/config/sections/files.jl to handle _csv_config settings and issue warnings for file names starting with an underscore.

File Parsing and Preloading:

• Modified _parse_model! and _parse_global_addons functions in src/parser.jl to exclude files starting with an underscore from preloading and to streamline file parsing logic. [1] [2]

CSV Reading Function:

• Enhanced _getcsv function in src/utils/general.jl to utilize the _csv_config settings for comment characters, delimiters, and decimal separators. Added error logging for CSV reading issues. [1] [2]

Testing:

• Added new tests in test/src/examples.jl to validate the new CSV format configurations and ensure proper functionality.

[sstroemer]

This is the first time we add an "experimental" feature to the YAML spec. This now starts with "_" (which we could re-use in the future), and we settled on potentially changing/discontinuing such "private" features of the YAML without the need for a major version bump.

[codecov]

Codecov Report

Attention: Patch coverage is 92.00000% with 2 lines in your changes missing coverage. Please review.

Project coverage is 50.20%. Comparing base (36ac0da) to head (39b3285).
Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
src/config/sections/files.jl	87.50%	1 Missing ⚠️
src/utils/general.jl	92.85%	1 Missing ⚠️

@@            Coverage Diff             @@
##             main      #53      +/-   ##
==========================================
+ Coverage   49.57%   50.20%   +0.63%     
==========================================
  Files         109      109              
  Lines        4303     4322      +19     
==========================================
+ Hits         2133     2170      +37     
+ Misses       2170     2152      -18     

Files with missing lines	Coverage Δ
src/parser.jl	81.57% <100.00%> (ø)
src/config/sections/files.jl	90.90% <87.50%> (-9.10%)	⬇️
src/utils/general.jl	53.95% <92.85%> (+2.22%)	⬆️

... and 1 file with indirect coverage changes

[sstroemer]

This was initially triggered by ait-energy/iesopt#27

[sstroemer]

We need to create a new iesopt release, so that this feature can be used without a custom .env file for Python users, refer to ait-energy/iesopt#64.

Documentation is already available!

[daschw]

Looks good! :)