WP-14838 data pre-process configuration: header/footer rows and column header [API] #48
Conversation
| # Only modifying for imports from csv connector for now as imports from s3 connector might have reasons for skipping empty rows. | ||
| # Could look into using csv.reader instead for cleaner code if s3 connector could also keep empty rows. | ||
| if options.get('is_csv_connector_import', False): | ||
| csv.DictReader.__next__ = next_without_skip |
There was a problem hiding this comment.
This is no longer an issue. Both original csv.DictReader.next and next_without_skip will skip empty lines (no commas), but keep lines with commas.
jayma91
requested review from
mjdoor,
ChrisLing1,
sunil-varicent,
JiawenHou and
rsantos-varicent
| 'type': 'object', | ||
| 'properties': {} | ||
| } | ||
| raise Exception('File is empty.') |
There was a problem hiding this comment.
Is this thrown just if there is no data rows in the file (since we throw another Exception('File is empty.') from csv_iterator.get_row_iterator if the file doesn't even have a header row)? If so, do we need to do that, since there's already complaints about ICM connector failing when data is empty, when they would rather it just import the empty data successfully?
jayma91
dismissed stale reviews from mjdoor, rsantos-varicent, and ChrisLing1
via
07cbc00
|
removed exception for empty file as we will have to revisit this to allow empty table import for icm. Update: import fails for empty file during sync for having empty schema. Seems better to keep error handling and remove later when fixing for empty file import. |
This reverts commit 07cbc00.
WP-14838 data pre-process configuration: header/footer rows and column header [API]
https://varicent.atlassian.net/browse/WP-14838
notes: