Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test for empty titles in DPR processor #643

Merged
merged 1 commit into from
Nov 30, 2020
Merged

Conversation

tholor
Copy link
Member

@tholor tholor commented Nov 30, 2020

When using DPR processor with use_fast=True and embed_title=True, we currently get an error if the "title" is an empty string. This is due to the handling of empty strings by FastTokenizers (was already resolved by huggingface/tokenizers#389).
Adding a test case here to cover this in future.

11/30/2020 06:53:05 - ERROR - farm.data_handler.processor -   Error message: Truncation error: Specified max length is too low to respect the various constraints
11/30/2020 06:53:06 - ERROR - farm.data_handler.processor -   Could not create sample(s) from this dict: 
 {'passages': [{'title': '', 'text': '"letters). Some ambiguity exists in the official code page assignment for the Kamenický encoding: Some dot matrix printers of the NEC Pinwriter series, namely the ""P3200""/""P3300"" (""P20""/""P30""), ""P6200""/""P6300"" (""P60""/""P70""), ""P9300"" (""P90""), ""P7200""/""P7300"" (""P62""/""P72""), ""P22Q""/""P32Q"", ""P3800""/""P3900"" (""P42Q""/""P52Q""), ""P1200""/""P1300"" (""P2Q""/""P3Q""), ""P2000"" (""P2X"") and ""P8000"" (""P72X""), supported the installation of optional font EPROMs. The optional ROM #2 ""East Europe"" included this encoding, invokable via escape sequence codice_3 with (n) = 23. While named ""Kamenický"" in the documentation, it was originally advertised by NEC as code page 867 (CP867) or ""Czech"". (However, it was never registered with IBM under that ID, as IBM registered"', 'label': 'positive', 'external_id': 'b9d68e52-72ed-414f-baed-99275dba4767'}]}

@tholor tholor changed the title Add test for empty titles Add test for empty titles in DPR processor Nov 30, 2020
@tholor tholor merged commit 2cc54be into master Nov 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant