Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUPPORT] Apply 'write.ignore.failed' when write data failed #12187

Open
fhan688 opened this issue Oct 31, 2024 · 2 comments
Open

[SUPPORT] Apply 'write.ignore.failed' when write data failed #12187

fhan688 opened this issue Oct 31, 2024 · 2 comments
Labels
feature-enquiry issue contains feature enquiries/requests or great improvement ideas

Comments

@fhan688
Copy link
Contributor

fhan688 commented Oct 31, 2024

Describe the problem you faced

In Flink engine, if exception occurs when task writing data, it will be ignored and the exception will be reported to StreamWriteCoordinator with write event, StreamWriteCoordinator will decide whether to commit when there is write failure according to 'write.ignore.failed'.

Whether we should apply 'write.ignore.failed' ahead when write failure occurs, thus throw an exception faster.

To Reproduce

Expected behavior

  1. promote 'write.ignore.failed' to hudi-client-common
  2. throw exception in write handles
  3. default value of 'write.ignore.failed' is false

Environment Description

Additional context

related PR:#12150

Stacktrace

@danny0405
Copy link
Contributor

@nbalajee Do you have some insights about promote the 'write.ignore.failed' as a basic option for all the engines? I kind of remember we had a discussion before and engine specific maintainance is prefferable.

@ad1happy2go ad1happy2go added the feature-enquiry issue contains feature enquiries/requests or great improvement ideas label Nov 7, 2024
@github-project-automation github-project-automation bot moved this to ⏳ Awaiting Triage in Hudi Issue Support Nov 7, 2024
@fhan688
Copy link
Contributor Author

fhan688 commented Nov 25, 2024

Whether we should involve more experts in this topic?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-enquiry issue contains feature enquiries/requests or great improvement ideas
Projects
Status: Awaiting Triage
Development

No branches or pull requests

3 participants