Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Increase admin max header data response size #615

Closed
1 of 20 tasks
katrogan opened this issue Nov 13, 2020 · 0 comments
Closed
1 of 20 tasks

[BUG] Increase admin max header data response size #615

katrogan opened this issue Nov 13, 2020 · 0 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@katrogan
Copy link
Contributor

katrogan commented Nov 13, 2020

Describe the bug
When admin returns a dump of error logs in a response this can exceed the gRPC trailing metadata limit which obscures the original source of the error.

Expected behavior
Admin should be able to propagate error messages fully to callers.

Flyte component

  • Overall
  • Flyte Setup and Installation scripts
  • Flyte Documentation
  • Flyte communication (slack/email etc)
  • FlytePropeller
  • FlyteIDL (Flyte specification language)
  • Flytekit (Python SDK)
  • FlyteAdmin (Control Plane service)
  • FlytePlugins
  • DataCatalog
  • FlyteStdlib (common libraries)
  • FlyteConsole (UI)
  • Other

To Reproduce
Steps to reproduce the behavior:

  1. Have a workflow with a task that outputs a reaaaaally big schema. Add a subsequent task with a different, but also long schema definition that consumes the output of the previous task.
  2. Attempt to register the workflow
  3. Be confused by the response
flytekit.common.exceptions.scopes.FlyteScopedSystemException: <_Rendezvous of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "received trailing metadata size exceeds limit"
debug_error_string = "{"created":"@1605080075.767410113","description":"Error received from peer ipv4:XX.Y.ZZ.12:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"received trailing metadata size exceeds limit","grpc_status":8}"

Screenshots
If applicable, add screenshots to help explain your problem.

Environment
Flyte component ALL

  • Sandbox (local or on one machine)
  • Cloud hosted
    • AWS
    • GCP
    • Azure
  • Baremetal
  • Other

Additional context

@katrogan katrogan added bug Something isn't working untriaged This issues has not yet been looked at by the Maintainers and removed untriaged This issues has not yet been looked at by the Maintainers labels Nov 13, 2020
@katrogan katrogan self-assigned this Nov 13, 2020
@katrogan katrogan added this to the 0.10.0 milestone Nov 13, 2020
eapolinario pushed a commit that referenced this issue Sep 28, 2023
* Handle batched TaskExecutionEvent reasons

Signed-off-by: Andrew Dye <[email protected]>

* Add tests

Signed-off-by: Andrew Dye <[email protected]>

* Update flyteidl version

Signed-off-by: Andrew Dye <[email protected]>

* Update to EventReason

Signed-off-by: Andrew Dye <[email protected]>

---------

Signed-off-by: Andrew Dye <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant