Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[java] BQ: use logical type in avro schema factory on write #33163

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

RustedBones
Copy link
Contributor

@RustedBones RustedBones commented Nov 19, 2024

when writing to BQ with avro, if the table schema contains DATE, TIME, TIMESTAMP columns, the default schema factory should create avro fields with matching logical type.

There is still an issue with DATETIME: BigQueryAvroUtils::toGenericAvroSchema favors generating schema for the reading side and generates an avro field with string(datetime) type. This can't be used on write (expecting long(local-timestamp-millis) or long(local-timestamp-micros)).

See note in doc

When exporting to Avro from BigQuery, DATETIME is exported as a STRING with a custom logical time that is not recognized as a DATETIME upon importing back into BigQuery.

@RustedBones RustedBones changed the title [java] BQ: use logical type in schema factory when enabled [java] BQ: use logical type in avro schema factory when enabled Nov 19, 2024
Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@RustedBones RustedBones changed the title [java] BQ: use logical type in avro schema factory when enabled [java] BQ: use logical type in avro schema factory on write Nov 19, 2024
Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.
R: @damondouglas for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant