Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-50194][SS][PYTHON] Integration of New Timer API and Initial State API with Timer #48838

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

jingz-db
Copy link
Contributor

@jingz-db jingz-db commented Nov 13, 2024

What changes were proposed in this pull request?

As Scala side, we modify the timer API with a separate handleExpiredTimer function inside StatefulProcessor, this PR make a change to the timer API to couple with API on Scala side. Also adds a timer parameter to pass into handleInitialState function to support use cases for registering timers in the first batch for initial state rows.

Why are the changes needed?

This change is to couple with Scala side of APIs: #48553

Does this PR introduce any user-facing change?

Yes.
We add a new user defined function to explicitly handle expired timeres:

def handleExpiredTimer(
        self, key: Any, timer_values: TimerValues, expired_timer_info: ExpiredTimerInfo

We also add a new timer parameter to enable users to register timers for keys exist in the initial state:

def handleInitialState(
        self,
        key: Any,
        initialState: "PandasDataFrameLike",
        timer_values: TimerValues) -> None

How was this patch tested?

Add a new test in test_pandas_transform_with_state

Was this patch authored or co-authored using generative AI tooling?

No

@jingz-db jingz-db changed the title Python new timer [SPARK-50194] Integration of New Timer API and Initial State API with Timer Implementation Nov 14, 2024
@jingz-db jingz-db marked this pull request as ready for review November 14, 2024 19:38
@github-actions github-actions bot removed the CORE label Nov 14, 2024
@jingz-db jingz-db changed the title [SPARK-50194] Integration of New Timer API and Initial State API with Timer Implementation [SS][PYTHON][SPARK-50194] Integration of New Timer API and Initial State API with Timer Implementation Nov 14, 2024
@jingz-db jingz-db changed the title [SS][PYTHON][SPARK-50194] Integration of New Timer API and Initial State API with Timer Implementation [SS][PYTHON][SPARK-50194] Integration of New Timer API and Initial State API with Timer Nov 14, 2024
@jingz-db jingz-db changed the title [SS][PYTHON][SPARK-50194] Integration of New Timer API and Initial State API with Timer [SPARK-50194][SS][PYTHON] Integration of New Timer API and Initial State API with Timer Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant