Y-Pydantic

[kafonek]

Hello ypy community. I wanted to share a learning project I'm working on in case it helps anyone else understand the Y ecosystem better. https://github.com/kafonek/y-pydantic are bindings to automatically update Pydantic models when changes to Y shared data types (YText, YArray, YMap) are observed, including nested type (e.g. YText's in YArrays).

The repo includes two Notebooks right now:

• Primer on vanilla ypy workflows (creating a doc, shared types, syncing between clients): https://github.com/kafonek/y-pydantic/blob/main/notebooks/01_y_py_basics.ipynb
• Motivation and demonstration of y-pydantic bindings: https://github.com/kafonek/y-pydantic/blob/main/notebooks/02_y_pydantic_intro.ipynb

I don't have a compelling use case for using y-pydantic in any sort of real app right now, it really is just a way to explore how ypy works. There's some brick walls I've run into that I would appreciate a chance to discuss with maintainers, I'm not sure what the best forum might be. Issues, chat, an ad-hoc Zoom we can schedule?

One of those brick walls is introspecting attributes on a YText object. There doesn't seem to be a way to access those besides observing changes to the YText object (on insert and retain) and saving them off outside the YText object. That falls apart in the example at the very end of the second Notebook when we sync two clients, one of which has a YArray with YText elements that have attributes. The observed YArray insert has the fully formed YText object, no updates to the attributes so those attributes on the YText elements (string characters) are effectively unobservable.

[davidbrochart]

Thanks for sharing @kafonek !
I was thinking about something along these lines too, this is great. One thing I'm worried about is that Y updates can temporarily break schemas, right?

[davidbrochart]

Maybe I read too fast, it seems that you're not using pydantic to do validation, IIUC?

[kafonek]

Thanks @davidbrochart . Right now the Pydantic validation is so open-ended that it's not really doing any validation, as YText is really just an list of Any, and YArray / YMap are containers (list/dict) of Any. In addition to that, one place Pydantic struggles is constrained container types (https://pydantic-docs.helpmanual.io/usage/types/#constrained-types), it only checks the constraint at model instantiation not when you mutate the list/dict/etc.

(screenshot edited to fix validation config syntax)

There's surely ways to get casting/validation working, serializing/re-instantiating models if need be. The ugly parts could be hidden away. Still, I don't have a "why do this" for y-pydantic yet, other than that it's fun to hack on. In one sense I suppose I was looking for answers to #77, "how do I introspect a whole YDoc?" The way it's written now though, y-pydantic doesn't help with that.