Upgrade Pydantic to v2

[kabirkhan]

Overview

Adds support for using Pydantic v1 or v2.

All relevant Pydantic functions are extracted to utils that support Pydantic v1 and v2 style.
CI Tests run with both Pydantic v1 and v2 but type checking focuses on Pydantic v2.

Besides the dependency upgrade there is 1 significant functionality change which fixes a bug where Pydantic models cannot currently be resolved from a registered function. More details in this comment: https://github.com/explosion/confection/pull/31/files#r1248253486

[kabirkhan]

This change might seem a bit weird but it came up because Pydantic now treats Pydantic dataclasses and BaseModels the same way during serialization.

In Pydantic v1, calling model.dict() with an instance of a dataclass would not JSON serialize that dataclass, it would just return that dataclass.

This allowed for our Optimizer README example to work and resolve that dataclass instance.

e.g. with pydantic v1 this worked still:

import dataclasses
from typing import Union, Iterable
import catalogue
from confection import registry, Config

# Create a new registry.
registry.optimizers = catalogue.create("confection", "optimizers", entry_points=False)


# Define a dummy optimizer class.
@dataclasses.dataclass
class MyCoolOptimizer:
    learn_rate: float
    gamma: float


@registry.optimizers.register("my_cool_optimizer.v1")
def make_my_optimizer(learn_rate: Union[float, Iterable[float]], gamma: float):
    return MyCoolOptimizer(learn_rate, gamma)


# Load the config file from disk, resolve it and fetch the instantiated optimizer object.
config = Config().from_disk("./config.cfg")
resolved = registry.resolve(config)
optimizer = resolved["optimizer"]  # MyCoolOptimizer(learn_rate=0.001, gamma=1e-08)

However, when swapped to pydantic v2, the final line here was resolving the optimzer to a dict.

...
optimizer = resolved["optimizer"]  # {"learn_rate": 0.001, "gamma": 1e-08}

This has actually always been a bug, because previously we could not make MyCoolOptimizer into a Pydantic model, it had to be a dataclass (or any other class that wasn't a Pydantic model).

With this code above, we could make MyCoolOptimzer into a Pydantic model and we'd get a Pydantic model back.

[kabirkhan]

So the other somewhat controversial change is the removal of checks for the Generator type. Pydantic supports validating Python generators via the Iterator and Iterable types, but not through the GeneratorType. This is also how you're supposed to type stuff in Python so I think this should just be a documentation update but wanted to call it out.

[kabirkhan]

This case can occur from the above change to pass through resolved pydantic models

[adrianeboyd]

Thanks for working on this, we have many users asking about it!

Just to start out with some comments about the CI and mypy...

[adrianeboyd]

(And please don't just remove python 3.6 from the CI like this!)

[svlandeg]

Nice work Kabir, thanks!

[adrianeboyd]

I had this installed in the background for testing and have run into a few problems with pydantic v1.10 (since you can't use v2 with thinc/spacy right now):

$ spacy init config -p morphologizer /tmp/morphologizer.cfg
ℹ Generated config template specific for your use case
- Language: en
- Pipeline: morphologizer
- Optimize for: efficiency
- Hardware: CPU
- Transformer: None
✘ Config validation error
nlp -> tokenizer	Promise(registry='tokenizers', name='spacy.Tokenizer.v1', args=[], kwargs={}) is not callable
{'lang': 'en', 'pipeline': ['tok2vec', 'morphologizer'], 'batch_size': 1000, 'disabled': [], 'before_creation': None, 'after_creation': None, 'after_pipeline_creation': None, 'tokenizer': {'@tokenizers': 'spacy.Tokenizer.v1'}}

Or with an existing config including a morphologizer with an internal tok2vec:

[components.morphologizer.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.morphologizer.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.morphologizer.model.tok2vec.encode.width}
attrs = ["ORTH", "SHAPE"]
rows = [5000, 2500]
include_static_vectors = true

Config validation error
cfg.model.tok2vec -> embed	instance of Model expected
cfg.model.tok2vec -> encode	instance of Model expected
{'@architectures': 'spacy.Tok2Vec.v2', 'embed': {'@architectures': 'spacy.MultiHashEmbed.v2', 'width': 256, 'attrs': ['ORTH', 'SHAPE'], 'rows': [5000, 2500], 'include_static_vectors': True}, 'encode': {'@architectures': 'spacy.MaxoutWindowEncoder.v2', 'width': 256, 'depth': 8, 'window_size': 1, 'maxout_pieces': 3}}

[adrianeboyd]

I don't currently understand what the differences are between the unit tests with catalogue (that pass) and the usage with thinc/spacy and pydantic v1.10 that fail above.

Let me temporarily add a CI test to make sure it's not something on my end.

[adrianeboyd]

I'm trying to think if there's a better way to define/manage the PYDANTIC_V2 setting so that it doesn't lead to so many typing errors...

[adrianeboyd]

Just a comment that this is only intended to be a temporary addition to the CI. confection needs a unit test that catches this issue.