Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the number of Schema to fundamental types #352

Open
tusharmath opened this issue Oct 4, 2022 · 2 comments
Open

Reduce the number of Schema to fundamental types #352

tusharmath opened this issue Oct 4, 2022 · 2 comments

Comments

@tusharmath
Copy link
Contributor

tusharmath commented Oct 4, 2022

Problem

Currently If one needs to match on Schema these are the number of options they need to deal with -

        a match {
          case Schema.Primitive(standardType, annotations) => ???
          case enum: Schema.Enum[_]                        => ???
          case record: Schema.Record[_]                    => ???
          case collection: Schema.Collection[_, _]         => ???
          case Schema.Transform(_, _, _, _, _)             => ???
          case Schema.Optional(_, _)                       => ???
          case Schema.Fail(_, _)                           => ???
          case Schema.Tuple(_, _, _)                       => ???
          case Schema.EitherSchema(_, _, _)                => ???
          case Schema.Lazy(_)                              => ???
          case Schema.Meta(_, _)                           => ???
          case Schema.Dynamic(_)                           => ???
          case Schema.SemiDynamic(_, _)                    => ???
        }

I think we can reduce them to some fundamental types that are completely orthogonal. For eg: Tuple, Option and Either can be implemented using Record and Enum. Similarly, Transform Lazy Meta Dynamic and SemiDynamic seem to leak implementation detail rather than just the schema of the type.

I propose that we reduce the public schema types to more fundamental ones only.

Would love to know what everyone thinks about it.

@jdegoes
Copy link
Member

jdegoes commented Oct 24, 2022

I'm onboard with this, conceptually. However, we do have some issues to resolve:

  1. Many formats have explicit support for Option (e.g. nullable columns in SQL, optional fields in protobuf, etc.). Codecs need the ability to know the meaning of null. Possibly typeId in Transform node could allow us to hold onto that.
  2. Meta and SemiDynamic should be gone. As for Dynamic schema, some backends can directly store JSON-like content. While we can describe JSON-like content with a recrusive schema, it may be more difficult for backends to process.
  3. Laziness is necessary for recursive schemas.

@vigoo
Copy link
Contributor

vigoo commented Oct 27, 2022

Another important thing is that if we delete some of these then it has to be gone from all the different zio-schema types. For example if we don't want to have an explicit Either then don't have it in Schema, in DynamicValue, etc. I had the opposite problem while working on zio-flow, that schema had Either and DynamicValue not, for example and then converting from/to dynamic values was loosing information.

Also keep in mind that transformations cannot be serialized so if there is any chance that two cases (let's say either vs an arbitrary sum type) needs to be differentiated that cannot be done only by transformations. Not sure if there are anything like this in the library.

Also it's possible that now that we have type IDs, some of these problems are easier to solve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

3 participants