PEP 544: Protocols

[ilevkivskyi]

This adds static support for structural subtyping. Previous discussion is here python/typing#11

Fixes #222

[JukkaL]

Left some comments based on a quick read-through.

[JukkaL]

It's a little unclear what 'completely optional' means here and whether the following sentences are related to this. This could perhaps mean some of these things:

• They are optional like everything defined in PEP 484: They don't affect Python runtime semantics and they are merely a feature in typing. Programmers are free to not use them.
• The standard library doesn't use protocols outside typing.
• Programmers are free to not use them even if they use type annotations.
• Python implementations don't need to support them.

This could rephrased like this to avoid the ambiguity:

... are completely optional:

• No runtime semantics will be imposed for variables or parameters annotated with a protocol class.
• Any checks will be performed only by third-party type checkers and other tools.
• ... maybe more

There is no intent to make protocols non-optional in the future.

[JukkaL]

This seems to imply that sequences no longer support iteration, since Sequence.__iter__ is not abstract and thus is not a protocol member. Is this intentional?

My view is that protocols should not have non-protocol attributes or methods. For example, none of the ABCs in collections.abc seem to have what I'd consider non-protocol members. Protocols should describe an interface, not an implementation. Default implementations are okay, but they should probably only use other members defined in the protocol.

If we want to support non-protocol members, I think we should restrict them to names with a double underscore prefix to make this super explicit. However, even this brings in some arguably unneeded complexity: self will now have a special type, since accessing non-protocol members is only safe through self. Example:

class C(Protocol):
    ...
    def f(self, other: 'C') -> None:
        self.__g()  # ok
        other.__g()  # not ok

    def __g(self) -> None:
        <something>

[ilevkivskyi]

This seems to imply that sequences no longer support iteration, since Sequence.__iter__ is not abstract and thus is not a protocol member. Is this intentional?

Sequence implements __iter__ therefore it will be accepted wherever Iterable is expected. But asking for __iter__ on Sequence indeed will fail.

This indeed looks problematic. Here is a solution that I could propose: we do not have such thing as non-protocol members. However, there will be two kinds of members, abstract and non-abstract. All of them should be implemented in order to consider a class as structural (implicit) subtype.

The reason to have non-abstract methods is that one can get them "for free" using explicit subclassing provided abstract methods are implemented in subclass. So that for larger protocols like Sequence or Mapping one doesn't need to do

def method(self):
    return super().method()

for every method.

This coincides with how ABCs work at runtime, so that this will be intuitive for someone who already worked with ABCs.

[ambv]

Protocols should describe an interface, not an implementation.

👍

Default implementations are okay, but they should probably only use other members defined in the protocol.

👍

[gvanrossum]

Here is a solution that I could propose: we do not have such thing as non-protocol members.

That's now specified below in the PEP right?

[ilevkivskyi]

That's now specified below in the PEP right?

Yes.

[JukkaL]

This is unclear. What is PInt?

[JukkaL]

Could this be rephrased as something like "Protocol types can be used in all contexts where any other types can used, such as in (examples). Generic protocols follow the rules for generic abstract classes, except for using structural compatibility instead of compatibility defined by inheritance relationships."

[JukkaL]

I'd consider leaving out All from this proposal, at least initially. This is something we can add later if there is a need, but currently I don't see evidence for this being important. Effectively the same behavior can be achieved through multiple inheritance. Besides, it would be a bit arbitrary if All only worked with protocols, and making All work with arbitrary types would be a much more complex feature, and not as directly related to this PEP.

[ilevkivskyi]

I think we could include this for three reasons:

• I could easily imagine a function parameter that requires more than one protocol, especially if the preferred style is to have several simple protocols, rather than few complex. By the way, some classes in collections.abc are simple intersections, for example Collection == All[Sized, Iterable, Container].
• It could be easily implemented for protocols. I also propose to use All instead of Intersection to emphasize it only works for protocols (mnemonics: variable implements all these protocols). On the contrary, I think that intersections of nominal classes would be very rarely used.
• If we decide to include this later, then people might simply be not aware of this feature (this is a minor reason, but still we need to take this into account).

[ddfisher]

I agree with Jukka -- I think it's confusing for All to work only with protocols, extending All to work with all types would add considerable complexity, and I'm not convinced there's a pressing need.

[ambv]

I was pushing for Intersection for the original PEP because I saw the need for being able to intersect ABCs. Given that protocols themselves solve this, and we can use multiple inheritance as Jukka is suggesting, I agree that it's wiser to leave this out.

I would still like to have it, but that can be a separate PEP. I agree it's complex to implement for concrete types, and better yet, for combinations of both.

[JukkaL]

I don't love the automatic isinstance check support, even though that's how the mypy Protocol class used to work. I don't think that they can be made to work reliably for arbitrary ABCs. Consider this example:

class P(Protocol):
    @abstractmethod
    def common_method_name(self, x: int) -> int: ... 

class X:
    <a bunch of methods>
    def common_method_name(self) -> None: ...

def do_stuff(o: Union[P, X]) -> int:
    if isinstance(o, P):
        return o.common_method_name(1)  # oops what if it's an X instance instead?
    else:
        return (do something with an X)

Maybe a type checker can warn about this example, but more generally the isinstance check could be unreliable, except for things where there is a common signature convention such as Iterable.

Another potentially problematic case:

class P(Protocol):
    x: int

class C:
    def initialize(self) -> None:
        self.x = 0

c1 = C()
c2 = C()
c2.initialize()
isinstance(c1, P)  # False
isinstance(c2, P)  # True

def f(x: Union[P, int]) -> None:
    if isinstance(x, P):
        # type of x is P here
        ...
    else:
        # type of x is int here?
        print(x + 1)

f(C())   # oops

Again, maybe we can live with this, but this could be pretty surprising. I'd argue that requiring an explicit class decorator would be better, since we can then attach warnings about problems like this in the documentation. The programmer would be able to evaluate whether the benefits outweigh the potential for confusion for each protocol and explicitly opt in -- but the default behavior would be safer.

[ilevkivskyi]

I'd argue that requiring an explicit class decorator would be better, since we can then attach warnings about problems like this in the documentation.

This is how I initially wrote this, since I intuitively felt that this could not be 100% reliable. Guido wanted to make it default, by now we have good examples to make this opt-in, so that I will revert this to class decorator.

[ddfisher]

I'm not sure I understand how a class decorator would solve the problem. Wouldn't that mean that some classes that implement the protocol (i.e. the decorated ones) will work properly with isinstance at runtime but others will not?

[ilevkivskyi]

classes that implement the protocol (i.e. the decorated ones)

The decorator is applied to protocol, not to implementation. This was in a previous draft. At runtime the decorator will add same __subclasshook__() to a protocol, that collections.abc classes have now. As Jukka pointed out, there are pros and cons of such approach, and a user could decide.

[ddfisher]

How would the __subclasshook__ work?

[ilevkivskyi]

Basically, if a class has all necessary attributes, then it is a subclass. This is why I mention below that "type checkers will narrow types after such checks by the type erased Proto (i.e. with all variables having type Any and all methods having type Callable[..., Any])" because we cannot reliably check the types of attributes at runtime.

This is how it is done in collections.abc see https://github.com/python/cpython/blob/master/Lib/_collections_abc.py#L72 and e.g. https://github.com/python/cpython/blob/master/Lib/_collections_abc.py#L93

[JukkaL]

What about Python 2.7?

[ilevkivskyi]

I think this could be used the same way for 2.7 - 3.5, modulo the fact that function type comments should be used in 2.7.

[JukkaL]

A better argument would be something like this: isinstance, as defined for classes that don't explicitly extend Protocol, is based on the nominal hierarchy. In order to make everything a protocol by default and have isinstance work would require changing the semantics of isinstance, which won't happen.

[JukkaL]

Additional reason: Many built-in functions only accept concrete instances of int (and subclass instances), and similarly for other built-in classes. Making int a structural type wouldn't be safe without major changes to the Python runtime, which won't happen.

[JukkaL]

Is it customary to include references to discussions that have happened about the PEP, such as in the typing issue tracker?

[ilevkivskyi]

Yes, this is a good idea.

[gvanrossum]

I wonder if we could do more deep linking to specific comments or subthreads? (No big deal if you think it's too much effort, but it might save some readers some time finding the right part of the long discussion in e.g. python/typing#11 -- including myself :-).

[ilevkivskyi]

OK, will do this.

[ilevkivskyi]

@JukkaL Thank you for helpful comments! I agree with almost all things. I will make corresponding changes in the text soon. The only thing where we do not agree is All, I still think we should have it.

Also we probably need to find a way how to indicate whether an abstract method has a default implementation or not in stubs.

[ddfisher]

As a general comment, I don't think it's a good idea to standardize this before having an implementation. An implementation would allow us to understand what's difficult to implement, what sorts of unexpected interactions it has with other type-system features, what parts are confusing, and what parts are practically useful. I think it will be very difficult to get an understanding of those things otherwise.

[ddfisher]

I'm worried this could be pretty confusing: you could have a class that explicitly implements some protocols and implicitly implements others. I don't know of any other language that allows this.

[ddfisher]

I agree with Jukka -- I think it's confusing for All to work only with protocols, extending All to work with all types would add considerable complexity, and I'm not convinced there's a pressing need.

[ddfisher]

I think this restriction makes sense, actually -- trying to construct a NewType from a protocol isn't cogent. You probably want an alias instead. Having NewType create a non-protocol subtype is surprising behavior and is not consistent with how NewType works in general, in my opinion. We shouldn't support this without a clear need.

[ddfisher]

I'm not sure I understand how a class decorator would solve the problem. Wouldn't that mean that some classes that implement the protocol (i.e. the decorated ones) will work properly with isinstance at runtime but others will not?

[ddfisher]

How do you propose to handle the backwards-incompatibility problem for all currently released versions of Python?

[ilevkivskyi]

The idea is that there should be no backward incompatibility. The classes in typing module (as well as in collections.abc) already behave like protocols at runtime:

class Whatever:
    def __iter__(self): return []
assert isinstance(Whatever(), Iterable)

as I mention below "Practically, few changes will be needed in typing since some of these classes already behave the necessary way at runtime", most of these will be changed only in typeshed.

Also, after I implement the changes proposed by Jukka, all code that passes mypy now, will still pass (unless I don't see some subtleties).

[ilevkivskyi]

@ddfisher Of course I am not going to wait until this PEP is accepted :-)
I think implementation and discussion of specification should go in parallel. This is a big feature, but I hope I will make a first PR quite soon.

[brettcannon]

@ilevkivskyi when you're ready for the first version to go in just let us know and we will commit it. Then you can have discussions go to your personal fork and just send PRs for updates as needed.

[ilevkivskyi]

@brettcannon I implemented most comments that appeared so far, but I would prefer to wait for input from @ambv

[ambv]

This is very well written. Good job! See comments, @ilevkivskyi.

Let's commit, @brettcannon.

[ambv]

Moreover, extensive use of ABCs might impose additional runtime costs.

Do we have any numbers? We know ABCs aren't free but I don't know if this is worth mentioning unless it's a major factor. The rationale for protocols to be the pythonic, dynamic and idiomatic version of ABCs is strong enough.

[ilevkivskyi]

class C(collections.abc.Iterable):
    def __iter__(self):
        return []

is 2x slower than

class C:
    def __iter__(self):
        return []

The numbers for typing.Iterable are even worse, but I have some ideas on how to improve those. The 2x slowdown is still not much, so I use "extensive" and "might". If you think it is not necessary we could remove this altogether.

[ambv]

It is also automatically extensible and works with additional, unrelated classes that happen to implement the required protocol.

Yes, this is the killer feature.

[ambv]

as a project name, use zope.interface (so I'd change it here).

When describing behavior (like on line 133), keep "Zope interfaces".

[ambv]

The main goal of this proposal is to support such behavior statically.

I'd put this point right in the rationale when you're already discussing ABCs.

to allow users achieving

Shouldn't it be "to allow users achieve"?

[ilevkivskyi]

http://ell.stackexchange.com/questions/11193/allow-to-infinitive-substantive-verb-ing says it probably should be "to achieve" here.

[ambv]

Protocols should describe an interface, not an implementation.

👍

Default implementations are okay, but they should probably only use other members defined in the protocol.

👍

[ambv]

Zope interfaces are a superset of Protocols defined here, just using an incompatible syntax to define them. They necessarily use Attributes because pre-PEP 526 there was no other way to annotate this. Arguably in the 3.6+ world zope.interface could (optionally?) adopt the Protocol syntax.

In which case to achieve interoperability we'd only need to figure out a way to inform the type checker that "Interface" (or rather "InterfaceClass") is essentially "Protocol". This way we could type check successfully (pretty much ignoring implements() and treating those classes as implicit implementations. I think this is perfectly fine as the goal is gradual typing: if the type checker stays silent in some cases, that's OK. It just never should raise false positives.

Adapters, invariants, and runtime implements/provides checkers are obviously left out. This is a non-goal.

[ambv]

type of x is P here

or rather a concrete type that implements P ;-)

[JukkaL]

The intention probably was that the static type of x is P.

[ambv]

Can you give an example of this unreliability? This sounds like a much stronger argument than the attribute assignment presented below (read comment about it).

[ilevkivskyi]

I added a shorter version of Jukka's example here.

[ambv]

This is not surprising to me and at runtime I can see this as being a useful feature. You can imagine:

class Client(Protocol):
    ...

class Connected(Protocol):
    connection: socket


...  # later in the code
isinstance(someClient, Connected)

Maybe we don't want to encourage this programming style but it definitely descibes some real world code, sigh.

[JukkaL]

The problem here is that the runtime and static view of the type of someClient differ, potentially resulting in runtime type errors uncaught by type checkers, as shown by the code below this line. Since protocols are mostly about static type checking, I think it's reasonable to make restrictions that are biased towards letting static checkers do a better job.

[ambv]

That's the one part I'm not sure about. Having every Protocol behave consistently at runtime would make it easier to use.

[JukkaL]

As discussed in the examples I gave, it would also make protocols more fragile and potentially surprising. My argument is that explicit here is better than implicit, even if it adds a bit of work for the programmer. Also, currently my working hypothesis is that protocols won't be very common in typical statically typed code -- they will be used every now and then, but not very frequently. If correct, the extra effort needed for adding the decorator would very little and basically insignificant. If this proves to be an incorrect hypothesis, it's much easier to add the missing implicit runtime check in the future than to remove an existing implicit runtime check, since the latter would break existing code. Basically, my reasoning is that if in doubt, leave it out from the first version until we have evidence that it would be useful.

[gvanrossum]

OK, I am actually convinced by @JukkaL's argument here. (But we should probably apply a similar reasoning to cut out some other esoteric ideas, like variables.)

[ilevkivskyi]

@brettcannon I think this could be merged now, unless there are some objections. I hope I addressed all points that were discussed. There may be some things that could be ironed out, but we can do this is additional PRs later.

[gvanrossum]

Let's merge it. That does not mean I am accepting the PEP but is ready for pyrhon-dev discussion.

…

--Guido (mobile)

On Mar 17, 2017 6:49 PM, "Ivan Levkivskyi" ***@***.***> wrote: @brettcannon <https://github.com/brettcannon> I think this could be merged now, unless there are some objections. I hope I addressed all points that were discussed. There may be some things that could be ironed out, but we can do this is additional PRs later. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#224 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMpSk7CC9zp0bIc8O1Nyi3PafVb3kks5rmzhAgaJpZM4MaIqF> .

[gvanrossum]

This is just the first half of my review. I skipped the background stuff and jumped right into the specification. I'll try to find time for the rest later this week.

[gvanrossum]

A subtlety here: you could explicitly subclass a protocol and still not implement it. At least, that's how I would interpret the behavior of Hashable (and a few others that have similar behavior) where you can set __hash__ = None in a subclass in order to not implement the protocol. This is useful for immutable subclasses of mutable classes.

[ilevkivskyi]

Yes. I will add a note that type checkers should understand setting attributes to None in the sense of data model docs: "Setting a special method to None indicates that the corresponding operation is not available."

[gvanrossum]

preferably -> typically

[gvanrossum]

/some or all parameters of/ -> /any parameters of a/

[gvanrossum]

New paragraph to start here.

[gvanrossum]

Hmm, I don't really like the idea of "trivial bodies". Remember that when explicitly subclassing an ABC, abstract methods can still be called, through super(). An abstract method that shouldn't be called ought to raise NotImplementedError.

In stubs, mypy already doesn't type-check trivial bodies (though I don't recall its exact definition of "trivial" -- it probably doesn't exactly match what you have here). If we want to specify that behavior for all checkers reading stubs (which would seem to make sense) it should go in PEP 484, not here.

Or perhaps I am missing a use case for these that you've found in real code?

[ilevkivskyi]

An abstract method that shouldn't be called ought to raise NotImplementedError.

OK, I think this is a reasonable requirement, I will update this part.

[gvanrossum]

Hm, this example makes this requirement feel like a (slight) nuisance -- in this particular example it's pretty clear that it's meant to define a protocol?

[ilevkivskyi]

It may be clear for a human (taking into account the name of subclass). But for a type checker there are two equally possible options: a user actually wanted a protocol, or just forgot to implement a protocol method. As you said above:

we don't want to accidentally have some class act as a protocol just because one of its base classes happens to be one

I think explicit is better than implicit applies here.

[gvanrossum]

Fair enough. Objection withdrawn.

[gvanrossum]

This seems to make sense, but does it really? There's no such rule for ABCs. What harm would there be in a protocol inheriting from a regular class? Hm, I guess there's ambiguity over whether the methods in that regular class all become protocol methods with default implementations. Hm... I guess it does make sense, but maybe an explicit example showing the rationale would help.

[gvanrossum]

I would just write this out rather than letting the reader guess to where exactly this is referring.

[gvanrossum]

will be -> are

[gvanrossum]

That sounds a little weird -- certainly an instance of a class that implements a protocol is a value of that protocol type? Maybe all you need is to add "exactly" somewhere to the sentence.

[gvanrossum]

OK, this is the rest of my review. Hopefully I didn't repeat myself too much or ignore your responses to the earlier review (honestly I'm too tired to look it up).

[gvanrossum]

If X is a protocol itself, "implements" gets a different meaning, right? I think in that case it just means that it must define all of P's methods right? Maybe it would be less ambiguous if you separated the two cases into separate bullets.

[ilevkivskyi]

Maybe it would be less ambiguous if you separated the two cases into separate bullets.

I agree.

[gvanrossum]

I think this deserves calling out, rather than just "Note".

[gvanrossum]

Hm, I'm not sure I like this example. Isn't the problem more that Traversible ought to be generic but isn't in this example?

[ilevkivskyi]

Isn't the problem more that Traversible ought to be generic but isn't in this example?

No, this "problem" (I would rather say "rule") applies to all recursive protocols. For example:

class Linked(Protocol):
    child: 'Linked'

class C:
    child: 'C'

x: Linked
x = C() # This should be OK, according to the "rule"

I think we just need a simpler example here.

[gvanrossum]

Agreed. The generics just distract. (Maybe we need a simple example first and then one showing how to combine this with generics?)

[gvanrossum]

I'd write "Quittable" -- just like it's "quitting" vs. "exiting". (English is never regular. :-)

[gvanrossum]

Can you find a different name for this? "Good job" is such an incredibly boring cliché...

[gvanrossum]

I think they should also be allowed for Python 3 (even 3.6+) and the text earlier that introduces these should mention this. (It's fine of course to recommend PEP 526 syntax and to use it in all examples except one showing the fallback syntax.)

[gvanrossum]

"In Go language" -> either "In Go" or "In the Go language"

[gvanrossum]

So this more or less assumes that the runtime isinstance check doesn't compare the arguments. But what if it did?

Or are you saying that in this case if P was decorated with @runtime the isinstance check would still be broken?

[ilevkivskyi]

Or are you saying that in this case if P was decorated with @runtime the isinstance check would still be broken?

I think yes, for example now isinstance(obj, Iterable) returns True for a class with __iter__ = 42. And we can't make @runtime much better because not all information is available statically. As Jukka said, it is not a big problem since static type checker will warn about this, but big enough to make support for isinstance() with protocols opt-in.

[gvanrossum]

OK, I'm convinced (also this review UI is very confusing, it seems we are having discussion on different subcommits of the same PR).

I still don't like the name @runtime though -- it's too generic. Can we brainstorm about a better name?

[gvanrossum]

OK, I am actually convinced by @JukkaL's argument here. (But we should probably apply a similar reasoning to cut out some other esoteric ideas, like variables.)

[gvanrossum]

I wonder if we could do more deep linking to specific comments or subthreads? (No big deal if you think it's too much effort, but it might save some readers some time finding the right part of the long discussion in e.g. python/typing#11 -- including myself :-).

[gvanrossum]

There's a couple more things that I should bring up.

It looks like the mypy core team (all Dropboxers) is going to be very busy with other features, to the point where we probably won't have time to even review an implementation of this PEP for the foreseeable future. Now, this is all subject to various priorities that may still change, and perhaps the implementation is super simple, but if it's anything like I expect, even a complete, finished implementation (and one that allows importing Protocol from mypy_extensions so it can be made to work with the stdlib versions of typing in earlier Python 3.5 and 3.6 releases) would still take up quite some time to review, time which we just may not have.

There's also the absolute requirement to send this PEP to python-dev for discussion. Since I'm not an author we don't need a BDFL-delegate -- we just need to be careful to listen to feedback. And while I am aware that my own time is a particularly scarce resource, I still don't want to delegate this particular topic.

[ilevkivskyi]

@gvanrossum

There's also the absolute requirement to send this PEP to python-dev for discussion.

OK, I will do this now.

[ilevkivskyi]

@gvanrossum

we probably won't have time to even review an implementation of this PEP for the foreseeable future.

This sounds a bit frustrating, but what do you mean by "foreseeable future", few years or few days?
Also, @ddfisher said above in #224 (comment) that "I don't think it's a good idea to standardize this before having an implementation", so that I thought that I should finish an implementation soon (even if not 100% complete).
I am a bit lost here.

[gvanrossum]

This sounds a bit frustrating, but what do you mean by "foreseeable future", few years or few days?

A few months at least, is my best estimate based on discussions we had in our project planning (not all [arts of the project are open source).

I agree it's frustrating -- it is for us too, but we need to acknowledge that we have limited velocity.

Also, @ddfisher said above in #224 (comment) that "I don't think it's a good idea to standardize this before having an implementation", so that I thought that I should finish an implementation soon (even if not 100% complete).

I believe David is more skeptical of this PEP than Jukka and I. But even so, the extra frustrating thing is that, indeed, the PEP process (by convention at least) prefers to have a reference implementation to support a proposal, without any guarantees that the reference implementation will ever be deployed. This is similar to the RFC-based IETF standardization process for internet protocols -- and the similarity is not coincidental, we modeled Python's PEP process after the RFC process (with which several core Python devs were highly familiar between 1995-2000).

I am a bit lost here.

I'm truly sorry. It is frustrating for all of us (as are my continued health issues that reduce my productivity dramatically).

[ambv]

@gvanrossum

we need to acknowledge that we have limited velocity.

What would it take for the project to nominate a few non-Dropbox core contributors?

A few months at least

This is not a huge obstacle for inclusion in Python 3.7, as long as we get to it sometime this year. Also, a reference implementation for the purpose of this PEP can be on a "branch" of mypy. In other words, for the PEP reviewers to consider it, it doesn't have to be merged. This was pretty common in the past.

@ilevkivskyi, I will provide preliminary review and use it internally before it gets merged (we are living on master anyway).

[ilevkivskyi]

OK, what about this way: I will have a branch in my fork of mypy, where I will build a reference implementation. People will be able to play with it (I will take care to merge mypy master frequently, so that the branch will be fully functional). And it is fine with me if it will be not merged for few months. To avoid a potential huge (and impossible to review) PR if/when the PEP is accepted, I will cherry-pick few smaller branches and make few sequential PRs.

@gvanrossum @ambv @JukkaL does this sound reasonable?

[gvanrossum]

It sounds reasonable (though I cannot make any hard promises).

CC @gnprice.