Proposal: typed yield expression, better generic function inference

[rerion]

tldr

This has exactly the same scope as #32523, except it has implementation proposal.

Search Terms

generator, generator function, yield, yield type, generic parameter typing

Setup

Proposal is to improve typing of yield expression, as explained here #32523.
In short with code like

function promiseRunner<Args extends any[], R>(mkGen: (...args: Args) => Generator<Promise<any>, R, any>): (...args: Args) => Promise<R> {
    return (...args) => {
        const gen = mkGen(...args);

        const innerRunner: (v?: any) => Promise<R> = (nextValue?: any) => {
            const next = gen.next(nextValue);
            return next.done ? Promise.resolve(next.value) :
                   Promise.resolve(next.value).then(innerRunner)
        };

        return innerRunner();
    }
}

type User = { id: string; age: number; };
declare function getUser(): Promise<User>

function *userAge(): Generator<Promise<any>, number, any> {
    // inferred type of (yield getUser()) is any
    const user = (yield getUser()) as User;

    return user.age;
}

even though developer KNOWS user should be of type User, she can't denote it in TypeScript.

Proposed changes

generic function return type inference

having function with signature like <T>(p: Promise<T>) => T and value of type Promise<number>
typescript can correctly infer type resulting from applying the value to function:

declare const unwrapPromise: <T>(t: Promise<T>) => T;
declare const awaitingNumber: Promise<number>;
const resOf = unwrapPromise(awaitingNumber); // inferred number

except when it can't:

type Fn<Args extends any[], Res> = (...args: Args) => Res;

type CallResult<Args extends any[], F extends Fn<any, any>> =
    F extends Fn<Args, infer G> ? G : never;

type UnwrapPromise = <T>(t: Promise<T>) => T;

type ExpectedStringGotUnknown =
    CallResult<[Promise<string>], UnwrapPromise>;

// typescript infers unknown, I infer string

I'd personally expect CallResult to work exactly like typescript call result inference.

typed yield

Developer should be able to inform typescript:
"yield has signature <T>(p: Promise<T>) => T inside this generator function"

this would be realised by providing supplementary type

type MonadicGenerator<TReturn, TUnwrap extends (yielded: any) => any> = Generator<
    Parameters<TUnwrap>[0],
    TReturn,
    ReturnType<TUnwrap>>

with usage like

// second argument is "type of yield" within generator
function *userAge(): MonadicGenerator<number, <T>(p: Promise<T>) => T> {
    const userP = getUser();
    const user = yield userP; // inferred type User, because
    // CallType<[typeof userP], <T>(p: Promise<T>) => T>  == User

    return user.age;
}

comments

type MonadicGenerator<TReturn, TUnwrap extends (yielded: any) => any> = Generator<
    Parameters<TUnwrap>[0],
    TReturn,
    ReturnType<TUnwrap>>

MonadicGenerator<number, <T>(p: Promise<T>) => T> 
== 
Generator<Promise<unknown>, number, unknown>

so the generator consumer facing type is correct -- we don't know Promise of what generator will yield.

This definition allows setting explicit relation between yielded value and next resumed value types, which typescript should be able to exploit, because it already does analogous things as seen here

declare const unwrapPromise: <T>(t: Promise<T>) => T;
declare const awaitingNumber: Promise<number>;
const resOf = unwrapPromise(awaitingNumber); // inferred number

The reason why CallResult doesn't work as intended is that generic parameter gets erased when the type is used as parameter.
I'd argue that generic functions are really primitive to javascript, since this is a thing:

function box(value) {
    return [value];
}

and natural signature for box is <T>(value: T) => [T], and it shouldn't be erased.

That said, even if this isn't implemented in general, it might be worth it for one special type,
consumed with particular syntactic feature.

[tadhgmister]

So we want a way of saying that if we yield a Promise<User> then yield would definitely return a User. This implies that the code that uses the generator when calling gen.next() produces a Promise< User > then we have to force the next call to gen.next() to pass in a User.

How could we possibly enforce this? It suggests that the actual interface of the generator changes every time you call next because what is allowed to be passed to the .next function changes every time it is called.

This works the opposite way as well, we could implement a state machine with a generator such that the values yielded is directly related to the value sent into the generator, meaning inside the generator the valid type to yield is constrained by the result of the previous yield.

So we actually have 2 call signatures, one for yield and one for gen.next which are essentially 2 sides to the same coin:

type VerySpecificGeneratorDefinition_ThatNeverReturns<
    Produce,
    Send,
    YieldSignature extends (yielded: Produce) => Send,
    NextSignature extends (sent: Send) => Produce
> = Generator<Produce, never, Send> & {next(...a: Parameters<NextSignature>): IteratorYieldResult<ReturnType<NextSignature>>}

except this still assumes the exchange is only based on what is passed in and out of the generator, what about a generator that produces a string then a number then a boolean then stops? So we would need a way of also representing that the generator state may change and what is produced by gen.next changes each time it's called....

So no, I'm pretty sure "Design Limitation" is still the appropriate tag for this request.

[rerion]

Hi, @tadhgmister, thank you for your response.

So we want a way of saying that if we yield a Promise then yield would definitely return a User.

Yes.

This implies that the code that uses the generator when calling gen.next() produces a Promise< User > then we have to force the next call to gen.next() to pass in a User.

No. This is not about forcing consumption of generator in any way, based on this typing.
Consumer (writer of say promiseRunner), still sees Generator<Promise<unknown>, number, unknown> type, and this is what compiler enforces.
Now the consumer SHOULD implement logic in the way You described, but as You noted it would be impossible for typescript to enforce it.

What I want, is for generator function author, to enforce correct typing when writing generator function, when consumer side is known beforehand.

---

So we actually have 2 call signatures, one for yield and one for gen.next which are essentially 2 sides to the same coin { ... } except this still assumes the exchange is only based on what is passed in and out of the generator, what about a generator that produces a string then a number then a boolean then stops?

Again,
this is about generator function author (not generator runner author) to be able to say:
"hey I know from external sources how this generator is going to be consumed, so please correctly infer type of resumed variable" not about actually saying what the generator will produce.

This has specific use case in mind:

1. First there is some known context for Generator, ie some code that will run it
2. From that we know the correct signature of yield within generator constructor we are writing.
3. So we would like to inform TypeScript about this correct signature.

There are examples of this in JavaScript (mobx flow, co, bluebird, redux-saga),
and TypeScript aims to give Javascript idioms types right?

---

My guess to what "design limitation" would be here is in my second comment in the original post:
it would require unreasonable amount of changes to not erase the generic parameter of a generic function.

[tadhgmister]

As a work around, I do want to mention that yield* otherGen() accurately type checks the return of the other generator, so if you are willing to replace x = yield v with x = yield* WRAPPER(v) we could pretty easily write a definition for WRAPPER that ensures correct typesafety. Playground Link

/**
 * creates a simple wrapper function that ensures typesafety on the yield statement for libraries.
 * So when you would do something like `const x = yield val` and the type of x depends on val somehow, you can do
 * ```typescript
 * type YieldSig = <T>(val: T)=>T // signaure of yield expression for your case.
 * const Yield = makeTypesafeYielder({} as YieldSig);
 *
 * function* example(){
 *     // we do yield* and it ensures typesafety
 *     const x = yield* Yield(val)
 * }
 * ```
 * @param psudoCallSignature not used, should be passed as `{} as Sig` where Sig is the call signature of the yield enforced by your library.
 */
function makeTypesafeYielder<Produce, Send>(psudoCallSignature: (y: Produce) => Send) {
    function* yielder(arg: Produce): Generator<Produce, Send, Send> {
        return yield arg;
    }
    return yielder;
}
type Y = <T>(p: Promise<T>) => T;
const Y = makeTypesafeYielder({} as Y);

type User = { id: string; age: number };
declare function getUser(): Promise<User>;

function* userAge() {
    // inferred type is correct.
    const user = yield* Y(getUser());
    return user.age;
}

As for you suggestion, this part seems very not sound:

Consumer (writer of say promiseRunner), still sees Generator<Promise<unknown>, number, unknown> type, and this is what compiler enforces.
Now the consumer SHOULD implement logic in the way You described, but as You noted it would be impossible for typescript to enforce it.

I understand you are only using your generators to pass to a function that will handle your generator in a particular way, but if the return signature is Generator<X, Y, unknown> then I could very easily write a consumer code that sends in totally invalid values. We can actually use the above technique to demonstrate that the accurate send type is the intersection of all expected send types, so if our first yield produces A then the second produces B then our send type is A & B since that is the only type that can be sent in that satisfies both scenarios:

// returns Generator<Promise<A> | Promise<B>, [A, B], A & B>
//                                                 look ^ here - intersection.
function* resolveBoth<A,B>(pa: Promise<A>, pb: Promise<B>){
    const a = yield* Y(pa);
    const b = yield* Y(pb);
    return [a,b] as [A,B];
}

So to accurately represent our case we would need to specify the call signature of the yield expression, but still infer the send types from the actual yields inside the generator. Except we can't do this because the only place we can write details about our generator is in the return as Generator<X,Y,Z> but this means that no types are inferred from the body, so again we can't do this because of design limitations.

I also want to point out that nearly every case that uses this (mobx flow, co, bluebird) including your example are being migrated to just use await since that does precisely what you intend. and for redux-saga it honestly looks like it would benefit from using yield* instead of just yield... like this is from the description of call(fn, ...args)

If the result is an Iterator object, the middleware will run that Generator function, [...]. The parent Generator will be suspended until the child Generator terminates normally, in which case the parent Generator is resumed with the value returned by the child Generator. Or until the child aborts with some error, in which case an error will be thrown inside the parent Generator.

This is literally describing exactly what yield* does, so if you know the function is a generator then yield call(fn, x) would identically translate to yield* fn(x) except you would keep type safety on the return. Not to mention the nice duality between yield* and await - within a Sega generator you call other Sega generators with yield* in the same way you call async functions with await.

---

Just to be clear I want this feature as much as you do. But the point of typescript is to give you nice type contracts, so to ensure a lot of contracts within a generator that aren't even attempted to be enforced by the consumer code seems like a real mistake to apply to the general case. And we can't make these contracts that are enforced on both sides because of design limitations. I hope my work around works for you, have a nice day. :)

[rerion]

First of all, Thank You!
I can live with the workaround, and I feel like I learned something new :)

---

That said

I understand you are only using your generators to pass to a function that will handle your generator in a particular way, but if the return signature is Generator<X, Y, unknown> then I could very easily write a consumer code that sends in totally invalid values.

True.
But it's no better currently.

// notice this also infers unknown as TYield
function *userAge() { // inferred Generator<Promise<User>, any, unknown> 
    const user = yield getUser(); // user is any

    return user.age; // so returned type is any
}

// this works for singular types
function *userAge(): Generator<Promise<User>, number, User> {
    const user = yield getUser();

    return user.age;
}
== equiv
function *userAge() {
    const user: User = yield getUser();

    return user.age;
}

// inferred Generator<Promise<User> | Promise<Account>, number, User & Account>
// the intersection may be correct, but impossible to satisfy
// can't really provide User & Account can I?
function *userAge() {
    const user: User = yield getUser();
    const account: Account = yield getAccount();

    return user.age + account.users.length;
}
// we end up with
function *userAge(): Generator<Promise<User> | Promise<Account>, number, User | Account> {
    const user = (yield getUser()) as User;
    const account = (yield getAccount()) as Account;

    return user.age + account.users.length;
}

const userAgeGen = userAge();
const fst = userAge.next();
const snd = userAge.next({} as Account); // no complaints
// oh no! I can send totally invalid values into our generator

So I argue, while my solution doesn't add any type safety consumer side, it doesn't remove it either. It's just the same.

But the point of typescript is to give you nice type contracts, so to ensure a lot of contracts within a generator that aren't even attempted to be enforced by the consumer code seems like a real mistake to apply to the general case. And we can't make these contracts that are enforced on both sides because of design limitations.

I didn't want to remove old Generator interface. I wanted to provide alternative, as in
"this function* thingey returns Generator<.....> or MonadicGenerator<....> (defaulting to Generator if ambient).
That's certainly not impossible.
The point is, my proposition wasn't about general case. It was about specific use case.

And why not add extra contracts if its possible, it's helpful (sometimes) and it doesn't break anything else.

---

I also want to point out that nearly every case that uses this (mobx flow, co, bluebird) including your example are being migrated to just use await since that does precisely what you intend

Picking promises as an example was a mistake.
Consider parser combinators:

type Parser<T> = (s: Source) => [T, Source]
type CoBind = <T>(p: Parser<T>) => T;

function defineParser<Args extends any[], Res>(
    definition: (...args: Args) => Generator<???> // Generator<Parser<unknown>, Res, unknown>
): (...args: Args) => Parser<Res> {
    // impl
    // 
}

Can't really use await, because await is just for async problems.

[tadhgmister]

my proposition wasn't about general case. It was about specific use case.

That is fair and I know you won't misuse the MonadicGenerator but you have to recognize that the moment a feature gets into the language people will be using it, someone will get errors because their consumer didn't do it's job and blame typescript for not informing them.

Picking promises as an example was a mistake.

No you are fine. It is a good example that most people can relate to, I was only countering the argument that this is a very common use case. Just because most of the actual use cases are being replaced with await that doesn't mean there aren't cases that absolutely do use this. redux-saga has made it very clear they need yield and my counter of "well the library is wrong, it should be using yield* instead" is as weak an argument as you can get. 😝

I have literally implemented a version of this, the goal being that if all await points were available synchronously then the function would just return synchronously. The idea is that I could write a function with this signature:

interface DreamArray<T> {
    find(predicate: (elem: T)=>boolean): T | undefined;
    find(predicate: (elem: T)=>Promise<boolean>): Promise<T | undefined>
    find(predicate: (elem: T)=>boolean | Promise<boolean>): T | undefined | Promise<T | undefined>
}

To accomplish this I ended up writing generators where every yield was possibly a promise and the send value was always the unwrapping of it. The difference is in my scenario I was responsible for both the producer and consumer side so I wanted both to be type safe. It didn't take long to realize that wasn't possible.

// the intersection may be correct, but impossible to satisfy
// can't really provide User & Account can I?

You see that is the beauty of it, if we accept that not both the producer and consumer are going to be nicely type safe in this scenario we have to choose one to be not type safe. So if the generator side writes itself to be totally type safe and the consumer can't possibly provide User & Account then we have to write the consumer with type casts (x as T) or explicit any variables. DONE! We have a producer which is type safe and a consumer which isn't.

Conversely if the generator does type casts (yield getUser()) as User then if things go wrong I'd start debugging assuming the issue is with the generator - either the signature it exposes or the internal handling is incorrect. (so we have a type safe consumer and non type safe producer)

The issue with your suggestion for MonadicGenerator is that it would let both the producer and consumer codes to be written without type casts or any variables and still be incorrect, so debugging would take a bit longer and would eventually come to the conclusion that the issue is with typescript. I'm not talking hypothetically here, I am describing my experience with this bug yesterday. You will notice that "don't allow users to declare that type" was the solution there and I'm afraid it's likely the solution here too.

To be clear, I can see the value of this kind of feature but I would want it to fit all these criteria:

• somewhere in the generator (preferably near the declaration) we specify the effective call signature of yield and have the yield expressions inside the generator type checked based on that
• the generator return type is still inferred from the body, so the produce is union of all yielded values and send is still intersection of all sends. (so that consumer must either support all yield points or be non type safe)
• The current behaviour is that if we define an explicit return type then nothing is inferred from the body, so we can't use the return type (or change that behaviour in a way that is still intuitive for the general case)
• the implementation effort and amount of new syntax should be proportional to how much the use case really needs it and how common this use case is.

If you can think of a way to meet all of these I will back you 100%, but if no one can come up with a good solution that meets this then that is kind of the definition of "Design Limitation".

Happy coding.😁

[rerion]

I know you won't misuse the MonadicGenerator but you have to recognize that the moment a feature gets into the language people will be using it, someone will get errors because their consumer didn't do it's job and blame typescript for not informing them.

Thats fair.

The issue with your suggestion for MonadicGenerator is that it would let both the producer and consumer codes to be written without type casts or any variables and still be incorrect, so debugging would take a bit longer and would eventually come to the conclusion that the issue is with typescript.

That's an excellent point. That scenario actually didn't occur to me, maybe because I rarely see those nullish types.

---

As a last try:

function *userAge(yield: <R>(p: Promise<R>) => R) { // return type implicit
    const user = yield getUser(); // correctly inferred because of yield pseudo-param
    const account = yield getAccount();

    return user.age + account.users.length;
}

where yield is optional fixed parameter, similar to this parameter.
(thats totally analogous -> this param is used to change how typescript sees this inside body,
yieldT would change how yield is seen).

Now there is no problem with using current rules for return type inference,

typeof userAge 
    == (yield: <R>(p: Promise<R>) => R) => Generator<Promise<User> | Promise<Account>, number, User & Account>
ReturnType<typeof userAge> 
    == Generator<Promise<User> | Promise<Account>, number, User & Account>

From user perspective this doesn't add much overhead, similarily as this parameter is mostly unused, unless you want to be strict about handlers.

I'm curious how would You feel about this :)

[tadhgmister]

yield is optional fixed parameter, similar to this parameter.

😮 Now that is a suggestion I can get behind! 🎉 👍 You will want to close this thread (since I have convinced you that your original suggestion wouldn't work) and open a new one for this. Yeah you have my support! I have a few thoughts on this below:

I do think that - at least for the initial implementation - the yield parameter would only serve the purpose of type contract and wouldn't be visible outside the generator, so the signature would still be ()=>Generator<...>. retaining it in the call signature like this is wouldn't actually affect how it is called so it's not necessary. (except maybe for yield* as I mention below, but I feel like that is a discussion for after someone on the team actually shows interest)

If we did in the future have something akin to self mutating methods (so that calling gen.next() changed the valid argument to the next call) we could use the yield declaration to affect the inferred Generator type so a consumer could be type safe. Probably a super long shot but point is that it would be compatible which is not true for many other suggestions I've seen.

Few points of validity:

• yield can only be used on generator functions, If a this parameter is present which should come first?
• the type of yield must be non optional and extends (p: any)=>any, possibly with overloads.
• a yield parameter and an explicit return type are not allowed together, since the whole point of yield is to infer the Generator generics from usage in the body.
  • this is a weird restriction that is unlike anything else in typescript, feel like this is a weak point in the suggestion
• yield expressions inside the generator are typed the same way as calling the pseudo yield parameter
• the returned generator type is inferred the same way we currently determine T and TNext (where T is union of all yielded values and TNext is intersection of explicit type annotations which are assigned to a yield expression) but we also have the capability to add a constraint on yielded values and infer the return types instead of relying on explicit type annotations.
• yield* expressions are... oh shoot how would this work?
  • If we totally don't care about possibly giving a hand to the consumer side in the future we can just say yield* aren't changed at all
  • in most use cases I'd expect the generator in yield* must have a compatible yield signature
• outside the generator body the yield parameter has no impact on the call signature of the original function.
• compiling would simply remove the yield parameter in the same way it removes a this parameter.

[rerion]

Sure.
I'm closing this, will open proper issue over the weekend.

Happy coding :)

[jonaskello]

For reference the new issue is in #36967