add extra parameters to the from_pretrained method

[not-lain]

I think we should update the from_pretrained method, for example, we can :

• add a token parameter there and pass the token explicitly
• revision parameter to load a tokenizer from a specific commit, pr or a branch.
• others ...

[bhavnicksm]

Hey @not-lain,

Thanks for opening an issue (^-^)

Just wanted to confirm some things:

• By token parameter, do you mean the hf_token?
• As far as I know, Tokenizers rarely change over revisions. Do you see it being used in a specific tokenizer case that changes often?

Thanks!

[not-lain]

Hi @bhavnicksm

By token parameter, do you mean the hf_token?

yess, exactly, the token can either be passed explicitly, or retrieved from the login credentials

As far as I know, Tokenizers rarely change over revisions. Do you see it being used in a specific tokenizer case that changes often?

To be 100% transparent, I haven't seen anything like that, but it could be a nice addition for developers.

Let me know how if you have any feedback about any of these parameters.

[bhavnicksm]

By token parameter, do you mean the hf_token?

yess, exactly, the token can either be passed explicitly, or retrieved from the login credentials

Hey @not-lain!

Thanks for the update! Yes, token makes sense for sure~ I'd prefer holding off on revisions for now since we don't see it being needed just yet... to keep the argument space a bit minimal. If it is required in the future, we can add it then!

Thanks! :)