Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add eval dataset support #4691

Merged
merged 8 commits into from
Jul 14, 2024
Merged

Conversation

codemayq
Copy link
Collaborator

@codemayq codemayq commented Jul 5, 2024

  1. merge load dataset and split dataset function

What does this PR do?

  1. add custom eval dataset support during training
  2. reconstruct the load dataset and split dataset functions

fix #2290
fix #3974

Before submitting

2. merge load dataset and split dataset function
@hiyouga hiyouga self-requested a review July 13, 2024 15:40
Copy link
Owner

@hiyouga hiyouga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hiyouga hiyouga merged commit 15b399a into hiyouga:main Jul 14, 2024
1 check failed
@dqgdqg
Copy link

dqgdqg commented Jul 14, 2024

Thanks for the valuable update. But now it is a little bit confusing.

Could you please give an example of **.yaml configuration to specify different train and val datasets instead of val_size?

@hiyouga
Copy link
Owner

hiyouga commented Jul 14, 2024

Thanks for the valuable update. But now it is a little bit confusing.

Could you please give an example of **.yaml configuration to specify different train and val datasets instead of val_size?

remove val_size and add eval_dataset

@dqgdqg
Copy link

dqgdqg commented Jul 17, 2024

It works. Thanks.

@Wolverhampton0
Copy link

还是报错:ValueError: Some keys are not used by the HfArgumentParser: ['eval_dataset']是为何?需要更新环境吗?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants