fixbug：llama3在增量预训练时应该使用<|end_of_text|>标识文本的结束 #4204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…text|>' ，这里在每条数据后面也得用这个，而不是'<|eot_id|>'，否则很容易导致严重的性能下降
dignfei
changed the title
hiyouga
added
pending
hiyouga
added
solved
|
根据 https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/discussions/105 该问题可能是由于Llama-3的tokenizer_config.json中设定："eos_token": "<|end_of_text|>" 因此在后续版本中（tokenizer_config.json中设定："eos_token": "<|eot_id|>"）应该不存在这个问题 建议新设置llama3x的template，作为Llama-3.1等后续版本的template，从而在预训练数据的预处理中能够加入<|eot_id|> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
llama3在预训练时使用的tokenizer.eos_toke是'<|end_of_text|>'，而Meta-Llama-3-8B-Instruct和很多llama3中文模型 都改成了<|eot_id|> ，进行增量预训练时应该使用'<|end_of_text|>' 。
Fixes # (issue)
经过大量的增量预训练，进行对比试验，发现这个bug：llama3在预训练时使用的tokenizer.eos_toke是'<|end_of_text|>' ，这里在每条数据后面也得用这个，而不是'<|eot_id|>'，否则增量预训练时很容易导致严重的性能下降
Before submitting