support ORPO

[hiyouga]

What does this PR do?

Support ORPO algorithm: https://arxiv.org/abs/2403.07691

Before submitting

• Did you read the contributor guideline?

[JianbangZ]

@hiyouga
请问怎么用ORPO文章里的这个数据集https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned
来用LLAMA-FACTORY训练ORPO？

这个数据集是如下format
{
"prompt": "Which books are still banned in Canada?",
"chosen": [
{
"content": "Which books are still banned in Canada?",
"role": "user"
},
{
"content": "As a responsible and ethical AI assistant, I must inform you that I cannot provide a list of banned books in Canada, as there are no books that are currently banned in the country. Canada has a strong commitment to freedom of expression and the right to access information, and there are no laws in place that restrict the publication or distribution of books.\n",
"role": "assistant"
}
],
"messages": [
{
"content": "Which books are still banned in Canada?",
"role": "user"
},
{
"content": "As a responsible and ethical AI assistant, I must inform you that I cannot provide a list of banned books in Canada, as there are no books that are currently banned in the country. Canada has a strong commitment to freedom of expression and the right to access information, and there are no laws in place that restrict the publication or distribution of books.\n",
"role": "assistant"
}],

"prompt_id": "aeccf551d9ba42fdf5f2044de43b8ce6e360fb523ace428317b81d804594e090",
"rejected": [
{
"content": "Which books are still banned in Canada?",
"role": "user"},
{
"content": "According to the Canadian Government’s Ban Affront website, there are still several books that are banned in Canada. These include The Begum’s Millionaire, The Education of Little Tree, The Harry Potter series, Lolita, 1984, and Lady Chatterley’s Lover. Some of these books are considered inaccessible due to their age, while others are still legally banned in certain parts of the country.",
"role": "assistant"
}
],
"score_chosen": 8.0,
"score_rejected": 5.0
}