Add new model (upstage/solar-pro-preview-instruct)

[remy-rec]

We have released a new model and would like to add it to the HELM leaderboard. We have confirmed that it works well on the local leaderboard.
If a merge is possible, could you let me know by when the merge will be completed?
You can check our model's performance at: https://www.upstage.ai/products/solar-pro-preview.

model_deployments.yaml

  - name: upstage/solar-pro-preview-instruct
    model_name: upstage/solar-pro-preview-instruct
    tokenizer_name: upstage/solar-pro-preview-instruct
    max_sequence_length: 4096
    client_spec:
      class_name: "helm.clients.huggingface_client.HuggingFaceClient"
      args:
        torch_dtype: auto
        trust_remote_code: true

model_metadata.yaml

  - name: upstage/solar-pro-preview-instruct
    display_name: solar-pro-preview-instruct (22B)
    description: solar-pro-preview-instruct (22B) is the most intelligent LLM on a single GPU ([blog](https://www.upstage.ai/products/solar-pro-preview))
    creator_organization_name: Upstage
    access: open
    num_parameters: 22000000000
    release_date: 2024-09-11
    tags: [TEXT_MODEL_TAG, LIMITED_FUNCTIONALITY_TEXT_MODEL_TAG]

tokenizer_configs.yaml

  # Upstage
  - name: upstage/solar-pro-preview-instruct
    tokenizer_spec:
      class_name: "helm.tokenizers.huggingface_tokenizer.HuggingFaceTokenizer"
      args:
        trust_remote_code: true
    end_of_text_token: "<|im_end|>"
    prefix_token: "<|startoftext|>"

[yifanmai]

Thanks for the suggestion! I'll discuss with the team internally and figure out what we want to do here.

[nayeon-upstage]

Hi @yifanmai ! I am Nayeon form Upstage. I respectfully request your team's consideration to include Solar in the leaderboard. Please let me know if you require further information or �tasks. Thank you!

[yifanmai]

Hi @nayeon-upstage, I've added Solar to the queue of models to add to the leaderboard. Would you already have results from internal runs that you could share?

[nayeon-upstage]

Thank you so much for your attention.Here is the results of solar-pro-preview-instruct(22b) uploaded to the local server through captures! It's a 22b model, but its performance is good compared to larger models. The file is compressed and exceeds 2G. If you need it as a file, I can send it to you via email. (please note that the latest template is not released, so it's a past bench LEADERBOARD_VERSION=v0.3.0)