🚀 New Release: cerbero-7b-openchat our latest SOTA model based on openchat3.5, delivering performance on par with or superior to ChatGPT 3.5!
🔥 The research paper unveiling the secrets behind cerbero-7b is now available on arXiv!
📢 cerbero-7b is the first 100% Free and Open Source Italian Large Language Model (LLM) ready to be used for research or commercial applications.
Try an online demo here
Built on top of mistral-7b, which outperforms Llama2 13B across all benchmarks and surpasses Llama1 34B in numerous metrics.
cerbero-7b is specifically crafted to fill the void in Italy's AI landscape.
A cambrian explosion of Italian Language Models is essential for building advanced AI architectures that can cater to the diverse needs of the population.
cerbero-7b, alongside companions like Camoscio and Fauno, aims to help kick-start this revolution in Italy, ushering in an era where sophisticated AI solutions can seamlessly interact with and understand the intricacies of the Italian language, thereby empowering innovation across industries and fostering a deeper connection between technology and the people it serves.
cerbero-7b is released under the permissive Apache 2.0 license, allowing unrestricted usage, even for commercial applications.
The cerbero-7b model has been rigorously evaluated across several benchmarks to demonstrate its proficiency in understanding and generating Italian text. Below are the summarized results showcasing its performance:
The Stanford Question Answering Dataset (SQuAD) in Italian (SQuAD-it) is used to evaluate the model's reading comprehension and question-answering capabilities. The following table presents the F1 score and Exact Match (EM) metrics:
| Model | F1 Score | Exact Match (EM) |
|---|---|---|
| cerbero-7b-openchat | 74.09% | 56.0% |
| cerbero-7b | 72.55% | 55.6% |
| Fauno | 44.46% | 0.00% |
| Camoscio | 37.42% | 0.00% |
| mistral-7b | 15.55% | 8.50% |
EVALITA benchmarks assess the model's performance in tasks like toxicity detection, irony detection, and sentiment analysis. The table below shows the F1 scores for these tasks:
| Model | Toxicity Detection | Irony Detection | Sentiment Analysis |
|---|---|---|---|
| cerbero-7b-openchat | 63.33% | 69.16% | 66.89% |
| cerbero-7b | 63.04% | 48.51% | 61.80% |
| Fauno | 33.84% | 39.17% | 12.23% |
| Camoscio | 38.18% | 39.65% | 13.33% |
| mistral-7b | 34.16% | 34.16% | 12.14% |
The name "Cerbero," inspired by the three-headed dog that guards the gates of the Underworld in Greek mythology, encapsulates the essence of our model, drawing strength from three foundational pillars:
-
Base Model: mistral-7b 🏗️ cerbero-7b builds upon the formidable mistral-7b as its base model. This choice ensures a robust foundation, leveraging the power and capabilities of a cutting-edge language model.
-
Datasets: Cerbero Dataset 📚 The Cerbero Dataset is a groundbreaking collection specifically curated to enhance the proficiency of cerbero-7b in understanding and generating Italian text. This dataset is a product of an innovative method combining dynamic self-chat mechanisms with advanced Large Language Model (LLM) technology. Refer to the paper for more details.
-
Licensing: Apache 2.0 🕊️ Released under the permissive Apache 2.0 license, cerbero-7b promotes openness and collaboration. This licensing choice empowers developers with the freedom for unrestricted usage, fostering a community-driven approach to advancing AI in Italy and beyond.
cerbero-7b is available in various flavors, each tailored for specific applications and use cases. Below is a table listing these versions along with their respective training datasets and base models:
| Model Name | Training Dataset | Base Model | Huggingface Model | Llama.cpp and Quantized Model |
|---|---|---|---|---|
| cerbero-7b | Cerbero Dataset | mistral-7b | link | link |
| cerbero-7b-openchat | Cerbero Dataset | openchat3.5 | link | link |
Each of these models brings its unique strengths to the table, making cerbero-7b a versatile tool for both research and commercial applications in the Italian language AI domain.
We are committed to continuously enhancing cerbero-7b. Our team plans to keep training and releasing new models as advancements in the 7b SOTA occur. This ensures that cerbero-7b remains at the forefront of AI technology, offering the most advanced and efficient solutions in the Italian language AI sector.
If you do not have enough RAM to fit the float32 model (for example when using Colab) we provide for each model a float16 version using the revision="float16" argument
model = AutoModelForCausalLM.from_pretrained("galatolo/cerbero-7b", revision="float16")cerbero-7b is a fully fine-tuned LLM, distinguishing itself from LORA or QLORA fine-tunes. The model is trained on an expansive Italian Large Language Model (LLM) using synthetic datasets generated through dynamic self-chat on a large context window of 8192 tokens
📢 Details on the Cerbero Dataset will be updated shortly!
cerbero-7b is trained on an NVIDIA DGX H100:
- Hardware: Utilizing 8xH100 GPUs, each with 80 GB VRAM. 🖥️
- Parallelism: DeepSpeed Zero stage 1 parallelism for optimal training efficiency.✨
The model has been trained for 1 epoch, ensuring a convergence of knowledge and proficiency in handling diverse linguistic tasks.
cerbero-7b supports 🤗Chat Templates using the tokenizer.apply_chat_template function.
The model is trained on full conversations using the following prompt format:
[|Umano|] First human message
[|Assistente|] First AI reply
[|Umano|] Second human message
[|Assistente|] Second AI reply
When crafting prompts, ensure to conclude with the [|Assistente|] tag, signaling the AI to generate a response.
Use [|Umano|] as stop word.
For example:
[|Umano|] Come posso distinguere un AI da un umano?
[|Assistente|]
While it's possible to include a brief system message at the start of your prompt, remember that the training data for cerbero-7b does not contain such system messages. Hence, it's recommended to minimize or avoid including them for optimal model performance.
You can load cerbero-7b (or cerbero-7b-openchat) using 🤗transformers
import torch
from transformers import pipeline
pipe = pipeline('text-generation', model="galatolo/cerbero-7b")
messages=[
{
"role": "system",
"content": "Conversazione tra un umano ed un assistente AI."
},
{
"role": "user",
"content": "Come posso distinguere un AI da un umano?"
},
]
prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
generated_text = pipe(prompt, max_new_tokens=128)[0]['generated_text']
print(generated_text)cerbero-7b is fully compatibile with llama.cpp
You can find the original and quantized versions of cerbero-7b in the gguf format here
from llama_cpp import Llama
from huggingface_hub import hf_hub_download
llm = Llama(
model_path=hf_hub_download(
repo_id="galatolo/cerbero-7b-gguf",
filename="ggml-model-f16.gguf",
),
n_ctx=4086,
)
llm.generate("""Questa è una conversazione tra un umano ed un assistente AI.
[|Umano|] Come posso distinguere un AI da un umano?
[|Assistente|]""")cerbero-7b is fully integrated with the ollama framework, enabling you to easily deploy and run our models in your environment. To use cerbero-7b within ollama, simply execute the following command:
ollama run galatolo/cerbero-7b
or ollama run galatolo/cerbero-7b-openchat for the openchat version.
📢 Attention: The released versions of
cerbero-7bslightly differ from those used in the paper. The training dataset for the released models was generated usingllama-65binstead ofllama-2-70b-chat, due to the more permissive license of thellama-65bmodel. Our tests indicate that both models produce datasets of comparable quality, and the resulting fine-tuned models demonstrate nearly indistinguishable performance
If you use cerbero-7b in your research, please cite our paper:
@article{galatolo2023cerbero,
title={Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation},
author={Galatolo, Federico A and Cimino, Mario GCA},
journal={arXiv preprint arXiv:2311.15698},
year={2023}
}