Zephyr-7b-beta

Badass Zephyr-7B-β is the second model in the Zephyr series of language models that are trained to act as helpful assistants. It is a fine-tuned version of mistralai/Mistral-7B-v0.1 that was trained on on a mix of publicly available, synthetic datasets using Direct Preference Optimization (DPO).

Model

Zephyr-7b-beta

Details

Developer

Huggingface

License

MIT

Model parameters

Pretraining tokens

Release date

Oct 2023

Supported context length

Price for prompt tokens*

$0.15/Million tokens

Price for response tokens*

$0.15/Million tokens

*Note: Data based on 11/14/2023

Model Performance Across Task-Types

Here's how Zephyr-7b-beta performed across all three task types

Metric

ChainPoll Score

QA without RAG

0.60

QA with RAG

0.71

Long form text generation

0.74

Model Info Across Task-Types

Digging deeper, here’s a look how Zephyr-7b-beta performed across specific datasets

Tasks	Insights	Dataset Name	Dataset Performance
QA without RAG	The model makes quite a mistakes which show bias and errors in factual knowledge.	Truthful QA	0.53
QA without RAG		Trivia QA	0.68
QA with RAG	Even with small size the model performs commendable which demonstrates great reasoning and comprehension skills. It struggles a bit on mathematical skills as it scores relatively low on DROP compared to other dataset.	MS Marco	0.9
		Hotpot QA	0.66
		Drop	0.55
		Narrative QA	0.73
Long form text generation	The model performs satisfactory at this task which shows good ability to generate long text without factual errors.	Open Assistant	0.74

💰 Cost insights

The model offers a decent balance of cost and performance. It is 13x cheaper compared to GPT3.5 and 6x cheaper compared to Llama 70b variant. We suggest using this instead of Mistral-7b-instruct-v0.1 & Llama-2-7b-chat.

LLMHALLUCINATIONINDEXLLM