lgkt / llama3-chinese-wangshenzhi

wangshenzhi出品，模型来源：https://hf-mirror.com/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-4bit，基于Meta-Llama-3-8B-Instruct

8B

255 Pulls Updated 4 months ago

4 months ago

b4a46dbb319f · 4.7GB

model
llama
·
8.03B
·
Q4_0
params
{"stop":["<|start_header_id|>","<|end_header_id|>","<|eot_id|>"],"temperature":0.6,"top_p":0.9}
system
You are a helpful assistant.
template
{{ if .System }}<|start_header_id|>system<|end_header_id|> {{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|> {{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|> {{ .Response }}<|eot_id|>

Readme

Llama3-8B-Chinese-Chat-v2.1-Q4_0（from wangshenzhi）

2024-5-19

Llama3-Chinese必要性： Llama3对中文支持并不好，经常会出现中文问题给出英文答案，本人收集了市场上Llama3的中文微调版本，量化并适配了modelfile，现分享给大家。

以下来自wangshenzhi

[May 6, 2024] We now introduce Llama3-8B-Chinese-Chat-v2.1! Compared to v1, the training dataset of v2.1 is 5x larger (~100K preference pairs), and it exhibits significant enhancements, especially in roleplay, function calling, and math capabilities! Compared to v2, v2.1 surpasses v2 in math and is less prone to including English words in Chinese responses. The training dataset of Llama3-8B-Chinese-Chat-v2.1 will be released soon. If you love our Llama3-8B-Chinese-Chat-v1 or v2, you won’t want to miss out on Llama3-8B-Chinese-Chat-v2.1!

Model Summary
Llama3-8B-Chinese-Chat is an instruction-tuned language model for Chinese & English users with various abilities such as roleplaying & tool-using built upon the Meta-Llama-3-8B-Instruct model.

Developed by: Shenzhi Wang (王慎执) and Yaowei Zheng (郑耀威)

License: Llama-3 License
Base Model: Meta-Llama-3-8B-Instruct
Model Size: 8.03B
Context length: 8K

This is the first model specifically fine-tuned for Chinese & English user through ORPO [1] based on the Meta-Llama-3-8B-Instruct model.

Compared to the original Meta-Llama-3-8B-Instruct model, our Llama3-8B-Chinese-Chat-v1 model significantly reduces the issues of “Chinese questions with English answers” and the mixing of Chinese and English in responses.

Compared to Llama3-8B-Chinese-Chat-v1, our Llama3-8B-Chinese-Chat-v2 model significantly increases the training data size (from 20K to 100K), which introduces great performance enhancement, especially in roleplay, tool using, and math.

[1] Hong, Jiwoo, Noah Lee, and James Thorne. “Reference-free Monolithic Preference Optimization with Odds Ratio.” arXiv preprint arXiv:2403.07691 (2024).

Training framework: LLaMA-Factory.

Training details:

epochs: 2
learning rate: 5e-6
learning rate scheduler type: cosine
Warmup ratio: 0.1
cutoff len (i.e. context length): 8192
orpo beta (i.e. \(\lambda\) in the ORPO paper): 0.05
global batch size: 128
fine-tuning type: full parameters
optimizer: paged_adamw_32bit

模型来源：https://hf-mirror.com/shenzhi-wang/Llama3-8B-Chinese-Chat-GGUF-4bit

QQ: 83649263