bespoke-minicheck

A state-of-the-art fact-checking model developed by Bespoke Labs.

7B

597 Pulls Updated 23 hours ago

yesterday

66607904c165 · 4.7GB

model
internlm2
·
7.74B
·
Q4_K_M
template
{{- if .Messages }} {{- range $i, $_ := .Messages }} {{- $last := eq (len (slice $.Messages $i)) 1 -}} <|im_start|>{{ .Role }} {{ .Content }} {{- if $last }} {{- if (ne .Role "assistant") }}<|im_end|> <|im_start|>assistant {{ end }} {{- else }}<|im_end|> {{ end }} {{- end }} {{- else }} {{- if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant {{ end }}{{ .Response }}{{ if .Response }}<|im_end|>{{ end }}
system
Determine whether the provided claim is consistent with the corresponding document. Consistency in this context implies that all information presented in the claim is substantiated by the document. If not, it should be considered inconsistent. Please assess the claim's consistency with the document by responding with either "Yes" or "No".
params
{"stop":["<|eot_id|>"],"temperature":0}

Readme

This is a grounded factuality checking model developed by Bespoke Labs.

The model takes as input a document (text) and a sentence and determines whether the sentence is supported by the document. In order to fact-check a multi-sentence claim, the claim should first be broken up into sentences. The document does not need to be chunked unless it exceeds 32K tokens.

bespoke-minicheck-howitworks.png

Bespoke-MiniCheck is the SOTA fact-checking model despite its small size.

Usage

The prompt template is as follows:

Document: {document}
Claim: {claim}

The response will either be Yes or No.

Examples

Prompt

Document: A group of students gather in the school library to study for their upcoming final exams.
Claim: The students are preparing for an examination.

Response

Yes

Prompt

Document: A group of students gather in the school library to study for their upcoming final exams.
Claim: The students are on vacation.

Response

No

Model performance

performance.png

The performance of these models is evaluated on our new collected benchmark (unseen by our models during training), LLM-AggreFact, from 11 recent human annotated datasets on fact-checking and grounding LLM generations. Bespoke-MiniCheck-7B is the SOTA fact-checking model despite its small size.

References

Website

Paper

LLM-AggreFact Leaderboard