Click here to download the project base paper on model artificial intelligence.
Abstract:
As large language model artificial intelligence improves, there is increasing interest in techniques that leverage these models’ capabilities to refine their outputs. In this work, we introduce Shepherd, a language model specifically tuned to critique responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them. At the core of our approach is a high-quality feedback dataset, which we curate from community feedback and human annotations. Even though Shepherd is small (7B parameters), its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation, Shepherd reaches an average win rate of 53-87% compared to competitive alternatives. In human evaluation, Shepherd strictly outperforms other models and on average closely ties with ChatGPT. We consider the title and the sub-title of a post as a question, its top-level comments as answers, and replies to these comments as critiques. Everything is associated with a community vote score, calculated by subtracting the total number of downvotes from the total number of upvotes. For the sake of clarity, we will refer the community to vote as question score, answer scores, and respect.
