OpenAI's innovative use of Prover-Verifier games to train powerful language models has had a positive impact in multiple ways. Firstly, it has significantly enhanced the readability of the language model's outputs. In practical applications, weaker models can verify the generated text, which provides a layer of assurance for accuracy. At the same time, it is also easier for humans to evaluate these texts.
The importance of readability in language model outputs is emphasized for their true usefulness to humans, especially when tackling tasks such as solving complex mathematical problems, where this characteristic is crucial. When powerful models focus solely on optimizing for correct answers, the solutions they generate can become obscure and difficult to understand. In such cases, the error rate of human evaluators in assessing these highly optimized solutions increases significantly.
Through the Prover-Verifier game, the text generated by powerful models can not only be effectively verified by weaker models but also be evaluated more efficiently by humans, a phenomenon known as increased readability. During the research process, by optimizing chain reasoning, performance was successfully enhanced while maintaining the ability of human evaluators to accurately assess solutions. By skillfully using pairs of strong and weak models, researchers trained powerful models to generate solutions that are easy for humans to understand, and throughout the training process, they carefully balanced the two key elements of performance and readability.