Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Every Sunday, NPR Hosts, New York Times’ crossword puzzle Guru will win thousands of listeners in a long-lasting segment called puzzle on Sunday. Can be solved in writing also Most likely Brainteasers are usually difficult for skilled competitors.
Therefore, some experts believe that the EU is a promising way to test the restrictions on the problem.
One New researchFrom Wellesley College, Oberlin College, University of Texas, Northeast University and the starting cursor created AI Benchmark from Puzzle episodes on Sunday. The team, as the tests of tests, such as the so-called substantial models, Openai’s O1, among others – sometimes “refuse” and provide answers to them.
“We wanted to develop a criterion with the problems that people can understand with common knowledge,” said Arjun Guha, one of the co-authors in the northeastern computer license and the research.
The AI industry is in one of a little benchmarking at the moment. Most of the tests used to use AI models, AI models, PHD levels and science questions, multiple skills such as skills that are not related to the average user. Meanwhile, many criteria – even The relatively recently broadcast criteria – They quickly approach the point of saturation.
The advantages of a public radio quiz game like a market puzzle is not a test for esoteric knowledge and explained Guge, who is unable to resolve such “Rote Memory”.
“I think that this is what makes these problems difficult to move on to a problem until you solve it is really hard – everything suddenly hit each other.” “It requires an idea and removal process.”
Of course no benchmark is perfect. The market puzzle is a US-based and English. The quizzes are possible to teach them and “cheat”, although the models are possible, although Guha said it does not see the proof.
“Every week, new questions are published and we can wait for the latest questions that we will not really look.” “We intend to keep the benchmark fresh and how the model performance changed over time.”
In the criteria, which consists of puzzle puzzles of researchers on Sunday, O1 and Deepseek’s thinking models like R1, over the rest. Provisional Models Comprehensive Fact – Check themselves before giving results Avoid some traps normally walking AI models. Trade, reasoning models are a little longer to come to solutions – usually up to a few minutes.
At least one model provides solutions that Deepseek knows that R1 is wrong for Monday puzzle questions. R1, Verbatim “I give up”, followed by an answer that was followed by an answer, which was followed by a wrong answer – this person may undoubtedly belong.
As the models give it a quick choice, just make it a strange choice, but to pull it back, and try to insult it better and fail again. They also forever “think” and make nonsense for answers or immediately reach a correct answer, but immediately continue to consider alternative answers.
“R1 is said to be ‘nervous,’ he said. It remains to be seen how the modernia can affect the quality of model results. “
The best-running model available on the assessment is O1, which is 59% of the recently released O3-mini Defined high “reasoning effort” (47%). (R1 hit 35%.) As the next step, researchers plan to expand their tests to additional models that will help identify areas where these models will be developed.
“You don’t need a doctoral student to be good in justification, so it should be designed to design thoughtful criteria that do not require doctoral level knowledge,” Guha said. “A benchmark with a wider speech can lead to better solutions in the future to understand and analyze the results of more researchers. In addition, the most modern models are growing in parameters, as it is increasingly placed in the settings, is not in the settings and are not accommodated in parameters we believe. “
