Google is using Anthropic’s Claude to improve its Gemini AI

According to internal correspondence seen by TechCrunch, contractors working to improve Google’s Gemini artificial intelligence are comparing its responses to results produced by Anthropic’s rival model, Claude.

Google would not say whether it had obtained permission to use Claude in the test against Gemini when reached by TechCrunch for comment.

As technology companies race to build better AI models, the performance of those models is often benchmarked against competitors. own models through industry benchmarks instead of seriously evaluating their competitors’ AI responses to contractors.

Gemini contractors tasked with evaluating the accuracy of model outputs must evaluate each response they see against multiple criteria, such as accuracy and detail. According to correspondence seen by TechCrunch, contractors are given up to 30 minutes per survey to determine whose answer is better, Gemini or Claude.

Contractors recently began seeing references to Anthropic’s Claude appear on an internal Google platform they use to benchmark Gemini against other unnamed AI models, the correspondence showed. At least one of the results submitted to Gemini contractors seen by TechCrunch clearly stated: “I am Claude created by Anthropic.”

One internal conversation indicated that contractors saw Claude’s answers as emphasizing safety more than Gemini’s. One contractor wrote, “Claude’s security settings are the strictest” among the AI ​​models. In certain cases, Claude would not respond to instructions that he considered dangerous, such as playing the role of another AI assistant. In another, Claude avoided answering the question, while Gemini’s response was cited as a “major breach of security” because it included “nudity and bondage.”

Anthropic commercial terms of service prohibit customers from accessing Cloud to “build a competing product or service” or “train competing AI models” without obtaining permission from Anthropic. Google is key investor anthropic.

When asked by TechCrunch, Shira McNamara, a spokeswoman for Google DeepMind, which runs Gemini, would not say whether Google had obtained permission from Anthropic to access Claude. When reached before publication, an Anthropic spokesperson had no comment by press time.

McNamara said DeepMind “compares model results” for evaluations, but did not train Gemini on Anthropic models.

“Of course, in accordance with standard industry practice, in some cases we compare model results as part of our evaluation process,” McNamara said. “But any suggestion that we’re using anthropic models to train Gemini is incorrect.”

last week, TechCrunch reported exclusively Google contractors working on the company’s AI products will now evaluate Gemini’s AI responses in areas beyond their expertise. Internal correspondence expressed concerns by contractors that Gemini might produce inaccurate information on highly sensitive topics such as health care.

You can safely send tips to this reporter on Signal at +1 628-282-2811.