Close
OpenAI's recent research highlights the limitations of large language models (LLMs) in software engineering tasks. The study, utilizing a benchmark called SWE-Lancer, tested models like GPT-4o and Claude-3.5 Sonnet on real freelance coding tasks from Upwork. Results showed that while these models can solve bugs, they struggle to understand the underlying issues, leading to frequent mistakes.
The research involved 1,488 tasks with a total payout of $1 million, divided into individual contributor and management tasks. Despite Claude 3.5 Sonnet performing the best, it only resolved 26.2% of issues correctly, indicating that LLMs cannot yet replace human engineers. The findings suggest that while LLMs can assist in coding, they still require significant improvement to be reliable in real-world applications.
• LLMs struggle to identify root causes of coding issues.
• Claude 3.5 Sonnet was the best-performing model in the study.
LLMs are AI models designed to understand and generate human-like text, but they struggle with complex coding tasks.
Benchmarking involves testing models against specific tasks to evaluate their performance, as seen with the SWE-Lancer dataset.
These tasks represent real-world coding challenges that LLMs were tested on, revealing their limitations in practical applications.
OpenAI is a research organization focused on developing advanced AI technologies, including LLMs like GPT-4o.
5 Sonnet, a model tested in the study.
Tech Xplore on MSN.com 13month
India Education Diary 14month
Isomorphic Labs, the AI drug discovery platform that was spun out of Google's DeepMind in 2021, has raised external capital for the first time. The $600
How to level up your teaching with AI. Discover how to use clones and GPTs in your classroom—personalized AI teaching is the future.
Trump's Third Term? AI already knows how this can be done. A study shows how OpenAI, Grok, DeepSeek & Google outline ways to dismantle U.S. democracy.
Sam Altman today revealed that OpenAI will release an open weight artificial intelligence model in the coming months. "We are excited to release a powerful new open-weight language model with reasoning in the coming months," Altman wrote on X.
Log in
Sign Up
AI Tools - Popular
AI Tools - Categories
GPTs - Categories
AI for Jobs
Key AI Highlights in this Article
Key AI Terms Mentioned in this Article
Companies Mentioned in this Article
