Product1 distinct publisher2 min readPublished
The company's own benchmark keeps every flagship model it tested below 60 percent completion across 2140 CRM task instances, and the fix its researchers report is domain knowledge somebody inside your org has to write down.
The Product Desk · Product desk
Compiled by The Product DeskSomething wrong?How this is made
Sort the failures by how loud they are. An agent that writes an illegal SOQL query gets an error back and sometimes repairs it, though Salesforce's researchers note that other attempted repairs produce further mistakes [8]. That one at least announces itself in the logs. The quiet failure is schema confusion: the agent treats an order item as a pricebook entry, or a lead as an opportunity, reads the wrong table, and hands back something shaped exactly like an answer [9].
The quietest case is ambiguity. Salesforce's own example is a support case transferred between two reps and whether it counts toward either rep's average handling time, both, or neither; the blog says agents usually assume one reading and carry on without noticing a choice was made [10]. Here is what teams tell themselves an agent will do with an underspecified task: ask. Here is what the execution traces show it doing: pick.
Put the ceiling against the denominator. At 60 percent completion, more than 856 of the 2140 task instances go unfinished [13]; at GPT-4o's rate, more than 1498 do [14]. Those are completion rates on scripted business tasks, which is a harder and more useful number than anything measured in sessions or clicks.
What Salesforke calls the remedy is domain-specific knowledge supplied in the prompt or wrapped as a tool [7]. Read the procedure finding as an operations statement rather than a prompt tip: instructions on how to do the work only exist if someone has written down how the work is done. The people who can write it are the admins and ops leads who already know which searches want SOSL rather than SOQL's fuzzy matching, which is precisely the fine-grained distinction the agents miss [11]. Salesforce sells Agentforce as agents that handle return and refund requests and draft tailored sales pitches [12]; the research underneath says the pitch arrives without your business context, and that context is your homework.
So sort candidate tasks on two axes before scoping anything. Axis one: is the correct procedure already written down somewhere a human follows it. Axis two: when the output is wrong, does it look wrong. Written procedure plus visible errors is where to start, because the agent is executing a runbook you can diff. Written procedure plus silent errors needs a checking step budgeted before the agent goes near a report. Unwritten plus visible errors is a documentation project wearing an AI budget line. Unwritten plus silent errors is the average-handling-time metric, and the honest move there is to write the procedure first, which is the work the drop-in agent was supposed to spare you.
Ranked by verification strength, evidence, and original report placement.
Salesforce's blog states that CRM tasks are difficult for current LLM agents to complete due to lack of training data coverage and LLMs' unfamiliarity with business context.
CRMArena-Pro, developed by researchers at Salesforce AI Research, consists of 22 tasks and 2140 task instances spanning categories such as workflow execution, policy compliance and information retrieval.
On CRMArena-Pro, GPT-4o is able to solve less than 30% of all tasks.
The reasoning model o1 still fails at just over 50% of all CRMArena-Pro tasks.
The best performing model out of 9 flagship models from various providers, Gemini-2.5-pro, struggles to achieve a completion rate of 60% on CRMArena-Pro.
Because Salesforce data manipulation requires SOQL and SOSL, agents sometimes produce queries with illegal syntax; they can correct some mistakes after observing the error message, but for others the attempted correction results in further mistakes.
Distinct publishers with included, body-backed reporting in this cluster.
1 article · August 31, 2026
Follow any of these and your For You feed starts watching them — no settings page required.
build
Adding a persona-driven user costs CRMArena-Pro's agents 23 points of success1 distinct publisher
invest
Salesforce's double digits, minus Informatica: agentic AI is real and still 2% of revenue1 distinct publisher
product
OpenAI starts charging some large accounts only for the jobs its model finishes1 distinct publisher
product
Salesforce's longer-dated backlog grows at half the rate of its cRPO headline1 distinct publisher
Evidence-backed comparisons of source perspectives and observed adoption signals. Read the methodology
Which Builder, Operator, and Investor concerns the observed source mix emphasized—not a truth score.
Evidence, demonstrated adoption, hype gap, incentives, and confidence are assessed independently, each on its own current evidence. How these are measured.
One vendor, no outside run
Every figure that matters — 22 tasks, 2,140 instances, the three model scores — traces to the company that built the benchmark and sells the agent platform, with no second party having reproduced any of it. The error analysis is specific enough to look like real log reading rather than marketing, but there is no per-model breakdown, no methodology detail beyond 'ReAct', and the summary table the post says quantifies its recommended fixes is not in the text we have.
Measured only in its author's harness
Nine flagship models have genuinely been put through CRMArena-Pro — inside Salesforce. Nothing here shows an outside lab, customer or evaluator running the benchmark, and nothing shows the how-not-what prompting recipe in production anywhere. Agentforce appears as product description and a pointer to default action libraries, with not one deployment number attached.
Sober data, modest title
A post filed under 'Tips and Tricks' is quietly reporting that the strongest model anyone has of nine cannot finish six in ten CRM tasks — that is understatement, not inflation. What pulls the other way is the cure: 'can greatly help' is doing work that no visible number supports here. The pessimistic half of this story is better evidenced than the optimistic half.
The cure is the product
Salesforce wrote the benchmark, employs the researchers, and sells the platform whose default action libraries and org metadata are offered as the way to close the gap. The conclusion that customers must supply procedure and domain knowledge is commercially convenient in the same breath as it is capability-damaging about models Salesforce does not make. That asymmetry is worth holding onto: the numbers cost the author something, the remedy does not.
Scores solid, remedy unmeasured
I would bet on the reported scores — vendors rarely lowball their own domain — and on the four failure modes, which are too idiosyncratic to be invented. What one source cannot settle is how much the procedural prompting actually recovers, or whether any of this survives outside Salesforce's schema and its two query languages.