ServiceNow's CoreAI team published AutoSynthData on October 2, 2026, a method for creating training data based on tasks an AI agent struggles to perform. Instead of adding more examples in general, the system uses the target model's failures and a stronger model's successes to choose where to focus. The published results concern experiments in the EnterpriseOps-Gym environment, not proven deployment in every organization.
In an experiment in the Hybrid domain, 2,000 examples were generated in about 18 hours. According to the researchers, training increased average Pass@1 by 7.2 percentage points, a relative improvement of 35%. In an ITSM experiment, 1,994 examples were generated in 66 hours, and average Pass@1 rose from 18.77% to 27.18%. The figures describe the published experimental setups; data generation time should not be compared between them without accounting for their differences.
Why an agent can explain well and still fail
An enterprise agent is not assessed solely on the last sentence it writes. If asked to update a service request, it needs to find the right request, understand the permissions and make the appropriate change. A convincing completion message does not fix a situation in which a different record was updated.
The EnterpriseOps-Gym dataset card describes a simulation environment with synthetic databases, working tools and the ability to reset the environment between experiments. Measurement examines the system's state after the agent's actions, using executable checks. The emphasis is on the outcome achieved, rather than requiring every agent to choose exactly the same sequence of actions.
The dataset includes 1,150 expert-created tasks across 8 enterprise domains, with 512 tools and 164 data tables. Those numbers explain why this is not an ordinary question-and-answer test: an action may affect information on which a later task depends.
For example, in a test environment, one could request a calendar change while retaining certain participants. A correct result must include both the requested update and the details that must not change. The mere existence of a new record is not enough. The example illustrates the nature of the test; it is not a new result reported by the researchers.
Failure becomes a subject to learn
According to the AutoSynthData publication, diagnostic tasks are translated into capability descriptions, from which new exercises are generated. The generator does not receive the original wording and trajectories of the evaluation tasks. Before an example is accepted for training, both its solution and verification mechanism are checked. This aims to avoid training on a task that looks plausible but cannot be performed.
The distinction between identifying a weakness and copying a test matters. If a model saw the same question in training, later success on it may reflect memorization of the example. Testing whether a capability has been learned requires changing the situation, names and request details, then seeing whether the same skill also works under the new conditions.
There is also a question about the checker itself. If an exercise asks for a task to be transferred to a particular team, a checker that verifies only a status change may approve an incorrect result. Conversely, one that requires specific wording may reject a valid result. Training data quality therefore also includes the quality of the definition of success.
This is a point that distinguishes producing large amounts of text from building useful learning material. The number of examples matters, but it does not reveal whether they are diverse, feasible and accurate. A large collection of nearly identical exercises can leave the same weakness untested.
The choice of test cases can also change the picture. If the validation set contains only tasks in which taking action is permitted, it will not adequately test the ability to stop when authorization is absent. If it contains only clear requests, it will not reveal how the agent handles an instruction that can be interpreted in two ways.
A team adopting an approach of training based on failures should therefore also document what did not enter the collection: which types of tasks were excluded, which tools were unavailable and which conditions were not tested. That information allows the success score to be read within its proper boundaries, instead of turning a local result into a general claim.
Earlier research explains the problem, not the state of every model today
The original EnterpriseOps-Gym paper was submitted in March 2026. Its authors tested 14 models and found gaps in planning and tool use. According to the abstract, providing ideal human action plans improved performance by 14 to 35 percentage points. The researchers viewed this as an indication that planning was a significant limiting factor in the environment they tested.
They also reported that agents struggled to refuse tasks that could not be completed. This is a different kind of failure: a system that does not know when to stop can take actions that do not move it toward a solution. However, results from March are not a current ranking of models launched since then, and should not be used to determine their performance today.
That background helps explain the new research question. If a weakness lies in planning actions, adding general facts to a model will not necessarily address it. Exercises may be needed in which steps depend on one another, information is missing or an action must not be performed. This is an interpretation of the problem's structure, not proof that every weakness is solved by the same method.
An improvement on a test must remain verifiable
In a separate background article from January 2026, Anthropic distinguishes between capability evaluations, which examine what an agent still struggles to do, and regression evaluations, intended to ensure that capabilities that already worked are preserved. The distinction helps interpret additional training: improvement on a focused task does not remove the need to check that the model has not regressed on other tasks.
The same article describes combining code-based checks, model-based evaluation and human review. Each approach has limitations: a rigid check can miss a valid solution expressed differently, and a model-based evaluator needs calibration. This is not independent validation of AutoSynthData, but engineering background for the question of how to evaluate this type of system.
For a team in Israel building an agent to work in Hebrew, the practical conclusion is to begin with a real, well-defined set of failures. It is possible to test, for example, confusion between similar names, identification of a date or an action performed in the wrong document. A separate test set is then needed, one that was not used to generate the training examples.
The new publication does not include dedicated proof of improvement in Hebrew or a commitment to business savings. It offers a systematic way to choose what the agent needs to learn and reports improvement in two experimental domains. The next important extension will be to see whether the improvement holds on new tasks, in different environments and with checks that also measure errors that were never intended to occur in the first place.