Agent Test Environment: Use Synthetic Data Before Connecting a CRM?

Test agents on synthetic company data first, then connect the real CRM. Follow three steps: test correctness and permissions in a simulated environment, recheck with real data in staging, then move to production. Testing directly against a production CRM can corrupt customer records in a single failed test.

What agent tests can you run in a simulated company environment?

On October 8, 2026, Eon released Era, a free tool. Choose an industry, a company size, and the systems to use. Era generates a virtual company with customers, opportunities, tickets, call logs, chats, and documents that match across systems. (Source: Eon blog)

1. The APIs match the real ones. Era returns data in the REST API formats used by Salesforce, HubSpot, Zendesk, Jira, Slack, and other vendors. Each system also has an MCP endpoint. After testing the agent, change the endpoint URLs to the production environment.
2. A new company for each test. Parameters determine the generated environment. The same parameters produce the same company. You can generate a company for each CI run and delete it when the run ends.
3. Permissions can be checked. Era’s GitHub README says tests can use assertions to check answers, tool choices, and whether a read-only agent wrote data. (Source: eon-io/era README)

Comparable scores require an answer key

In September 2026, the Eon team published a paper on arXiv. Code calculates each question’s answer from the generated data. Scoring uses exact matches. 9 models each answered the same 33 questions 3 times. Accuracy ranged from 42.4% to 76.8%. Averages by question type: simple filtering 92.6%, cross-system 39.6%, multi-hop 3.7%. (Source: arXiv 2609.09853)

Another paper from the same team tests “hidden knowledge.” Examples include a promise made to a customer on a phone call, or which of several records the company actually uses. The evidence is in the systems, outside the record named in the question. Of 12 agents, the best answered 18 of 24 attempts correctly. 4 models got at most 6 correct in each of the two procedures. (Source: arXiv 2609.30055)

For companies, these results show that an agent with CRM access can still miss business context across systems. Include questions that test this before launch.

Synthetic data cannot replace staging

Era’s documentation states that the environment is 100% synthetic. It cannot replace a staging system with real data. Run another check in your own staging environment before launch.

Synthetic data tests whether workflows are correct and permissions stay within their limits. Incorrect field entries, historical dirty data, and customers with the same name only appear in real data.

Division of work before launch:

1. Simulated environment: functional questions, cross-system questions, and read-only permission assertions. Run them after every change.
2. Staging: de-identified real data. Spot-check answers.
3. Production: start on a small scale. Begin with read-only access, then allow writes.

Rerun the simulated tests whenever you change prompts, switch models, or add tools. Tools like Era can run in CI automatically on every commit.

If the system under test has no API and the agent can only click through the interface, see Can a Computer-Use Agent Safely Operate Your Legacy ERP? for account isolation and screenshot records.

For decisions to make before scaling, see AI Agent Security Governance Must Come Before Scaling. For engineering guardrails, see Engineering Guardrails for AI Coding: From Writing Code to Constraining It.

Agent testing: FAQ

Can agent launch tests use the production CRM directly?

This is not recommended. An agent may write or delete the wrong records during testing. Get it working in a simulated environment first, then recheck in staging.

Do scores from a simulated environment represent production performance?

They only reflect workflows and permissions. Check how messy the real data is in staging.

Where does agent test data come from? Do I have to generate synthetic company data myself?

You can generate agent test data yourself or use an existing tool. Eon’s Era can generate synthetic data for a whole company based on industry and size. Data stays consistent across systems. Eon says it is free for developers. If you generate your own data, at least make sure customers, opportunities, and tickets match across several systems. Otherwise, the tests cannot detect cross-system problems.