How to automate synthetic monitoring that stays worth maintaining

Synthetic monitoring runs a scripted journey against production on a schedule — sign in, search, add to basket, check out — and tells you when it stops working. It catches the failures that no server-side metric sees: the button that does nothing, the form that silently rejects, the third-party script that breaks the page.

It also has the highest maintenance cost of any monitoring you can run, because a script that drives an interface depends on an interface that changes. Deciding what to script, and accepting how few of them you can afford, is the entire discipline.

What we'll cover

• What it catches that nothing else does
• Why these break so often
• Choosing what to script
• Building it with CodeWords
• Writing a check that survives a redesign
• Test data in production
• Alerting without crying wolf
• Measuring the journey, not only whether it worked
• Making it survive
• Limits worth knowing about
• What to build first
• Frequently asked questions

What it catches that nothing else does

Client-side failures. A script error breaking a button produces a perfectly healthy server, and only something driving a browser notices.

Third-party breakage. A payment widget, a chat tool, or an analytics script failing to load can block a page, and none of it is in your logs.

The journey rather than the endpoint. Each step working individually does not mean the sequence works, and the sequence is what a customer does.

Rendering and interaction, which server responses cannot describe — content that arrives and never displays looks like success everywhere else.

Real network conditions, including DNS, certificates, and CDN behaviour from outside your infrastructure.

Absence of traffic. A journey nobody has attempted since a release is a journey nobody has tested, and synthetic checks exercise it regardless of demand.

Why these break so often

Being honest about this up front is what prevents an abandoned suite.

Selectors change. A class name, a label, a button's position — any redesign breaks scripts that depend on them.

Timing. The check acted before the page finished, which is intermittent, environment-dependent, and the single most common cause of a false failure.

Unexpected interruptions. A cookie banner, a survey prompt, a maintenance notice — anything the script was not told about stops it.

Content changes. An assertion on text fails when marketing rewords the heading.

Test accounts degrade. A password expires, a trial ends, an account accumulates state that changes what it sees.

None of these is a fault in the product, which is why a noisy suite trains people to assume a failure is the monitor rather than the site — and then it is, once.

Choosing what to script

Very few. Three to five journeys for most products. A suite of forty is a suite nobody maintains, and an unmaintained check is worse than none because it eventually reports nothing.

Pick by revenue and by blast radius. Sign-in, the primary transaction, and whatever the business would notice within an hour.

Not the long tail. Edge cases belong in a test suite that runs before deployment, not against production every five minutes.

One journey per check, so a failure names what broke rather than requiring somebody to read a log to find out which step.

Keep each short. A twelve-step script has twelve chances to break and tells you less than three three-step ones.

Review the list quarterly and remove what no longer matters, since journeys outlive their importance and the maintenance continues regardless.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, so a check can assert against your own systems and route failures to the people who can act.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which journey, what a successful outcome looks like, and how often.
3. Authorize the connections it needs, including wherever the alert should go.
4. Describe the exceptions: a slow step, an unexpected dialog, a failure during a known deployment.
5. Break something deliberately in a staging environment and confirm the check notices.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Writing a check that survives a redesign

The practices that separate a suite people maintain from one they turn off.

Use dedicated test attributes. An element marked explicitly for automation survives every visual change; a class name does not. This is the single highest-leverage habit here, and it requires cooperation from whoever writes the interface.

Fall back to accessible roles and labels where attributes are not available, since those change less than styling and are meant to be stable.

Never match on text a marketer might edit, and never on position.

Wait for conditions, not durations. Waiting until an element is present is stable; waiting two seconds is a timing bug waiting for a slower day.

Dismiss known interruptions explicitly and fail on unknown ones, so a new banner is reported rather than silently worked around.

Capture a screenshot on failure, which is the most useful diagnostic available because the failure is visual.

Test data in production

The awkward part, since these checks run against the real system.

Use a dedicated account, clearly marked, never a real customer's.

Exclude it from analytics and reporting, or your conversion numbers include a robot completing checkout every five minutes.

Decide about writes. A check that places an order creates real data that somebody has to clean up, and a check that never writes cannot verify the transaction that matters.

Clean up deliberately, and monitor the cleanup, since debris accumulates and eventually confuses somebody looking at real data.

Mind side effects. A checkout check may send an email, charge a card, or notify a fulfilment team, and each needs suppressing for the test account rather than tolerating.

Rotate its credentials like any other, and alert distinctly when the check fails on authentication — which otherwise looks like the site being broken.

Alerting without crying wolf

Require consecutive failures from more than one location before alerting. A single failure is normal.

Retry immediately on failure, once, before counting it — most false failures are transient.

Alert differently for slow and for broken, since the responses differ.

Suppress during deployments, with a window, rather than letting every release produce alerts.

Track the false alarm rate per check. A check that alarms falsely more than occasionally is costing more attention than it protects, and the honest response is to fix it or delete it.

Report recoveries, so nobody is left assuming a problem is ongoing.

Measuring the journey, not only whether it worked

A check that already drives the whole journey is collecting timing data, and most teams discard it.

Record the duration of each step, not only the total, so a slow journey points at which part.

Track it over time. A step that has doubled since last month is a regression nobody reported, because it is still working.

Compare against the same hour on the same weekday, since load patterns make raw comparisons misleading.

Watch the trend before the threshold. A step creeping upward for three weeks is a better signal than the day it finally breaches.

Correlate with releases. A step that slowed by forty per cent the day after a deployment has told you where to look.

Report it monthly to whoever owns the journey, since performance degradation is gradual and nobody notices it from inside.

Making it survive

Run the check from outside your infrastructure, so an outage does not take the monitoring with it.

Treat scripts as code — in version control, reviewed, with the same care as tests.

Update them with the interface. A redesign that breaks three checks should fix them in the same change, which requires whoever ships the redesign to know they exist.

Delete rather than mute. A muted check still runs, still costs, and still looks like coverage while catching nothing.

Report the outcome. Checks run, passed, failed, false alarms, and duration trends.

Limits worth knowing about

Frequency costs money, per check per run, and a five-minute interval across many journeys adds up faster than people expect.

Synthetic traffic is not real traffic. It exercises one path with one account, and it will not find a bug affecting one customer's data.

Browser checks are slower and more fragile than API checks, so use them only where the browser is the point.

Geographic coverage is limited to where your provider runs, which may not match where your customers are.

They do not replace real-user monitoring, which sees what actual people experience including the segments a script never represents.

What to build first

One check for your single most important journey, from outside your network, with test attributes agreed with whoever owns the interface, alerting after two consecutive failures. One reliable check beats a suite of twelve flaky ones, and the attribute agreement is what determines whether it is still working in six months.

Two habits make the difference. Ask for dedicated test attributes rather than matching on class names or text, since that one change removes most of the maintenance. And track the false alarm rate per check — a check that cries wolf is costing more attention than it protects, and the right response is to fix or delete it rather than tolerate it.

Frequently asked questions

How many synthetic checks should I have?

Three to five journeys for most products. A large suite is one nobody maintains, and an unmaintained check eventually reports nothing while still looking like coverage.

Why do these break so often?

Because they depend on an interface that changes. Selectors, timing, unexpected banners, and reworded content all break a script without anything being wrong with the product — which is why test attributes matter so much.

What is the single best practice?

Dedicated test attributes on the elements your checks interact with. They survive redesigns that break class names and text matching, and agreeing them with whoever builds the interface removes most of the ongoing maintenance.

Should checks write data in production?

It depends what you need to verify. A read-only check cannot confirm the transaction works; a writing check creates data and side effects that need suppressing and cleaning up. Decide deliberately rather than by default.

How do I stop test traffic polluting analytics?

Exclude the test account explicitly, everywhere — analytics, conversion reporting, and any operational dashboard. A robot completing checkout every five minutes otherwise appears in numbers people quote.

When should I delete a check?

When it alarms falsely more than occasionally and you are not going to fix it, or when the journey no longer matters. Muting leaves it running, costing money, and appearing as coverage while catching nothing.

Do synthetic checks replace real-user monitoring?

No. They exercise one path with one account on a schedule; real-user monitoring sees what actual people experience, including the segments and devices a script never represents. The two answer different questions.

What should I do with the timing data?

Record per step rather than per journey, track the trend, and correlate with releases. A step that has slowly doubled is a regression nobody reported, because it never failed — it only got worse.

How do I spot gradual degradation?

Compare against the same hour on the same weekday and watch the trend rather than the threshold. Performance decays slowly enough that nobody inside notices, which is exactly what a monthly report to the journey's owner is for.

Related reading

• Automating API health checks
• Automating website uptime monitoring
• Shift-left testing