AI automation platform comparison: how to run your own

A ranked list of platforms is out of date within months and was usually written by somebody with a position. Even an honest one is a comparison against a generic set of requirements, which is not the set you have.

What survives is the method. Deciding what you are comparing on, weighting it before you see the products, and testing the things that only reveal themselves under failure — that process produces a defensible answer for your situation and can be re-run when the situation changes.

What we'll cover

• Why published comparisons are limited
• Weight the criteria before looking
• The criteria worth weighting
• Building it with CodeWords
• Designing the proof of concept
• Scoring without fooling yourself
• The costs that appear later
• Making the decision defensible
• Who should be in the room
• Making it survive
• Limits worth knowing about
• What to do first
• Frequently asked questions

Why published comparisons are limited

They age quickly, since pricing and capability change every few months.

They are rarely disinterested, and even independent ones are frequently affiliate-funded.

They compare against generic requirements, which differ from yours in ways that matter.

They test the demo path, not the failure path, which is where the differences are.

They cannot know your constraints — your existing systems, your data residency requirements, your team's skills.

Use them to build the shortlist, then do your own work, which is the only part that produces a decision you can defend.

Weight the criteria before looking

Write and weight the criteria first, which is the single most important discipline in this process.

Doing it afterwards is rationalisation, since a product you liked will suggest criteria it happens to satisfy.

Get agreement on the weights from everybody who will be affected, before the demos, when it is still an abstract conversation.

Keep it short. A twenty-criterion matrix is theatre; six to eight weighted properly is a decision.

Distinguish requirements from preferences. A requirement disqualifies; a preference scores.

Record it, because the weights are what makes the eventual decision explicable to somebody who was not there.

The criteria worth weighting

The connectors you actually need, at the operation level rather than the logo level.

Failure handling: retries, partial completion, replay and visibility.

Data residency and processing, including sub-processors and whether your data trains anything.

Cost at projected volume, with the unit definition understood.

Who can build and who can break, meaning permissions, environments and credential handling.

Exit: export of definitions and history, and what happens to your connections.

Support responsiveness, which matters disproportionately and is testable during a trial.

Building it with CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: the workflow you chose for the evaluation, with its real edge cases.
3. Authorize the connections it needs.
4. Describe the exceptions: a failing step, a rate limit, malformed input, an unavailable system.
5. Compare the result against the same workflow built on the other candidates.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Designing the proof of concept

Use the same workflow on every candidate, or you are comparing tasks rather than platforms.

Choose a genuinely awkward one, since easy workflows do not discriminate.

Include the edge cases you actually have, not the ones a vendor suggests.

Have the same person build it, or you are measuring familiarity.

Break it deliberately on each platform and record what happened and what you were told.

Time the build, and time a change to it afterwards, since changing is what you will do repeatedly and building is what you do once.

Scoring without fooling yourself

Score against the criteria, not against overall impression, which is what the weighting was for.

Score independently before discussing, since the first opinion voiced anchors everybody else.

Record why, not just the number, or the score is unreviewable.

Notice a disqualifying result and stop, rather than letting a weighted average carry a product past a requirement it failed.

Be suspicious of a tie, which usually means the criteria did not capture what you actually care about.

Sleep on it. Enthusiasm decays over a week and the notes are what remain.

The costs that appear later

Migration off whatever you are using now, which is real work rarely counted.

Training and ramp, including the people who will maintain it rather than the ones who chose it.

The second tool you will still need for something this one does not do.

Growth in usage, which changes the cost structure rather than scaling the rate.

Maintenance of what you build, which is proportional to the number of automations and not to the platform.

The exit, whose cost you are deciding now even though you will pay it later.

Making the decision defensible

Write it down: the criteria, the weights, the scores, the reasons.

Name the thing you gave up, since every choice has one and pretending otherwise damages credibility when it surfaces.

Say what would change the decision, which is what makes a review possible rather than a re-argument.

Set a review date, because this category moves and a decision from two years ago may no longer be right.

Share it with the people affected, which prevents the decision being relitigated by everybody who was not consulted.

Keep the record, since the same evaluation will be requested again and re-running it from scratch is the expensive option.

Who should be in the room

An evaluation run entirely by the team that wants the tool produces a predictable answer, and one run by procurement alone produces a different predictable answer.

The person who will maintain it matters most and is frequently not consulted until after the decision.

Somebody from security or data protection, early, since a residency or sub-processor answer can disqualify a candidate and finding that late wastes everybody's effort.

A representative of the people whose work it changes, because adoption is decided by them regardless of what is purchased.

Finance on the cost structure, not the headline price, since the unit definition is where the surprises are.

Not everybody on everything. A large evaluation committee produces a compromise choice that nobody advocates for afterwards.

Name the decision-maker at the start, so the process informs a decision rather than substituting for one.

Making it survive

Do not migrate everything at once, whatever the decision.

Keep the evaluation artefacts, including the proof-of-concept workflows, which are the basis of the next review.

Watch the assumptions, particularly volume and pricing, which are the two that change.

Revisit on a trigger — a pricing change, an acquisition, a capability gap — rather than only on a date.

Report what it actually costs after six months against what was projected, which is how the next evaluation gets better.

Limits worth knowing about

The category is changing quickly, so a thorough evaluation has a shelf life.

Vendor roadmaps are not commitments and should not carry weight in scoring.

Acquisitions change products, sometimes substantially, and are not predictable.

Your requirements will change too, frequently within the term you are signing.

No platform will satisfy everything, and a comparison that finds one has understated somebody's requirements.

What to do first

Write and weight your criteria before you look at a single product. It takes an afternoon, it forces the disagreements out while they are still abstract, and it is the difference between a decision you can explain and a preference you justified afterwards.

Two habits make the difference. Build the same awkward workflow on every candidate and deliberately break it, since the happy path does not discriminate between platforms and the failure path is what you will live with. And name what you gave up in the written decision, because every choice has a cost and acknowledging it is what makes the record credible when that cost eventually surfaces.

Frequently asked questions

Why not rely on a published comparison?

They age within months, are rarely disinterested, compare against generic requirements rather than yours, and test the demo path rather than the failure path. Use them to build a shortlist, then do your own work.

What is the most important step?

Writing and weighting the criteria before looking at any product. Doing it afterwards is rationalisation, because a product you liked will suggest criteria it happens to satisfy.

How many criteria should there be?

Six to eight, weighted, with requirements separated from preferences — a requirement disqualifies while a preference scores. A twenty-criterion matrix is theatre.

How should a proof of concept be designed?

The same genuinely awkward workflow on every candidate, including your real edge cases, built by the same person, deliberately broken on each — and time both the build and a subsequent change, since changing is what you will do repeatedly.

How do I score without fooling myself?

Score independently before discussing, since the first opinion voiced anchors the rest. Record reasons rather than only numbers, and stop when a candidate fails a requirement rather than letting a weighted average carry it through.

Which costs are usually missed?

Migration off the current tool, training for the people who will maintain it, the second tool you will still need, usage growth changing the cost structure, and the exit — whose cost you are deciding now and paying later.

Should vendor roadmaps affect the decision?

No. They are not commitments, and acquisitions change products in ways nobody predicts. Score what exists.

How often should this be revisited?

On a trigger — a pricing change, an acquisition, a capability gap — as well as on a date. The category moves quickly enough that a thorough evaluation has a real shelf life.

Who should be involved in the evaluation?

The person who will maintain it, somebody from security or data protection early enough that a residency answer can disqualify a candidate, a representative of the people whose work changes, and finance on the cost structure rather than the headline price.

Should the whole committee decide?

Name a decision-maker at the start. A large committee deciding by consensus produces a compromise nobody advocates for afterwards, which is how a tool ends up with no owner.

Should the evaluation include building on each platform?

Yes, the same awkward workflow on each, built by the same person. Timing the second change rather than the first build matters most, since changing a workflow is what you will do repeatedly.

What if two candidates score the same?

Be suspicious of it. A tie usually means the criteria failed to capture what you actually care about, which is worth resolving before deciding rather than breaking the tie arbitrarily.

Related reading

• AI workflow automation software: the questions that separate vendors
• AI workflow tools and what each layer does
• AI workflow software: where software fits