Jenkins and CodeWords: automating around the build server
What belongs in a Jenkinsfile and what does not, build failure triage across jobs, flaky test detection, and keeping an ageing controller honest.
Aymeric Zhuo11 min read
On this page
Jenkins instances accumulate. A pipeline added for a service that was decommissioned two years ago, a job somebody set up to run nightly that nobody has looked at since, plugins nobody is confident about updating. The build server is frequently the oldest continuously running piece of infrastructure a team has, and the least examined.
Automation around Jenkins divides cleanly. Anything about one build belongs in the Jenkinsfile. Anything about patterns across builds, across jobs, or across time belongs outside it, because a pipeline has no natural place to stand and look at the whole instance.
What we'll cover
- What belongs in the Jenkinsfile
- What the Jenkins API reaches
- Connecting it to CodeWords
- Eight automations worth building
- Flaky tests, which nobody owns
- Building it so it survives
- Limits worth knowing about
- What to build first
- Frequently asked questions
What belongs in the Jenkinsfile
In the pipeline: building, testing, quality gates, artifact publishing, deployment, and anything that should fail the build. These are properties of one commit and belong where they can stop it.
Outside the pipeline: anything spanning builds or jobs. Trend reporting, flaky test detection, cross-job dependency analysis, cleanup of jobs nobody runs, and notifications shaped for people rather than for the build.
The anti-pattern is a post-build step that posts to chat, updates a tracker, files a ticket, and calls two external APIs. It works, the controller now holds credentials for all of those, and every job that copies the pattern copies the credentials.
Keeping external integrations outside the controller is also a security improvement, because a Jenkins controller is a machine that executes code by design and is therefore a poor place to concentrate credentials for everything else you own.
What the Jenkins API reaches
Jobs can be listed, read, triggered, enabled, and disabled, with their configuration available as XML.
Builds can be listed and read with their result, duration, timestamps, causes, and parameters. The build history is the raw material for every trend automation here.
Console output can be read, which is what makes failure triage possible.
Test results are exposed where a test reporting plugin is configured, including which tests failed and for how long they have been failing.
Queue state shows what is waiting and why, which is where you find out that a job has been blocked on an offline agent for a day.
Nodes and agents can be listed with their status, which is the other half of that answer.
Plugins can be listed with versions, which supports the maintenance reporting most instances badly need.
Nearly everything is reachable through the remote access API with a suffix on the URL, which makes Jenkins unusually easy to automate against once you have a token.
Connecting it to CodeWords
CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.
- Open CodeWords and start a new automation.
- Describe what should happen in plain language to Cody, the automation builder: which jobs, which condition, and what should happen elsewhere.
- Authorize the connection with a Jenkins user and API token, using an account with read access rather than an administrator where the automation only reads.
- Describe the exceptions: a job with no build history, a build still running, an agent that is offline.
- Run it in report-only mode first and read what it would have said.
You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.
Eight automations worth building
Failure triage with context. A failed build reported with the job, the commit, the author, and the relevant part of the console output rather than a link to it. The default notification says something failed; the useful one says what and whose.
Flaky test detection. Tests that pass and fail without the code changing, identified across builds and reported with how often. Covered in its own section because it is the highest-value thing here.
Build duration trends. Which jobs are getting slower, over weeks rather than felt impressions. Build time degrades gradually and is noticed only when it becomes unbearable.
Queue and agent health. Jobs waiting unusually long, agents offline, and executors saturated, reported before somebody asks why the build is slow today.
Abandoned job detection. Jobs with no successful build in months, jobs that have not run at all, and jobs pointing at repositories that no longer exist. Every long-lived instance has these and nobody removes them.
Plugin and version reporting. Which plugins are installed, which are outdated, and which have known issues, as a monthly report. Jenkins maintenance slides because nobody has the list.
Cross-job dependency detection. Which jobs trigger which, so a change to a shared job is understood before it is made rather than after.
Deployment records. What was deployed, when, from which commit, written somewhere durable, so the question of what changed has an answer later.
Flaky tests, which nobody owns
A flaky test is worse than a failing one. A failing test blocks work and gets fixed. A flaky test teaches people to rerun the build, and once that habit exists, real failures get rerun too.
Detect them by result history rather than by intuition. A test that has both passed and failed on the same commit, or that fails intermittently across builds with no related change, is flaky. That is computable from build history and is not something anybody will spot reliably by feel.
Rank by disruption, not frequency. A test failing one run in fifty is annoying. One failing one in four blocks the team daily and is worth fixing this week.
Report them with the evidence. How often, since when, and on which builds. A flaky test report without the pattern attached gets argued with rather than acted on.
Make the list visible and owned. Flaky tests persist precisely because they belong to nobody. A weekly list with names against them changes that more than any tooling.
Resist auto-retry as the answer. Retrying flaky tests makes the build green and hides the signal. It is sometimes the pragmatic short-term choice, and it should be recorded as a deliberate decision rather than adopted quietly, because a suite with retries everywhere has stopped telling you anything.
Building it so it survives
Read-only by default. Most useful Jenkins automation only reads. Triggering and configuring jobs are separate permissions and should be granted separately.
Handle the still-running build. An automation reading the latest build will sometimes find one in progress, with no result yet. Treat that as a state rather than a failure.
Paginate properly. Build history is long. An automation reading only the most recent results computes a trend from a window that flatters.
Report the outcome. A weekly summary is both the useful output and the thing that tells you the automation still runs.
Do not let an automation modify job configuration unless that is specifically what it is for and somebody has agreed to it. Job configuration on an old instance is frequently load-bearing in ways nobody remembers.
Limits worth knowing about
Performance on large instances is a real constraint. Requesting the full API tree for every job on an instance with thousands of builds is a heavy request. Use the tree and depth parameters to ask only for the fields you need.
Console logs can be very large, so read incrementally or read the tail rather than pulling everything.
Authentication needs an API token, not a password, and the token carries the permissions of the user it belongs to.
Plugin availability determines what is exposed. Test results in particular depend on a reporting plugin being configured, so an automation expecting them will find nothing on jobs without it.
Instances vary enormously. Jenkins is old and configurable, so an automation that works on one instance may need adjusting for another, particularly across controller versions.
What to build first
Flaky test detection, running read-only. It needs no write access, it produces a list that is immediately actionable, and it addresses the problem quietly costing the most: a team that has learned to rerun builds rather than trust them.
Two habits make the difference. Run it in report-only mode for a cycle and read the output, since the first version usually ranks by the wrong thing. And have it report weekly even when the list is short, because a silent report cannot be distinguished from a broken one.
Frequently asked questions
Why not put this in the Jenkinsfile?
Because none of it is about one build. Flaky tests, duration trends, and abandoned jobs are patterns across builds, and a pipeline has no place to stand and observe the whole instance. Keeping external credentials off the controller is a second, separate reason.
Can an automation trigger builds?
Yes, with a token that has permission. Whether it should depends on what the job does. Triggering a nightly regression run is reasonable; triggering a deployment is a decision to make deliberately rather than to discover you made.
How do I detect flaky tests reliably?
From build history: tests that have both passed and failed without the relevant code changing, or that fail intermittently across consecutive builds. Rank by how often they disrupt rather than by raw failure count, since one failure in four is a different problem from one in fifty.
What permissions does the automation need?
Read access for nearly everything worth building. Add build-triggering permission only if it triggers, and never give it configuration rights unless reconfiguring jobs is specifically its purpose.
How do I stop the automation overloading the controller?
Use the tree parameter to request only the fields you need, avoid fetching full console logs where a tail will do, and schedule heavy analysis outside working hours. Jenkins API requests are heavier than they look on large instances.
Can it find jobs nobody uses any more?
Yes, and it is one of the more satisfying reports to run: jobs with no successful build in months, jobs that have never run, and jobs pointing at repositories that no longer exist. Report them for a decision rather than disabling automatically, since an annual job looks abandoned for eleven months of the year.
Does this work with Jenkins running behind a firewall?
It needs network reach to the controller, so an internally hosted instance needs the automation to be able to reach it. Where that is not possible, the inverse pattern works: have Jenkins push events outward rather than having the automation pull.
How do I report build trends without overloading the controller?
Pull the summary fields you need with the tree parameter rather than full build objects, cache what you have already read, and run the analysis on a schedule rather than on demand. Build history is large and the API will happily give you all of it slowly.
Can it help with a Jenkins migration?
Yes, and this is one of the better uses. An inventory of every job, when it last ran, when it last succeeded, and what triggers it is the document a migration actually needs, and almost nobody has it. Producing it read-only before anybody plans the move saves arguing from memory.
What should the weekly report contain?
The flaky test list ranked by disruption, jobs whose duration has grown materially, anything that has not succeeded in a month, and agents that were offline. Four things, one page. A longer report gets skimmed, which defeats the purpose of having built it.
Can it tell me whether a build failure is mine?
Partly. Correlating the failure with the commits in that build narrows it, and distinguishing an infrastructure failure from a code failure narrows it further. That combination gets most failures to the right person, which is the practical goal rather than perfect attribution.