Automate log analysis: application errors and exceptions
Grouping exceptions that are the same bug, spotting failure modes that are genuinely new, correlating errors with releases, and routing each one to the team that owns it.
Aymeric Zhuo10 min read
On this page
Application logs have a particular problem that infrastructure logs do not: the same underlying bug produces thousands of superficially different lines. Different user IDs, different request paths, different timestamps, different values in the interpolated message. A person reading them sees a wall. A grep for "error" returns everything and tells you nothing.
The job of automated application log analysis is to turn that volume into a short list of distinct problems, ordered by how much they matter, each routed to whoever can fix it. This page covers how. For syslog, host metrics, and infrastructure-level correlation, see automating server log analysis.
What we'll cover
Why searching does not scale
Search answers questions you already know to ask. The failures that hurt are the ones nobody thought to search for.
Three specific limitations come up repeatedly.
Variable data defeats string matching. Failed to process order 88213 for user 4471 and Failed to process order 88219 for user 3902 are the same bug and share no useful search string. Matching on a prefix works until someone changes the message format.
Volume hides proportion. A search returning twelve thousand matches tells you there are errors. It does not tell you that eleven thousand are one known issue and the remaining thousand are four distinct problems, one of which started this morning.
Absence is invisible. The most serious failures often produce fewer logs rather than more. A queue consumer that died stops logging entirely, and no search for errors will ever find it. Detecting this requires knowing what should be present.
Automation addresses all three by working on aggregates over time rather than on matching text.
Grouping errors that are the same bug
This is the foundational step, and everything downstream depends on getting it right.
The aim is a fingerprint: a stable identifier that is identical for two occurrences of the same bug and different for distinct bugs.
Normalize the variable parts. Replace numbers, UUIDs, email addresses, file paths, and quoted values with placeholders. Failed to process order <NUM> for user <NUM> groups the two examples above correctly and takes a handful of regular expressions.
Prefer the stack trace where you have one. For exceptions, the exception type plus the top few frames of the trace is a far better fingerprint than the message. It survives message rewording, which is common, and it distinguishes the same exception type thrown from different places, which matters.
Trim the trace deliberately. Framework and library frames are usually noise. Taking the topmost frames from your own code gives a fingerprint that stays stable across dependency upgrades.
Handle the awkward cases explicitly. Errors with no trace, errors from third-party services, and errors whose message is entirely dynamic need their own rules. A default fingerprint of the logger name plus severity catches these adequately.
Grouping typically reduces tens of thousands of lines to a few dozen distinct problems, which is the point at which a person can actually read the output.
Deciding what actually matters
Once grouped, the groups need ordering. Raw count is a poor guide, since the noisiest error is frequently the most harmless.
Weigh four signals together.
Distinct users affected. Far more meaningful than occurrence count. One user retrying forty times is one problem; forty users hitting it once each is considerably worse.
Whether it is new. A group not seen before this week deserves attention regardless of volume, because nobody has assessed it.
Rate of change. A group that has quadrupled since yesterday is more interesting than a larger group that has been flat for months.
Where it sits. An exception in payment handling outranks one in a background report generator, and this is business knowledge that has to be supplied rather than inferred.
The output worth producing is short: the five or six groups that meet a threshold on these combined, each with its count, affected users, first seen date, trend, and a representative example. Anything longer goes unread, which returns you to where you started.
Spotting genuinely new failures
New failure modes are the highest-value output, and the naive implementation is noisy enough that people switch it off.
Keep a register of known fingerprints with first and last seen dates. New means absent from the register, which is a simple lookup rather than a judgement.
Give new groups a settling period. Reporting a fingerprint the instant it appears means reporting every transient blip. Waiting for a small number of occurrences within a window removes most of that noise at the cost of a few minutes' delay.
Expect a burst after every deploy. New code produces new fingerprints, including harmless ones from changed message text. Correlating with deploys, covered next, separates the genuine new failures from renamed old ones.
Watch for disappearance too. A fingerprint that occurred steadily and then stopped is either fixed or hidden behind something that has stopped running. The second case is the one worth knowing about, and nothing else will tell you.
This is also where a language model earns its place in the pipeline. Given a fingerprint, a representative stack trace, and the surrounding log lines, a model produces a plain-language summary of what appears to have gone wrong and which component is implicated, which is genuinely useful triage on an error nobody has seen before. Keep it to summarizing and suggesting: the grouping, counting, and thresholds should stay deterministic, because those need to produce the same answer every time.
Correlating with releases
Most new application errors are caused by a deployment, which makes this the highest-return correlation available.
Record deploys in the same timeline. Service, version, time, and who deployed it. Nearly every CI system can emit this, and without it the correlation is guesswork.
Compare windows around each deploy. Error groups in the hour after a release against the hour before, and against the same period on previous days to allow for normal daily variation.
Report the delta, attributed. A message naming the service, the version, the new error groups that appeared, and the rate change on existing ones is the single most useful automated output in this whole area. It reaches the person who just deployed while they still have the change in mind.
Watch the slow ones too. Not every regression appears immediately. An error group that starts climbing four hours after a release, as a cache expires or a scheduled job runs, is easy to miss and easy to catch with a longer comparison window.
Routing to the team that owns it
An alert reaching everyone reaches nobody, and the routing is what determines whether the system is used after month two.
Map by code path. Service names, module names, and stack trace paths usually indicate ownership already. A mapping from those to teams handles most errors without anyone thinking about it.
Include enough context to act. The fingerprint, counts, affected users, first seen, trend, a representative trace, and a link to the full logs. An alert that requires opening three tools before the problem is even understood gets postponed.
Route by severity, not everything to one place. New errors in critical paths to a channel people watch; everything else to a digest. Mixing the two trains people to ignore the channel, which costs you the critical alerts.
Suppress what is acknowledged. A known issue being worked on should stop alerting. Without suppression, people mute the channel, and then the next genuine alert arrives in a muted channel.
Send a digest even when quiet. A short daily summary confirming the pipeline ran, with the top groups and anything new, keeps the absence of alerts meaningful.
Describing this is more practical than building it from parts. On CodeWords you describe what should happen in plain language: where the logs come from, how to group them, what counts as new, what deserves an alert, and who receives it. Cody, the automation builder, builds it, connects it to your logging and chat tools, and deploys it. Automations connect to more than 3,000 integrations. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.
Logging so that analysis is possible
Much of the difficulty in log analysis is created upstream, and a few habits in the application make everything downstream easier.
Log the error with its context, once. An exception logged at three levels of the call stack becomes three fingerprints for one bug. Catch, add context, and log at the boundary where you can actually say something useful.
Keep variable data in fields, not in the message. Order processing failed with order_id and user_id as separate fields fingerprints perfectly and stays searchable. The same information interpolated into the message string requires normalization to group and is fragile when the wording changes.
Include a correlation identifier. A request or trace ID carried through every log line for one operation turns scattered entries into a story. This is the single most valuable field for investigating anything non-trivial.
Use severity levels consistently. If warnings are used for things nobody acts on, the level stops carrying information, and every analysis rule has to work around it. Agreeing what each level means is a half-hour conversation that pays off indefinitely.
Log the successful path at low volume too. Knowing that something ran successfully is what makes its absence detectable. A start-and-finish pair for scheduled work costs almost nothing and catches the silent death that error logging never will.
Never log secrets or personal data. Redact at the point of logging rather than relying on downstream filtering, since the log has already been written by then and is probably in three places.
Frequently asked questions
Do I need structured logging first?
It helps considerably and is not a prerequisite. Structured logs make fingerprinting more reliable and the fields directly available. Unstructured logs can be parsed adequately with normalization rules, and moving to structured logging is worth doing anyway for reasons beyond this.
How is this different from an error tracking tool?
Error trackers do the grouping and alerting well and see only what your application reports to them. An automated workflow can combine that with deploys, business context, support tickets, and your own severity rules. Many teams run both, with the workflow adding correlation the tracker cannot see.
Will a language model hallucinate log analysis?
It can, which is why the counting and thresholds should be deterministic and the model confined to summarizing and suggesting. A summary that is occasionally wrong is acceptable when the numbers beside it are always right, and the trace is there for anyone who wants to check.
How far back should I keep logs for this?
Enough history to establish what normal looks like, which usually means several weeks. The fingerprint register should be kept longer, since it is small and it is what makes "new" meaningful.
What about logs containing personal data?
Redact on ingestion rather than downstream, so the sensitive values never reach the analysis. Fingerprinting works on normalized messages in any case, and normalization already replaces most of the fields that would be sensitive.
Should every new error produce an alert?
No, and doing so is the most common reason these systems get ignored. Apply a settling threshold, weight by affected users and code path, and send the rest in a digest. Interrupt people only for what warrants interruption.
How long does it take to see value from this?
The grouping alone usually pays for itself within the first run, because seeing tens of thousands of lines resolve into a few dozen distinct problems is immediately clarifying. The new-failure detection needs a couple of weeks to build a register before it becomes trustworthy.