AWS S3 and CodeWords: automating around object storage

S3 is where files go to wait for something to happen to them. Somebody uploads a CSV that needs importing, a supplier drops a file that needs parsing, a scanner deposits a document that needs reading, an application writes an export that needs sending somewhere else.

The bucket does not do any of that. It holds the object and, if you have configured it, tells you the object arrived. Everything after that notification is the automation, and the interesting design decisions are about how you react rather than about storage.

What we'll cover

• Reacting to objects arriving
• What the S3 API reaches
• Connecting it to CodeWords
• Eight automations worth building
• Permissions, where this goes wrong
• Cost and lifecycle
• Building it so it survives
• Limits worth knowing about
• What to build first
• Frequently asked questions

Reacting to objects arriving

Three ways to know a file has appeared, and the choice shapes everything else.

Event notifications fire on object creation and deletion, delivered to a queue, a topic, or a function. This is the right default: it is immediate, it does not cost anything to wait, and it scales without you thinking about it.

Polling a prefix means listing objects on a schedule and comparing against what you have already seen. Simple, and it costs a list operation every interval whether or not anything changed, and it is always slightly late.

A scheduled sweep over everything in a prefix, regardless of whether it is new. Occasionally what you actually want — for reconciliation or reprocessing — and a poor substitute for events.

Use events where you can. Use polling when you cannot configure notifications on a bucket you do not own, which is common when a partner drops files into somewhere you were given access to.

What the S3 API reaches

Objects can be listed, read, written, copied, and deleted, with prefixes acting as the organizing structure. There are no real folders, only keys that contain slashes, which matters more than it sounds when you are designing prefixes.

Metadata can be read without downloading the object, which is the single most useful capability for cheap automation: size, type, timestamps, and any custom metadata you set on upload.

Presigned URLs grant time-limited access to a specific object without exposing credentials, which is how you let someone upload or download without giving them anything permanent.

Multipart upload handles large objects, and matters because incomplete multipart uploads cost money while being invisible in a normal listing.

Versioning, where enabled, keeps previous copies, which changes what deletion means and is worth knowing before you write anything that deletes.

Storage classes determine cost and retrieval time, and objects can be transitioned between them by lifecycle rules rather than by your automation.

Connecting it to CodeWords

CodeWords connects to more than 3,000 integrations, and the connection is made once and reused.

1. Open CodeWords and start a new automation.
2. Describe what should happen in plain language to Cody, the automation builder: which bucket and prefix, what arrives there, and what should happen to it.
3. Authorize the connection with credentials scoped to the specific bucket and prefix rather than to your account.
4. Describe the exceptions: a file in the wrong format, a partial upload, an object that is larger than expected.
5. Test with a copy of a real file in a test prefix before pointing anything at production paths.

You describe the outcome; Cody builds it, connects it, and deploys it. The free plan covers light use, with Pro at $39 per month and Business at $100 per month as usage grows; details are on the pricing page.

Eight automations worth building

Import on arrival. A CSV lands, gets validated, and its rows reach the system that needs them, with a report of what was accepted and what was rejected. The reporting half is the part usually missing, and it is the half that matters when a supplier sends a malformed file.

Document processing pipelines. A scan arrives, gets prepared, has its fields extracted, and the structured result is filed. Keep the extraction result alongside the original, since the original is the evidence.

Media handling. Images resized and converted on upload, video transcoded, thumbnails generated, results written back under a predictable key.

Partner file exchange. Watch a prefix a partner writes into, process what appears, and move it to an archive prefix so the landing area stays a reliable signal of what is outstanding.

Scheduled exports. Assemble a file, write it to a prefix, and notify whoever consumes it, with a presigned URL if they are outside your account.

Integrity checking. Verify that expected files arrived, that their sizes are plausible, and that nothing has been sitting unprocessed. Missing files are the failure nobody alerts on because absence is invisible.

Orphan and incomplete cleanup. Find incomplete multipart uploads and objects nothing references, and report them. These accumulate silently and cost money indefinitely.

Access reporting. Who read what, from where, using access logs, for buckets holding anything sensitive.

Permissions, where this goes wrong

More S3 incidents come from permissions than from anything else, and the same few mistakes recur.

Scope to the prefix, not the bucket. An automation that processes uploads in one prefix needs access to that prefix. Granting the whole bucket is easier and gives the credential far more reach than the task requires.

Read and write are different needs. Most processing automations read the input and write the output somewhere else. That is read on one prefix and write on another, not full access to both.

Delete is a separate decision. Very few automations should be able to delete. Moving a processed file to an archive prefix is a copy and a delete, which is worth being deliberate about rather than granting casually.

Never make a bucket public to solve an access problem. Presigned URLs exist precisely for this, are time-limited, and are scoped to one object.

Rotate credentials and revoke them when an automation is retired. An access key created for a job that finished two years ago is exactly the credential that appears in an incident report.

Cost and lifecycle

Storage is cheap enough that nobody watches it, which is how it becomes expensive.

Requests cost as well as storage. An automation listing a large prefix every minute is paying for those list operations forever, and at volume they can exceed the storage cost of what they are listing.

Use lifecycle rules rather than automations for transitions. Moving objects to colder storage after a period is a bucket configuration, not something your automation should implement.

Incomplete multipart uploads cost money and do not appear in a normal listing. A lifecycle rule to abort them after a few days is the single best-value configuration available, and almost nobody sets it.

Retrieval from cold storage has a cost and a delay. An automation that needs to read archived objects should know that before it tries.

Versioning multiplies storage. With versioning on, deleting does not reduce what you pay. That is the point, and it needs a lifecycle rule to expire old versions or the bucket grows forever.

Building it so it survives

Make it safe to process the same object twice. Event delivery is at-least-once. Track processed keys, or make the processing naturally repeatable.

Read the metadata before the object. Checking size and type first means an unexpected 4GB file produces a sensible rejection rather than an exhausted automation.

Write outputs to a different prefix from inputs. Writing results back into the watched prefix creates an automation that triggers itself, which is a memorable way to discover recursion.

Move processed files out of the landing area. A prefix that only contains unprocessed work is a status report you get for free.

Report the outcome with counts. Files seen, processed, rejected. A pipeline that silently stops looks identical to a quiet week without this.

Limits worth knowing about

Listing is paginated, and an automation reading only the first page works perfectly in testing and silently under-processes in production.

Consistency is strong for reads after writes, so the historical advice about waiting before reading a newly written object no longer applies, though anything relying on listing to reflect a write immediately deserves testing.

Object size limits apply per object and per part in a multipart upload, which matters for anything generating large exports.

Request rates scale by prefix. Very high-volume workloads benefit from spreading keys across prefixes, which is a design decision made at the start rather than retrofitted.

Cross-region transfer costs money. An automation running in one region against a bucket in another is paying for every byte, which is easy to miss until the invoice.

What to build first

Integrity checking on whichever prefix matters most: are the files that should have arrived actually there, are they a plausible size, and is anything sitting unprocessed. It needs read access only, it usually finds something on the first run, and it protects everything you build afterwards.

Two habits make the difference. Run it alongside whatever happens now for a cycle and compare, since the differences are usually genuine discoveries. And have it report that it ran even when it found nothing, because a silent check cannot be distinguished from a broken one.

Frequently asked questions

Should I use S3 events or poll the bucket?

Events, wherever you can configure them: immediate, cheaper, and they scale without thought. Poll when the bucket belongs to somebody else and you cannot add a notification, which is the common case for partner file drops.

How do I handle very large files?

Read the metadata first and decide before downloading. For genuinely large objects, process in a streaming fashion or in ranges rather than pulling the whole thing into memory, and treat anything above a threshold you set as a case that needs a different path.

Can I give a customer access to one file?

Yes, with a presigned URL, which is time-limited and scoped to that object. This is the right answer to nearly every "how do I share this" question, and it is much better than the alternatives people reach for, which usually involve loosening bucket permissions.

What permissions does the automation need?

The least that works: read on the input prefix, write on the output prefix, and delete only if it genuinely moves files. Scope to prefixes rather than buckets, and revoke the credential when the automation is retired.

Why is my bill higher than the storage I can see?

Three usual suspects: incomplete multipart uploads, which are invisible in normal listings; old versions retained by versioning; and request costs from something listing frequently. A lifecycle rule handles the first two and reviewing your polling intervals handles the third.

How do I stop an automation triggering itself?

Write outputs to a different prefix from the one you are watching, or filter the notification by prefix or suffix. Self-triggering recursion is the classic S3 automation failure and it is entirely preventable at design time.

What happens if the same event is delivered twice?

Your automation runs twice, because delivery is at-least-once by design. Either track which keys you have processed or make the processing naturally repeatable, such as writing to a deterministic output key that simply gets overwritten.

Should the automation move files after processing?

Yes, into an archive prefix, because a landing prefix that only holds unprocessed work is a free status report. Moving is a copy followed by a delete, so it needs delete permission on the source, which is worth granting deliberately rather than as part of a broader policy.

How do I handle files that arrive in the wrong format?

Move them to a quarantine prefix and report them rather than failing the run. A supplier sending a malformed file is a business problem that needs a human conversation, and an automation that simply errors leaves nobody with the information to have it.

Related reading

• Box and CodeWords
• CodeWords integrations and templates