S3 answers 403 for a missing key

For three weeks the [Binderdex](https://www.binderdex.com) social planner kept suggesting the same Pokemon set. Nine Instagram posts, two TikToks, cross-posts on every channel, and the planner still came back with the same answer every morning. The reason was embarrassing once I saw it: the set cooldown only read the TikTok ledger, which held three posts. Everything else we published was invisible to it.

Yesterday we rebuilt that memory across four merged PRs, and the interesting failure lived in the most boring part: a daily snapshot of trending hashtags, stored as one object per day in S3. The planner reads the newest snapshot to rank what's hot, so it needs a job that runs without a human awake. That's where S3 lied to me. Well, it didn't lie. It answered a question I hadn't asked carefully.

The 403 that means "no such key"

The snapshot writer runs under a scoped-down role: it may list only under two specific prefixes. That restriction has a side effect I didn't expect. When the code does a GET on a key that doesn't exist, S3 answers 403 instead of NoSuchKey, because admitting the key is absent would leak information about what you can't see. And the very first run of every new day hits exactly that path: today's object doesn't exist yet.

My first version treated any 403 on the read as "missing, start fresh". That's wrong in a way that would have hurt later. A 403 is also what you get for a key that genuinely exists but that the role can't touch. Silently treating a real permission bug as an empty day means the job runs "fine" while writing garbage. So the rule became: a 403 counts as missing only when an exact-prefix LIST proves the key absent. A 403 on a key the LIST can see still fails, loudly.

Here's the shape of the read, simplified from what shipped:

async function readDay(key: string): Promise {
  try {
    return await get(key);
  } catch (err) {
    if (!is403(err)) throw err;
    // A scoped-down role gets 403 for missing keys too.
    const exists = (await listPrefix(dirOf(key))).includes(basename(key));
    if (exists) throw err; // real permission problem: fail loud
    return null;           // proven absent: brand-new day
  }
}

The LIST is the whole trick. Existence has to be proven by a call that's allowed to see the answer, otherwise you're interpreting silence and silence has two meanings.

Two writers, one day key

The second review pass caught a nastier one. The write path was read-merge-write ending in an unconditional PUT. Two runs on the same day (the scheduled job and a human poking the planner by hand) could each read the day's object, merge their own tags in, and clobber each other. The fix is conditional writes: PUT with If-Match carrying the ETag you merged from, If-None-Match when you're creating a new day, and on a precondition failure, re-read, re-merge, retry, three attempts before giving up.

for (let attempt = 0; attempt < 3; attempt++) {
  const current = (await readDay(key)) ?? emptyDay();
  const merged = mergeTags(current, todaysTags);
  try {
    await put(key, merged, preconditionFor(current));
    return merged;
  } catch (err) {
    if (isPreconditionFailed(err)) continue; // someone wrote first
    throw err;
  }
}
throw new Error("day key kept changing under us");

What the planner does now

The user-facing upshot: the cooldown is finally honest. Every channel's published history feeds it now, your own channel's posts hold a set for three weeks, other channels' for two, and when something is on hold the planner names the post that blocked it. A new picker command takes a channel name and prints its ranked subjects with the reasoning next to each one. The first real production run moved Monday's pick off the stale set onto something that hadn't been posted anywhere in weeks.

Two honest gaps: nothing automated calls the picker yet (that's the next step), and the scheduled job itself still needs its infrastructure deploy plus one manual run before I'll call it shipped. The code and the 190-odd tests around it are done; the last mile isn't. If you keep a daily-keyed object in S3 behind a scoped role, I'd check your missing-key path tonight. Mine was broken in a way that only showed up on day one, which is exactly when nobody is watching.