TaxpayerScope
Status
active
Date
September 23, 2026
Methodology version
4

0006: Raw source archive object key layout

Context

INFRA-08 requires the raw source archive S3 bucket to have versioning, encryption, and lifecycle policies enabled, and PRD §24 requires that a silent government file replacement be detectable. Phase 1’s D-01/D-02 (.planning/phases/01-source-data-verification/01-CONTEXT.md) deliberately deferred the canonical S3 key layout to Phase 2: Phase 1’s local archive is a flat, git-tracked directory keyed only by source_id, which is adequate for a one-time verification pass but was never meant to be the layout Phase 3’s collector uploads against.

Phase 3 builds the collector that populates this bucket (ROADMAP Sequencing Note: “implemented in Phase 3 against the layout fixed here”). This record fixes that layout before Phase 3 exists, so the collector has a committed, tested contract to write to rather than inventing one under time pressure. data/sources/fl-edr-sources.json’s 144 committed manifest records — each carrying source_id, filename, original_url, agency, retrieved_at, file_size, and sha256 (PIPE-03’s field contract) — supply the county, series, hash, and filename every archive key in this layout is built from.

Decision

  1. The archive key shape is:

    raw/florida/edr/{county_slug}/{series}/{sha256_prefix}-{original_filename}

    where series is expenditures or revenues, and sha256_prefix is the first twelve characters of the file’s lowercase hex SHA-256 digest.

  2. The object name — not just the object’s containing path — carries the content hash. A re-downloaded source file whose bytes changed produces a different sha256_prefix and therefore lands as a new object beside the original rather than overwriting it in place. This is the D-20 through-line: prefer the option that removes the failure mode over the option that merely detects it after the fact.

  3. pipeline/adapters/florida/archive_keys.py implements this as a pure function, raw_archive_key(county_slug, series, sha256, original_filename) -> str, that raises ValueError on any malformed input rather than producing a plausible-looking key. A bad key written once occupies a permanent slot in the archive.

Rationale

Why content-addressing the object name, not the whole path. A key that varies only by directory (for example, a retrieval-date directory) still lets a same-named file overwrite a prior version’s object name within a differently-dated directory tree, and does nothing to stop an overwrite if the upload path is ever re-run against the same date. Putting the hash directly in the file name itself means the object identity is the content identity: two different byte strings for the same government/series pair are structurally two different keys, full stop, with no separate bookkeeping (a manifest lookup, a date comparison) required to notice.

Why the county/series tree is kept, rather than flattening the path. This is the organization Phase 1’s manifest already uses (source_id: fl-edr-{county}-{series}), and it is what a human investigating an incident actually wants when browsing the bucket directly: “show me every file ever archived for Orange County’s expenditures series” is a prefix listing, not a manifest query.

Why the original filename is retained in the key. Provenance questions — “what did the government actually name this file when they published it?” — are answerable by reading the key alone, without a separate manifest lookup. This matters during an incident (a challenged figure, a suspected replacement) when the manifest itself might be exactly what is under scrutiny.

Why PRD §6’s year-partitioned sketch is deliberately not followed. PRD §6 sketches a raw tree shaped /raw/florida/edr/2015/ 2016/ …, implying one file per fiscal year. Phase 1 established directly — by downloading and inspecting all 144 source files — that Florida’s EDR office publishes one multi-year workbook per county per series, spanning roughly FY2005–FY2025 in a single .xlsx file. There is no year boundary to partition on: a single archived file’s content covers roughly twenty fiscal years at once, so a year-keyed prefix has nowhere to put it without either duplicating the same file under twenty different year prefixes (defeating the point of content addressing, and making a single re-download look like twenty separate changes) or arbitrarily picking one year to key it under (which is simply wrong — the file is not “the 2015 file”). This is stated here as the evidence-based divergence it is, not left as a silent gap between the PRD’s sketch and what actually gets built.

Rejected alternative: pure flat content addressing (raw/sha256/<hash>). This is the strictest form of integrity and gives perfect deduplication — identical bytes always produce the identical key regardless of which county or series they came from. It was rejected because it makes the bucket unbrowsable: every provenance question (“which county is this?”, “which series?”, “what did the government call it?”) requires a manifest lookup, including during an incident — precisely the moment a human most wants to just look at the bucket and see something meaningful, not cross-reference an external JSON file object-by-object.

Rejected alternative: retrieval-dated snapshots (raw/florida/edr/{county}/{series}/{retrieved_date}/{filename}). This is closest to PRD §6’s year-partitioned spirit and trivially answers “what did we have on file as of date X?” for an audit. It was rejected because identical re-downloads — which are the common case, since most re-runs will not find a changed government file — would duplicate identical bytes on every single collector run, growing the archive without bound for files that never actually changed. It also fails PRD §24’s actual requirement: the key itself says nothing about the file’s content, only about when it was fetched, so detecting whether two dated snapshots hold the same or different bytes still requires comparing hashes out-of-band.

Source Evidence

The seven-field manifest contract every key in this layout is built from, from data/sources/fl-edr-sources.json (144 records, committed in Phase 1):

{
  "source_id": "fl-edr-alachua-expenditures",
  "filename": "alachuacountyexpenditures.xlsx",
  "original_url": "https://edr.state.fl.us/Content/local-government/data/revenues-expenditures/cntyfiscal/alachuacountyexpenditures.xlsx",
  "agency": "Florida EDR",
  "retrieved_at": "2026-09-22T22:27:13.482802+00:00",
  "file_size": 283875,
  "sha256": "649e962a8736900319ff7f06ad7dabb2dd67265ba5ba820fbb66543a8d4c1442"
}

Applying this layout to that record produces:

raw/florida/edr/alachua/expenditures/649e962a8736-alachuacountyexpenditures.xlsx

The multi-year-workbook finding this record rests on is Phase 1’s own source enumeration (.planning/phases/01-source-data-verification/01-CONTEXT.md D-01/D-02): 132 county-fiscal-page files (66 counties × 2 series) plus 2 Duval municipal-page files, each one workbook per county/series pair, not one file per fiscal year.

The bucket properties this key layout is written into are Task 2 of this plan (infrastructure/infra/template.yaml’s RawArchiveBucket): versioning enabled, AES256 server-side encryption, and a lifecycle rule (raw-archive-tiering, Prefix: raw/) that transitions to STANDARD_IA at 30 days and GLACIER_IR at 180 days, with no expiration rule for current objects — D-07’s no-expiration stance, cross-referenced here because it is what makes this key layout’s guarantee durable: a key that can never be overwritten is only meaningful evidence if the object behind it can also never be deleted.

Methodology Version

This decision takes effect in methodology_version: 1. It has no prior version to supersede — this is the first record of the raw archive’s canonical S3 key layout, closing the layout question Phase 1’s D-01/D-02 explicitly deferred here.

Revision (2026-09-24)

This revision supersedes methodology_version: 1 with methodology_version: 2. The decision recorded above is carried forward unchanged into methodology_version 2; the changes this version-2 coordinated bump makes are recorded in 0001-total-expenditures-definition.md, 0005-population-and-cpi-series.md and 0008-total-revenue-definition.md. This revision recorded by Task 3, 03-06-PLAN.md.

Revision (2026-09-24)

This revision supersedes methodology_version: 2 with methodology_version: 3 (ADR-0009, 04-01-PLAN.md). The decision recorded above is carried forward unchanged into methodology_version 3; the changes this version-3 coordinated bump makes are recorded in 0009-category-and-revenue-classification.md. This revision recorded by Task 3, 04-01-PLAN.md.

Revision (2026-09-25)

This revision supersedes methodology_version: 3 with methodology_version: 4 (ADR-0010, 05-01-PLAN.md). The decision recorded above is carried forward unchanged into methodology_version 4; the changes this version-4 coordinated bump makes are recorded in 0010-statewide-publication-rules.md. This revision recorded by Task 3, 05-01-PLAN.md.