🧱 Estate site (public twin) · index · reuse · features · data layers · handoff status

DATA_LAYERS_CENSUS.md — the data we own but never counted

_Created: 06-07-2026 · Last updated: 29-10-2026_

Purpose. A QA/gap-finding census of the data layers across the ~85 canonical repos, built by harvesting the filesystem (not the curated inventory). Where FEATURES_INDEX.md answers *"what did we consciously register as an asset?"*, this file answers the opposite: what large data is sitting on disk that was never raised to asset level, never counted, and never analysed? It is a defect-finder, not a catalogue — every row is meant to prompt "should this be backed up / registered / deduplicated / turned into a statistic?".

Scope vs. the kosha manifest (ruling F7, 26-08-2026). kosha/data/manifest/datasets.json is the registry for published, fetchable derived datasets — the ones another repo or a human is meant to pull down and consume. This census stays the human survey of large on-disk data, including data that is not publishable (rights-uncertain, personal, or simply not worth shipping) — a giant belongs here whether or not it will ever earn a manifest row.

> Harvest method: os.walk over the org clone root, data extensions only (.jsonl .json > .tsv .csv .txt .db .sqlite .conllu .xml .parquet …), 1 MB threshold for "large". > Tracked-vs-local classified by git ls-files membership. Committed re-run: > tools/data_layers_harvest.py > (H1989, 31-07-2026) — replaces the original throwaway harvest.py + classify2.py. > Latest snapshot: reports/DATA_LAYERS_HARVEST_SNAPSHOT_2026-08-22.md > + data/data_layers_harvest_2026-08-22.json > (H3316, OxAlpha x-preview-f-free; previous: reports/DATA_LAYERS_HARVEST_SNAPSHOT_2026-07-31.md > + data/data_layers_harvest.json). > Sizes below mix the 31-07-2026 re-harvest (headline / §1) with older per-file notes > that still carry their measurement dates; re-verify before acting on a single path.

At a glance

• 22-08-2026 re-harvest (H3316, OxAlpha x-preview-f-free): 31.58 GB large files

(≥1 MB) across 157 top-level dirs; 1,599 large files; tracked 11.08 GB (35%) vs local-only 20.50 GB (65%) — local-only *absolute* down ~1.2 GB and its share down 2 pts since 31-07. FEATURES_INDEX basename hit 36 / miss 1,563. Every anonymous no-consumer accumulation now named — see §6.

• 31-07-2026 re-harvest (H1989): 32.3 GB large files (≥1 MB) across 158 top-level

dirs under GitHub/ (scope wider than the original ~90 — see Method); 1,646 large files; FEATURES_INDEX basename hit 38 / miss 1,608.

• Tracked 10.6 GB (33%) · local-only 21.7 GB (67%) — local-only *share* slightly down from

the 06-07 ~70% / 19.2 GB framing, but absolute local-only grew (~+2.5 GB) with new giants (SpellCheck DTA zip, spoken-sanskrit segments, nagari.db, nkrya-parallel).

• Historical 06-07 baseline (kept for delta): ~27.5 GB large / 1,343 unregistered / 8.3 GB tracked.
• One file — archive_stopword.sqlite — is still ~10.5 GB, larger than every tracked repo combined.
• ✅ H691 provenance sweep (11-07-2026, Fable 5 claude-fable-5): every *then*-listed §2

giant has SHA256 + live counts + schema + rebuild recipe in the kosha manifest (kosha PR #52). Of the old “19.2 GB single-copy” headline, only corpus_lexicon.jsonl (~277 MB on 31-07) is genuinely single-copy-and-not- rebuildable — new 31-07 giants (DTA zip, segments.jsonl, …) still need the same regenerate-vs-backup classification. Update 25-10-2026: the corpus_lexicon half of this sentence is superseded by the §2 row below (H3316/H3389): it is no longer single-copy, and the current state is recorded there with a dated verification.

• ✅ GAPS §3 graduated 31-07-2026 → FINDINGS §261 —

the “never counted” claim is closed; residual unknowns are listed there, not a blank estate.

The headline QA finding is not "we lack statistics." It is that the majority of our derived data still sits local-only — statistics (mostly done) and provenance (H691 for the old giants) come first; new local-only arrivals need the same treatment.

1 · Data footprint per repo (tracked vs local-only)

31-07-2026 re-harvest (large files ≥1 MB only). Full table: reports/DATA_LAYERS_HARVEST_SNAPSHOT_2026-07-31.md.

_SpellCheck jumped from ~0.3 GB (06-07) to 2.8 GB because external_src/dta/dta_lingattr.zip (2.53 GB) is now on disk — the largest new local-only arrival. Full tail (AP, PWK, CORRECTIONS, Systema, …) is in the snapshot report._

2 · 🔴 Local-only giants — single-copy, unbackuped, uncounted

Every file below is not tracked by git (gitignored or never added). If this disk dies, they are gone. None is registered as an asset. Ranked by size.

Exact counts (measured 06-07-2026; rows marked H691 re-measured 11-07-2026 with SHA256s recorded in the kosha manifest):

Two recurring defects visible in this list:

1. Duplicate pipeline variants — the assembled_cards.jsonl → .renou.jsonl →

.renou.bhs.jsonl → .renou.bhs.wl.jsonl series (four ~200 MB stages, ~810 MB) is content-contained: all four carry the identical 120,173 rows, earlier stages fully — only the final stage need survive (~607 MB). The per-dict {code}.renou (dot) vs {code}_renou.bhs.wl (underscore) families are NOT byte-redundant — they diverge (12-07-2026) ruled that divergence a correction**: the dot-file is canonical and the _renou* chain over-attests (it predates the DCS min-support filter), so the ~379 MB underscore chain is safe to delete as superseded (not merged, not renamed).

MG's Yandex Disk (rclone yadisk:, WebDAV) scanned read-only: 13,963 files · 224.53 GB · 64 folders. Research giants: ААЗализняк-архив 108.8 GB (txt 266 + srt 200 + mp3 261), Dark_library 23.6 GB (368 pdf), Sanskrityatina workshop 26.7 GB (55 dirs), Panini 10.1 GB, kRtam own-corpus (Параллели 621 CSV 4.2 GB, Lazarus dictionary src, Композиты). Publishing: Frish multivolume chrestomathy 27 toms InDesign 8 GB. Full classification + Top-10 reusable candidates: reports/YADISK_INVENTORY_07-09-2026.md. Landing = derived-only (H1333 precedent).

3 · Statistics to add, per layer (and where they live)

The question "what statistics, where" resolves to one table. Each row is a stat that is currently not computed, the QA question it answers, and its home.

Priority order for QA value: (1) the local-only-backup gap in §2, (2) ~~the "keys with no RU gloss" gap-list (directly feeds Tier-1 RussianTranslation)~~ ✅ done 11-07-2026 (H685), (3) the <ls> citation graph (highest publishable novelty), (4) ~~the code-duplication census (measures the sanskrit-util dedup payoff)~~ ✅ done 12-07-2026 (H688).

4 · Actionable gaps (mirror to GTD)

• ✅ Back up corpus_lexicon.jsonl (277 MB / 290,543,363 B) — **local dual-drive mirror

done 31-07-2026 (H1998, Grok 4.5 grok-4.5): SHA-256 9f3d852f1f1424c275af2cc1823dab1b561e649320e597d3cab013068ccc4072 identical on SanskritLexicography/…/src/corpus_lexicon.jsonl and D:\Backups\csl-data-layers\SanskritLexicography\RussianTranslation\src\. Also mirrored the irreplaceable ReverseDictionary master (266820-reverse-Gasuns.txt). Script retiered: tools/backup_census_priority.py defaults to IRREPLACEABLE-only + --local-mirror (regenerable giants no longer pollute the default run). 🔴 Yandex / off-machine leg still open** — needs local YANDEX_DISK_* env (H235-era @DO). Do not priority-backup archive_stopword.sqlite or dta_lingattr.zip (rebuild / re-fetch). Update 22-08-2026 (H3316): a third copy of the master landed as a tracked file in gasyoun/pwg-ru-data (corpus/corpus_lexicon.jsonl, SHA-256 identical) — the "irreplaceable single-copy" status is retired; remaining risk is only the off-machine leg.

• 🟢 Prune superseded Renou variants — **{code}_renou* underscore chain ✅ DELETED

12-07-2026 (~362 MB freed, MG authorization; superseded per dot-file canonical, chain over-attests). Also ✅ DELETED 12-07-2026 (~580 MB freed, MG authorization):** assembled_cards stages 1–3 (content-contained, verified 120,173-row parity + field-superset keeper; only the final assembled_cards.renou.bhs.wl.jsonl survives, Net ~942 MB freed across both. Small .chunk/.perf/.smoke/.test/.quarantine fixtures kept.

• ✅ Register the tracked-but-uncounted assets in FEATURES_INDEX.md — done 11-07-2026

(H694, Fable 5 claude-fable-5): CORRECTIONS/ngram/allngramtxt.txt (6,656,616 n-grams, 82.3 MB → F43), VisualDCS/derived-data/Kompozity/names.csv (168,880 compounds, 90.7 MB → E42), the csl-observatory correction_events_{all,typed,final}.csv trio (52,498 events × 3, 178.0 MB → E41); SL PR #350 + kosha manifest rows (kosha PR #53, ids correction-events-log, kompozity-names-compound-splits, cdsl-headword-ngram-oracle).

• ✅ VisualDCS/derived-data/Paralleli-v-tekstah-korpusa-SRC/PARA/Stopovye/

(102 files, 1.17 GB, the largest single derived-data bundle in the repo) — checked here 07-07-2026 (H291): it is git-tracked (113 objects incl. a split .7z), so it does not belong in § 2's unbackuped-giants table; instead registered as a bundle row (stopovye-parallel-passages) in kosha/data/manifest/datasets.json.

5 · The 27-07 no-consumer accumulation census (72 catalogued items) — reconciled 30-07-2026

Source: ASK_BATCH_STAGING_FABLE_ROADMAP_HARVEST_2026-07.md § "The data that is piling up — 72 catalogued accumulations" (27-07-2026, Opus 5 1M claude-opus-5[1m], /ask-batch): data on disk with no downstream consumer found, harvested as a side-output of the Fable-tier roadmap sweep. Ten of the sweep's minted handoffs (H1754–H1763) drain six of these rows in aggregate, but the source doc does not map which six — the mint list and the accumulation list are reported as separate totals, not a row-to-row crosswalk, so that mapping cannot be reconstructed after the fact.

Honest enumeration limit. The source doc lists 72 as a *count*, not 72 named rows — its prose names roughly a dozen items with enough path specificity to re-stat individually (the rest were only visible in the harvesting session's per-agent workflow journals, which are ephemeral subagent transcripts, never committed, and not recoverable from this session). Per this handoff's scope fence ("apply the most conservative reading that satisfies the acceptance criteria, log it, press on"), the reconciliation below covers every item the manifest names concretely, re-stated against the filesystem as of 30-07-2026 rather than copied from the 27-07 figures. The remaining ~60 items named only in the manifest's size/count roll-up prose cannot be individually rowed without re-running the harvest — that gap is recorded here rather than silently closed.

Dedup verdict: 1 of 12 named items (ORS-FAQ's Telegram export) was already a §2 row — no duplicate added. 1 of 12 (Uprava's telegram-zabota-export/result.json) has gained a consumer since 27-07 and is marked consumed rather than dropped. The remaining 10 are new rows above.

What this does not close: the ~60 accumulations visible only in the manifest's size/count prose (e.g. the aggregate "six with no consumer" inside RuWritingStyles-corpus's PDFtoTXT pairs, or SanskritGrammar's ten pwg_* directories counted as one bucket) are captured at the bucket level above, not the individual-file level the acceptance criteria originally envisioned — that finer grain no longer exists outside this reconciliation. Re-running the harvest (/ask-batch's underlying os.walk + no-consumer cross-ref, same method as §"Method & reproduce" below) is the only way to recover it. Closed 22-08-2026 — see §6.

6 · The 22-08 accumulation naming pass (H3316) — every anonymous accumulation named

The §5 honest-enumeration limit is closed. The fresh harvest (At a glance, 22-08) was re-classified by tools/data_layers_accumulations.py over the committed inventory data/data_layers_harvest_2026-08-22.json: 217 of the 1,599 large files are both local-only and FEATURES_INDEX-unregistered; adaptive directory bucketing names them as 67 accumulation rows — each with a producing repo, a consumer verdict, and a keep/prune-candidate flag; 0 unadjudicated. Mechanical facts: reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.tsv; adjudication: reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.verdicts.json — the table below is regenerable from those two (--verdicts --md-out), not hand-typed. Scope fence held: this pass classifies only; nothing was deleted or moved. Deletion candidates above are proposals for a later MG review sheet.

Dual-run provenance (claim-holder adjudication, 22-08-2026). Two independent OxAlpha (x-preview-f-free) sessions executed H3316 concurrently — the claim holder (14:13Z) and a second session that worked in the same worktree without checking the live claim. Both ran full harvests 12 minutes apart with byte-identical totals (1,599 large files, 33,904,216,315 B; tracked 11,894,462,164 B vs local-only 22,009,754,151 B), diverging only in dir count 156 vs 157 (one empty-of-large-files dir appeared between runs) — an inter-run parity check, recorded here as validation. The landed pass is the second session's (fbadc9ea); this paragraph and the two corpus_lexicon.jsonl updates above are the claim holder's adjudication layer. One verdict refinement for MG's review sheet: dta_lingattr.zip is not agent-regenerable but IS human-refetchable — the upstream public zip it came from is named in detectors/extract_dta_pairs.py (dta-lingattr-tei_*.zip from deutschestextarchiv.de/download); backup-vs-refetch stays MG's call. The −1.2 GB local-only drop since 31-07 is explained by an external prune of SamudraManthanam's local layer measured on disk this pass: nkrya-parallel bulk removed (was ~0.9 GB local-only on 31-07; remainder tracked), web/corpus.db zeroed to 0 B (gitignored generated search DB), web/offline-packs/ emptied to .gitkeep (rebuild recipe: web/scripts/build_offline_pack.py; prod status H2392). Zero-deletion proof for both layers: git diff --name-status origin/main..HEAD shows A/M entries only.

Post-H691 giants — regenerate-vs-backup verdicts

None of the three real giants carries an H691 kosha-manifest row (checked live: 107 manifest rows, zero basename hits) — they are post-H691 arrivals, exactly the gap the 31-07 At-a-glance flagged.

H3180 backup disposition (23-08-2026). All three post-H691 giants now have a weekly restic estate copy (tags local-data-layer,w4, snapshot e47ee898): dta_lingattr.zip and the nagari-2005-2026/ mbox primary are included; segments.jsonl is included as cheap insurance despite its REGENERABLE flag (regeneration needs live fetches). One conflict recorded rather than silently resolved: §2 says archive_stopword.sqlite is "fully regenerable … no backup needed" (H691/H1989 manifest ruling) while this section's table flags it "KEEP · backup-priority (top)" — Wave 4 followed the older regenerable ruling and excluded the 11 GB file, so the estate does not carry it until MG rules which row wins (@DECIDE, logged in docs/BACKUP_W4_LOCAL_DATA_LAYER_RUN_2026-08-23.md). Same run also excluded the dcs_full.sqlite.gz twin (PRUNE-candidate row above),

Full classification table (67 rows)

> ⚠ Verdict update 05-09-2026 (H4177, MG chat vote on 8 prune items): the table below > predates the vote. Canonical verdicts now live in the regenerated > reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.md > (+ .verdicts.json sidecar, same date): items 1–6 APPROVED-DELETE (Windows-side, SHA-proofed; item 1 conditional on the H3389 nonlocal backup — met), item 3 after one builder smoke, > item 7 segments.jsonl LEAVE AS IS, item 8 archive_stopword.sqlite BACKUP IT > (H691-vs-H3316 conflict resolved in favor of backup-priority; restic-include @DO open). > Audit: review/weekly/archive/decisions_applied_2026-09-05_data-noise-prune-vote.md.

[generated-table: DATA_LAYERS_ACCUMULATIONS_2026-08-22 · regen: tools/data_layers_accumulations.py --json data/data_layers_harvest_2026-08-22.json --tsv reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.tsv --max-files 28 --max-depth 4 --verdicts reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.verdicts.json --md-out reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.md]

Method & reproduce

Committed (H1989, 31-07-2026): tools/data_layers_harvest.py — single stdlib walker that sizes large data files, classifies tracked-vs-local via git ls-files, and optionally marks FEATURES_INDEX basename registration.

python tools/data_layers_harvest.py --root C:\Users\user\Documents\GitHub ^
  --json-out data/data_layers_harvest.json ^
  --md-out reports/DATA_LAYERS_HARVEST_SNAPSHOT_YYYY-MM-DD.md

Exclusions (name heuristics, not a hard allow-list): *-h###* worktrees, Uprava-* / SanskritLexicography-* / IndologyScholars-* / SanskritGrammar-* sidecars, temp_*, claude-config-* / codex-config-*, utility clones (Retrieval-based-Voice-Conversion-WebUI, SOCKS5-VPS), plus node_modules / vendor / .git walk-prunes. The 06-07 throwaway pair (harvest.py + classify2.py) is retired — do not re-derive it.

Committed (H3316, 22-08-2026): tools/data_layers_accumulations.py — stdlib classifier over the harvest JSON: keeps local-only + unregistered files, buckets them at adaptive directory granularity (--max-files 28 --max-depth 4), merges an authored verdict sidecar (longest-prefix match), and emits the TSV + classification markdown table of §6. The harvest's snapshot header now takes --model-run (was hardcoded to one lane and mis-stamped provenance).

python tools/data_layers_accumulations.py \
  --json data/data_layers_harvest_2026-08-22.json \
  --tsv reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.tsv \
  --max-files 28 --max-depth 4 \
  --verdicts reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.verdicts.json \
  --md-out reports/DATA_LAYERS_ACCUMULATIONS_2026-08-22.md

Related hubs: FEATURES_INDEX.md (what's registered) · PROJECT_INTERLINKS.md (who feeds whom) · SHARED_CODE.md (the actions from §4) · GAPS §3 → FINDINGS §261.

_Dr. Mārcis Gasūns_

7 · Process-mining event layer (H4394, 08-09-2026)

New owned layer: the hashed per-case business event log produced by tools/pm_extract_events.py from the Systema prod DB (read-only, tunnel path of pull_samskrte_db.py).

Contours and PII classes: docs/PROCESS_MINING_SOURCES_2026-09.md; frozen gate/thresholds: docs/PROCESS_MINING_PREREGISTRATION_2026-09.md (pre-registered 2026-09-08T16:02:08Z, BEFORE first extractor run).

8 · Image layer registration (H5507, 26-09-2026)

9 · PD machine-readable text layer (H6718, 15-10-2026)

The Poona Dictionary's machine-readable text and its committed derivations, registered per the 14-10-2026 PD grill (Q4). Headline numbers follow

> Scope caveat, on every number: the digital text exercises the letter-*a* canon only up > to ~apaca- — the fascicles issued 1976–2009 (vols. 1.1–9.x). The printed record has > since reached Vol. 35 / 6,056 pp by 2019, still inside letter *a*; that post-2009 print > canon is *additional* material the text does not carry.

The data/pd/ directory has since gained three AOB declared-canon TSVs (H6707, 14-10-2026) — pd_aob_declared_works.tsv, pd_aob_fuzzy_short_adjudication.tsv, pd_aob_siglum_title.tsv — plus the H6923 denominator audit (pd_aob_declared_canon_audit.json, 17-10-2026: what 1,419 / 1,118 / 1,464 each measure, band 7.78–8.32 %) and the H6720/H6941 reconciliation (pd_aob_entries.tsv, 1,430 printed entries with their match route and exercise state; pd_aob_sigla_reconciliation.tsv, 2,954 classed discrepancy rows; pd_aob_sigla_reconciliation.json, three presence measures + the 269-of-681 printed-text confirmations), and its README carries the per-file descriptors; the per-file row counts above are from that README.

H6949 (18-10-2026) added the quotation-level unit, a different unit from the sigla every other row here counts: pd_quotations_by_entry.tsv (107,630 rows — {#...#} blocks, quotations, siglum loci, further references, attestations per entry), pd_quotations_vs_pwg.tsv (4,093 rows — the same lemmas in PD and PWG, both measured with the same rule) and pd_quotations_summary.json (totals, top-20, the PWG comparison block, input hashes). The unit itself is defined in reports/PD_QUOTATION_UNIT_2026-10-18.md; the scope caveat above binds these numbers too — they are a floor, not a description.

H7458/C2 (30-10-2026, PR #666) added four committed derived layers: pd_locus_fields.tsv (391,444 explicit pairs), pd_locus_coverage_by_entry.tsv (107,630 rows; 105,177 = 97.72 % with a pair), pd_locus_rejections.tsv (diagnostic classes and examples), and pd_locus_fields_summary.json (protocol, metrics, input hashes and exact reconciliation of all 28,446 non-pair markers). Regenerate all four together with python3.13 scripts/pd_locus_fields.py --pd <pd.txt> and verify with --check; H6949/C1 artifacts stay unchanged. Report: PD_LOCUS_FIELDS_C2_2026-10-29.md.

Generated by Uprava tools/build_public_site.py from an explicit three-hub allowlist under a deny-rail leak guard (E031 wave 4 / H8046, 11-11-2026). Private paths are withheld by construction, never by review after publication.