Data Auditing and Source Strategy
This document records the strategy as of 2026-09-16. The underlying notes were extended on June 29 to cover name normalization, source sampling, duplicate records, protected-profile membership cleanup, and academic-year inference. Proposed features and criteria below are distinct from completed data corrections.
1. Model Boundary
V0.1 would not add a domain_event table. Real-world facts could already be represented by Membership, PersonalBest, RaceResult, and related business tables. A separate event table would duplicate entry and create synchronization work, while there was no defined requirement yet for a dynamic feed, news events, or a user-facing timeline.
Business facts: Person / Organization / Membership / PersonalBest / RaceResult / Competition...
Data-maintenance process: AuditLog
If an athlete career timeline became necessary, the first approach would be to aggregate existing business facts rather than maintain a second event ledger in advance.
2. AuditLog: Changes to Data, Not Events in the World
AuditLog answers: When, why, by whom, and how was our data changed? It can record a new athlete, profile change, new or corrected personal best, race result, school correction, verification-status change, entity merge or split, source replacement, or bulk import. It is not athlete news, article content, or a general operations log unrelated to business records.
The reason vocabulary was:
| Reason | Meaning |
|---|---|
initial_entry | First entry |
source_update | New data at a source |
source_correction | Correction after a source conflict, using a more authoritative source |
stale_source | Earlier source became outdated |
manual_error | Human entry error |
format_normalization | Format normalization |
translation_update | Translation update |
verification_update | Verification status changed |
entity_merge | Entities merged |
entity_split | Entity split |
import | Bulk import |
system | System action |
3. Explore One Athlete Before Bulk Import
The next step was to examine one athlete in depth, such as Asahi Kuroda, to test the model and page design. The sample should reveal whether basic details, affiliations, school history, personal bests, Hakone appearances, other race results, source conflicts, and display decisions could be expressed adequately. A personal best, for example, needed room for its source, date, race, and career stage.
The proposed order was: inspect one athlete; adjust fields exposed by real records; document source priorities; validate against a few more athletes; design CSV/import scripts; and only then bulk-enter rosters from three schools. This was a proposal to avoid locking an untested model into a complex pipeline.
4. Person-First, Race-Frontier Collection
Collection should not remain an endless deep dive into one person. Starting from a valuable athlete, every discovered race, affiliation, and source should identify adjacent work without requiring immediate completion. For example, entering Kuroda's result on leg 5 of the 102nd Hakone Ekiden also reveals athletes in that race, including Shinsaku Kudo. The collector would create a task to complete all leg-5 results and, separately, a task for Kudo's profile, then choose the next item by value and evidence quality.
This resembles breadth-first exploration, but priorities come from source reliability, page value, and data gaps rather than mechanical traversal.
Task Types and States
At minimum, tasks should cover athlete-profile completion, race or edition completion, source verification and conflict resolution, and correction of existing facts. Completed tasks should retain their state instead of being physically deleted:
| State | Meaning |
|---|---|
pending | Awaiting work |
in_progress | Being handled |
done | Completed |
blocked | Evidence is insufficient to decide |
skipped | Explicitly deferred |
History explains why a record was prioritized, which nodes were checked, and why data remains missing. It could also support a later administrative interface or import process.
Prioritization and Public Display
Priority should consider source authority; visible gaps in an existing page; useful connections through a race leg, university, school, or corporate team; model-boundary tests; and relevance to frequently visited pages. A simple scale was P0 for an immediate page or data dependency, P1 for a valuable adjacent node, P2 for worthwhile but non-urgent work, and P3 for a lead to retain.
The proposed home-page component could show five recent public tasks in pending or in_progress, favoring P0 and P1. It should use visitor-facing language such as “Complete all results for leg 5 of the 102nd Hakone Ekiden” and “Complete Shinsaku Kudo's profile.” Internal notes, unresolved source conflicts, and weak leads should stay private. Once done, an item would leave the home page but remain in history.
A task asks what to complete next and why. AuditLog records what actually changed and why. Creating a task need not create an audit entry; completing it should create one if business data changed. A blocked source search can stay a blocked task without an audit entry. Correcting a stored fact after a source conflict needs both an audit entry and the conclusion on the task.
5. Verification Means Profile Completeness
DataStatus should convey more than whether any record exists. For Person.status, verified meant the athlete's V0.1 core profile met the current completion standard. A reliable source for a single personal best or result does not make the whole profile complete.
| Status | Interpretation |
|---|---|
verified | Entity or fact meets the current version's completeness standard |
pending | Preliminary data exists but core fields lack evidence, rely on weak sources, or await review |
conflicting | Sources disagree and no decision is yet justified |
missing | A needed displayed field or fact is known to be absent |
An athlete should remain pending while important work is open: missing birth date, birthplace, reading, race time, personal-best date or venue, major non-Hakone results, or provenance for displayed facts. Moving a profile to verified required at least: confirmed name, display name, romanization or reading; confirmed current or latest main affiliation; an explainable high-school, university, and corporate-team history where relevant; no unexplained core biographical gaps; sourced core personal bests with date/race/place where possible; key Hakone records with edition, leg, time or explicit missing-time note, leg rank, and source; evidence or an explicit pending marker for important displayed facts; and no major unresolved conflict that would mislead visitors.
A shell profile created from one race result stays pending. Representation of a team in one race is not proof of long-term membership. A personal best from an entry list or media summary needs confirmation. Athletes added while completing leg 5 of the 102nd Hakone Ekiden would usually remain pending initially. Kudo, for example, would remain pending even with an official university personal best and an official Hakone result if birth details, hometown, or earlier Hakone times were still missing.
Temporary Race Teams
Membership holds long-term or career affiliations such as school, university, corporate team, or club. A temporary representative team belongs on the race result only. For example, RaceResult.organizationId = 関東学生連合 should not create a Membership for the athlete: the Kanto Student Union is an event-specific Hakone team, not a long-term affiliation. If the real university is unknown, current affiliation should be empty or pending.
The 100th Hakone Ekiden did not field a 関東学生連合 team. A result attached to that team in that edition was therefore an erroneous import to audit and delete, not merely an uncertain value. Student-union source labels such as 東京大学大学院 or 東大大学院 were normalized to 東京大学 at this stage to complete the university timeline.
Protected Profiles and Old Seed Memberships
An import would not replace all Membership rows on a protected athlete profile, preserving hand-curated facts. This protection could leave estimated early seed-source memberships alongside later sourced records; a prior entity merge could also move old memberships onto the surviving person.
The correction policy was to fix the data itself rather than hide duplicates in the UI: remove seed memberships superseded by real evidence, retain the sourced timeline, write an AuditLog entry with source_correction, and use a dedicated maintenance script rather than manual record-by-record edits.
The first confirmed case was asahi-kuroda. An initial cleanup retained a wrong timeline from a later import; a second correction set Tamano Konan High School to 2019-04-01 ~ 2022-03-31, Aoyama Gakuin University to 2022-04-01 ~ 2026-03-31, and GMO Internet Group to 2026-04-01 ~ present. Removing duplicate protected-profile records and correcting an inferred academic timeline are different repairs. Both need their own audit trail.
Infer Academic Years From Race Dates
When a source gives only the athlete's grade at the race, such as an NTV Hakone leg page, grade is a snapshot. School and university Membership dates must be derived from the academic year containing the race, never from a fixed expression such as 2026 - grade.
Hakone is usually held in January, within the Japanese academic year that began the preceding April and ends that March. Thus a January 2025 race belongs to an academic year ending in 2025; a third-year university runner's timeline would be 2022-04-01 ~ 2026-03-31. An importer should use Race.startsAt, then CompetitionEdition.startsOn, and only then a conservative date based on the edition year.
A repair script should not require the race-result source ID and membership source ID to match. The result may come from official records while the affiliation came from NTV or an early seed. Match on person, race context, and academic-year rule instead of source ID alone.
6. Source Priority
Different websites can report the same fact. The general policy was to prefer official over media sources, current over stale sources, official event results over reporting, and university/team sites over fan sites. Fan sites can suggest leads, but critical facts need independent confirmation. Media profiles can be referenced without copying their article text or images.
| Fact | Suggested source order, highest first |
|---|---|
| Current affiliation and roster | University/team site; official Hakone entry material; NTV Hakone page; media profile such as Monthly Athletics Online; fan/data site |
| Hakone entry and appearance | Hakone official source; Kanto student athletics federation; NTV Hakone page; media; fan/data site |
| Personal best | JAAF/official results; university/team site; official Hakone entry material; NTV Hakone page; media profile; fan/data site |
| Ordinary race result | Event official result; JAAF/regional association/student federation result; university/team recap; media; fan/data site |
| High school | University/team site; official Hakone entry material; NTV Hakone page; official high-school ekiden material; media profile; fan/data site |
Names and Romanization
displayNameJa should preserve the Japanese name with one ASCII space between family and given name. Internally, displayNameRoman uses GivenName FamilyName. The order printed on a source page is evidence, not an automatic slug rule.
Before importing a new source type, sample two or three known athletes, identify whether it commonly writes Family Given or Given Family, and record the conclusion in the source notes or local collection guide. If order varies within one source, import conservatively: reuse known entities first and avoid creating a new person from romanization alone. NTV, entry lists, and profile pages can differ; romanization is a matching aid, not a primary identity key.
Known duplicates caused by earlier name-order mistakes should be treated as duplicate people rather than separate athletes. Choose the surviving slug, migrate linked records, write AuditLog with entity_merge or format_normalization, and confirm that routes, source mappings, and results have moved before deleting the duplicate.
7. Sample Collection Checklist and Next Steps
For Kuroda, the suggested sample covered Japanese name, reading and romanization; birth date and hometown; height and weight where reliable; high school; university, faculty, department and grade; 5,000 m, 10,000 m, and half-marathon personal bests with date, event, place and career stage; Hakone entries and appearances; Izumo and All-Japan University Ekiden appearances; major seasonal results; sources, priority decisions and open questions.
Collection need not wait for a finished page, but bulk collection should wait for real samples to test the model. The near-term actions were to create the sample checklist, record sources and priorities per fact, establish the four task types, consider a public list of five pending updates, add fields exposed by the sample, make data status and provenance clearer on the page, and design CSV import only after the model stabilizes. These are proposed actions except for the specific corrections identified above.