All articles

Tasuki Keifu: Data Governance Agent Technical Design

Personal projectsRead in Chinese

Data Governance Agent Technical Design

Updated November 18, 2026 against current source and the PRD. This replaces historical build plans. Evolution belongs in the optimization history. Contract: event-governance/1; rule constant: 2026-11-13.2; normalization policy: whole-group-cross-check/8. Constants do not capture every later patch; also record the actual commit/image.

1. Ownership

The main site owns Prisma, migrations, business pages and cache invalidation. The Agent repository owns the governance-v2 CLI, runner, Graph, research adapter and sole controlled BusinessWriter. The personal site hosts articles and its current read-only observation entry. Legacy diagnostics and batch imports are not the V2 pipeline.

Prepare Competition, Edition, Race and Source deterministically before submission. Input does not require manually resolved person IDs. Do not invoke legacy importRacePayload for another identity decision. LLM, web and SMTP calls stay outside business transactions.

2. Intake and persistence

A strict Zod manifest contains version, requestKey, eventEditionId, mode (dry_run/apply), agentMode (full/membership_only) and batches. Identical keys/input reuse a task; changed input under the same key is rejected. Apply after dry-run uses a new key.

Each batch contains unitType (leg/heat/group), unitKey, unique nonnegative sequence, sourceId, raceSlug, organizations, rows, optional teams/sourcePayload. Rows may provide names, kana, romanization, birth date, organizationKey, Membership hints, marks/ranks, grade, notes, hometown, PBs and source identifiers/URLs. Missing facts are valid; duplicate row/organization keys and unknown references are rejected, never silently filtered.

sourceKey represents a real stable identifier, not a name or unstable row number. Default person observation keys combine raceSlug and row key. Adapt converts legacy payloads and preserves originals without calling their writer.

Store hash-addressed gzip payloads on a persistent absolute directory; SQL retains references. Reads validate hashes, paths/symlinks and a 64 MiB decompression cap. Checkpoints live in governance_checkpoint. Dry-run persists plans only; later batches do not see earlier simulated business writes.

3. Models

RaceResult.organizationNameSnapshot preserves the source label separately from organizationId. unknown/club/company remain accepted input labels but cannot create formal Organizations or Memberships; business retirement does not mean enum values vanished from the schema. Regional selections/organizers do not imply Membership. Only junior/high school, university and confirmed corporate_team representation generates implicit Membership observations.

Membership dates carry source/inferred/unknown basis, unknown/ongoing/ended endState, dateEvidence and optional employment/team_registration exclusivityGroup. An unverified old date is not source evidence; a missing end is not ongoing. Business entities do not store Agent execution state.

Governance tableResponsibility
EventImportTaskIdempotency, input hash, mode, summary, lease and lifecycle
GovernanceBatchUnit/order, payload reference/hash, row count and summary
BatchAttemptAttempt number, graph thread, input fingerprint, metrics/errors
AgentExecutionThree responsibility groups under an attempt
GovernanceFindingSubject, severity, resolution, reason and evidence
GovernanceChangeControlled operation, target/version, idempotency, patch/evidence, audit reference
NotificationDeliveryEvent/batch notification, deduplication, lease and retries

Governance UUIDs are distinct from business foreign keys. SQL migrations enforce lifecycle and ownership constraints. Finding status is not business verification status.

4. Graph

load_batch_context
 → organization_rules → organization_light_batch
 → person_rules → person_light_batch
 → prepare_cross_check → normalization_cross_check → validate_cross_check
 → organization_adjudication → organization_network_research
 → affected_person_reassessment → normalization_deep_research
 → finalize_normalization
 → govern_membership (conditional) → verify_person_facts (conditional)
 → aggregate_governance_result
 → runner persists plan → BusinessWriter commits

No back edges: one cross-check, at most one organization adjudication and one affected-person reassessment. Nodes can perform no calls. Person and organization conclusions cannot circularly prove one another.

Identity routing

Reuse source mappings, canonical names, shortName and NameVariant first, with NFKC, whitespace and school-form normalization. Alias prefiltering and node normalization are not completely unified; alias types do not have a full weighting model. Do not claim exhaustive alias recall.

exact_name_kana_historical_organization requires a unique exact written-name/kana candidate, a sourced earlier-or-same-date record from another event, identical canonical organization and no academic/competing-affiliation contradiction.

unique_person_under_unconfirmed_organization requires no confirmed organization mapping, input type unknown/club/company, exactly one row under that organization key, exactly one recalled same-normalized-name candidate and no known birth-date conflict. It reuses the person without creating an organization/Membership, not all same-name identities indiscriminately.

Remaining organization/person items each receive one whole-group non-web lightweight request. personAssessments explains each candidate's nameRelation and samePersonPlausibility and must cover allowedIds exactly. Kana is a recall signal, not merge proof; transcription errors need explanations, not a matching-character count. A match citing only the current row without independent target evidence or an explicit equal birthday is downgraded per item while valid peers continue.

Cross-check uses shared deduplicated catalogs, all source rows and organization overview, relevant evidence and truncation notices. personKeys must be current row keys, not candidate/provisional IDs. Unknown references, foreign targets and malformed structures fail validation.

Unresolved serious organizations receive single-object web verification limited to match/distinct/unresolved. Affected people are reassessed once; medium/high plausible identity competition or substantive conflict can enter individual research. Candidate count alone does not trigger web search. Current DNS rows skip person lightweight/heavy research, using safe rules or independent source people; historical DNS can support registration identity.

Membership and facts

Confirmed allowed organization types create implicit hints when absent: schools enrolled, corporate teams affiliated. Unknown dates are valid, including DNS records. Reuse unchanged relationships and preserve distinct periods. Reliable grade/date anchors map to Japan's April school year; infer known high-school/university windows in either direction, fill blanks and let sourced dates correct inference.

Unexplained exclusive/time conflicts may call membership_conflict research. API/decode errors remain technical failures. Validate all suggested date repairs before applying any of them.

Facts run only for supplied hometown/PBs and currently skip DNS and unresolved identities. Require comparable marks and source-quality thresholds; daily PB improvement does not revoke erroneous PBs or replace better marks with slower ones.

Identity-unresolved rows currently defer Membership/facts while retaining people/results. Report this limitation and arrange follow-up after identity resolution rather than claiming complete downstream governance.

Membership-only

membership_only restores identities from source mappings, unique normalized names and current race-result person mappings, using confirmed business organization types. It builds only Membership plans and necessary conflict research, without repeating normalization or race writes. Missing identities/organizations create membership_unresolved rather than nonexistent person IDs.

5. Model protocol, budgets and diagnostics

Configure model/route explicitly, independently of the current Codex session. The existing route uses pro gpt-6.1-sol; equal model names under different account groups are not equivalent. The adapter uses /responses, message-array input, streaming and structural JSON schema, followed by full local Zod/scope checks. Only a completed successful response can authorize a decision. Confirmed web conclusions require actual web_search output and traceable URLs/quotes.

Default heavy budget is three logical requests per batch. Non-web normalization/cross-check/adjudication share five. Six request kinds have at most three attempts, with separate persistent retry reservations; recovery does not reset them. Retry transient network/timeout, 408/409/429/5xx, incomplete responses and repairable output errors. Do not blindly retry normal unresolved, refusals, missing configuration/budget, oversized input or 400/401/403.

GOVERNANCE_RESEARCH_TIMEOUT_MS is application timeout, default 25000 ms; deployed overrides must be inspected rather than inferred from examples. Production tuning raised a request timeout to 240 seconds; verify actual environment/image on deployment. General research input cap is 24000 characters, normalization/adjudication 512000, response reading 4 MiB. No verified per-request hard token or monetary cap is claimed.

Physical request logs include callId/X-Client-Request-Id, stage, start, attempt, timeout, input UTF-8 bytes/hash, HTTP status, upstream/response IDs when available, error phase, usage and exhaustion. A gateway may not support lookup by client ID. Limit HTTP error reads to 32 KiB and persist sanitized bounded summaries. Distinguish failed/incomplete/missing-completion SSE, network, JSON/schema and scope errors. Never expose full prompts, response bodies or credentials; unknown failed-request usage is not zero cost.

6. Transactions and recovery

Persist Plan before applying. A separate-privilege writer validates lease fencing, target versions and controlled operations, then commits entities, source mappings, AuditLog, applied changes and cache invalidation in one batch transaction. Merges/splits migrate scoped relations and retain audits. LLMs cannot issue arbitrary SQL or field patches.

Tasks/batches: queued/running/completed/completed_with_unresolved/failed; partial is only an extension slot, not half-commit permission. Attempts: running/succeeded/failed/superseded. Changes: planned/applied/superseded. Apply also requires the write gate; dry-run never writes business records.

Technical failure stops later batches but preserves committed predecessors. Retry resumes unfinished work, reuses completed checkpoint plans or replans stale context. Post-commit crashes finish bookkeeping without duplicating results/Memberships. Fencing blocks stale workers. Track LLM retries separately from batch retries.

7. Reporting and mail

importedRows counts committed apply results; plannedRows includes dry-run. repairs counts applied repair operations rather than unique people. Do not double-count writes on retries; do accumulate all physical model attempts and tokens. Unconfigured prices are unknown, and estimates do not cover unconfigured search costs.

Only serious unresolved or technical failures queue batch alerts; terminal event states queue summaries. Titles identify the event and relevant leg/heat, and failed runs retain stage/call statistics. Delivery uses stable deduplication, leases and backoff, independent of business commits. SMTP is not end-to-end exactly once.

Authenticated read-only observation fails closed when configuration is missing. Bounded queries expose readable states, stages, attempts, metrics, sources and minimal diffs; truncated lists are not whole-event totals. Do not expose payloads, prompts, checkpoints or secrets. Further UI changes are maintained separately.

8. Deployment and operations

Deploy main-site migrations/grants and checkpoint initialization before Agent/observer updates. Runtime connections use the main business database, not the legacy Agent database. Protect environment configuration and credentials outside Git/articles.

Use the existing Compose payload volume, business database network, health checks and memory safeguards. Verify running image/configuration and service health after deployment; a pushed commit alone is not deployment proof. Publishing documentation does not submit an event.

The governance-v2 CLI supports adapt, submit, run, status, retry, worker, work-once, deliver, mail-worker, setup-checkpoints and cleanup-checkpoints. Use full for ordinary imports and membership_only for relationship repair. Consult repository runbooks for commands; their historical narratives do not replace live checks.

Default checkpoint cleanup removes successful/superseded attempts seven days after task completion and retains failed recovery points. Raw inputs, governance audits and notifications lack a complete automated retention policy. Stopping workers does not undo committed data. Application rollback retains additive schemas; data corrections use audited compensation rather than an unaudited one-click rollback.

9. Verification and limits

V2 has run production incremental imports and Membership reruns. Final unresolved=0 for the reviewed 333rd task includes manual decisions and corrections, not exclusively automatic Graph results.

Validate changes through type checks, rule/research/name regression and isolated PostgreSQL integration tests. Report checks actually run instead of reusing old test counts. Never inject faults against production.

Known limits include incomplete alias harmonization, deferred downstream work for unresolved identities, incomplete retention/cost coverage and uncertain public evidence/model judgments. Record each new event's inputs, versions, decisions and failures; one manually closed event does not prove general accuracy.

Tasuki Keifu: Data Governance Agent Technical Design | Xuhuan / Gelatoni