G09: Systems analysis, contracts, and architecture decisions

Prerequisites and observable goals#

The central question: what behavior are we designing?#

“Build a scalable AI service” is not a specification. It does not say who sends inputs, which schema is allowed, how long results may take, what happens to duplicates, or whether stale output is acceptable. Start with actors, objects, invariants, and failure semantics. Technology selection follows those constraints rather than replacing them.

Our running system accepts a small two-feature scoring job and records one result associated with an immutable model artifact. The educational version uses local synthetic data. A game developer can recognize telemetry-like feature names, but the contract is equally applicable to a scientific batch tool or an internal analysis workflow. It is not a real anti-cheat or safety-critical decision system.

Functional and quality requirements#

A functional requirement describes a behavior: “Submitting a valid new job ID stores a pending job and its delivery record together.” A quality requirement constrains how behavior is delivered: “Pending work is bounded by configured storage policy” or “Results can be reproduced from the recorded model hash and feature order.” A vague adjective such as “fast” should be replaced by a workload, metric, threshold, and measurement method before it becomes an acceptance criterion.

Write nonrequirements too. The reference does not provide external authentication, public internet serving, consensus, GPU inference, multi-tenant quotas, model retraining in production, or secure handling of real personal data. Stating these boundaries prevents a correct classroom implementation from being mistaken for a deployable business system.

For each requirement, define a failure observation. An invalid shape should produce a documented rejection before storing a job. A duplicate ID with a changed payload should be a conflict, not a silent overwrite. A worker crash should leave a recoverable pending delivery rather than an apparently completed job with no result.

A state machine reveals illegal transitions#

Define job states: absent, pending, recorded-at-consumer, acknowledged-at-producer. The middle state can exist after a crash even when the producer does not yet know about the consumer record. This is why a single distributed Boolean “done” is often insufficient.

The diagram describes logical states spanning two stores. It does not claim one atomic transaction across them. A retry can observe the consumer record and then complete the final acknowledgement. The contract should reject a transition that changes the model or payload under the same job identity.

Interfaces should carry the needed evidence#

A job record needs its identity, canonical payload, payload digest, model identity, and delivery status. A model artifact needs its schema version, feature order, preprocessing parameters, learned weights, decision threshold, and training provenance. A result needs the job identity, probability, decision, and model digest. A probability alone is not enough to reproduce which model produced it.

Canonical JSON in this project gives stable bytes for hashing by sorting keys and avoiding NaN. A cryptographic digest detects changes relative to a trusted expected digest; it does not authenticate the publisher by itself. A malicious actor can modify both content and an untrusted digest. Authentication and trust distribution are separate future deployment requirements.

An interface contract should distinguish syntax validation, semantic validation, and resource bounds. A two-number array is syntactically simple, but nonfinite values, swapped feature order, and extreme magnitudes are semantic hazards. Depth, length, and batch limits prevent valid-looking inputs from demanding unbounded work. Python JSON parsing of a small explicit format is safer for this classroom artifact than loading arbitrary executable object graphs, but input parsing alone does not make an exposed service secure.

Decomposition by reasons to change#

Separate data generation, preprocessing, model training, artifact validation, scoring, and delivery. Training can change while the artifact contract stays stable. Transport can change while the scoring function remains deterministic. A database migration can alter storage representation without altering the mathematical prediction. These are interface boundaries with testable contracts, not an argument for making every function a network microservice.

For this small project, a few modules in one repository are simpler than separate services. More deployment units create more partial failures, version coordination, observability needs, and operational overhead. A monolith is not automatically poorly modular; a microservice system is not automatically well separated. Choose distribution only when an actual scaling, isolation, ownership, or deployment requirement justifies it.

An architecture decision record, worked#

Decision. Use a local transactional outbox and consumer deduplication for educational reliable scoring.

Context. A producer and worker may stop between writes. Repeating a request must not create another recorded result. We need a reproducible local test with no paid services.

Alternatives. An in-memory queue is simpler but loses pending work when the process exits. One shared database transaction is simpler if both producer and consumer effects truly belong to one store, but hides the cross-boundary retry problem being studied. A managed queue is operationally realistic in some settings but adds credentials and service-specific delivery assumptions beyond the lab.

Consequences. The outbox makes pending work explicit and testable. It requires acknowledgement cleanup, poison-message policy, idempotency retention rules, schema/version compatibility, and bounded storage. The reference demonstrates core atomicity and redelivery but leaves those production policies as extensions.

Evidence and reversal condition. Inject a crash after consumer commit and before producer acknowledgement, reopen the databases, retry, and verify one recorded effect. Reconsider the architecture if throughput, multi-writer contention, data-retention rules, or external side effects invalidate the small local design.

Failure analysis and observability#

A failure-mode table should connect a fault to an invariant, a detection method, and a recovery action. For corrupt payload bytes: hash/schema checks reject before effect; record a permanent error and investigate rather than retry forever. For a busy database: bounded timeout returns a transient error; retry under a limit. For a missing model version: do not substitute the newest model silently; reject or wait for the exact artifact according to policy. For consumer commit followed by worker death: detect the pending producer record and retry safely.

Logs explain individual events; metrics summarize populations; traces connect operations across components. None substitutes for correctness. Attach correlation IDs that reveal state transitions without recording raw sensitive payloads. A dashboard showing zero exceptions does not prove no jobs vanished; measure accepted, pending, applied, rejected, and acknowledged counts with compatible definitions. Reconciliation checks can detect gaps that individual request logs miss.

Independent practice#

F foundation C standard course work T transfer S optional stretch L runnable lab or project. Attempt the complete set before opening its solutions.

Smallest useful hints#

For .01, a testable target can be proposed without claiming it was achieved. For .03, the same input need not give the same result under different model parameters. For .06, a missing invariant is not offset by a high score in an unrelated category.

Complete solutions: G09#

Compare the first step where your work differs. A corrected answer is not yet an independent success.

Open every solution

Readiness and continuation#

You are ready for a larger design review when you can produce a counterexample to your own proposal, explain one rejected alternative fairly, and connect every critical requirement to evidence. Apply these artifacts to GP2. Do not modify a production system as part of this exercise; use a separate synthetic repository and a design review before any future integration.