C14: Software engineering, testing, debugging, and defensible measurement

A. Prerequisites and observable goals#

Level: Degree core and continuous practice; engineering evidence develops across every project.

B. The central question: can another engineer safely change your work?#

A program is not finished when it produces one desired output. Other people need to understand its boundary, reproduce its build, identify failures, and change it without silently violating assumptions. Requirements describe externally observable behavior and constraints. An interface assigns responsibility: which inputs are valid, who owns a resource, what errors mean, and which state can change. An implementation is one way to satisfy that contract.

Cohesion concerns how closely a module’s responsibilities belong together. Coupling concerns dependencies between modules. Hiding database calls inside a sorting function makes tests and failure reasoning unnecessarily entangled. Conversely, splitting every three-line operation into a separate abstraction can make the control flow harder to see. Prefer boundaries aligned with different reasons to change: parsing a file, computing statistics, and formatting output are natural separate concerns in J01.

An architectural decision should name its context, alternatives, choice, and consequences. “Use a queue because queues are scalable” is not an argument. “Use a bounded queue of 128 jobs so producers cannot consume unlimited memory; block interactive producers for at most a deadline, then reject with a recoverable error” is a testable design decision with a stated tradeoff.

C. A test is an experiment with an oracle#

A unit test exercises a small behavior in isolation. An integration test checks components together, such as a transaction against a real temporary database. An end-to-end test exercises a whole user-visible path. Property-based testing generates many inputs and checks a general property, such as sorted output preserving the input multiset. Performance tests measure a defined workload against a baseline or budget. None of these categories replaces a correctness argument.

An oracle determines what counts as correct. For shortest paths on small grids, a simple BFS is a useful independent oracle for A*. Reusing the same flawed heuristic code in both the implementation and the oracle creates correlated mistakes. For an antiderivative, differentiating the candidate is a separate check. For a simulator, a known analytic solution is more informative than whether its animation looks plausible.

Choose ordinary, boundary, invalid, and adversarial cases. Empty input, one element, duplicate keys, exact capacity, missing records, maximum allowed sizes, partial data, and repeated requests often reveal assumptions hidden by typical examples. A regression test should fail on the old defect and pass on the fix. A test that never reaches the faulty branch gives little protection even if it is named after the bug.

D. Debug from evidence, not from edits#

First reproduce the problem with the smallest input that still shows it. Record expected and actual behavior. Form a specific hypothesis about the first incorrect state, then inspect or instrument that boundary. Change one relevant cause, rerun the reproduction, and add a regression test. A debugger’s call stack and variable inspection can reveal state transitions; logs provide evidence across process boundaries. Avoid changing five unrelated things and attributing improvement to the last edit.

Suppose a parser reports a wrong p95. Before rewriting the percentile function, inspect the sorted data, chosen quantile convention, and row filtering. With ten values, nearest-rank p95 uses rank $\lceil0.95(10)\rceil=10$, the maximum. Another interpolation convention may return a different value without either implementation violating its own definition. The defect may be an unstated requirement rather than an off-by-one error.

Use version control to preserve focused changes and inspect differences. A useful commit explains one coherent change and its reason. Code review asks whether the implementation matches its contract, handles failures, and has meaningful tests; it is not merely a search for style violations. Continuous integration runs the documented build and checks in a clean environment. Passing CI is evidence about those checks, not proof that every supported platform or workload is correct.

E. Measure the operation you intend to compare#

A benchmark needs a workload, input distribution and size, implementation revision, environment, build settings, timing boundary, repetitions, and correctness validation. Keep setup outside a timed region only when the real question excludes setup. If an index is rebuilt every query, reporting query time without build time misrepresents that use case.

Use a monotonic elapsed-time clock for durations. Warm-up can reduce one-time effects but should be reported rather than silently discarded until results look good. Alternate candidate order when drift is possible. Preserve samples, not only a best run. Report medians or another justified summary and the spread; paired observations are useful when comparing under similar conditions. A statistically detectable change need not matter operationally, and a noisy null result is not proof of identical performance. Review A04 for uncertainty and effect size.

If a benchmark performs no observable work, an optimizing compiler may remove it. Consume results and verify equivalent outputs. Debug builds and sanitized builds are useful for diagnosis but usually inappropriate as the only performance comparison. Conversely, release tests using assert must keep assertions enabled or use a test framework whose checks remain active. This companion explicitly undefines NDEBUG for its C++ test targets so release tests actually check their assertions.

F. Worked engineering change: invalid latency values#

Initial requirement: compute mean and p95 for a CSV of latency values. Revised requirement: reject nonfinite and negative values and report the source row. A clean design parses into validated records, computes statistics from that trusted representation, and formats one JSON result. Parsing errors produce a nonzero exit code and a diagnostic on standard error; successful machine-readable JSON stays on standard output.

Acceptance criteria include a finite positive fixture, zero latency, a malformed number, NaN, a negative value, an empty file, and an incorrect header. The computation tests use explicit small lists independent of CSV. Integration tests invoke the command line and inspect exit status and output channels. Refactoring the parser should not change the statistics API, while changing the percentile convention requires a deliberate output contract update.

Observability adds logs for events, metrics for aggregated quantities, and traces for linked spans across components. These have different costs and privacy implications. Prefer recording a row number or opaque request identifier over copying sensitive input. Dependencies should be pinned or constrained deliberately, scanned and updated through a documented process, and reviewed for licensing and maintenance risk. A lockfile or successful import is not a security audit.

G. Independent practice#

F foundation C standard course work T transfer S optional stretch L runnable lab or project. Attempt the complete set before opening its solutions.

H. Complete solutions: C14#

Compare the first step where your work differs. A corrected answer is not yet an independent success.

Open every solution

"""C14.05: monotonic-clock measurements with explicit repetitions."""
import math
import statistics
import time
from collections.abc import Callable

def measure(operation: Callable[[], object], repeats: int = 9) -> dict:
    if type(repeats) is not int or repeats < 3:
        raise ValueError("at least three repetitions required")
    operation()  # warm-up, deliberately excluded from measurements
    samples = []
    last = None
    for _ in range(repeats):
        start = time.perf_counter_ns()
        last = operation()
        samples.append(time.perf_counter_ns() - start)
    return {"unit": "ns", "samples": samples,
            "median": statistics.median(samples), "last_result": last}

def nearest_rank(values: list[float], p: float) -> float:
    if not values or not 0 < p <= 1:
        raise ValueError("nonempty values and 0 < p <= 1 required")
    if any(not math.isfinite(x) for x in values):
        raise ValueError("finite values required")
    ordered = sorted(values)
    return ordered[math.ceil(p * len(ordered)) - 1]

if __name__ == "__main__":
    print(measure(lambda: sum(range(10000))))

I. Retention and readiness#

Retrieve A04 paired comparisons, C02 invariants, and C10 race reasoning. Advance when another person can reproduce a failure and a benchmark from your written instructions. Apply this module’s reporting structure to J01–J05 rather than treating documentation as a final decorative step.