G16: Research literacy, reproducibility, and independent engineering evidence

Prerequisites and observable goals#

The central question: what would justify this claim?#

A paper, benchmark, or polished repository presents claims supported by a particular combination of reasoning and evidence. Reading productively means reconstructing that support, not collecting impressive conclusions. First identify the object being studied: an algorithm, implementation, model, dataset, physical system, or organization. Then write the central claim with its quantifiers, assumptions, and comparison point.

“Correct for every input satisfying P” calls for an argument covering that domain. “Faster on these seven workloads” is an empirical statement about named workloads and measurement procedures. “Useful for users” may require usability, error-cost, accessibility, or field evidence beyond runtime. These claims should not inherit credibility from one another automatically. A valid proof about an ideal algorithm does not certify a buggy implementation, and a passing implementation test does not prove the theorem.

Use a reading sheet: problem and motivation; objects and notation; assumptions; main result; proof mechanism or experimental design; baseline; limitations; and one possible counterexample or follow-up. Decode an unfamiliar symbol in its immediate definition before searching broad summaries. Reproduce a small derivation or trace from the method section. Reading the abstract alone can locate a topic, but does not establish that you inspected the method or its conditions.

From a broad ambition to a bounded investigation#

For an engine subsystem, a broad question such as “Can this be faster?” needs a specified task and budget. A bounded question is: “For immutable point snapshots and these recorded query distributions, does a grid reduce end-to-end query time relative to a scan while returning exactly the same sorted IDs?” The specification states what cannot change. Workload generators, seeds, input sizes, and output checks become part of the experiment.

The independent variable is the intended method change. Outcomes may include latency, memory, candidate count, and correctness. Threats to interpretation include different result ordering, omitted allocation cost, warmed caches for only one method, changed compiler flags, and timing a smaller workload. Establish the same output contract before timing. Use an uncomplicated reference as an oracle when feasible, while independently testing any predicate shared by both implementations.

A negative result is useful when it narrows a hypothesis: a grid may lose on tiny workloads because setup dominates, or on dense clusters because filtering removes little work. Reporting only the best workload hides where the method is unsuitable. Predeclare representative cases and include at least one workload likely to challenge the claimed advantage. This is not pessimism; it is method selection evidence.

Reproducible computation and repeated scientific evidence#

For this book, computational reproduction means rebuilding and rerunning the stated code and inputs. independent corroboration means testing the claim using new implementation, data, or investigators as appropriate. Terminology varies across fields; state the operation rather than relying on a label alone.

Save source revisions, dependencies, compiler options, operating system, workload parameters, random seeds, commands, outputs, and the timing boundary. A hash helps detect a changed file; it does not prove who produced it or that its content is true. Synthetic fixture generation reduces privacy risk but does not establish fidelity to real use. Different numerical backends or thread schedules can introduce nondeterminism even when a seed is fixed.

A deterministic rerun yielding the same values checks reproducibility of the computation. It does not create a new independent sample from the target population. Repeated timings of one fixed input estimate aspects of runtime variation on that setup, not the distribution of every future workload. Distinguish process repetitions, workload repetitions, seeds, users, and deployment sites as different possible experimental units.

A paired measurement with an explicit estimand#

Suppose old runtimes are $(10,100,20)$ ms and corresponding new runtimes $(9,95,21)$ ms. Define difference as new minus old, obtaining $(-1,-5,1)$ ms. The mean difference is $-5/3$ ms and median is $-1$ ms. Negative values favor the new method under this definition. One workload became slower. The ratio of total times is $130/125=1.04$; the mean of per-workload speedup ratios is a different statistic. Choose the summary based on the operational question instead of selecting the largest-looking number after measuring.

Pairing matters because two measurements on the same input share workload difficulty. The provided lab alternates which method runs first and checks outputs before accepting a timing. Alternation reduces a simple order bias; it does not eliminate thermal changes, contention, frequency scaling, caching, garbage collection, or all drift. A serious performance study may require randomized order, isolated processes, more workload variation, and controlled system settings. Report those limits rather than claiming isolation that was not performed.

The reference uses perf_counter_ns, records nanoseconds, and includes the whole called function but not its input generation. It checks equivalent results outside the timing interval. Its default comparison, Python’s built-in sum against an explicit Python loop on integers, is an instructional harness, not a novel algorithm benchmark. A method’s speedup must not be asserted before reading the actual measurements.

Uncertainty and selection are part of the design#

A confidence interval describes a procedure and its assumptions; it is not generally a posterior probability that a fixed unknown parameter lies in this particular computed interval. A p-value is not $P(H_0\mid\text{data})$. It measures a tail probability under a specified null model and test statistic. Practical importance also needs an effect size, uncertainty, and cost context. A statistically detectable one-microsecond change may not matter to a task dominated by network delay.

If twenty independent true-null tests each have a .05 false-positive probability, the chance of at least one false positive is

Independence is part of this numerical calculation. Correlated searches require different reasoning, but they do not make unlimited result selection harmless. Predeclare the main comparison, report the family of comparisons, use an appropriate multiplicity procedure when inferential claims require it, and label later exploration as exploration. Do not suppress failed configurations or change stopping rules because the current result looks favorable.

A bootstrap interval also depends on the resampling unit. Resampling individual sessions when the independent units are players can underestimate uncertainty. G04 supplies a paired resampling lab and explains its bounds. For a small number of fixed benchmark workloads, the interval may describe a narrow empirical resampling model rather than a broad hardware-performance population.

An evidence-based path from this book to stronger work#

For systems competence, produce a bounded implementation with a resource budget, invariant, fault tests, measurements, and a documented failure mode. For language/compiler work, provide semantics, a differential test oracle, verifier conditions, and counterexamples rejected by the verifier. For ML engineering, preserve the data contract, split units, fitted preprocessing, selection rule, held-out results, and deployment limitations. For research-oriented study, reconstruct a published claim and ask a sharply delimited question whose answer would change understanding, then choose a method that can actually address it.

Advanced scientific, embedded, aerospace, or medical software needs domain assumptions, validation standards, and review beyond this companion. A physically plausible animation is not a validated simulation. A mathematical convergence theorem does not prove a controller safe under unmodeled actuator limits. These boundaries identify further work, not a reason to stop learning.

Independence, collaboration, and an honest portfolio#

Keep the first unaided attempt, not just the final polished artifact. Use AI after recording a sticking point: request a small hint, propose your own repair, then explain the completed method without copying. For code review, ask for counterexamples and missing tests as well as style. Record what was generated, what you changed, what you independently derived, and what remains unverified. Accurate attribution is stronger evidence than presenting a reference implementation as original work. Research ethics guidelines emphasize transparency and responsible reporting. [G-ICLR]

A portfolio document should state the task, constraints, design, central correctness argument, build command, tests, measured workload, results, limitations, and your contribution. Include a short decision log explaining a rejected alternative. In an oral defense, predict a changed input, identify an assumption that could fail, repair one unfamiliar bug, and explain what the tests do not establish. Interview exercises are one assessment format; sustained reproducible work demonstrates other capabilities.

Advance when fresh work shows adequate setup, reasoning, execution, verification, and communication for the next dependency. Do not average away a dangerous gap: excellent code formatting cannot compensate for a missing authorization boundary. Schedule retrieval and transfer on another day, and keep one primary advanced direction instead of starting every track simultaneously.

Independent practice#

F foundation C standard course work T transfer S optional stretch L runnable lab or project. Attempt the complete set before opening its solutions.

Smallest useful hints#

For .01, start with “in these two observations.” For .03, compute the probability of no false positives first. For .04, distinguish the committed effect from the missing reply and later acknowledgment.

Complete solutions: G16#

Compare the first step where your work differs. A corrected answer is not yet an independent success.

Open every solution

"""G16: reproducible measurement records, not a universal performance conclusion."""
import hashlib
import json
import math
import platform
import random
import statistics
import sys
import time


def digest(path):
    h=hashlib.sha256()
    with open(path,"rb") as f:
        for block in iter(lambda:f.read(65536),b""):
            h.update(block)
    return h.hexdigest()


def paired_trials(old,new,workload,repeats=7):
    if type(repeats) is not int or not 2 <= repeats <= 1000:
        raise ValueError("2..1000 pairs required")
    expected=old(workload)
    if new(workload) != expected:
        raise AssertionError("methods disagree before timing")
    records=[]
    for i in range(repeats):
        times={}
        for name,fn in (("old",old),("new",new)) if i%2 == 0 else (("new",new),("old",old)):
            start=time.perf_counter_ns()
            result=fn(workload)
            elapsed=time.perf_counter_ns()-start
            if result != expected:
                raise AssertionError("timed output mismatch")
            times[name]=elapsed
        records.append({"pair":i,"first":"old" if i%2 == 0 else "new",**times})
    differences=[r["new"]-r["old"] for r in records]
    return {"python":sys.version,"platform":platform.platform(),"units":"ns",
            "timing_boundary":"function call only; output check afterward",
            "records":records,"median_paired_difference":statistics.median(differences)}


def manual_sum(data):
    total=0
    for x in data:
        total+=x
    return total


if __name__ == "__main__":
    rng=random.Random(21)
    workload=[rng.randrange(-100,101) for _ in range(10000)]
    print(json.dumps(paired_trials(manual_sum,sum,workload),indent=2))

Readiness and a bounded continuation#

Read a primary paper in one chosen specialization, reproduce one nontrivial derivation or experiment, and write the strongest supported claim and the most important unresolved limitation. This is an unassigned continuation beyond the completed project count. A credible research question grows from the precise gap between those two statements, not from the size of a model or the prestige of an employer.