G11: Ethics, security boundaries, and accountable decisions

Prerequisites and observable goals#

The central question: who bears the cost when this system is wrong?#

Correct code can implement an inappropriate policy perfectly. A game moderation model might execute its arithmetic without errors while unfairly excluding legitimate players. A telemetry collector can faithfully record events that should never have been collected. Engineering competence therefore includes deciding what should be built, what evidence permits its use, and who can challenge its decisions.

Describe the proposed decision before discussing the model. “Flag a session for review” is not the same action as “permanently ban an account.” Their reversibility, false-positive costs, explanation requirements, and acceptable evidence differ. List the affected users, operators, data contributors, bystanders, and future maintainers. Include people who never consented to be training examples. No single accuracy score represents all their interests.

The ACM chapter summary highlights avoiding harm, respecting privacy, honest representation, and professional review. The ICLR code additionally provides a research-oriented account of transparent reporting and attention to wider impacts. These are reference points for deliberation, not a substitute for contextual judgment. NIST’s voluntary AI RMF organizes risk work around governance and the mapping, measurement, and management of risks. This chapter uses that practical distinction, without asserting legal compliance or certification. [G-ETHICS][G-ICLR][G-NIST]

Turn a broad concern into a testable risk statement#

A useful statement has a condition, an undesirable effect, an affected party, a detection method, and a mitigation. For example: “When a legitimate player’s network loses packets, the detector may interpret repeated movement corrections as cheating; the player could be excluded; evaluate on authorized synthetic loss traces and reviewed labels; restrict the output to an appealable review flag until error behavior is understood.”

This is more useful than writing “AI bias: high” in a spreadsheet. It connects a mechanism to a consequence. The mitigation must be tested: a review queue that receives more alerts than humans can inspect merely moves the failure. G07’s queueing and admission reasoning applies. A reviewer who sees only the model’s verdict may not provide an independent check. Record what evidence the reviewer can examine and whether the person affected can appeal.

Distinguish intended use, plausible misuse, and accidental misuse. A debugging log can accidentally become a permanent store of private information. An administrative endpoint can be misused if authorization is checked only in the user interface. A legitimate retry can duplicate an action if identity is not enforced at the storage boundary. These are related to, but not interchangeable with, malicious attacks.

Confusion matrices: a count before a percentage#

Let $Y\in\{0,1\}$ be a reviewed target and $\widehat Y=\mathbb 1[p\ge t]$ a threshold decision from a score $p$. Define true positives TP, false positives FP, false negatives FN, and true negatives TN by comparing these two binary values on the same observations. Then

Precision conditions on the decision being positive. Recall conditions on the target being positive. False-positive rate conditions on the target being negative. They cannot be substituted for one another, just as $f(a)$ and $f'(a)$ cannot.

For targets $(1,0,1,0)$ and predictions $(1,1,0,0)$, all four counts equal one. Accuracy, precision, recall, and FPR are each $1/2$, but their equality is accidental: the denominators and interpretations remain different. If there are no actual positives, recall has denominator zero. The reference returns None, not zero or one. A missing denominator is missing evidence, not excellent or terrible performance.

The Brier score is $n^{-1}\sum_i(p_i-y_i)^2$. It uses probabilities rather than just the decisions, and smaller is better on the evaluated observations. It does not establish that probabilities remain calibrated under a changed population. The report retains counts and a threshold, because a rounded rate without its denominator conceals sample size and class balance.

Worked comparison: equal error rates need not mean equal precision#

Suppose group A has 100 positives and 900 negatives, and group B has 500 positives and 500 negatives. These are artificial group labels, not inferred traits. A classifier has recall $.8$ and FPR $.1$ in each group. Its expected counts are

Thus precision is $80/170=8/17$ for A but $400/450=8/9$ for B. Equal conditional error rates do not force equal precision when prevalence differs. The difference follows from changed denominators; it does not by itself tell us which policy is ethically preferable.

Conversely, imposing one equality can change another measure. Which errors matter depends on the decision and its consequences. A model may also use unreliable or historically biased labels, omit relevant populations, or evaluate only people who received an earlier decision. A numerical audit should start questions rather than close them. Describe how labels were obtained and what cannot be observed.

Privacy and security are design constraints, not labels#

Collect the minimum information needed for a specified purpose, limit retention and access, and distinguish identifying information from pseudonymous keys. Hashing a small predictable identifier does not make it anonymous: an attacker can hash all candidates and compare. Even a large random pseudonym can link repeated actions when retained across systems. A digest in GP2 binds bytes to an expected value; it does not authorize collection or authenticate an untrusted publisher.

Authentication establishes an identity under a credential scheme. Authorization decides whether that identity may perform a particular operation on a particular object. Check the latter at the enforcing boundary, not only when drawing a button. Least privilege restricts permissions so a compromised component cannot access everything. Secret management separates credentials from source and logs. This book contains no real credentials and uses local synthetic data.

Threat modeling begins with assets, actors, trust boundaries, entry points, and assumptions. For the scoring project, assets include model integrity and correct job/result association. An untrusted artifact must not be loaded as arbitrary executable objects. A JSON parser with a whitelist reduces one risk but does not make a public endpoint production-safe; size limits, rate limits, authentication, monitoring, and operational review still matter. Use maintained cryptographic libraries for real cryptography rather than designing a classroom cipher.

An accountable decision record#

Record the use case, stakeholder concerns, data sources, alternatives, expected benefits and harms, validation evidence, residual risks, responsible owner, appeal path, and conditions that stop deployment. An acceptable conclusion can be “do not automate this action.” Another can be a reversible, limited pilot with clear monitoring. Do not fabricate a numerical risk score to make the conclusion look objective.

For a small game studio, a reasonable educational boundary is offline synthetic session analysis with no automatic account action. A future real system needs a separate review of collection permissions, user communication, game-specific adversarial behavior, appeals, and release operations. Passing the lab cannot substitute for those decisions.

Independent practice#

F foundation C standard course work T transfer S optional stretch L runnable lab or project. Attempt the complete set before opening its solutions.

Smallest useful hints#

For .01, make a two-by-two count table before dividing. For .03, ask how many candidate inputs exist. For .05, separate being logged in from permission to modify that particular account.

Complete solutions: G11#

Compare the first step where your work differs. A corrected answer is not yet an independent success.

Open every solution

"""G11: descriptive group error audit. No automatic fairness or deployment verdict."""
import math


def binary_report(y, probabilities, threshold=.5):
    if not y or len(y) != len(probabilities) or not 0 <= threshold <= 1:
        raise ValueError("nonempty matching vectors and threshold in [0,1] required")
    if any(type(t) is not int or t not in (0,1) for t in y):
        raise ValueError("strict binary labels required")
    if any(not math.isfinite(p) or not 0 <= p <= 1 for p in probabilities):
        raise ValueError("probabilities must be finite in [0,1]")
    pred = [int(p >= threshold) for p in probabilities]
    tp = sum(t == 1 and v == 1 for t,v in zip(y,pred))
    fp = sum(t == 0 and v == 1 for t,v in zip(y,pred))
    fn = sum(t == 1 and v == 0 for t,v in zip(y,pred))
    tn = len(y)-tp-fp-fn
    ratio = lambda a,b: a/b if b else None
    return {"n":len(y), "tp":tp, "fp":fp, "fn":fn, "tn":tn,
            "precision":ratio(tp,tp+fp), "recall":ratio(tp,tp+fn),
            "fpr":ratio(fp,fp+tn), "selection":(tp+fp)/len(y),
            "accuracy":(tp+tn)/len(y),
            "brier":math.fsum((p-t)**2 for p,t in zip(probabilities,y))/len(y)}


def audit(y, probabilities, groups, threshold=.5):
    whole = binary_report(y, probabilities, threshold)
    if len(groups) != len(y) or any(type(g) is not str or not g for g in groups):
        raise ValueError("one nonempty group label per observation required")
    reports = {}
    for g in sorted(set(groups)):
        idx = [i for i,v in enumerate(groups) if v == g]
        reports[g] = binary_report([y[i] for i in idx],
                                   [probabilities[i] for i in idx],threshold)
    return {"overall":whole, "groups":reports}


if __name__ == "__main__":
    print(audit([1,0,1,0],[.9,.6,.4,.2],["A","A","B","B"]))

Readiness and continuation#

Before advancing, explain who is affected by an error, what a reported rate conditions on, and what would make you refuse deployment. Continue with privacy engineering, participatory design, usable security, accessibility, and domain-specific governance with appropriate specialists. Use G16 to report uncertainty honestly, and revisit this chapter whenever a project changes from prediction to real action.