Technology, Data, and Delegated Decisions

Technology does not remove moral judgment; it redistributes it. A software team chooses the target. Data collection defines what can be seen. A threshold encodes a tradeoff. An interface nudges a reviewer. Procurement assigns power to a vendor. A complaint process determines whether an error can be corrected. By the time an automated recommendation appears, dozens of human decisions have already shaped it.

The NIST AI Risk Management Framework describes AI risk as socio-technical and organizes ongoing work around GOVERN, MAP, MEASURE, and MANAGE ¹. Its useful practical lesson is broader than AI: assess the entire context of use, not just a component’s laboratory performance.

Begin with legitimacy, not capability

The first question is not “Can we build it?” Ask:

1. What decision or behavior will this system influence?
2. What legitimate purpose does that serve?
3. Who authorized the purpose and under what rules?
4. Is the data use compatible with what people were told and could reasonably expect?
5. Is a less intrusive method adequate?

A face-matching tool might be accurate enough to identify staff entering a secure lab. That does not justify using it to infer emotional state, track lawful association, or monitor every public visitor. Function creep occurs when a system built for one purpose acquires new uses without renewed justification, consent, or safeguards.

The Universal Declaration’s protections for privacy, expression, association, equal protection, and effective remedy provide public boundary questions even when local law differs ². Ask which rights the use can affect, who owes protection, and how a person can obtain a remedy.

TRACE the system

Use TRACE at design, purchase, deployment, and review:

• Task: What exact decision is supported? What is outside scope?
• Rights and relationships: What claims, expectations, dependencies, and power differences are involved?
• Accuracy in context: Which error matters, for whom, at what threshold, and compared with what baseline?
• Control: Who can inspect, override, pause, appeal, and repair?
• Evolution: How will behavior, data, users, and purposes change after deployment?

Task

Avoid inflated labels. “Predict student success” may actually mean “rank applicants by similarity to past students who completed a program.” The narrower description exposes assumptions and prevents the output from acquiring undeserved authority.

Rights and relationships

Consent is not magical permission. A person may click “agree” with no realistic alternative, no comprehension, or no bargaining power. The Belmont Report analyzes informed consent through information, comprehension, and voluntariness ³. Although written for research, that three-part check is a valuable analogy for data practices. Do not claim legal equivalence; use it to ask whether apparent agreement is ethically meaningful.

Accuracy in context

Always write a confusion matrix in ordinary language:

• a false positive flags someone who should not be flagged;
• a false negative misses someone the system was meant to find.

Their costs are different. A fraud screen tuned to catch nearly everything may freeze many legitimate payments. A lenient screen may reduce inconvenience while allowing costly abuse. Overall accuracy can hide poor performance for a smaller group. Compare error rates and consequences across relevant conditions, then ask whether the categories themselves make sense.

Accuracy is necessary only when the task is legitimate. A perfectly accurate tool for an impermissible purpose remains impermissible.

Control

“Human in the loop” is not a safeguard unless the human has information, time, competence, authority to disagree, and a way to record why. If reviewers approve 500 cases an hour, see only the score, or are punished for overrides, the human may be ceremonial. Measure override rates, disagreement reasons, and whether appeals change outcomes.

Assign named owners for five verbs: inspect, interrupt, explain, correct, compensate. If every participant says the vendor or another department owns one of them, accountability has evaporated into the workflow.

Evolution

Deployment changes the world measured by the system. People adapt to scores. Staff stop recording details the tool ignores. High-risk labels may trigger surveillance that generates more recorded incidents, reinforcing the original label. Monitor data drift, outcome drift, use outside scope, and emerging workarounds. Set review and retirement dates before launch.

Use proportional safeguards

The burden of evidence should rise with severity, scale, opacity, and irreversibility. A recommendation for playlist order needs less process than a decision affecting housing, medical access, employment, liberty, or essential benefits. Use four deployment levels:

1. Sandbox: synthetic or historical data; no effect on people.
2. Shadow: compare outputs with current practice; outputs do not decide.
3. Pilot: limited scope, explicit monitoring, easy rollback, supported appeal.
4. Scale: only after predefined evidence and governance thresholds are met.

This is not automatically slow. A staged pilot can reveal failure sooner than a full launch and make learning cheaper.

Define a stop rule: for example, “Pause if severe false positives exceed two in any week, if appeal reversal exceeds 15 percent, if use expands beyond approved departments, or if required monitoring data are missing.” A metric without an action threshold is observation, not governance.

Worked case: prioritizing housing complaints

A city receives more housing complaints than inspectors can visit. A vendor offers a model that ranks complaints by likely code severity. Historic inspections supply the labels.

TRACE reveals several risks. The task should prioritize inspections, not declare a landlord guilty. Historic data reflect where residents knew how to complain, trusted government, had internet access, and where inspectors previously went. A neighborhood with few complaints may be safe or unheard. False negatives can leave families in danger; false positives can burden owners and displace tenants if enforcement is blunt.

A defensible pilot combines the model with random sampling of low-ranked cases, phone and walk-in reporting, multilingual outreach, inspector review with reasons, rapid escalation for explicit hazards, and an appeal/correction channel. Evaluate time to severe-hazard discovery, error by neighborhood and reporting channel, tenant displacement, and inspector disagreement - not just agreement with old labels.

Counterarguments

“Humans are biased too.” Correct. The baseline is not an ideal human. Compare the proposed system with actual practice, including human inconsistency and delay. Automation may improve some outcomes while creating new scale, opacity, and dependence.

“Transparency reveals trade secrets or enables gaming.” Full source-code disclosure is not always necessary. Affected people can still receive the decision’s purpose, principal factors, limitations, review path, and meaningful reason. Independent auditors can inspect more under controlled access.

“No system can be perfectly fair.” Perfection is not the threshold. The obligation is to define acceptable performance and rights protections, test them, disclose residual risk, and stop when the evidence fails.

Practice: write a deployment card

Choose a tool you use or might buy. In one page, fill TRACE; name both false-positive and false-negative harms; state the baseline; select sandbox, shadow, pilot, or scale; assign the five control verbs; and write two stop rules. End with the strongest reason not to deploy. If the only answer is price, repeat the analysis.

Pocket summary

Treat technology as a loop of human choices. TRACE the task, rights and relationships, accuracy in context, control, and evolution. Pilot when harm is uncertain, make appeal real, and decide in advance what evidence will stop the system. Delegation can move a decision; it cannot move away responsibility.