Detail of Astronomer by Candlelight by Gerrit Dou. An astronomer studies a book and celestial globe by candlelight.

Separate capability from judgment

An intelligent system still needs a clear objective and a way to examine its results.

A system can perform a task well while pursuing the wrong result. Its capability does not decide which outcome the company should value. People remain responsible for the goal and the tradeoffs it contains.

A fluent answer is not proof of truth. Confidence of presentation and correctness are separate questions. Evaluate the result through evidence appropriate to the task.

Automation should extend useful judgment rather than hide its absence. Define the standard, observe the behavior, and change the system when it fails.

Back to the library