Key takeaways
- The visible screen and happy path are a small part of an internal tool's work.
- Every integration creates dependencies, permissions, failure modes, monitoring, and change responsibilities.
- Exceptions and approvals should be designed before launch, not discovered as support tickets.
- Maintenance includes detecting drift, reconciling effects, testing repairs, communicating change, and deciding when to retire.
- A lower build threshold only helps when the ownership threshold falls with it.
Map the visible tip and four submerged layers
The system below the interface
Define
Trigger · Data model · Rules · Finish state
Control
Identity · Permissions · Approvals · Secrets
Operate
Exceptions · Logs · Retries · Fallback
Own
Monitoring · API drift · Repair · Retirement
A credible build plan assigns every submerged responsibility to a person, system, or review gate. If one is unnamed, it has not disappeared; it has become unowned work.
The joke in xkcd's “Automation” works because writing code becomes debugging, rethinking, and ongoing development while the original task remains. The deeper issue is not developer optimism alone. The screen was scoped; the operating system around it was not.
An internal tool becomes affordable when these layers can be reused or managed, not when they are ignored.
Layer one: define the work
| Responsibility | Question |
|---|---|
| Trigger | What creates exactly one item, and how are duplicates recognized? |
| Data model | Which facts, sources, versions, and states travel with it? |
| Rules and judgment | What is settled, what varies legitimately, and what is policy? |
| Finish state | What observable business outcome closes the work? |
| Examples | Which normal, boundary, and exception cases prove the specification? |
Layer two: control authority and data
- Authenticate people and services; authorize records, fields, functions, and actions separately.
- Use least privilege, separate environments, controlled secrets, and explicit revocation.
- Put human approval before consequential or irreversible effects and show the evidence needed to decide.
- Define retention, deletion, export, masking, and prohibited data in prompts, logs, and notifications.
Layer three: operate beyond the happy path
Google's monitoring guidance makes the basic point that without monitoring you cannot tell whether a service is working. For a business workflow, “the function returned” is not enough. Observe whether the expected business effect occurred once and the item reached a valid finish state.
| Failure | Required path |
|---|---|
| Missing or invalid input | Stop before effect; identify the missing fact and owner |
| Transient dependency error | Bounded retry with backoff, budget, and exhausted path |
| Timeout after possible effect | Reconcile destination state by business key before retry |
| Unhandled exception | Create visible, owned work with evidence and a clock |
| Harmful effect | Contain, notify, preserve evidence, and use authorized rollback or compensation |
Layer four: own change, repair, and retirement
- 01Detect driftTriggerWatch failure patterns, dependency notices, policy changes, access changes, and user corrections.
- 02Classify the changeHuman approvalDecide whether implementation broke, a dependency changed, or the real process needs relearning.
- 03Patch or relearnAI judgmentPropose a versioned change or return to supervised examples; never hide changed behavior inside a failed run.
- 04Regression-check and approveHuman approvalReplay known cases, review changed authority and data use, then confirm release.
- 05Retire deliberatelyHuman approvalExport required data, preserve evidence, revoke credentials, remove schedules, tell users, and close ownership.
Bound the iceberg instead of pretending it is not there
| Concern | Bounded first version |
|---|---|
| Users | One team and named roles |
| Data | Minimum fields and one source of truth |
| Actions | Drafts or reversible internal updates |
| Exceptions | Three common routes and a general owned queue |
| Availability | Documented manual fallback instead of high-availability architecture |
| Change | One owner, versioned release, small regression set, and review date |
Limitations and when not to use this
- The iceberg is an inventory, not a requirement to build enterprise-scale infrastructure for every small tool.
- Managing a responsibility through a provider does not remove your organization's policy, data, approval, and outcome accountability.
- This page does not estimate a specific project's cost or claim maintenance is automatic or included in a commercial plan.
Sources
- Automation — xkcd Accessed 14 August 2026
- Eliminating Toil — Google, The Site Reliability Workbook Accessed 14 August 2026
- Monitoring Distributed Systems — Google Site Reliability Engineering Accessed 14 August 2026
Run the pre-mortem
Take one proposed tool and name its likely process, data, permission, exception, reliability, and ownership failure before launch.
Run the pre-mortemUli Prantz
Builds and operates all-agents
Uli Prantz builds all-agents, the process-automation platform this site documents. He writes about the operational side of automating recurring business work: where deterministic code beats model judgment, where it does not, and where a human still has to approve.