LAB-002
Automation Rescue
A reference implementation showing how a fragile, already-live automation is audited, diagnosed, repaired, and hardened, with the reliability layer, documentation, and monitoring it should have shipped with the first time.
- Failure modes found
- 6
- Typical count identified during a mid-complexity audit
- Systems touched
- 4
- n8n, CRM, error queue, monitoring dashboard
- Reliability scenarios
- 7
- Each with detection, response, and outcome
- Documentation artifacts
- 3
- Runbook, architecture diagram, credential registry
- Data reconciliation
- Included
- Duplicate and partial record cleanup as part of the rescue
Synthetic environment, not client production metrics. Actual audit findings and repair time vary based on the age, complexity, and documentation state of the existing automation. Metrics here are illustrative of a typical mid-complexity case, not guaranteed outcomes.
Reference implementation using synthetic data only. The modelled company is Vantage Retail Group (fictional). No real client data, client names, or real performance results are presented. All metrics are modelled with stated assumptions. Modelled scenario: Vantage Retail Group has a two-year-old order-sync automation built by a freelancer who is no longer reachable.
What changes after the rescue
Error visibility
A workflow fails silently. No one notices until a customer complains or a report looks wrong weeks later.
Duplicate records
Retried webhooks and repeated triggers quietly create duplicate CRM records over months.
Credentials
API keys are hardcoded directly into workflow nodes, shared across multiple automations with no rotation plan.
Ownership
The person who built it left. No one else understands the logic or feels safe changing it.
Data integrity
Months of accumulated duplicate and partial records make reporting unreliable.
API changes
A vendor changes their API response format. Bad data flows downstream undetected for weeks.
Monitoring
No visibility into whether the automation is even running correctly on any given day.
How the rescue is structured
Eight stages from audit to ongoing monitoring. Click any stage to see what it does and why it is designed that way.
The failure modes a rescue actually fixes
These are the defects most commonly found in a fragile, unmonitored automation, and what the rebuilt version does instead.
Pick a failure and watch the run replay: where it breaks, and how the system recovers or hands over to a person.
Injected failure
Legacy workflow fails with no notification
- Run
- FAIL (failure)
- Error Queue
- Alert
- Ops Review
- Detection
- Audit of historical execution logs reveals runs that errored out with zero alerting configured, the original defect being fixed.
- Response
- Every node in the rebuilt workflow is wrapped with explicit error handling that routes to an error queue and fires an alert.
- Outcome
- A failure is now a visible event with an alert, not a discovery made weeks later during a data reconciliation.
Don't just read the failure modes, run them.
An interactive Shopify → HubSpot sync simulation. Pick a scenario: API outage, duplicate customer, a field edited on both sides, and watch the run log handle it step by step. Fictional data, not connected to a live store or CRM.
Where humans stay in the loop
A rescue involves judgment calls that should never be automated. These are the points where the process deliberately holds for a human decision.
Ambiguous historical data
When reconciling duplicate or partial records, any case where the correct resolution is unclear is held for a human decision rather than guessed.
Credential rotation
Rotating a credential shared across multiple systems requires a scheduled window and sign-off, since it can affect workflows beyond the one being rescued.
Schema-breaking API changes
When a vendor API changes format in a way the workflow cannot safely interpret, data is held for review rather than passed through with a best guess.
Irreversible data merges
Merging two customer or company records is a one-way operation. It requires explicit approval before being applied.
Scope changes mid-rescue
If the audit uncovers a second, unrelated fragile workflow, that is scoped and approved separately rather than silently expanding the engagement.
Reference artifacts
Synthetic representations of what an audit and rebuild produce. All company and person data is fictional.
{
"finding_id": "AUDIT-014",
"severity": "high",
"defect": "no_error_handling",
"node": "Sync Order to CRM",
"impact": "Silent failure, no alert, no retry",
"occurrences_last_90d": 6,
"recommended_fix": "Add retry (3x, backoff) + error queue route"
}{
"node": "HTTP Request",
"name": "Sync Order to CRM",
"parameters": {
"credentialSource": "secretStore",
"idempotencyKey": "={{ $json.order_id }}"
},
"retryOnFail": true,
"maxTries": 3,
"onError": "continueErrorOutput"
}14:02:01 INFO order.received id=ord_5521 src=shopify 14:02:01 INFO idempotency.check result=new 14:02:02 WARN crm.write.timeout attempt=1/3 14:02:04 INFO crm.write.retry attempt=2/3 14:02:05 INFO crm.write.ok contact=hs_44210 14:02:05 INFO order.complete total_time=4.1s
Put your own numbers in
Illustrative model only. Adjust inputs to match your context. Not client data.
Monthly manual hours
8 failures × 75 min ÷ 60
Firefighting hours potentially recovered
85% automation assumption
Monthly labour value
At $40/hr
Estimated annual value
Illustrative only · not client results
A person reads every brief and replies with questions or next steps.