- CRM create/update
- Outbound messaging or notifications
- Code, configuration or deployment changes
- Ticket, record or document mutation
- Agent-to-tool workflows with approvals
- Multi-step workflows with retry/recovery logic
Primary engineering engagement
Decide how much authority this workflow has earned.
The audit answers one operational question: does this action-capable workflow have enough evidence, confirmation and recovery control for the authority it already holds — or the authority you are about to give it?
Object of the audit
We audit the action chain, not the model in isolation.
The risk-bearing object is the workflow that connects a model to data, permissions, tools and an external effect. A model answer may be imperfect without causing an incident; a write-capable action chain can turn ambiguity into a real change.
- Pure text-generation quality review
- General model benchmarking
- Certification requests
- Unbounded production exploitation
- Legal compliance opinions
- Requests that require public disclosure of private systems
Decision model
Four planes. One authority decision.
The workflow should gain authority only when all four planes tell a coherent story. The audit looks for places where capability has moved ahead of proof or recovery control.
Authority plane
What can act, on which object, with whose approval, under which scope, and which transitions must remain impossible.
Evidence plane
What evidence must be present and fresh at decision time: inputs, provenance, approvals, tool state, timestamps and version context.
Effect plane
How the intended external change is independently confirmed instead of inferring success from internal completion or a tool response.
Recovery plane
How uncertainty is contained: fail closed, retry safely, compensate, roll back, resume from a known state or escalate to a human owner.
BitEvo Doctrine in practice
The vocabulary maps directly to audit tests.
These are not slogans layered on top of the method. Each term points to a concrete question the audit must answer.
Authority Budget
Measure the permission surface created by actions, objects, integrations, approvals and retry paths. Then ask whether the evidence chain is strong enough to justify that surface.
Evidence Before Effect
For critical actions, identify the evidence that must be present and fresh before execution. Post-hoc observability is useful, but it is not an authorization gate.
False Green
Test for states that appear healthy or successful while required evidence, freshness or external confirmation has diverged from the decision state.
Failure plan
Test the assumptions that justify authority.
The 10–20 scenarios are agreed during scope. The point is not random breakage; it is to remove one assumption at a time and observe whether the workflow still knows enough to act.
01Duplicate or retry execution
02Stale or missing source evidence
03Ambiguous approval state
04Tool response ≠ external effect
05Version or configuration drift
06Interrupted run and resume
07Authority applied to the wrong object
08Evidence unavailable at decision time
Finding anatomy
A finding is complete only when it changes the owner decision.
Accepted findings are written so another engineer can challenge the assumptions, reproduce the path and see why the evidence supports a specific operational decision.
Trigger
The exact condition that makes the failure path reachable.
Authority
What the workflow was allowed to do, to which object, under whose approval and with which prohibited transitions.
Evidence
What the workflow knew at decision time — including freshness, provenance, approvals and state required to justify action.
Effect
What actually changed outside the orchestration layer, including partial, missing and duplicate effects.
Recovery
What the system does when the action cannot be trusted: stop, retry, compensate, rollback or escalate.
Owner decision
The bounded next action supported by the evidence: expand authority, constrain it, repair the workflow or retest.
Evidence discipline
Severity is not confidence, and confidence is not authority.
Observed consequence, reachability, reproducibility, evidence quality and control leverage are kept distinct. A dramatic theory with weak evidence does not outrank a narrower failure that is repeatable and tied to a real external effect.
Decision package
The output tells an owner what the workflow may do next.
The deliverables are organized around action, not theatre: what authority exists, what evidence supports it, what failed, what must change and what can be retested.
01Executive report
Decision-level summary of the tested workflow, accepted findings, limitations and owner actions.
02Authority / effect map
The bounded action, object, approval, integration and external-effect relationships in the agreed scope.
03Test inventory and scenario results
The agreed scenarios, execution status and evidence-backed result for each tested path.
04Reproducible evidence pack
Minimum evidence needed to reproduce accepted findings and separate observations from assumptions.
05Finding cards with impact, evidence, limitations, and confidence
Decision-relevant findings without collapsing severity, confidence or authority into one score.
06Prioritized repair backlog
Repair actions ordered by decision impact and tied to the evidence and affected authority boundary.
07One retest
Re-run against the same agreed finding criteria after relevant remediation is available.
08Evidence manifest / hashes where applicable
Evidence inventory and integrity identifiers where the engagement artifacts support them.
Real failure → repair → adversarial retest.
Before offering this audit externally, BitEvo ran the method against its own agent/control workflows. The internal self-audit found bounded authority and evidence failures, rejected an early repair whose own authority logic could be bypassed, forced additional repair cycles and retested the resulting controls.
The public case covers authority crossing, contradictory no-effect receipts and caller-controlled effect semantics while keeping customer_case=false, independent_certification=false and production_wide_security_claim=false.
Open internal dogfood caseClaim ceiling: bounded internal evidence only. It does not establish universal no-bypass, provider-wide enforcement, fleet-wide production adoption, certification or absence of unknown defects.
Rules of Engagement
Authorization is part of the evidence chain.
Before testing: written scope, allowed tests, prohibited actions, access approver, data classification, minimum necessary data and safe replay conditions.
Public intake: prepares a scope brief only. It does not authorize testing and must not contain API keys, passwords, tokens, private keys, wallet seeds, production credentials or customer secrets.
Default boundary: staging/test workflow. Production penetration testing is not included by default.
Prepare Primary Audit scope