Practical lab guide
MicroBank 7: break, recover, and close the lab
Verify a stopped consumer, preserved database data, and scoped infrastructure cleanup.
Docker
PostgreSQL
Predict before you try
Checkpoint: why not rerun --create to check recovery?
Try it: Break only your Ledger consumer →On this page
- Start with a known successful case
- Break only your Ledger consumer
- Interpret the interruption without creating extra work
- Verify persistence before tearing down
- Review cleanup in reverse ownership order
- Publish the learning evidence
- Sources and practice status
- Author verification record
- Floci starter execution record
Start with a known successful case
Complete steps 1–6 and retain evidence/first-case.json. The project milestone is the implemented local API flow, not a production banking system. Keep a short runbook with source SHA, image IDs, Floci/provider versions, startup sequence, probe command, shutdown procedure, and known application gaps.
Break only your Ledger consumer
Stop the Ledger service in this named Compose project, then submit a new synthetic case:
./scripts/study/compose.sh stop ledger
python3 scripts/study/probe.py --create --case-file evidence/recovery-case.json
./scripts/study/compose.sh logs --tail=60 accounts
./scripts/study/compose.sh start ledger
./scripts/study/compose.sh ps
python3 scripts/study/probe.py --verify evidence/recovery-case.json
The create operation should save its case then fail verification while Ledger is unavailable. If case creation itself failed, inspect Accounts before continuing; do not assume a file was written. After Ledger starts and becomes responsive, rerun the read-only verification until the bounded check succeeds or investigate the error. Trace the outbox and queue while it is stopped. Do not receive/delete messages from the live consumer queue just to inspect them: that changes visibility and can interfere with the exercise.
This is a deliberate service interruption, not an automated chaos framework. A healthy process alone does not prove the queued transaction reached the Ledger.
Interpret the interruption without creating extra work
When Ledger is stopped, Accounts may still accept and publish a new request. A failed verification then means the complete result was not observed within the probe's bounds; it does not prove Accounts rejected the write. This is why you retain the created case and verify it after restarting the consumer.
The expected recovery is that Ledger processes the queued event and the saved case's checks pass. If that does not happen, inspect one handoff at a time: Accounts outbox, topic/subscription, queue visibility and policy, consumer logs, then Ledger data. Preserve the case identifier so every observation refers to the same operation. Reading messages through a competing consumer can change visibility, so prefer appropriate queue attributes and logs during diagnosis.
Checkpoint: why not rerun --create to check recovery? It creates another account and deposit rather than checking the interrupted case. Why can a health endpoint pass while verification fails? Process health does not establish consumption or the expected account balance.
Verify persistence before tearing down
./scripts/study/compose.sh restart accounts ledger
./scripts/study/compose.sh ps
python3 scripts/study/probe.py --verify evidence/first-case.json
python3 scripts/study/probe.py --verify evidence/recovery-case.json
Wait for service readiness and repeat the verification after transient startup failures. Both cases should retain their independent account balances and matching entries. Named volumes preserve database files, while SPRING_JPA_HIBERNATE_DDL_AUTO=update avoids the source's local create-drop behavior. This is restart evidence, not a tested backup/restore or a production data-durability guarantee.
For a normal pause, use ./scripts/study/compose.sh stop. On resume, start Floci/databases first, verify or reconcile Terraform, regenerate the Ansible configuration, and then start the services. This pinned Floci profile enables persistent storage. Verify the same case after a restart; a recreated queue alone does not prove message persistence.
Review cleanup in reverse ownership order
Stop application publishers/consumers before destroying their dependencies. Keep Floci running until Terraform finishes:
./scripts/study/compose.sh stop accounts ledger
terraform -chdir=infra/study/terraform plan -destroy -out=cleanup.tfplan
terraform -chdir=infra/study/terraform show cleanup.tfplan
terraform -chdir=infra/study/terraform apply cleanup.tfplan
terraform -chdir=infra/study/terraform state list
./scripts/study/compose.sh down
Inspect the destruction plan before applying it. It should contain only this lab's resources. If you added objects to the practice bucket, remove only the disposable objects you own before destroying it. state list should be empty after successful destruction. Retain the state/lock files until this has been verified.
The normal down command retains the named database and emulator volumes. When deliberately discarding all synthetic lab data, run ./scripts/study/compose.sh down --volumes for this project. This erases its named volumes; it is not part of a persistence or rollback check. Do not use broad Docker pruning. Remove only the lab image tags you no longer need, and keep source and sanitized evidence.
Publish the learning evidence
Extend the MicroBank evidence/runbook fixture from Systems Foundations, or save a sanitized Markdown report:
| Evidence | What it demonstrates |
|---|---|
| Versioned Compose profile and image IDs | Explicit packaging, networking, and runtime inputs |
| Reviewed Terraform plan and no-change follow-up | Resource ownership and change review |
| Ansible second run | Configuration convergence |
| Probe tests and actual local result | Bounded automation and the implemented API contract |
| Consumer interruption and recovery record | Diagnosis through asynchronous dependencies |
| Restart verification and cleanup record | Deliberate lifecycle and data handling |
| CI run and deployment note | Traceability from source to selected artifact |
Keep a visible backlog: backend authorization, input validation, concurrent idempotency constraints, settlement consumption, real notification service, frontend balance integration, database migrations, and stronger automated tests. Record observations instead of marking these features complete. Synthetic personal information, tokens, raw environment files, and state do not belong in the published evidence.
Continue to Advanced DevOps for Kubernetes, GitOps, observability, progressive delivery and broader resilience exercises. Real AWS/EKS/GKE adapters need a separate identity, networking, security and cost design. The foundations implementation stays local.
Sources and practice status
Compose lifecycle commands↗, Terraform destroy planning↗, MicroBank application source↗.
These guides supply implementation files and acceptance steps. Documentation review, static checks, unit tests, and a site build are not substitutes for running the full stack with the chosen Floci image/authentication and recording the results. Keep reviewed status until that execution evidence is available.
Author verification record
On 30 September 2026, the seven probe unit tests and CLI failure cases passed. The extracted Ansible playbook rendered disposable local fixture files, preserved mode 0600, reported changed=0 on its second run, and rejected a non-local queue contract. Compose configuration parsing confirmed the intended six services and loopback publications. Terraform 1.12.2 validated the configuration with the signed AWS provider 6.66.0. Python/Bash snippets passed syntax checks.
That earlier documentation review did not start containers or verify hosted CI/Auth0. The newer downloadable Floci starter is covered by the scoped execution record below.
Floci starter execution record
On 27 October 2026, the pinned Compose teaching starter was executed on macOS arm64 with Linux containers: Floci 2.1.0, Docker 29.8.2, Terraform 1.16.4 / AWS provider 6.66.0, Ansible core 2.21.4 and Python 3.14.7. Resource creation, configuration, two API builds, a sequential repeated-key deposit, same-case restart persistence, and stopped-Ledger recovery were checked. The starter includes the Ledger timestamp-serialization fix found during that run. See setup and limitations.
This evidence covers the local backend flow. Auth0 UI, hosted Actions, Kubernetes adaptation, real AWS authorization, concurrency and production durability require their own execution evidence.
Prove what you learned
Inspect the result
Retain selected source/profile, case identity, expected balance and matching-entry count.
Explain it in your own words
Explain why your observation supports the stated outcome, and name one thing it does not prove.
Try a small variation
Repeat the same check after the documented restart or recovery; preserve the original case instead of creating a replacement.
Your evidence should show
- Record the context/revision or label the work as a paper or prepared fixture.
- Compare the prediction with the actual result from “Break only your Ledger consumer”.
- Retain selected source/profile, case identity, expected balance and matching-entry count.
- Explain the transfer challenge and retain the same verification question.
Hint
Follow the case identifier across the outbox, message route and Ledger result.
Answer guide: compare after predicting
It creates another account and deposit rather than checking the interrupted case. Why can a health endpoint pass while verification fails
Compare this with your actual observation. A paper answer is useful reasoning practice; record execution separately.
Recall private networking, persistent volumes and bounded Python checks. Revisit the previous lesson.
Save the prediction, result and explanation ↓Your notes and evidence
Record observations, questions, or links to your work. Keep credentials out of your notes.
Back up or restore this path
Progress and notes stay in this browser. A backup contains only this learning path.