Worked Examples & Case Studies
6 real-world failure scenarios, correct architectural responses, and the exam-style takeaway for each. Read the situation, form your answer, then reveal the analysis.
📋 Situation
A platform team supports 14 services. Each service has a copied pipeline file. Build failures are inconsistent — some pipelines use outdated tasks and different test thresholds. Fixing one pipeline does not fix the others.
🔍 Analysis
The failure pattern indicates duplicated logic drift. Without a shared template, each service evolves independently and diverges from the intended standard. This is the template problem.
✅ Correct Design
- Create shared step/job templates for restore, build, test, and security scan in a common repository.
- Parameterise service path, runtime version, and test filters.
- Enforce shared quality gates and a standard artefact naming convention.
- Pin consuming pipelines to a tagged template version for controlled updates.
💡 Why This Wins
It cuts maintenance overhead, improves consistency, and gives predictable CI behaviour. One fix propagates to all 14 services.
📋 Situation
A canary deployment reaches 20% traffic. Error rate increases from 0.2% to 2.4% and checkout conversion drops measurably. The team is considering whether to continue the rollout to gather more data.
🔍 Analysis
Both technical (error rate) and business (conversion) indicators show degradation. Canary is designed to contain blast radius. Continuing the rollout violates the fundamental purpose of a canary gate.
✅ Correct Action
- Stop promotion immediately — do not increase the canary percentage.
- Trigger automatic rollback to the previous stable version.
- Review release annotations in Application Insights to correlate deployment with metric change.
- Patch the defect and re-run the canary with tighter health gates.
💡 Why This Wins
Canary gates exist to stop bad releases before they reach 100% of users. Expanding the canary when signals are degrading is the worst possible response.
📋 Situation
Auditors ask who approved a high-risk production release and which policy checks passed before it deployed. The team has email chains and Slack messages, but no immutable approval logs. The audit is at risk of failing.
🔍 Analysis
Informal evidence (emails, chat) is not auditable. It can be deleted, forged, or simply unavailable. The compliance failure is a process design failure — evidence was never captured in a durable, machine-readable form.
✅ Correct Design
- Enforce identity-bound approvals in the pipeline using Azure Pipelines approval gates — the approver's identity and timestamp are automatically recorded.
- Configure pre-deployment gates that call ITSM APIs to verify approved change requests.
- Archive gate outcomes with timestamps in an immutable log.
- Link work items, commits, artefacts, and deployment records via Azure DevOps.
💡 Why This Wins
Automated evidence is auditable, reliable, and defensible. It cannot be retroactively altered and is captured every time, not just when someone remembers to document.
📋 Situation
A team moves to CI/CD and ships more frequently. However, incident volume rises, MTTR worsens, and the change failure rate climbs to 28%. The team believes the solution is to reduce deployment frequency.
🔍 Analysis
Delivery speed is not governed by reliability thresholds. Reducing deployment frequency is the wrong instinct — it does not fix the underlying quality problem, it just reduces opportunities to see it. The issue is that releases are not gated on SLO health.
✅ Correct Design
- Define SLIs (e.g., request success rate, p99 latency) and set SLO targets.
- Calculate the error budget per deployment period.
- Tie release promotion to SLI health — if the error budget burn rate is high, freeze risky releases.
- Invest in better test coverage and pre-deploy health checks rather than slowing deployment.
💡 Why This Wins
It balances delivery speed with service reliability. When the error budget is healthy, teams can move fast. When it's degraded, the system automatically constrains risky changes.
📋 Situation
Production differs from test in ways that cause unexpected failures after deployment. Investigation reveals the team has been applying hotfixes directly in the Azure portal to production infrastructure. The IaC repository no longer reflects what is actually running.
🔍 Analysis
IaC is not being enforced as the source of truth. Manual portal changes create state drift. The next pipeline run will either overwrite the manual changes or fail with a conflict, causing further problems.
✅ Correct Design
- Enforce IaC-only changes via PR and pipeline — block direct portal access where possible using Azure Policy or RBAC restrictions.
- Run terraform plan / what-if in the pipeline to detect drift before applying.
- Use drift detection in the CD pipeline to alert on out-of-band changes.
- Conduct an immediate reconciliation to bring the IaC repo in sync with actual state.
💡 Why This Wins
It restores environment consistency and removes the source of surprise failures. Future hotfixes go through the IaC pipeline, maintaining a reviewable, auditable change trail.
📋 Situation
A pipeline log accidentally reveals a database connection string. Investigation shows the secret was stored as a plain pipeline variable and was echo'd during a debug step. The log has been visible to the entire engineering team for three weeks.
🔍 Analysis
Secrets are handled as plain variables rather than runtime secure retrieval. The secret must be treated as compromised immediately — it has been exposed for three weeks. Two problems exist: the current exposure and the underlying architecture that allowed it.
✅ Correct Response (in order)
- 1. Rotate the secret immediately — issue a new credential and revoke the exposed one.
- 2. Audit access logs — check database audit logs for any unexpected connections using the exposed credential.
- 3. Move secrets to Azure Key Vault — link to a variable group or use managed identity for runtime retrieval.
- 4. Enable log masking — ensure secret variables are marked as secret so they are masked in all future logs.
- 5. Remove the debug echo step and audit other pipelines for similar patterns.
💡 Why This Wins
It addresses both the immediate exposure (rotate/revoke) and the structural problem (move to Key Vault). Rotation without architecture change means the next log exposure will repeat the same incident.