Home / Projects / Runbook Assistant
Case Study AI · Cloud · DevOps

DevOps Runbook Assistant

An AI assistant that answers IT troubleshooting questions from a team's own runbooks, safely, with sources, on Microsoft Azure.

Runbook Assistant Grounded · 2 sources
Our IIS app pool keeps stopping. What should I check?

Rapid-fail protection is likely stopping the pool after repeated crashes [1]

  1. Check Event Viewer for IDs 5002 / 5011
  2. Review the pool identity and its permissions
  3. Capture a crash dump of w3wp.exe [2]
Risky command: iisreset restarts every site on the server.
Type
In-house product build
Domain
IT operations & DevOps
Platform
Microsoft Azure (UK South)
AI approach
RAG with an agent workflow
Delivery
10 milestones, prototype to production
01 — The challenge

Critical know-how, buried when it's needed most

Operations teams keep their hard-won knowledge in runbooks: how to fix a crashing IIS application pool, an unreachable EC2 instance or a failing deployment pipeline. During an incident, finding the right page and the right step takes time, and generic AI chatbots don't know your environment.

Worse, they can invent commands that sound plausible but are wrong.

02 — Our solution

An assistant that only answers from your runbooks

It finds the most relevant sections, checks that they really answer the question, and replies with step-by-step guidance and the sources it used. If the runbooks don't cover a question, it says so instead of guessing.

Around the AI sits everything a business needs to trust it in production: guardrails, automated quality testing, monitoring and a secure, fully scripted cloud deployment.

Quality you can measure

Proven by tests, not assumed.
0
Automated tests
0
Evaluation questions
0
Guardrail layers
0
Infrastructure as code
How it works

From question to trusted answer

Every question goes through the same four steps, and each step is traced so the team can see exactly why an answer was given.

  1. 01

    Load the runbooks

    A data pipeline validates the runbooks, splits them by section and indexes them for both keyword and meaning-based search. Only changed files are re-processed.

  2. 02

    Understand the question

    Guardrails check the question first. Then the assistant works out the topic (networking, CI/CD, security and more) to narrow the search.

  3. 03

    Find & verify

    Hybrid search finds the best-matching sections. A grader checks they're relevant; if not, the question is rephrased and searched again.

  4. 04

    Answer with sources

    The reply gives clear steps, cites the runbook sections it used and warns about risky commands before anyone runs them.

Key features

Built to be trusted in production

Grounded answers

Answers come only from your documentation, with the sources shown, so engineers can check every step.

Guardrails

Secrets such as passwords and keys are masked, prompt-injection attempts are blocked, harmful content is filtered and usage is rate-limited.

Risky-command warnings

Commands that can cause outages, such as server restarts or deletions, are highlighted with a clear warning.

Self-correcting search

If the first search misses, the assistant rephrases the question and tries again before answering.

Measured quality

An evaluation suite of real and adversarial questions scores every version, so improvements are proven, not assumed.

Reliable data pipeline

Runbooks are validated before indexing, and updates are incremental, idempotent and safe to re-run.

Cloud architecture

Secure, scalable and fully automated on Azure

The whole environment is defined in Terraform and deployed with scripted, reviewable steps. No passwords or API keys live in the code or the container image.

Microsoft Azure · UK South Provisioned with Terraform
UsersSecure HTTPS chat in the browser
Azure Container AppsThe assistant — scales to zero when idle
Container Apps JobLoads and updates the runbooks on demand
Data & AI
Azure AI SearchHybrid keyword + vector search
AI modelsLanguage model and embeddings
Security
Azure Key VaultAPI keys, read via managed identity
Managed identitiesLeast-privilege access
Operations
Container RegistryVersioned images
Log AnalyticsLogs & monitoring
Budget alertsCost control

All of it provisioned with Terraform, with remote state protected by Entra ID.

Technology

The tech stack

AI & orchestration
LangChainLangGraphLangSmithAzure AI Search
Application
PythonStreamlitDocker
Azure platform
Azure Container AppsAzure Container RegistryAzure Key VaultManaged IdentitiesLog Analytics
Delivery & quality
TerraformPytest

Want an AI assistant for your team's knowledge?

Runbooks, policies, product manuals or support articles: we can build a secure assistant on your own documentation and your cloud.