THE BLUEPRINT

DRAFTING…

The Public-Sector AI Deployment Cycle: A From-Scratch Playbook for Zero-Maturity Teams | The Blueprint

When to use this

Use this playbook when your team has zero existing DevOps maturity and needs to build a complete, governed AI deployment pipeline from first principles — no version control, no CI/CD, no containerization, no model registry.

What you'll have at the end

The Cycle Owner completes the full deployment cycle end-to-end for their first AI system, holding: a completed classification intake, a working CI/CD pipeline, a registered and evaluated model, signed-off governance gate documentation, and a live monitoring dashboard.

Stage 0 — Classification Intake and Four-Dimension Branching

Before any tooling is installed, resolve four branching dimensions that determine the team's path through every subsequent stage: data sensitivity (public/sensitive/regulated), deployment track (pilot fast-track/production), model paradigm (traditional ML/GenAI-LLM), and provenance path (in-house trained/externally sourced). This is the Classification Intake — the single decision point that shapes every fork in the Deployment Cycle.

Tips

Complete all four dimensions in a single intake session before touching any tooling — partial classification leads to rework when a later stage discovers a mismatch. Use the AI Risk Classification Tool to resolve the data sensitivity dimension, as its output (public/sensitive/regulated) determines gate strictness at Stages 6, 8, and 10.
Record the four resolved dimensions in a Classification Intake Document and store it in the repository root as `CLASSIFICATION.md` — this becomes the single source of truth for all branching decisions downstream.
If the deployment track is pilot fast-track, explicitly fence it: tag the repository branch as `pilot/*` and document in `CLASSIFICATION.md` that lighter gates apply only to this branch, preventing pilot-grade process from leaking into production deployments.
The four dimensions are independent: a regulated-data GenAI model sourced externally on a pilot track triggers different gates at different stages than a public-data ML model trained in-house for production. Resolve each dimension separately and document the combination.

Common mistakes

Starting Git or Docker setup before classification is complete — teams that skip intake inevitably rebuild pipelines when they discover regulated data or a proprietary license mid-deployment.
Treating pilot and production identically — either over-gating pilots into stagnation or under-gating production into unmanaged risk. The Deployment Track dimension exists precisely to apply proportionate process.
Resolving only data sensitivity and ignoring provenance or paradigm — each of the four dimensions gates different stages, and an unresolved dimension forces an assumption that may be wrong.

Stages 1–3 — Foundational Pipeline: Git, Docker, and CI

Build the foundational pipeline from a blank slate. Stage 1 installs Git and establishes the repository structure. Stage 2 containerizes applications with paradigm-specific Docker configurations. Stage 3 constructs a CI pipeline using self-hosted Jenkins or GitHub Actions. These three stages create the backbone that every subsequent stage depends on.

Tips

Install Git (`sudo apt-get install git` on Ubuntu/Debian) and create a monorepo with this minimum structure: `/src`, `/models`, `/tests`, `/configs`, `/docs`, `/infra` — place `CLASSIFICATION.md` from Stage 0 at the repo root. Initialize with `git init` and make the first commit immediately to establish the baseline.
For traditional ML, use `python:3.11-slim` as the Docker base image and install `scikit-learn`, `pandas`, and `mlflow` as build-stage dependencies. For GenAI, use the same base but add `transformers`, `accelerate`, and a lightweight API server (`uvicorn` with `fastapi`). Create separate Dockerfiles: `Dockerfile.ml` and `Dockerfile.genai`.
For CI, choose self-hosted Jenkins if your team needs on-prem runner support or has air-gapped requirements (`docker run -d -p 8080:8080 jenkins/jenkins:lts`). Choose GitHub Actions with a self-hosted runner if your repository is already on GitHub and you need faster setup. Either way, the CI pipeline runs on every push.
Configure CI to run three jobs on every push: lint (`ruff check .` or `flake8 src/`), unit tests (`pytest tests/ -v --cov=src`), and Docker build verification (`docker build -f Dockerfile.ml .`). Do not add deployment steps yet — CI and CD are separated by design; CD arrives at Stage 9.

Common mistakes

Stage 4 — Provenance Fork: In-House Training vs Pre-Trained Model Intake

The pipeline forks by Provenance Path. The in-house branch integrates the training pipeline with MLflow experiment tracking, logging hyperparameters, metrics, and artifacts for full reproducibility. The pre-trained branch performs intake validation including model card review, licensing checks, and vendor or provider assessment. The fork is determined by the classification intake at Stage 0.

Tips

For the in-house branch: integrate MLflow tracking by wrapping your training loop in `mlflow.start_run()` and logging params with `mlflow.log_params()` and metrics with `mlflow.log_metrics()`. Store the MLflow tracking server URI in an environment variable (`MLFLOW_TRACKING_URI`), not hardcoded in source.
For the pre-trained branch: require a model card or equivalent documentation before intake — if the provider cannot supply one, treat the model as unvetted and escalate to the security stakeholder. Review the model card for intended use, training data description, known limitations, and evaluation methodology.
For pre-trained models: verify the license is compatible with public-sector use before integration. Apache 2.0, MIT, and BSD are safe. GPL may restrict distribution. Proprietary or unspecified licenses require procurement review at Stage 8 and may block deployment entirely.
Document the provenance decision in `docs/PROVENANCE.md`: for in-house, record training dataset sources, hyperparameter ranges, compute infrastructure, and training date; for pre-trained, record provider name, model version, license, intake date, assessment outcome, and model card reference.

Common mistakes

Stage 5 — MLflow Model Registry with Paradigm-Specific Artifact Handling

Set up the MLflow model registry as the single source of truth for what is deployed. For traditional ML, register serialized models with batch scoring configurations. For GenAI, register API-gated inference endpoints with prompt templates and version management. The registry entry must exist before the evaluation gate at Stage 6.

Tips

Start the MLflow tracking server with `mlflow server --backend-store-uri sqlite:///mlflow.db --default-artifact-root ./mlruns --host 0.0.0.0 --port 5000`. For production, replace SQLite with PostgreSQL (`postgresql://user:pass@host:5432/mlflow`) for concurrent access and durability.
For ML: register models with `mlflow.register_model('runs:/<run_id>/model', '<model_name>')` and attach a `batch_scoring_config.yaml` as a registered artifact specifying input schema, batch size, expected latency, and scoring endpoint format. This config is what the evaluation gate and deployment stage consume.
For GenAI: register the API endpoint configuration (provider, model identifier, API version, rate limits) as an MLflow model entry rather than a serialized file. Version-manage prompt templates as separate artifacts with semantic versioning (e.g., `prompt-v1.2.0`) so prompt changes are auditable alongside model changes.
Use MLflow model stages (`None`, `Staging`, `Production`, `Archived`) to track deployment status. Enforce a hard rule in CI: no model can be deployed unless its registry stage is `Production`. Check this programmatically with `client = MlflowClient(); client.get_latest_versions('<model_name>', stages=['Production'])`.

Common mistakes

Stage 6 — Evaluation, Validation, and First Security Sign-Off

The evaluation and validation gate branches by Model Paradigm for different test strategies and risk profiles. This is the first Governance Gate where a security stakeholder must provide formal Security Sign-Off before the model can proceed to infrastructure provisioning. The pilot fast-track path applies a proportionately lighter gate, fenced to prevent leakage into production.

Tips

For ML: run a held-out test set evaluation with task-appropriate metrics (accuracy, F1, RMSE) plus fairness checks across protected attributes if the model makes decisions about people. Log all results to MLflow as artifacts using `mlflow.log_artifact('evaluation_report.json')` so they are traceable to the model version.
For GenAI: run prompt-based evaluation suites covering accuracy, hallucination rate, toxicity, and refusal appropriateness using open-source frameworks like `promptfoo` or `deepeval`. Define a minimum of 20 test prompts per use case and log the full input-output pairs as evaluation artifacts.
For the Security Sign-Off: present the Classification Intake Document, provenance documentation, evaluation results, and the MLflow registry entry to the security stakeholder. Record sign-off in `docs/SECURITY_SIGNOFF_STAGE6.md` with the stakeholder's name, date, and conditions (if any). This file is checked by CI at Stage 9 before deployment can proceed.
For pilot fast-track: the gate requires evaluation results and a brief security review but does not require full fairness audits, procurement diligence, or transparency documentation. Document this lighter gate explicitly in the `pilot/*` branch so CI can verify the correct gate level was applied.

Common mistakes

Stages 7–8 — Infrastructure Provisioning and Pre-Deployment Governance Gate

Stage 7 provisions infrastructure in the hybrid on-prem or cloud environment using basic infrastructure-as-code. Stage 8 is the pre-deployment Governance Gate — a blocking checkpoint covering transparency documentation, auditability artifacts, data stewardship verification, and procurement diligence. Deployment cannot proceed without Security Sign-Off from the designated stakeholder.

Tips

For Stage 7: use Terraform (open-source) to define infrastructure as code. Create separate modules for on-prem resources (`/infra/on-prem/`) and cloud resources (`/infra/cloud/`). Store Terraform state in a backed-up remote location (e.g., an S3 bucket with versioning or an on-prem NFS share), never in the Git repository.
For Stage 8 transparency documentation: produce a public-facing model summary in `docs/MODEL_SUMMARY.md` describing what the model does, its intended use, known limitations, data sources, and contact information for the responsible team. Public-sector accountability requires this to be accessible to citizens and oversight bodies.
For Stage 8 auditability: verify that the MLflow registry entry, provenance documentation, CI pipeline history, evaluation results, and Terraform configuration are all versioned and retrievable. An auditor must be able to trace any deployed model back to its training data, code commit, evaluation results, and infrastructure state.
For Stage 8 procurement diligence: if the model is pre-trained or uses a hosted API, attach the vendor assessment, license verification, and data processing agreement to the governance gate package. Even open-source models with hosted inference endpoints have data processing implications that require procurement review.

Stage 9 — Extending CI to CD for Production Release

Extend the CI pipeline built in Stage 3 to continuous deployment for production release. This stage automates the path from a passed Governance Gate to a live deployment, with explicit rollback procedures. The pilot fast-track path uses the same CD mechanism but targets a staging environment or separate namespace.

Tips

Add a deployment stage to your CI pipeline that triggers only after Stage 8 sign-off is verified. In Jenkins, add a stage that checks `fileExists('docs/SECURITY_SIGNOFF_STAGE8.md')` before proceeding. In GitHub Actions, use a job conditional: `if: hashFiles('docs/SECURITY_SIGNOFF_STAGE8.md') != ''`.
Tag Docker images with the Git commit SHA (e.g., `docker build -t myapp:abc1234 .`) and deploy only SHA-tagged images. Never deploy `latest` tags in production — they make rollback impossible because you cannot determine which version was running when an issue occurred.
Implement a rollback procedure: maintain the previous production Docker image tag in a deployment manifest file (`/configs/deployment_manifest.json`) and define a CI job that can redeploy it with a single manual trigger. Document the rollback procedure in `docs/ROLLBACK.md` and test it at least once before going live.
For pilot fast-track: deploy to a staging environment or a separate Kubernetes namespace (`kubectl apply -n pilot-namespace`) rather than production infrastructure. The CD pipeline configuration is identical but the target is fenced, and the deployment manifest records which environment received the deployment.

Common mistakes

Stage 10 — Post-Deployment Monitoring with Prometheus/Grafana

Establish post-deployment monitoring using Prometheus and Grafana, branching by Model Paradigm. For traditional ML, monitor model drift and data quality. For GenAI, monitor API latency, log prompts and responses, and implement content-safety monitoring. The pilot fast-track path applies lighter monitoring with the same dashboard structure so teams can graduate from pilot to production by deepening, not rebuilding.

Tips

Install Prometheus and Grafana using Docker Compose for simplicity. Define both services in a `docker-compose.monitoring.yml` file with persistent volumes (`prometheus_data`, `grafana_data`) so metrics and dashboards survive container restarts. Expose Prometheus on port 9090 and Grafana on port 3000.
For ML: instrument the scoring endpoint with Prometheus metrics for prediction distribution (`model_prediction_value` histogram), input feature distributions (`model_input_feature` gauge), and request latency (`model_inference_duration_seconds` summary). Configure Grafana alerts for distributional drift using population stability index (PSI) thresholds — alert when PSI exceeds 0.2.
For GenAI: log every prompt and response as structured JSON to a log store with fields for timestamp, prompt text, response text, model version, and latency. Expose Prometheus metrics for API latency (`genai_api_duration_seconds`), token usage (`genai_tokens_used` counter), and error rate (`genai_errors_total` counter). Run responses through an open-source toxicity classifier (e.g., `Detoxify`) and alert when toxicity scores exceed a defined threshold.
Create a single Grafana dashboard per deployment with panels for latency, error rate, and paradigm-specific metrics (drift for ML, content-safety for GenAI). Export the dashboard JSON from Grafana and store it in the repository at `/monitoring/dashboards/` so it is version-controlled, reproducible, and shareable across deployments.

Before you start

Executive sponsor for AI deployment with budget authority
Designated platform or DevOps engineer assigned as Cycle Owner
Security stakeholder with formal sign-off authority identified and available
Identified AI use case with basic data source availability
Access to a deployment environment (on-prem server or cloud tenant)

Templates Included

DOC

Foundational Pipeline Build Reference: Stages 1–5

pro · Read in platform

The Public-Sector AI Deployment Cycle: A From-Scratch Playbook for Zero-Maturity Teams

Stage 0 — Classification Intake and Four-Dimension Branching

Stages 1–3 — Foundational Pipeline: Git, Docker, and CI

Stage 4 — Provenance Fork: In-House Training vs Pre-Trained Model Intake

Stage 5 — MLflow Model Registry with Paradigm-Specific Artifact Handling

Stage 6 — Evaluation, Validation, and First Security Sign-Off

Stages 7–8 — Infrastructure Provisioning and Pre-Deployment Governance Gate

Stage 9 — Extending CI to CD for Production Release

Stage 10 — Post-Deployment Monitoring with Prometheus/Grafana

Company

Account