on
AI-generated Kubernetes manifests: a practical workflow for safety and effectiveness
AI can speed up writing Kubernetes manifests, but speed without safety is like playing a power chord without tuning — loud, fast, and likely to hit a sour note. I picked a recent, practical topic that sits squarely under “Using AI to generate Kubernetes manifests safely and effectively”: how to combine AI-assisted manifest generation with policy-as-code (shift-left checks) and supply‑chain verification (signatures/attestations) to get fast, reproducible, and enforceable deployments. This is timely because policy engines and signature tooling have continued to evolve, and there are recent feature and security updates that change what you should do in practice. (kyverno.io)
What this article covers
- Why you need policy and signing when you use AI to generate manifests
- A compact, practical workflow you can adopt (shift-left -> CI checks -> cluster admission)
- Concrete checks and snippets to get you started
- Where to be careful (real risks and a recent Kyverno CVE to be aware of)
Why policy + signatures matter with AI-generated manifests AI can produce YAML fast, but it can also introduce unsafe defaults (privileged containers, hostPath mounts, wide RBAC rules), non-deterministic values, or metadata lacking provenance. Two systemic mitigations are essential:
-
Policy-as-code (validate before and at admission): Run the same declarative rules in CI and in-cluster admission so manifests are checked consistently. Kubernetes has been moving toward in-cluster CEL-based validating admission primitives and policy engines like Kyverno are adding new policy types to make this easier to adopt. (kubernetes.io)
-
Supply-chain verification (signatures & attestations): Ensure images and important artifacts are signed and that your policies require verification. Sigstore / cosign has become a de-facto standard for signing images and bits of build provenance; Kubernetes docs and many vendors now show workflows for verifying signed artifacts during release and in-cluster. (docs.sigstore.dev)
A compact, practical workflow Think of this like a simple three-part song: Compose (AI + templates), Play (CI checks), and Record (sign & enforce). Below is a pragmatic flow you can adopt.
1) Compose — deterministic AI generation
- Use prompts + templates (Helm, kustomize, Jsonnet) rather than asking the model to “invent” full manifests each time.
- Seed the model with constraints (e.g., “no hostPath, images must be pinned by digest, CPU and memory limits required”).
- Produce machine-readable provenance: include a small annotations block (generator, prompt hash, timestamp, Git commit) so you can trace generated YAML back to CI or a human review.
2) Shift-left checks in CI
- Lint and schema-validate: run kubeval / kube-score / yamllint to catch invalid kinds or mis-typed fields.
- Policy tests: use conftest / OPA (Rego) or Kyverno CLI to run your organization’s policies on generated YAML before merge.
- Example: run Kyverno’s CLI to validate a manifest in CI so it fails the pipeline before merge. Kyverno is also adding policy types that make it simpler to express image verification and other validations. (kyverno.io)
3) Sign and attest build outputs
- Build your images and sign them with cosign (or your chosen PKI/KMS). Store attestations (SBOM, build provenance) alongside images.
- Verify signatures during CI as a gate: if an image is unsigned or the attestation is missing, fail the pipeline.
- Quick verify example:
# verify an image signature with cosign cosign verify registry.example.com/my-app@sha256:<digest> \ --certificate-identity="ci@your-org.example" \ --certificate-oidc-issuer="https://accounts.google.com"This pattern is widely documented in the Sigstore / cosign docs and Kubernetes verification guides. (docs.sigstore.dev)
4) Enforce in-cluster at admission
- Use Kubernetes ValidatingAdmissionPolicy (CEL) or an admission policy engine like Kyverno/Gatekeeper to enforce the same rules you ran in CI. Keeping the same rules both places prevents “CI pass, cluster fail” drift. Kubernetes has native CEL-based validating admission primitives and Kyverno has integrations that can generate admission policies for you. (kubernetes.io)
- Example CEL rule (minimal): deny HostPath in a Deployment spec
```yaml
apiVersion: admissionregistration.k8s.io/v1alpha1
kind: ValidatingAdmissionPolicy
spec:
matchConstraints:
resourceRules:
- apiGroups: [“apps”]
apiVersions: [“v1”]
resources: [“deployments”]
validations:
- expression: “!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(v, !has(v.hostPath))” message: “hostPath volumes are forbidden” ```
5) GitOps + human-in-the-loop review
- Keep generated manifests in Git so they go through PR review; make signatures and provenance visible in the PR.
- Use GitOps tooling (Argo CD / Flux) to apply from a signed Git commit; insist on image digests (not tags) in the manifests your GitOps controller reads.
Where to be careful — actual security incidents and caveats
-
Toolchain bugs matter: a recent Kyverno vulnerability (CVE-2025-29778) affected image signature verification logic in some versions; keep policy engines up to date and subscribe to CVE/advisories for your tools. That CVE shows how a bug in verification logic can allow unexpected artifacts through. (wiz.io)
-
Keyless signing tradeoffs: Sigstore’s keyless (fulcio + rekor) model is convenient, but you must think about identity constraints (certificate identities, OIDC issuers) in verification to avoid accidental acceptance of wrong signatures. Document and pin allowed identities in policy.
-
AI hallucinations: models may invent fields or deprecated API versions. Always run schema and admission checks; prefer CRD/Server-side apply validation where available.
Wrapping up — priorities for the next sprint
- Add policy-as-code checks to the CI stage that runs right after manifest generation.
- Require image signatures + attach attestations; verify them in CI and with in-cluster policies.
- Keep policy engines and signing tools updated; subscribe to advisories.
- Invest five hours in templating: deterministic prompts + templates reduce surprises by a large margin.
If you treat AI as a drafting tool and fold it into a reproducible, policy-driven pipeline, you get both speed and safety. It’s like letting a synth riff inspire a track, but keeping a producer — policy and signing — in the control room to make sure the final cut is polished, auditable, and safe to release.
Further reading / references
- Kyverno release and new policy types (ValidatingPolicy, ImageValidatingPolicy). (kyverno.io)
- Kubernetes ValidatingAdmissionPolicy (CEL-based admission). (kubernetes.io)
- Sigstore / cosign signing documentation and usage. (docs.sigstore.dev)
- Kubernetes guide for verifying signed artifacts and example cosign usage. (kubernetes.io)
- Kyverno CVE and advisory details (CVE-2025-29778). (wiz.io)