Linux+ Cloud and Virtualization: KVM/libvirt, cloud-init, and Hybrid Operation
- Describe the KVM + libvirt stack and manage VMs with virsh and virt-install
- Decide when a workload wants a VM (isolation) versus a container (density)
- Explain how cloud-init provisions an instance on first boot, and re-test it with cloud-init clean
- Pull instance identity and secrets from platform metadata at 169.254.169.254 instead of baking them into the image
- Apply the cattle-not-pets mindset to run the same automation on-prem and in the cloud
Why a server should be disposable
Frame the whole lesson around the cattle-not-pets mindset before any tooling.
For decades, a server was a pet. You named it
apollo, you logged in over months to hand-tune its config, and when it got sick you nursed it back to health because nobody remembered exactly how it was built. That works untilapollodies at 3 a.m. and you have no way to rebuild it.The cloud-era model is cattle. Each instance is one of an interchangeable herd: it's described entirely by an image plus a definition file (infrastructure-as-code), so if one misbehaves you don't nurse it — you destroy it and stamp out an identical replacement from the same recipe.
This lesson is the toolset that makes cattle possible on Linux: KVM/libvirt to run virtual machines, cloud-init to provision them automatically on first boot, and instance metadata to feed each one its identity and secrets at runtime.
The Linux virtualization stack: KVM + libvirt
Name the layers of the stack and what each one is responsible for.
Linux virtualization is a layered stack, and the exam expects you to know which layer does what:
- KVM (Kernel-based Virtual Machine) is a module inside the Linux kernel that turns the kernel itself into a Type-1 (bare-metal) hypervisor — guest VMs run directly on the hardware's CPU virtualization extensions, not on top of a host OS.
- QEMU emulates the virtual hardware each guest sees (disk, network card, display).
- libvirt is the management layer — a stable API and daemon (
libvirtd) that sits in front of KVM/QEMU so you have one consistent toolset regardless of the hypervisor underneath. - `virsh` is the command-line client you type at, and `virt-install` is the tool that builds a brand-new VM.
Think of it like an airport: KVM is the runway and control tower built into the ground itself, QEMU is the individual aircraft, and libvirt is the booking-and-dispatch system you actually interact with — you never talk to the runway directly.
Check yourself
Which layer of the stack gives you one consistent management API across hypervisors — KVM, QEMU, or libvirt?
Driving VMs with virsh
Inspect and control the VM lifecycle with the everyday libvirt verbs.
virshis to VMs whatsystemctlis to services: a single command with a verb set you learn once. Start by looking before you touch — list what exists and its state.Example — not runnable on your lab box
virsh list --allId Name State ---------------------------- 3 web01 running - db01 shut off
- list
- Show domains (libvirt's word for a VM/guest).
- --all
- Include stopped VMs too. Without it, only running domains appear — so a shut-off VM looks like it doesn't exist.
Two VMs are defined. web01is running and has a live Id (3);db01is shut off and shows-for Id because a stopped domain has no runtime id. The classic beginner trap is running plainvirsh list, seeing onlyweb01, and concludingdb01was never created.Example — not runnable on your lab box
virsh start db01Domain 'db01' started
- start
- Boot a defined-but-stopped domain. Power-on equivalent.
- shutdown
- Ask the guest OS to power off gracefully (sends ACPI signal — needs a cooperating guest).
- destroy
- Force-stop immediately, like yanking the power cord. Does NOT delete the VM despite the scary name.
startpowers on the guest. The mirror command isvirsh shutdown db01, which asks the guest to halt cleanly — prefer it overdestroy, which is a hard power-cut you reach for only when the guest is wedged. Re-runvirsh list --allanddb01flips torunning.Check yourself
You ran
virsh listand your VM is missing, but you're sure you created it. What flag shows stopped VMs too?Building a VM with virt-install — and when NOT to
Provision a new VM, and decide VM-vs-container by isolation versus density.
virshmanages VMs that already exist; `virt-install` stamps out new ones. It defines the virtual hardware and points the guest at an OS source so it can boot and install.Example — not runnable on your lab box
virt-install --name web02 --memory 2048 --vcpus 2 --disk size=20 --os-variant ubuntu22.04 --cloud-initStarting install... Domain creation completed.
- --name
- The domain name you'll later pass to virsh.
- --memory
- RAM for the guest, in MB (2048 = 2 GB).
- --vcpus
- Number of virtual CPUs the guest sees.
- --disk size=20
- Create a 20 GB virtual disk for the guest's storage.
- --os-variant
- Tells libvirt which OS this is so it picks optimal virtual hardware/drivers.
- --cloud-init
- Attach cloud-init data so the VM provisions itself on first boot (next step).
This single command defines a 2-vCPU, 2 GB, 20 GB VM and registers it with libvirt — afterward virsh list --allshowsweb02. Crucially, every line here is declarative: the same flags rebuild an identical VM on any KVM host. That's cattle, expressed as a command.But should this even be a VM? A VM carries its own full kernel and OS, which is heavy. A container shares the host's kernel and ships only the app and its libraries, which is light. The choice is a single trade-off:
- VM → isolation. Its own kernel means a compromised or crashing guest can't easily reach the host or its neighbours. Pick a VM for hostile multi-tenant workloads, for a different OS/kernel than the host, or when strong security boundaries are required.
- Container → density. Sharing the kernel means near-zero overhead, so you pack far more workloads per host and they start in milliseconds. Pick a container for many instances of your own stateless services.
Check yourself
You need to run 200 instances of your own stateless web service as cheaply as possible. VM or container?
cloud-init: provisioning on first boot
Explain what cloud-init does, when it runs, and how to re-test it.
A freshly stamped image is generic and identical to every other copy. cloud-init is what turns that generic image into this specific instance — and it does it automatically on the instance's very first boot.
On that first boot cloud-init reads user-data (a config you supply, usually YAML) and applies it: it sets the hostname, creates users, installs their SSH keys, installs packages, writes files, and runs arbitrary scripts. Think of it as the instance's new-hire onboarding: the laptop comes off the shelf identical for everyone, and onboarding personalizes it — accounts, access badges, software — on day one.
Example — not runnable on your lab box
cat user-data.yaml#cloud-config hostname: web02 users: - name: deploy sudo: ALL=(ALL) NOPASSWD:ALL ssh_authorized_keys: - ssh-ed25519 AAAAC3Nz... deploy@laptop packages: - nginx runcmd: - systemctl enable --now nginxThe #cloud-configfirst line is required — it's the marker cloud-init looks for. This file sets the hostname, creates a passwordless-sudodeployuser with an SSH key (no password ever set), installs nginx, and enables it. Notice there's no secret in here that the image itself stores — the SSH public key is safe to ship; the matching private key never leaves the operator's laptop.Example — not runnable on your lab box
sudo cloud-init clean --logs- clean
- Wipe cloud-init's state markers so it treats the NEXT boot as a first boot and re-runs provisioning.
- --logs
- Also remove cloud-init's logs, for a fully clean slate when testing.
Silent success. After clean, the next reboot makes cloud-init run its full first-boot sequence again against your currentuser-data— the supported way to iterate on a provisioning config without rebuilding the image from scratch.Check yourself
You changed your cloud-init user-data and rebooted, but the instance is unchanged. Which command makes it re-run provisioning on the next boot?
Where identity and secrets come from: instance metadata
Pull config and credentials from the platform at runtime instead of baking them in.
How does cloud-init know it should be
web02and notweb57? It asks the platform, not the image. Every cloud (and many on-prem setups) exposes an instance metadata service at a fixed link-local address: `169.254.169.254`. Link-local means it's only reachable from the instance itself and never routes off the box — so it's a private channel between the running instance and the platform.The instance queries that endpoint at boot to learn who it is — its hostname, its user-data, and short-lived credentials the platform mints for it. The principle: identity and secrets come FROM the platform AT runtime; they are never baked into the image.
curl -s http://169.254.169.254/latest/meta-data/hostname instance-id local-ipv4 public-ipv4 iam/
- -s
- Silent: suppress curl's progress meter so only the response body prints.
The metadata endpoint returns a directory-like listing of what this instance can ask about itself. Drilling in — e.g. curl -s http://169.254.169.254/latest/meta-data/hostname— returns the value for that one key. Theiam/branch is where the platform hands out temporary, auto-rotating credentials, so the instance gets access without any long-lived key stored on disk.Check yourself
At what well-known link-local IP does an instance reach its cloud metadata service?
Try it in your live terminal
Detect your own environment with safe, read-only probes on the real sandbox.
You have a real root shell. These are read-only probes — they only report what's present, they change nothing. Run them in order to see which parts of the stack this box has. It's fine if a tool isn't installed; the
|| echofallback just tells you so instead of erroring.systemd-detect-virt 2>/dev/nullReports whether you're inside a VM or container and which technology (e.g. kvm,lxc,docker) — ornoneon bare metal. This is the fastest way to answer "am I virtualized, and how?".virsh --version 2>/dev/null || echo 'no libvirt'If libvirt is installed you get a version number, confirming this box can host KVM guests; otherwise the fallback prints no libvirt. The2>/dev/nullhides the error message so the fallback reads cleanly.cloud-init --version 2>/dev/null || echo 'no cloud-init'Prints the cloud-init version if this image is cloud-provisionable, else no cloud-init. On a real cloud instance you'd also runcloud-init statusto see whether first-boot provisioning finished.
- KVM is the in-kernel Type-1 hypervisor; libvirt is the management layer you drive with
virsh(list/start/shutdown) andvirt-install. - VM = isolation (its own kernel), container = density (shared kernel) — choose on which boundary the workload needs.
- cloud-init provisions an instance from user-data on first boot only; reset with
cloud-init cleanto re-test. - Identity and secrets come from the platform at runtime via metadata at
169.254.169.254— never baked into the image. - Treat instances as reproducible cattle (image + IaC), so the same automation runs on-prem and in the cloud.
| Dimension | Virtual machine (KVM) | Container |
|---|---|---|
| Kernel | Its own, full guest kernel | Shares the host kernel |
| Isolation | Strong — hardware-enforced boundary | Weaker — process-level on shared kernel |
| Density / overhead | Heavier; fewer per host; boots in seconds | Light; many per host; starts in ms |
| Manage with | virsh, virt-install (libvirt) | container runtime (see containers lesson) |
| Pick when | Hostile tenants, different OS, hard isolation | Many copies of your own stateless service |
