Two schools of thought in infrastructure management have different answers to the same question: when the configuration of a server needs to change, what do you do?
Configuration management answer: connect to the server and change it. Tools like Ansible, Puppet, and Chef automate this process.
Immutable infrastructure answer: build a new server with the desired configuration, route traffic to it, terminate the old one. The server itself is never modified after provisioning.
Both work. They make different trade-offs and fit different contexts.
Mutable Servers and Configuration Management
In the configuration management model, servers are long-lived. They’re provisioned once, then continuously updated throughout their lifetime.
Ansible playbook example:
- name: Configure application server
hosts: app_servers
become: yes
tasks:
- name: Install Java 21
package:
name: java-21-openjdk
state: present
- name: Deploy application JAR
copy:
src: "{{ jar_file }}"
dest: /opt/app/application.jar
owner: app
mode: '0755'
notify: Restart application
- name: Update application config
template:
src: application.yml.j2
dest: /opt/app/application.yml
notify: Restart application
handlers:
- name: Restart application
systemd:
name: myapp
state: restarted
What configuration management does well:
- Granular updates: change one config file without replacing the entire server
- Works well with long-lived infrastructure
- Familiar to operators who know Linux
- Lower bandwidth requirements (send changes, not entire images)
- Supports heterogeneous environments (different OS versions, different configurations per server)
The drift problem: over time, servers in the same role diverge. Manual changes are made during incidents and never encoded in the playbooks. Package updates happen at different times. The playbook represents what should be configured; the actual server is what has been configured over time. These diverge.
Configuration management tools reduce drift but don’t eliminate it. A server that’s been running for two years and has survived dozens of playbook runs, emergency changes, and OS patches is a snowflake — unique and hard to reproduce.
Immutable Infrastructure
In the immutable model, once a server (or container, or VM image) is provisioned, it is never modified. Changes require building a new image and replacing the old infrastructure.
The cycle:
- Change the build definition (Dockerfile, Packer template, AMI builder)
- Build a new image
- Deploy the new image (rolling deployment, blue/green, canary)
- Terminate old instances running the old image
# Dockerfile — fully declarative, reproducible
FROM eclipse-temurin:21-jre-alpine
RUN addgroup -S app && adduser -S app -G app
WORKDIR /app
COPY --chown=app:app application.jar application.jar
USER app
EXPOSE 8080
ENTRYPOINT ["java", "-jar", "application.jar"]
# Terraform — describes the desired state
resource "aws_autoscaling_group" "app" {
launch_template {
id = aws_launch_template.app.id
version = "$Latest"
}
min_size = 2
max_size = 10
desired_capacity = 3
}
To “update” the application, you update the Dockerfile, build a new image, update the launch template, and trigger a rolling replacement of instances.
What immutable infrastructure does well:
- No drift: every instance was built from the same image at the same time
- Reproducibility: the image is exactly what’s in the Dockerfile
- Rollback: restore the previous image version (no state on the servers to worry about)
- Simpler operations: no need to run and maintain configuration management tools
- Auditability: every image is built from version-controlled source, buildable from scratch
The costs:
- Every change requires a new build and deployment cycle, even for configuration changes
- Image sizes: full OS + runtime + application is larger than a diff
- Stateful workloads don’t fit (databases, file servers)
- Build pipeline required for all changes
Containers: Immutable Infrastructure Mainstream
Containers (Docker, containerd) made immutable infrastructure the default model for application servers. A container image is inherently immutable — you can’t “update” a running container; you replace it.
# Kubernetes Deployment — immutable by nature
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: app
image: myapp:v2.3.1 # Specific version, not "latest"
# Every version bump requires a new image
The Kubernetes deployment model is immutable: update the image tag, and Kubernetes rolls out new pods, terminates old ones. The old pods are identical to what was deployed before — they haven’t been modified in place.
Configuration that changes independently of code (feature flags, environment-specific settings) goes into ConfigMaps and Secrets, not into the image:
# Configuration separate from the image
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
DATABASE_URL: "jdbc:postgresql://postgres:5432/mydb"
FEATURE_NEW_CHECKOUT: "true"
This preserves immutability (the image doesn’t change) while allowing configuration updates without rebuilding.
Where Configuration Management Still Wins
For all the advantages of immutable infrastructure with containers, configuration management still has legitimate use cases:
Bare-metal servers and VMs that aren’t container hosts: containers abstract the OS layer; bare-metal workloads don’t have this abstraction. Replacing a bare-metal server to change a configuration is expensive (physical provisioning, rack space, hardware lifecycle).
Databases and stateful services: you can’t replace a running PostgreSQL server the way you replace an application container. Data must stay in place. Configuration management (Ansible for PostgreSQL configuration, pg_upgrade for version upgrades) is the appropriate model.
Long-lived infrastructure that predates containers: legacy applications that can’t be containerized may still require configuration management.
Hybrid environments: a Kubernetes cluster running on EC2 instances — the Kubernetes workloads are immutable, but the EC2 nodes that host Kubernetes may be managed with configuration management for OS configuration.
The Practical Reality
Most production environments are hybrid:
- Application services run in containers on Kubernetes (immutable)
- Database servers run on managed services (RDS, Cloud SQL) with configuration through provider APIs
- Kubernetes nodes themselves may use immutable node images (Bottlerocket, Flatcar) or configuration management
The key decision is at the application tier — the part that changes most frequently. Immutable infrastructure with containers is the right default for application servers. It’s operationally simpler, less prone to drift, and aligns naturally with modern CI/CD.
Configuration management remains appropriate for infrastructure that can’t be containerized and stateful services that can’t be replaced wholesale.
Operational Implications
Debugging: in a mutable model, you can SSH to a server and inspect its state. In an immutable model, you rely on logs and observability — the container may be gone by the time you’re debugging. This requires investing in observability before you encounter the incident.
Incident response: in a mutable model, hotfixes can be applied directly to running servers. In an immutable model, hotfixes require going through the build pipeline. This requires a fast pipeline and possibly a break-glass procedure for true emergencies.
Security: immutable infrastructure has better security properties — no SSH access in production, no ability to make changes outside the controlled pipeline, drift is structurally impossible. Configuration management requires maintaining SSH access and the attack surface that comes with it.
The shift to immutable infrastructure is largely the right direction for application services. It aligns with how containers work, how Kubernetes works, and how modern CI/CD pipelines work. The configuration management mental model doesn’t disappear — it just moves to the build process, where it operates at a higher level of abstraction and produces more reliable outcomes.