Drift detection
Last updated
Was this helpful?
Last updated
Was this helpful?
In infrastructure-as-code, the concept of drift represents the difference between the desired and the actual state of the infrastructure managed by your tool of choice - , , , etc. In practice, there are two sources of drift.
The first source covers changes directly introduced by external actors - either humans or machines (scripts). If an on-call SRE changes your database parameters otherwise controlled by Terraform, you've introduced drift. If an external script updates your Kubernetes cluster in a way that conflicts with its Pulumi definition, it's drift as well.
The other source of drift comes from the dependency of your resources on external data sources. For example, if your load balancer only expects to receive traffic from , you may want to restrict ingress to a predefined range of IPs. However, that range may be dynamic, and your IaC tool queries it every time it runs. If there's any change to the external data source, it's showing up as drift, too.
In the first scenario, drift is an unwanted by-product of emergencies or broken processes. In the latter, it's both desired and inevitable - it's proof that your otherwise declarative system responds to external changes. In other words - drift happens, so deal with it. 😎
Spacelift comes with a built-in mechanism to detect and - optionally - reconcile drift. We do it by periodically executing on your stable infrastructure (in Spacelift, we generally represent it by the ) and checking for any changes.
To get started, create a drift detection configuration from the Scheduling section of your stack settings. You will be able to add multiple cron rules to define when your reconciliation jobs should be scheduled, as well as decide whether you want your jobs to trigger ( jobs) in response to detected drift:
We generally suggest turning reconciliation "on" as it ensures that you get the most out of drift detection. Reconciliation jobs are equivalent to manually triggering and obey the same rules and constraints. In particular, they respect their stacks' auto-deploy setting and trigger plan policies - see for more details.
However, if you choose not to reconcile changes, you can still get value out of drift detection - in this case, drifted resources can be seen in the Resources view, both on the stack and account level. Also, drift detection jobs trigger like regular runs, where they're clearly marked as such (driftDetection field).
With drift detection enabled on the stack, are quietly executing in the background. If they do not detect any changes, the only way you'd know about them is by viewing all runs in the Account > Runs section and filtering or grouping by drift detection parameter - here is an example:
But once your job detects drift (and you've enabled ), it triggers a regular tracked run. This run is subject to the same rules as a regular tracked run is. For example, if you set your stack not to , the run will end up in an state, waiting for your decision. The same thing will happen if a produces a warning using a matched warn rule.
The only real difference between a drift detection job and one triggered manually is that the run section of your policy input will have the drift_detection field set to true - and this applies to both and policies. You can use this mechanism to add extra controls to your drift detection strategy. For example, if you're automatically deploying your changes but want a human to look at drift before reconciling it, you can add the following section to your body:
