Build1 publisher3 min readPublished
The drift job that catches a console fix needs write access to the state lock
Terraform reads an emergency console change as a config mismatch and reverts it on the next scheduled apply. A dev.to writeup builds the CI job that catches it first, and explains why the plan-only role still needs lock writes.
The Engineer · Build desk
What happened
- Someone adds an inbound security group rule in the AWS console during an incident, nobody writes it into Terraform, and three weeks later a scheduled apply reverts it silently or fails on an unexpected diff.
- The dev.to writeup's detection job runs terraform plan -refresh-only -detailed-exitcode, because a plain plan's exit code 2 fires for real drift and for unapplied .tf edits alike.
- Remote state with locking is a precondition, including an S3 backend with use_lockfile = true from Terraform 1.10, and the older dynamodb_table argument is deprecated as of Terraform 1.11.
- A detection role holding only s3:GetObject and read-only DynamoDB permissions fails on every single run with a lock-acquisition error instead of returning a clean read-only plan.
- The pipeline classifies what the plan finds, opens a PR or issue for human review, and remediates automatically only for narrowly scoped, low-risk cases.
Compiled by The EngineerSomething wrong?How this is made
Why it matters
- cost The platform team pays before the first useful report: remote locked state, a pinned CLI, a committed provider lockfile, scheduled triggers, PR credentials and two separate cloud roles.
- exposure Noise is the real risk to review quality. Once diffs are always non-empty, engineers stop reading them, and a genuinely dangerous change passes unnoticed.
- decision A detection role kept strictly read-only leaves two outcomes: widen it to allow lock writes, or accept a job that errors on every schedule.
- precedent With dynamodb_table deprecated, new drift pipelines get built on native S3 locking, and the legacy lock table becomes a migration item rather than a supported default.
Terraform has no field for intent. It compares the config against refreshed state, sees that the security group no longer matches, and treats the difference as something to correct [4]. So the remediation path has two ends: the console edit gets written into the `.tf` files, or someone reverts it deliberately. A scheduled job that reports the difference and stops there still leaves the next apply free to revert it.
With `-detailed-exitcode`, Terraform returns 0 for no changes, 1 for an error and 2 when changes are found, and that code is the branching point for everything downstream [7]. The post warns that the workflow has to capture the code carefully, because GitHub Actions runs step scripts with `bash -e` [17].
Locking is mandatory for a reason that has little to do with the plan damaging state. `plan` never writes to the state file; only `apply` does [10]. Locking exists to stop a collision from corrupting anything, and a lock collision just fails the operation with an error [11]. Local state has no locking and no shared visibility across a team, so the post rules it out for scheduled runs that might overlap a human-triggered plan or apply [21].
The permission list is where the read-only instinct backfires. Acquiring the lock needs `dynamodb:PutItem` and `dynamodb:DeleteItem` on the lock table under the legacy backend. Under native S3 locking it needs `s3:PutObject` and `s3:DeleteObject` on the lockfile object [12]. Read-only in IAM and read-only in Terraform are not the same permission set. The CI platform also needs scheduled triggers, such as GitHub Actions `schedule:` or GitLab CI pipeline schedules. It needs credentials for opening PRs or issues, and plan-only cloud credentials for the detection run itself [20]. Apply-capable credentials go in a separate role, gated behind approval and introduced later in the pipeline [19].
Two other prerequisites shape what the diff shows. Pin the CLI with `required_version` and commit `.terraform.lock.hcl` [14]. A scheduled run that resolves a newer provider version than the last manual plan fills the diff with provider-attribute changes that have nothing to do with real infrastructure drift [15]. Providers also backfill attributes Terraform never wrote: a default security group rule, a provider-assigned ARN suffix, KMS key rotation metadata. The next apply will happily fix them. The more mundane sources are another team tagging resources through the AWS SDK and auto-scaling groups rewriting instance counts [3].
The post publishes no measurements: drift frequency, false-positive rate, the count of how many findings its classification step routed correctly are all missing [23]. It is one engineer's design, first published on kuryzhev.cloud and republished on dev.to [2]. For it to transfer, your state has to be remote and locked and your CLI and provider versions pinned. The automatic remediation rules have to be narrow enough to run unattended on the low-risk cases the author restricts them to [18].
What to watch
- A later Terraform release that removes dynamodb_table outright would force backend edits on teams still using the legacy lock table.
- A published post-mortem in which a scheduled apply reverted an incident fix would move this from design advice to measured cost.
- Any writeup that publishes its allow-list for automatic remediation together with the false-positive rate behind it.