How to review an AI-written feature flag rollout

When an agent puts a change behind a feature flag, review the two behaviors it creates. Check what runs with the flag off, what runs with it on, and what happens when the flag value cannot be read. Then ask who can turn it on, how they will observe the rollout, and how they will turn it off. A flag can limit exposure, but it does not make the new path correct.

Suppose a PR adds a new checkout price calculation behind a flag. The agent writes the new calculation and wraps the call in an if. The obvious diff is small. The review still has to cover which customers see each price, whether both paths use the same inputs, and what happens to an in-flight checkout if the flag changes.

Define the behavior on each side of the flag

Start with the request that led to the PR. Is the flag there for a gradual rollout, an internal test, or a fast disable switch? Those goals lead to different defaults and ownership. Ask the author to write down the expected behavior for off, on, and unavailable. For a price calculation, compare the amount shown to the customer with the amount charged and recorded. A test that only checks the new helper misses the boundary between the helper and the checkout flow.

Read the flag evaluation where the application actually makes the decision. Check its default value, the identity or account used for targeting, and whether a cached value can change mid-request. If the new path writes data that the old path cannot read, switching the flag off may not restore the old behavior. Follow the changed data shape through readers, background jobs, and stored records before calling it reversible.

In Review Assist, compare the proposed rollout with problem.user_asks, approach.requirements, and any assumptions in the committed Intent Document. The guided tour can point you to changed hunks. It cannot tell you whether every path through the flag is safe.

Test the flag matrix, including failure

A practical feature flag testing matrix starts with off and on at the service boundary, then adds a failed or delayed flag lookup. For each state, check the observable result and any side effect. In the checkout example, that means the returned total, the persisted order amount, and any downstream event. Run these tests with the targeting context the rollout will use, not just a hard-coded Boolean in a unit test.

Look for behavior that differs across instances or over time. If the flag is evaluated once when a request starts, confirm that every later step uses that decision. If it is evaluated again before writing an order, a customer might see one price and be charged another. If the flag provider is unavailable, the fallback must be explicit and tested. A safe fallback is a product decision tied to the data and operation, not a universal default.

Read verification.performed and verification.not_verified for the actual head commit. A passing suite is useful only if it exercised the states and side effects you care about. The guide to untested paths shows how to turn a missing case into a focused review request.

Make the rollout and rollback executable

Ask for the first audience, expansion rule, metric or log that signals trouble, and the person who can change the flag. For the price example, a rollout plan might start with internal accounts, compare calculated totals across old and new paths without charging twice, and expand only after mismatches are understood. The exact thresholds belong to the team operating checkout. The PR should at least name the signal and the decision owner.

Check that turning the flag off reaches the old behavior without another deploy. If the new path has already changed data, queued jobs, or external calls, a toggle alone may not undo the effect. Ask what must be repaired and how the team will detect a partial rollback. If schema or backfill work travels with the flag, follow the database migration review too.

Finally, give the flag an exit condition. Who will remove the old branch after rollout, and what evidence says it is safe to do so? A permanent two-path implementation increases the number of states future changes must handle. If the flag is deliberately long-lived, say why and identify its owner.

Ask for the missing proof

A useful review comment is: “The new price path is covered when the flag is on. Please add a test for the off path and a failed flag lookup at checkout, show which account key controls targeting, and state what happens if we disable the flag after new-price orders have been written. Who watches mismatches during the first rollout?”

Approve when both paths and the failure path have a defined result, the evidence covers the side effects, and the rollout has an owner with a usable rollback. Review Assist helps you locate the agent's intent and claimed verification; its coverage check confirms that changed hunks were explained. The decision about whether the flag is safe to ship still rests on the code, tests, and rollout plan.