Is AI actually changinghow you build?
Fifteen questions you can answer without reading a line of code. Each one says who to ask or what to look at. One rule: if you’ve only been told something exists, that’s “Partly there”. Not sure? Say so. It becomes a question for your engineering lead.
Tools on desks
Your stage: 0, Tools on desks.
AI does the typing
- Every engineer has a paid AI coding tool and uses it on real work every week.How to check: Ask for the licence count against engineering headcount, and last month’s usage report from the tool’s admin dashboard. Works when licences cover everyone and usage shows nearly everyone active each week.
- There’s a written rule for how AI-written work is reviewed before it reaches customers.How to check: Ask to see the rule, then ask for one recent piece of work where it was followed. Works when you’ve read the rule and seen the example.
- Your team can tell you what share of new code is written by AI, as a number from a tool, not a guess.How to check: Ask what share of last month’s code was AI-written, and where that number comes from. Works when you get a number with a named source.
Guardrails
- Nothing reaches customers until automatic checks pass, and nobody, founders included, can skip them.How to check: Ask when the checks last blocked a release, and who can override them. Works when there’s a recent real block, and the answer to “who can override” is no one, or a logged emergency process.
- There are written limits on what AI must never touch, such as customer data, payments and live systems, enforced by software rather than trust.How to check: Ask for the list, and ask how a breach would be stopped automatically. Works when the list exists and the answer describes something that blocks, not a policy people are trusted to follow.
- When a release breaks something, the fix includes a new automatic check so it can’t break the same way again.How to check: Pick the last three incidents and ask what check was added for each. Works when all three have one.
Proof, not trust
- Your most important customer journeys, such as sign-up, payment and the core workflow, are tested automatically before every release.How to check: Ask for the list of journeys covered, and the last time a test caught a problem before a customer did. Works when your top journeys are on the list and there’s a recent catch.
- Every shipped change comes with proof it works that a non-engineer can look at: a recording, a screenshot or a test result.How to check: Ask to see the proof for three changes shipped last week. Works when you opened all three and understood what you were looking at.
- The same mistake rarely reaches customers twice.How to check: Ask for last quarter’s incident list and count the repeat causes. Works when there are none, or one at most with a written explanation.
Digital team members
- AI takes work from the same priority list as your people and runs recurring jobs on a schedule, including when nobody is online.How to check: Ask for Monday’s summary of what was done over the weekend, and the list of scheduled AI jobs. Works when real work was done while nobody was online, and the jobs are written down.
- You know what AI costs each month and what it saves, and you’ve seen both in writing.How to check: Ask for monthly AI spend beside the trend in people-time per shipped change. Works when it fits on one page, shows at least four weeks of trend, and the trend is falling.
- Shipping doesn’t depend on one person. If your CTO were away for two weeks, releases would continue safely.How to check: Ask who approved last month’s releases. Works when no single person approved more than half of them.
Learns from its mistakes
- Customer outcomes, such as reported defects, outages and time to fix, are tracked next to delivery speed, and both are improving.How to check: Ask for one scorecard with both. Works when you’ve seen it, with at least a month of trend.
- When something goes wrong, the system proposes a change to its own process, and a person approves it.How to check: Ask for one recent example, from the failure through to the change that was made. Works when you’ve seen one end to end.
- There’s a written, narrow policy for what AI may ship without a person approving it, and someone audits it.How to check: Ask to see the policy and the last audit. Works when both exist and the audit is recent.
Start with the first question.
Answer from what you’ve seen yourself. If you’ve only been told, that’s “Partly there”.
You’re paying for AI and getting faster typing. Nothing about how work is checked or shipped has changed, so neither has your speed or your risk.
What stands between you and Stage 1.
- 1Can you show me last month’s usage report for our AI coding tool, against headcount?
- 2Can you send me our written rule for reviewing AI-written work, and one recent change where we followed it?
- 3What share of last month’s code was written by AI, and which tool does that number come from?
Send your engineering lead a link with your answers. When they finish the engineering edition, they’ll see exactly where you disagree. Your answers travel in the link itself; nothing is stored on a server.