The single best test is to ask a candidate to walk through a workflow they put into production: what broke, how they noticed, and what they changed. People who have only built a course demo cannot answer that. Use the questions below to hear the difference, then pay them for a short test on your real process. A one-page version you can score in the room is the skills checklist.
Rekruuto vets before it introduces an AI Implementation Specialist. You should still ask these questions yourself. Score each answer out of five, and set your pass mark before you read the work.
What are the 21 interview questions?
Ask them in order if you can. Stop early if the production stories are thin. Each line is the question. The notes under the five groups say what a solid answer contains.
- Walk me through one automation that ran in production for at least a month. What was the trigger?
- What broke in that workflow, how did you find out, and what did you change?
- Which parts were rules, and which parts used a model? Why?
- Show me the runbook a colleague used when you were away.
- What did you refuse to automate, and why?
- How do you hear about a failure before a customer does?
- What do you log, and who reads it?
- How do you cap model or task usage so a loop cannot run up a bill overnight?
- Describe a time the output looked fluent and was wrong. What check caught it?
- How do you test a change without sending it to real customers?
- How do you turn a messy verbal process into steps and exceptions?
- What do you ask before you open the tool and start building?
- How do you write a handover a non-technical owner can follow?
- Tell me about a stakeholder who wanted full automation and you talked them into a review step.
- How do you prioritize a backlog when everything is called urgent?
- Whose accounts do the workflows live in, and what happens if you lose access tomorrow?
- How do you share access without sending a password in chat?
- What data should this workflow never see?
- If you left on Friday, what would we need on Monday to keep the live workflows running?
- How do you decide a human must approve an output before it is sent or paid?
- What would you deliver in the first 30 days in our business, and what would you refuse to promise?
What questions reveal real production experience rather than a course?
Questions 1 to 5. A good answer names a real trigger, a real failure, and a change that shipped. A red flag is a tour of a tutorial, a screenshot with no date, or a story that never left a personal account.
Listen for nouns. “The Shopify order tag”, “the HubSpot lifecycle stage”, “the finance inbox rule”. Vague nouns (“the AI”, “the automation”) usually mean the work was a demo. Ask who else used it. If the answer is “just me, for the course”, score it as practice, not production.
A good answer also includes something they would do differently. People who have operated a workflow have a scar. People who have watched a video have a feature list.
Use your normal hiring template for the rest of the role. A virtual assistant job description template is a starting point for hours, tools, and reporting. Do not let it replace the production questions.
How do you test error handling, monitoring, and cost control?
Questions 6 to 10. A good answer names an alert, a log, a spend cap, and a place to test. A red flag is “I check it when someone complains” or “the model is very accurate now”.
You are not looking for a particular vendor. You are looking for a habit. They should be able to say where a failed run shows up, who is notified, and how a new version is tried on old examples before it touches customers. They should also be able to say what a runaway loop would cost and how they stop it. Provider prices change. The habit of putting a cap on does not.
If they have never seen a wrong-but-fluent answer, they have not read enough outputs. That is disqualifying for a seat that drafts text.
What red flags should you watch for?
Across the whole interview, these are the patterns that waste a trial week.
- Every example is a personal demo, a hackathon, or a certificate.
- They cannot describe a failure.
- They want admin access to your accounts before they have described the process.
- They promise a percentage of time saved. You did not ask for a forecast, and they cannot know your baseline.
- They will not work in your accounts, only in theirs.
- They talk over the question about human review.
- They cannot write a short explanation without jargon.
None of these is a statistic. They are reasons to end the conversation. A polished portfolio site does not cancel them.
Questions 11 to 15 cover thinking and communication. A good answer starts with questions, not a tool. They can turn a rambling description into steps, exceptions, and an owner. They can say no to a stakeholder without being theatrical. A red flag is building in the first sentence, or a backlog ordered by whatever the last message said.
What’s a good paid test for an AI automation candidate?
Pay them for two hours on a messy process from your business. The point is how they scope, not how much they click.
Send this brief the day before, with any private data removed.
Here is a real process, written badly on purpose. In two hours: ask us the questions you need, write the steps and the exceptions, say what you would not automate, and either build a tiny safe slice or write the build plan if two hours is not enough to build safely. Do not email a customer. Do not move money. Do not use a personal account for our data.
Score it out of five on each line. Set the pass mark before you read the work, and use the same sheet for every candidate.
| Line | 1, weak | 5, strong |
|---|---|---|
| Questions asked | None, or only about the tool | About exceptions, owners, and what must not happen |
| Steps written | A tool list | Trigger, steps, exceptions, owner |
| Judgment | Automates the risky bit first | Refuses or gates the risky bit |
| Safe slice or plan | A demo in their account | A plan or a slice inside a test space you control |
| Explanation | You cannot follow it | A colleague who was not on the call can follow it |
Pay the agreed fee even if you do not hire. You asked for work. Two hours is short on purpose. A candidate who needs your production passwords for a test is not ready.
How do you check data security and account ownership?
Questions 16 to 21. A good answer puts workflows in your company accounts, uses a named user, and can list data the workflow should never see. A red flag is a shared password, a personal Gmail owner, or “trust the model vendor”.
Ask them to describe Monday morning after they resign. The workflows should still run, the runbook should be in your drive, and a second person should be able to pause them. Zapier’s own help center explains how to transfer Zaps when someone leaves a team. If your stack is Zapier, the candidate should recognize that problem even if they have not read that page.
The wider habits are in data security for remote teams. Use them as the house rules, and use question 19 as the interview version.
Question 20 is the values question. You want a person who will slow a launch down. Question 21 is the planning question. Compare the answer with the 30, 60 and 90 day plan. Someone who promises five production agents in week one has not heard the constraints.
After you hire, the person who leaves later is the subject of who maintains the automations. Ask question 19 now so that article stays theoretical.
Frequently asked questions
What is the one question that matters most?
Ask them to walk through a production workflow: what broke, how they noticed, and what they changed. A course demo does not survive that question.
How many questions are there?
Twenty-one, in five groups: production proof, failures and cost, process and communication, security and ownership, and judgment about what not to automate.
Should the test be unpaid?
No. Pay for the two hours. The test is scoping and a safe slice, not a free build of your system.
How should I score the answers?
Score each line out of five. Set the pass mark before you read the work, and use the same sheet for every candidate.
Does Rekruuto replace this interview?
No. Rekruuto vets before an introduction. You should still ask about production failures, account ownership, and human review.
Get matched with your hire
If you would rather not run a search, get matched with your hire and use this list on the AI Implementation Specialist introduction. If you have not decided the work is ready for a specialist, take the AI automation scorecard first. The 1-week trial with replacement or refund is the paid-test idea extended to a working week, with a refund path if the match is wrong. Read the current trial terms on that page.