How to choose AI tutoring software: 12 questions to ask

What should AI tutoring software actually do?

AI tutoring software uses a language model, usually with speech in and out, to teach a student one to one. For an organization (a school, a test-prep company, a course business, an L&D team) it has three jobs: teach your material without making things up, actually teach rather than wait for questions, and tell you what each student understood.

Most demos show a good conversation. Few show the other two jobs, and that is where products differ. The 12 questions below are grouped into six areas. Send them to every vendor in writing before the demo, then ask each one to show the answer live on a chapter you pick. Score each answer 0 (red flag), 1 (partial) or 2 (clear yes, shown live). A red flag on accuracy or data should rule a vendor out whatever the total.

Content and accuracy: is it grounded in your material?

1. What exactly does the tutor teach from? There are three possible answers: the vendor’s own curriculum, the model’s general knowledge (sometimes plus the web), or the material you upload. If you teach your own syllabus, with your notation, your worked-example style and your exam format, only the third is a fit. Ask which formats it ingests (scanned PDFs, recorded video, slide decks, question banks) and what happens to equations and diagrams. Then hand over your messiest chapter, not the vendor’s sample.

2. Can a teacher review a lesson before a student sees it? If every session is generated fresh, there is nothing to review in advance; you can only audit transcripts after students have seen the mistakes. Ask whether there is a preview, whether a teacher can correct it, and whether the correction then holds for every student. Ask too what happens to reviewed lessons when the vendor switches to a newer model.

3. What happens when a student goes off-syllabus, or asks for something that is not in the material? A grounded tutor says the material does not cover it, answers from what is there, and returns to the lesson. Ask whether it browses the web and whether you can turn that off. Our post on keeping an AI tutor grounded has a 10-minute probe test you can run during the demo.

Teaching: does it teach, or only answer?

4. Does it teach a planned lesson, or only respond to questions? A chatbot serves the students who already know what to ask. A tutor opens the chapter, explains a concept, asks a question and moves on only when the student is ready. Ask the vendor to start a lesson and stay silent for the first two minutes. See what the software does on its own.

5. How does it check understanding, and what happens after a wrong answer? Look for a question after each concept, graded against an expected answer or rubric, a hint before the answer, a limit on how many times it loops, and a later revisit of concepts the student missed. This matters more than it sounds. In a field experiment with nearly 1,000 high-school math students in Turkey, Hamsa Bastani and colleagues at the University of Pennsylvania found that a GPT-4 chat interface raised scores on practice problems by 48%, yet students who had used it scored 17% lower than a control group once access was removed. A version prompted to give hints instead of answers largely removed that harm (the study was published in PNAS in 2025). Our list of techniques for checking understanding online shows what good checks look like.

6. Does it remember each student? Across sessions, the tutor should keep where the student stopped, which concepts they have mastered and which mistakes they made, then resume with a short recap. “Each chat starts fresh” means every session repeats the diagnosis and nothing builds up for the student or for you.

Student experience: voice, board, languages and devices

7. How does the student actually interact with it? A text chat is the minimum. Stronger products teach aloud, let the student answer by voice or by typing, and build a board step by step so a worked example unfolds instead of arriving as a wall of text. Check that formulas render properly, that diagrams are drawn rather than described, and that the student can slow the pace. Test on a mid-range phone with a real student, because long pauses make a spoken lesson hard to follow.

8. Which languages, devices and accessibility needs does it support? Ask for a live demo in every language you need rather than a list of supported languages. Ask where students use it: inside your existing platform, in the vendor’s app, or through a separate login. For accessibility, ask for a conformance report against the Web Content Accessibility Guidelines and test with a keyboard and a screen reader. US public schools and colleges have a deadline: the Department of Justice’s ADA Title II rule requires state and local government web content and mobile apps to meet WCAG 2.1 Level AA, and an April 2026 interim final rule moved compliance to April 26, 2027 for entities serving 50,000 people or more and April 26, 2028 for smaller ones. Check with your counsel on how it applies to you.

Control and reporting: what can you set, and what comes back?

9. What can the organization control? Useful settings include tone or strictness, pace, the teacher persona and voice, which courses are switched on, session length, and whether the tutor may hand over full solutions. You should also be able to read the full transcript of any session. If something goes wrong with a student, “we can’t see the conversation” is not an answer you can give a parent or a client.

10. What does it report, and can you export it? The useful report is learning data: mastery per student per concept, the weak concepts across a class or batch, minutes taught and a session summary a teacher can read in a minute. Engagement dashboards (logins, messages sent) do not tell you who needs help. Ask for a CSV or API export so the data outlives the contract.

Data and safety: who owns it, who trains on it, and what about minors?

11. Who owns your content and your students’ data, is any of it used to train models, and how are minors protected? Get written answers to each part. Who owns uploaded content and the lessons generated from it? Are content or transcripts used to train or fine-tune any model, the vendor’s or a third-party provider’s? Which AI providers receive data, and under what terms? How long is data kept, and how do you delete a learner or the whole account? Who inside the vendor can read transcripts? Which security certifications does the vendor hold (SOC 2, ISO 27001), and if none, what practices will it document for you? If the product clones a teacher’s voice or face, how is consent recorded and how is it revoked?

Pricing and total cost: what will a real term cost?

12. What will one real term cost, all in? AI tutors are sold per seat (per enrolled student per month or year), per learner-minute of use, or as a platform fee plus usage. Give each vendor the same scenario, for example 200 students using it 30 minutes a week for 12 weeks, and ask for the full bill. Then ask about minimum commitments, annual prepayment, setup or content fees, extras (avatars, custom voices, extra languages), overages, platform fees if the tutor only runs inside the vendor’s app, and what it costs to export your data when you leave. Seat pricing charges for students who never log in; usage pricing charges only for use but needs a cap. Our AI tutor cost guide works through the arithmetic and a break-even calculation.

The AI tutoring software scorecard

Copy this into a spreadsheet, one column per vendor. Score 0, 1 or 2 per row, and treat any red flag in rows 1–3 or 11 as a stop.

How to run a two-week AI tutor pilot

A demo shows the vendor’s best case. A pilot shows yours. Two weeks is enough to test accuracy, participation, report quality and real cost. It is not enough to prove learning gains: the National Student Support Accelerator at Stanford describes high-impact tutoring as sessions three or more times a week for ten or more weeks. Treat the pilot as a gate to a longer, measured rollout.

1. 1

   Days 1–2: pick one chapter and one cohort

   Choose a chapter students find hard, 20–50 students, and one teacher who will review the lessons and read the reports.
2. 2

   Days 2–3: review before launch

   The teacher reviews every generated lesson and logs each error found. That count is your accuracy baseline.
3. 3

   Day 3: run the probe test

   Ask off-syllabus questions, give a confident wrong answer, push back on a correct one, and ask for the answer outright. Record what happens.
4. 4

   Days 4–12: schedule the sessions

   Three sessions a week of 20–30 minutes, on the timetable, not “optional”. Give students a pre-quiz of 10 questions on the chapter.
5. 5

   Day 8: read the reports

   Does the weak-concept list match what the teacher sees in class? Read five transcripts in full, including one from a struggling student.
6. 6

   Days 13–14: decide on evidence

   Run the post-quiz, total the cost, survey students and the teacher, and compare everything with the targets you set on day 1.