Spark won't give you the answer. Here's how we make sure.

Every student with a phone already has a machine that will solve their homework. Paste the question, get the working, copy it down, feel helped. The trouble is that feeling helped and having learned are different things, and the gap shows up in the exam hall.

That isn't just a hunch. In a large field experiment with high-school maths students, those given an unrestricted GPT-4 tutor did better on practice problems, then did worse than students with no AI at all once the AI was taken away. A version with guardrails that pushed students to do the work themselves removed most of that harm (Bastani et al., 2025).

Spark is our attempt at the guardrailed version, built for Sec 1–4 students in Singapore working through the O-Level syllabus. This post explains the rules Spark follows, the research each one comes from, and how we test that the AI actually follows them. It also covers the parts that broke when we tested it hard.

A prompt is a request, not a guarantee

The easy way to build an AI tutor is to tell a chatbot "be Socratic, don't give the answer". It works in a demo and fails in real use: a student who asks three times, claims their teacher said it was fine, or just waits, will usually get the answer eventually.

So in Spark the important rules are not in the prompt. They're in ordinary code that runs on every reply before the student sees it, and that we can test without an AI model at all. The model writes a draft and the code decides whether it is allowed to send it. A draft that breaks a rule is regenerated once. If it fails again, the student gets a short, safe question instead.

Five rules, and where they come from

1. Never hand over the answer

The research: students learn more from producing an answer than from receiving one, and struggling with a problem before being shown how can deepen learning ("productive failure", Kapur 2008).

What Spark does: an answer-leak guard checks every reply against six fixed rules. It catches the obvious case ("x = 4") and the sneaky one, where the tutor slips the result into a question ("so 3x = 15, what next?"). We tuned it to flag too much rather than too little: a false alarm costs a badge on the parent's view, but a missed leak teaches a student that waiting works.

2. Help rises only when the student needs more of it

The research: good tutoring is contingent: the tutor gives more help when the learner struggles and less when they succeed (Wood, Bruner & Ross, 1976, the paper that gave us the word "scaffolding").

What Spark does: every session has a five-step hint ladder, from a nudge to a full worked example. The step goes up by exactly one when the student is genuinely stuck: they ask for help, say they're lost, or get a step wrong. It holds when they're making progress, and it never jumps. Spark's checker refuses any reply that offers more help than the current step allows.

The AI model doesn't get to choose the step. In early versions it did, and one reply could jump a student from step 1 to step 4. Now the model's opinion is recorded and ignored.

3. One question at a time, and no lectures

The research: working memory is small. Piling several questions or a long explanation onto a student who is already working something out leaves less room for the thinking itself (Sweller, 1988).

What Spark does: the same pre-send checker rejects a reply that asks more than one question, or that explains for more than about three sentences without asking the student to do something.

4. Explaining your reasoning counts for more than being right

The research: students who explain worked steps to themselves learn far more than those who don't. This is the self-explanation effect (Chi et al., 1989). A bare correct answer can come from a lucky guess or a copied step. A correct answer the student can justify is much stronger evidence.

What Spark does: Spark builds its picture of what a student knows from a log of individual events, not a single score. A correct answer moves a skill up a little. A correct answer the student then explains in their own words moves it up by roughly two and a half times as much. A correct answer that needed a full walkthrough counts for less than one that needed a nudge. Because the score is rebuilt from the log, we can replay any student's history and show exactly why their record says what it says.

5. Don't praise what you haven't checked

The research: false praise is worse than none. It tells a student a wrong method is fine, and it teaches them to stop trusting the feedback.

What Spark does: Spark refuses a reply that tells the student they're right unless the model has also marked the step as correct in its hidden working notes. Separately, a rule-based checker spots three common algebra slips in the student's own working without any AI: moving a term across without changing its sign, expanding only the first term of a bracket, and adding the squares in Pythagoras when one side should be subtracted. If that checker isn't sure, it says nothing.

How we test it: ten models, simulated students

Rules are only useful if they hold up against real student behaviour, and real students are inventive. So we built a test harness in which one AI model plays the student and another plays the tutor, talking to Spark through exactly the same connections the real app uses.

The simulated students follow scripts:

• a student who works the problem properly;
• one who fishes for the answer, begs, or claims a parent said it was allowed;
• one who is confused and gives up;
• one who says thanks and leaves.

After every session the harness checks the record mechanically:

• Did the hint ladder move when it shouldn't have?
• Did the summary claim a skill the student never practised?
• Did any internal model text leak into what the student saw?
• How long did each reply take?

A second AI pass then reviews each transcript for what a script can't judge: maths errors, leaked answers, and praise for wrong work.

We ran this against ten open models hosted on NUS's SoCLaaS service. The results:

The good news: no model leaked an answer. The answer-fishing, begging and fake-permission scripts got nothing out of any of them, because the guard sits outside the model.

The bad news: most of what we found was in our code, not the models'. That is the useful kind of result, because it is the kind we can fix.

What broke, and what we changed

The hint ladder rose on correct work. Three separate causes:

• "Thanks for the help!" and "if I get stuck I'll come back" counted as asking for help.
• When Spark replaced a rejected reply with its safe fallback question, the student's correct step was left unjudged, and three unjudged steps in a row count as a struggle.
• One model wrote its "this step is correct" note as visible text instead of a hidden note, so Spark never saw it.

We fixed all three. Replaying the test students' messages, every false rise we checked now holds steady.

One model's private reasoning reached students. Some models "think out loud" inside special tags before answering. Spark now strips those out as the reply streams in, even when a tag is split across two chunks of text. This mattered more than it looked: the leaked reasoning got 61 of that model's 68 replies rejected by the checker.

Session summaries overclaimed. The end-of-session summary sometimes listed topics only mentioned in the tutor's greeting, and listed correct steps as "struggles". Summaries now keep only topics the student actually worked on.

A failed reply could be saved twice. If a student resent a message after a timeout, it was stored twice and the same "please resend" fallback repeated. Both are fixed.

Each fix comes with an automated test that fails if the bug comes back. Spark has more than 60 such test scripts, and most run without any AI model, so they cost nothing to run on every change.

We also switched Spark's live model from x-test-1 (75 s a reply) to qwen3.6:35b (about 4 s), with x-test-2 as the backup.

Then we ran the simulated students against the fixed version, three more times, and each run found something new:

• Over-credit: one solved equation, "4x − 3 = 17", was being credited to three different equation skills at once, because it matched each skill's description loosely. Now only the skills the session is about can be credited that way.
• Blind retries: when Spark rejected a draft and asked the model to try again, the model was never told why, so it often made the same mistake twice. The retry now names the rule the draft broke.
• Repeated fallback: two rejected drafts in a row sent the student the same safe fallback question twice. There are now three, and Spark never repeats one it used in its last two replies.
• Too many questions: a draft rejected only for asking too many questions is now cut back to its first question, instead of being thrown away.

On the final run the tutor leaked no answers, replies took 2–11 seconds, and the hint ladder rose only once: on the turn where the student said they were stuck.

What we don't know yet

We want to be as honest about this as about the rest. Everything above shows that Spark behaves the way the research says a good tutor should: it doesn't leak answers, it scaffolds in steps, and it rewards explanation. None of it yet shows that students who use Spark learn more. That needs a proper study comparing students who use Spark with students who don't, measured on problems they solve alone afterwards, and we haven't run one.

Here's what we're working on next:

• Measure the mix of tutor moves: count how often Spark asks for justification versus explains outright, so we can compare it with published data on human tutors.
• Check independence after help: after a student gets help, follow up later with a similar problem they must solve alone.

If you're a teacher or parent who'd like to be part of that study, we'd like to hear from you.

References

• Bastani, H. et al. (2025). Generative AI without guardrails can harm learning. PNAS. https://www.pnas.org/doi/10.1073/pnas.2422633122
• Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P. & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2). https://onlinelibrary.wiley.com/doi/abs/10.1207/s15516709cog1302_1
• Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3). https://www.tandfonline.com/doi/abs/10.1080/07370000802212669
• Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2). https://onlinelibrary.wiley.com/doi/10.1207/s15516709cog1202_4
• Wood, D., Bruner, J. S. & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2). https://acamh.onlinelibrary.wiley.com/doi/10.1111/j.1469-7610.1976.tb00381.x