Moving Beyond Clicks: How Multimodal Inputs Enable Better Learning

Every teacher who has used an edtech program has seen the student dashboard that shows steady progress, levels cleared, stars earned, a bar inching toward green. However, when you look at their next assessment or hear them read aloud, the in-app progress has not translated into higher capability.

When I was teaching, that's why I was reluctant to put my students on most programs. Progression within the app did not equal development of mastery offline.

Carl Hendrick explains why in an essay that circulated widely among educators last winter. One of his examples will be familiar to anyone who teaches early reading. An app shows the word "elephant" next to a picture of an elephant. The child taps the right answer by matching the picture, without decoding the word. The app logs a success, but the child hasn't learned to read "elephant."

Hendrick argues that learning only happens reliably under certain conditions. The first is that "the target skill must be the only path to success." If a student can succeed another way, many will. That isn't laziness; people naturally take the easier route. His other conditions follow from the same logic. Students shouldn't move on until they've mastered what comes first. And they need to produce answers, not just recognize them. (He calls these conditions instructional invariants.)

He blames incentives for why most apps ignore these rules. The features that make learning work also make apps harder, and harder apps hurt engagement numbers. I think he's right. But there's a second reason, and it matters for what's changing now.

For most of edtech's history, a student could tell the software only one thing: where they tapped. That covers multiple choice, drag and drop, and matching a picture to a word. A tap tells you a student picked the right answer, but not how they got there.

That limit stopped even well-intentioned teams from following Hendrick's rules. If the only evidence is a click, some path to the right answer will skip the skill. You can't require mastery before moving on if you can't tell whether mastery happened. And choosing from a list is recognition, not production.

What's changed

Most of the conversation about AI in schools is about what AI produces: lesson plans, reading passages, chatbot tutors that talk back. Skepticism there is reasonable. Generated content can be wrong, and a tutor that explains something well doesn't mean the student understood it.

Much less attention goes to the other direction: what AI makes it possible for students to show. Speech recognition can listen to a child read a word aloud and tell whether they produced the right sounds for the letters or guessed. A student solving a math problem with a stylus can have their steps evaluated, not just their final answer.

In these uses, AI isn't teaching. It's listening and checking. That's a narrower job than writing lessons, and I'd argue a more useful one, because it gives software evidence of the skill itself.

That evidence helps with each of Hendrick's conditions:

• The production skill becomes the only path. You can't must read out loud, accurately, to progress.
• Mastery checks become trustworthy. Holding a student back is only fair if the signal behind it is reliable.
• Students produce instead of recognize. Reading aloud and writing out work are production by nature.

How we're building

At Project Read, our phonics tutor asks students to read aloud and to spell words with their finger on a touchscreen. We assess each response down to the grapheme-phoneme correspondence. Students don't advance in the UFLI scope and sequence until they've shown they can apply the focus skill. There's no way to click through.

UFLI has studied the tutor's effectiveness, and the results are forthcoming. The data is very promising, particularly for students starting below the 50th percentile of fluency. We'll link to the study when it's published.

The best assessment of foundational skills has always meant listening to the child read out loud, or analyzing their spelling. While software can save time, until recently it has used poor proxies (clicking multiple choice responses) to assess those same skills. Multimodal AI models change that status quo, enabling verification of these skills by listening and analyzing handwriting. I’m hopeful about the newfound possibilities to save teachers time, while finally providing reliable data on student decoding and encoding skills.