Sunday evening, December 2022. A French teacher is grading her twenty-second essay. She opens the twenty-third, and something seems off. The essay is flawless. Too flawless. The transitions are impeccable, the vocabulary is rich—there’s nothing to correct, and therefore nothing to teach.
Her first question that evening: “Did this student cheat?” It was the question on everyone’s mind that winter. ChatGPT had been launched a few weeks earlier and would reach 100 million monthly active users by January 2023. By July of that same year, only one country in the world had issued regulations on generative AI.
Two camps emerged. On one side, a ban: blocking the tools and returning to paper-based exams. On the other, a cautious integration, adopted as early as 2023 by several universities, including those in the Russell Group: “Rather than seeking to ban their use […], [students] and staff must have an understanding of the effective, ethical, and transparent use of these tools.”
Both sides were debating the same issue: permission. The real question arose later, when schools began to equip their communities for real: the question of the design and integration of these tools.
At Nexapp, we often say that technology must serve people. UNESCO says much the same thing in its Guidelines for Generative AI in Education and Research: it is human capacity and collective action—not technology—that determine our ability to address the fundamental challenges facing societies.
Let’s go back to our teacher. She eventually decides, like many others, that she might as well learn how to use it, and asks the tool to explain a grammar concept to her class. The response comes out smoothly, with proper punctuation. It begins with “Of course” and cites three studies that don’t exist.
The first problem is hallucination: an LLM generates text by predicting the next word based on learned patterns. Without a document-search mechanism connected to it, there is no way to verify that its response corresponds to reality. It is a machine that mimics without understanding what it is saying, to borrow the image used by Bender and his colleagues in 2021.UNESCO puts it more soberly: “Generative AI is based neither on observation of the real world nor on other fundamental aspects of the scientific method, nor is it aligned with human or social values.”
Our teacher, on the other hand, spots the false references because she knows her subject. The risk is higher among young learners who lack a solid foundation of prior knowledge on the subject at hand. Those who need help the most are the least able to detect when they’re being given the wrong kind of help.
The next step is obvious: just verify the information.
The second problem is opacity. Verifying requires understanding where the answer comes from, and that is precisely what a model does not provide. An LLM contains billions of parameters, and even when its weights are public, examining them does not explain why it produced that particular response. We can explain the general approach, but not the specific result. That’s why opacity isn’t just a moral issue: Nazaretsky’s research shows that when people who use these systems don’t understand how a system arrived at a result, they are less likely to want to use and adopt it.
Our teacher isn’t going to write to the department. She’ll just stop opening the tool, without telling anyone. A silent abandonment—a common and costly outcome—because it leaves no trace in the metrics.
But while she closes the app, her students are opening it.
The third problem is homogenization. Students encounter models trained on the dominant data from the web—which is predominantly Western—whose answers align with the most common norms. UNESCO warns that a reliance on AI suggestions can lead to a standardization of answers that undermines the value of critical thinking and inquiry.
However, a school that produces nothing but average answers has ceased to do its job.
The fourth problem—and the most serious one— is dependency. There is something worse than answers that all look alike. We’ve already mentioned this on this blog in relation to development teams: by constantly following the blue line on the GPS, you eventually lose your sense of direction. What applies to a programmer applies doubly to a seventh-grade student, because for them, effort isn’t a cost—it’s the very mechanism of learning itself. UNESCO is very clear on this point: the use of generative AI must be prevented “when it would deprive learners of the opportunity to develop their cognitive abilities.” This isn’t just a warning. It’s a design requirement.
Four problems, then, and not a single one that can be solved by changing the model:
Bad news for those who are just waiting for the next model. And this is where software engineering truly shines. The following four initiatives address this.
We saw proof of this at a client’s site. Nexapp, in collaboration with Collège Sainte-Anne, developed a proofreading assistant named Emilia, designed to analyze texts according to the criteria of the Quebec education system. According to metrics reported by the Emilia team, French proofreading is completed three times faster.
But the most telling figure lies elsewhere—in the turnaround time. Before, grading took longer, and feedback didn’t come until two weeks later, by which time the student had forgotten their essay. Now, the feedback reaches them while the material is still fresh in their memory. That’s when learning happens.
This benefit doesn’t come from the model’s power. It stems from an architectural choice: Emilia suggests a correction, and the teacher validates, adjusts, or refines it based on the concepts covered in class. Pedagogical control never leaves the classroom. The AI suggests; the human decides. Take away that decision, and all you’re left with is an automatic grader that nobody wants—just like the tool our teacher had closed.
This shift—from the power of the model to the design of the interaction—also applies to pedagogy. If a machine can write an essay in a matter of seconds, the essay no longer proves anything. What we assess must therefore change: no longer just the ability to produce a text, but what it contains and the process that led to it. This is what UNESCO refers to as the professional competencies needed to work with AI.
At Laval University, the guidelines for using AI in assessments already require a mandatory declaration of usage, with the queries used and the generated content included as an appendix. The prompt is part of the result.
Other institutions had already responded, without waiting for ChatGPT. Design-Based Education, championed in particular by the Finnish university HAMK, has multidisciplinary teams of students work on real-world challenges from the workplace, with an outcome that is unknown at the outset. In a project like this, the question “What if a robot wrote that?” loses much of its bite.
That’s not to say it’s a miracle, though: monitoring and validating student learning places a much heavier burden on faculty. What if the key question were: how do we shift from teaching to guiding?
This is a question that a software development company like Nexapp knows firsthand. Managing ambiguity, breaking down a vague problem into iterations, leading a team without dictating the solution, and making progress visible even as the destination keeps shifting—this is the daily reality of an agile team. We don’t know how to teach French—even if we have Mylène Truchon, a developer with a master’s degree in literature. What we do know how to do, above all, is equip a technological transition of this kind. It remains to be seen how to orchestrate it through software.
UNESCO is explicit about the method: the use of generative AI should be designed jointly by teachers, learners, and researchers, following a six-perspective framework that ranges from subject-matter relevance to ethical risks. It is worth noting that there is no mention of building a model. All the meaningful work takes place between existing models and the classroom.
This work is divided into four areas of focus.
From our professional perspective, this framework describes a product discovery workshop. Short cycles, rapid feedback loops, prototypes tested by real teachers in real classrooms. And integration with existing systems, such as Moodle, Canvas, and existing portals, rather than yet another tool to open every morning. Every additional tool adds to the cognitive load of a teaching staff that already has more than enough on its plate. This is one of the main reasons why software ends up unused—even before quality issues come into play.
Two of these problems—hallucination and homogenization—have the same cause: the model draws on its own accumulated knowledge rather than on school standards. There is a costly way to correct this and a reasonable way.
UNESCO devotes a section to EdGPTs—foundation models fine-tuned with smaller amounts of high-quality educational data. This approach is viable, and projects are already underway, such as EduChat by East China Normal University and MathGPT by the TAL Education Group. When UNESCO published its guidelines, these efforts were still in their infancy.
No one should therefore wait for the perfect educational model before taking action—and fortunately, we don’t need one. Retraining a model is expensive and remains out of reach for virtually all educational institutions. Anchoring an existing model to local knowledge, on the other hand, is a matter of routine integration—what is known as document anchoring, or RAG: we connect the generator to the school’s corpus—its curriculum and grading rubrics—and require it to cite the sources on which it relies.
This access to a school’s corpora and student work requires making decisions up front: what data is necessary for a defined educational purpose, where it is processed, how long it is retained, and whether the provider can reuse it. In Quebec, Bill 25 regulates part of this, to be validated on a project-by-project basis. Confidentiality is not a feature added at the end of a project; it is an architectural constraint.
The advantage? The model remains general-purpose and replaceable should a better one emerge. It is the context that becomes local and autonomous. We’re moving from a passive, non-critical acquisition process to strategic planning. This grounding also prevents educational responsibility from shifting to a vendor: teaching staff remain in control of their own instruction.
Anchoring does not make the model explainable; no one will ever see its internal reasoning. It makes its answers verifiable, which is not the same thing—but it is more than enough.
From a product perspective, this results in interfaces that show the actual path leading to a suggestion, the evaluation criteria applied, and the distinction between what is extracted from a document and what is generated. Our teacher can then challenge a grading decision, knowing exactly what it’s based on—which is essential for her to continue using the tool the following week.
Anchoring reduces errors without eliminating them. The system may retrieve the wrong document, rely on an outdated corpus, or distort a valid source. This is where the engineering work comes into play: evaluation datasets, a rule to refrain from scoring when the corpus does not contain the answer, logging, and regression tests following model updates.
That leaves the most serious problem—dependence—and none of the three previous approaches addresses it. Neither co-design, nor grounding, nor verifiability determines what the student does in their own mind. It all comes down to interaction.
A tool that provides the answer short-circuits the effort. A tool that asks questions to guide the student’s thinking stimulates that effort. In practice: require the student to make an attempt before offering a suggestion; limit the amount of information the assistant provides at once; log and prioritize the learning process over the result; and transform a good Socratic prompt into reusable software code rather than leaving it in a Word document.
But we still need to decide what to measure: the time to feedback, the percentage of suggestions that teachers reject, actual usage after sixty days, and the number of attempts a student makes before asking for help. An educational tool succeeds not because it makes a strong impression during a demonstration, but because it improves a defined outcome without hindering autonomy.
This last area is also the least well-defined: everything is still decided at the design stage. Three profiles are emerging, however, which UNESCO cautiously classifies as “potential but unverified uses.”
|
Profile |
Role |
Student Prerequisites |
Risks to Avoid |
|---|---|---|---|
|
The socratic challenger |
Ask the student about what they already know, to help them go further |
Sufficient age, prior knowledge, ability to identify a false argument |
The echo effect: standard answers instead of opposing viewpoints |
|
The math or code tutor |
Immediate feedback, at the student’s own pace |
Genuine motivation and the foundations of language |
Feedback that is sometimes incorrect, and computational thinking that no longer develops |
|
The research advisor |
Helping to frame a question, suggesting a method |
Knowing the subject well enough to spot a fabricated source |
Fictitious publications, and copy-and-paste replacing trial and error |
The right-hand column reiterates, one last time, the problems from the very beginning: hallucination, homogenization, and the disappearance of effort. A better model would mitigate some of these issues, though likely never completely. The rest hinges on the software that surrounds the model—on what it allows, what it displays, and what it refuses to do on the student’s behalf.
UNESCO puts it bluntly: AI tools should be designed to expand or enhance human intellectual capacities and social skills, not to weaken them, conflict with them, or usurp them.
So, let’s return one last time to our teacher. Her question, in 2022, was, “Did this student cheat?” In 2026, her question has changed. She wants to know whether the tool on her desk makes her better able to do her job, or whether it simply shifts her work elsewhere. And this question isn’t decided at the model level. It’s decided through a series of software decisions: who has the final say, what the system allows to be verified, what knowledge it draws upon, and how much effort it leaves to the student.
The current model will change several times over the system’s lifetime. These decisions, however, must survive each of those changes.
As our colleague Alexandre Rivest writes, the essence of software development lies in its ability to combine technical expertise with humanity. From now on, the same will be true for learning as well.
Would you like to evaluate an AI use case at your institution or validate the architecture of an existing educational solution? Nexapp supports organizations in designing, securing, and integrating AI solutions tailored to their actual needs. Contact us.