Two AIs face a language they do not know

Introduction

What does a generative model do when it encounters words whose meanings it genuinely does not know? An experiment using Crestignac’s vocabulary reveals two very different strategies and raises an essential question: how can we distinguish, within an answer, what comes from the source, from inference or from invention?

A deliberately simple experiment

The same image was presented to two artificial intelligences, with no additional information.

It depicts a demonstration by young Crestignac residents being interviewed by journalists. Several placards are visible, and two demonstrators speak:

« Fako, that mocks the foam off my style: let them balibulase every morning, we’re not counting two or three Crestins.»

and:

« Fako, that oozes foujarmine that smells of a minister’s skull.»

For a model unfamiliar with Crestignac, several elements are therefore unknown: Fako, balibulase, Crestins, foujarmine and the expression to smell of a minister’s skull.

Yet the image provides a wealth of context: a demonstration, uniforms, journalists, demands, young people speaking and a reference to an institution.

In other words, the model does not know what the words mean, but has enough clues to construct a plausible interpretation.

This is precisely where the experiment becomes interesting.

A constructed corpus rather than a string of invented words

The vocabulary used in this experiment is not a sequence of randomly generated words designed to trip up the models.

It comes from the Dictionnaire de la Duchesse, Crestignac’s dictionary:

https://dico.crestignac.com

Its distinctive feature is that it develops gradually alongside the world itself. A word may emerge in an article, a situation, a character or a usage. It is then defined, reused, sometimes given variant forms, and gradually connected to other concepts.

The Dico is therefore not built before Crestignac: it takes shape alongside it.

Usage produces words; the words are recorded in the dictionary; they can then reappear in other situations and contribute to new usages.

This process gradually produces internal coherence.

A Crestin is not merely a word that sounds as though it belongs to Crestignac: it is a unit of currency. Phaco / Fako has a particular use in the speech of young Crestignac residents. A foujarmine has its own definition. Likewise, « to smell of a minister’s skull» belongs to a set of expressions whose meanings have developed within this world.

A few entries provide direct access to this source of truth:

Dictionary: https://dico.crestignac.com

Phaco: https://dico.crestignac.com/mot/phaco

Phacochérisme: https://dico.crestignac.com/mot/phacocherisme

The other terms cited in the experiment should be linked to their Dico entries at publication, wherever their canonical URLs are available.

This construction is essential to the experiment.

An AI can produce an interpretation that differs from the dictionary’s and nevertheless build something remarkably coherent around it.

What it produces is not necessarily absurd.

It produces a different coherence.

The presence of coherence in the model’s answer therefore does not demonstrate that it has recovered the coherence of the corpus.

AI A: acknowledging uncertainty

The first AI identifies the problem fairly quickly.

It suggests some interpretations based on context, but notes that several terms are unfamiliar and that it cannot assign them a definite meaning.

It is then given Crestignac’s web address and told about its dictionary.

It searches for the terms, finds several definitions and corrects part of its initial analysis.

Not everything is perfect. Some associations remain incorrect, and a few interpretations still go beyond what the source actually says.

But its strategy changes: the initial uncertainty becomes information to be addressed.

AI B: immediately filling in the world

The second AI adopts a very different strategy.

Faced with the same unfamiliar terms, it immediately tries to connect them to a linguistic system it already knows.

From this first hypothesis, it gradually builds an entire explanatory framework: supposed word origins, meanings, usages and cultural context.

The whole is remarkably coherent.

The problem is that the initial premise is false.

The words belong to Crestignac’s vocabulary, and several have precise definitions in its dictionary.

The AI has therefore not recovered their meaning.

It has built a system capable of giving them one.

And that system is coherent enough to make the error difficult to detect.

Access to the source does not necessarily solve the problem

The second part of the experiment is perhaps the most interesting.

Once directed to the actual source, AI B abandons its initial hypothesis. It consults the corpus and retrieves several accurate details.

One might think the problem solved.

It is—but only partly.

Because between the pieces of information actually present in the corpus, the model sometimes continues to establish relationships that are not there.

It no longer necessarily fabricates the data itself.

It fabricates the links between the data.

This is an important distinction when discussing grounding.

Connecting a model to a document repository, a search engine, a RAG system or a domain-specific source does not guarantee that its entire answer comes from that source.

The source may provide elements A and B.

The model may then decide that A implies B, even though this relationship has never been established in the corpus.

Plausibility and truth

This experiment deliberately uses a constructed vocabulary.

That is what makes it particularly observable: because the meanings have been defined in an identifiable corpus, it becomes possible to compare what the model says with what actually exists in the source.

But the mechanism observed extends far beyond Crestignac.

A generative model is extremely effective at producing a coherent continuation from incomplete elements.

This is precisely one of its strengths.

It is also one of its challenges.

An explanation can be perfectly structured without being grounded.

The more coherent, seemingly well-documented and linguistically convincing it is, the harder it becomes for the user to distinguish retrieved information from an inferred relationship or an invented detail.

From a correct answer to a traceable answer

In occasional use of artificial intelligence, checking whether an answer seems correct may be enough.

In a professional system, this approach quickly reaches its limits.

When AI is involved in reviewing a case file, synthesising documents, supporting decisions or making use of business data, another question becomes essential.

It may no longer be enough to ask:

« Did the AI give a good answer?»

We must also be able to ask:

« Can I trace what, in its answer, comes from the source, from inference… or from invention?»

This distinction directly concerns grounding, hallucination, traceability and, ultimately, the trust we can place in a generative system.

The challenge, then, may not simply be to build AIs that answer correctly.

It is also to build systems in which we can know why we consider their answer correct.

Illustration: the image of the Crestignac demonstration used in the test, showing the two young interviewees and their speech bubbles.