MosaicLeaks tests secret leakage in AI agents

With MosaicLeaks, Hugging Face shines a spotlight on a blind spot in AI agents

The race for AI agents has long been framed as a performance competition: better planning, better tool execution, better ability to navigate the web, summarize documents, or orchestrate complex workflows. But as these systems move beyond demos to connect to real environments — messaging systems, internal documents, knowledge bases, CRMs, research tools — another question is becoming central: can an agent keep a secret?

That is precisely the ground targeted by MosaicLeaks, a benchmark presented by Hugging Face in a publication titled “MosaicLeaks: Can your research agent keep a secret?”. The initiative focuses on a very concrete risk of the agent era: the leakage of sensitive information when a system has rich context, access to tools, and instructions that are sometimes contradictory or indirectly malicious.

The subject is far from theoretical. In companies, agent deployments are accelerating around use cases where the value comes precisely from access to internal data: document research, support team assistance, note preparation, competitive intelligence, decision support, exploration of private corpora. The more useful an agent is, the more connected it is to sensitive sources. And the more connected it is to those sources, the more foundational the question of confidentiality becomes.

Hugging Face’s choice to formalize this problem in the form of a benchmark is revealing of a market shift. The evaluation of models and agentic systems is no longer limited to task success. It must also measure what an agent must not do: reveal a password, expose a confidential excerpt, transmit information present in its context but absent from the user’s legitimate request, or allow itself to be manipulated by an instruction hidden in an external source.

For a player like Hugging Face, historically associated with open source, datasets, models, and benchmarks, this publication fits into a logical continuity. The company has contributed significantly to making AI system evaluation more transparent and more reproducible. With MosaicLeaks, it applies that evaluation culture to a security issue that is becoming a priority as the industry pushes agents into production at scale.

The benchmark’s appeal also lies in its framing. It is not just a matter of asking whether a model “hallucinates” or whether it correctly follows an instruction. It is about examining how an agent behaves when several layers of information coexist: a user objective, a working context, consulted documents, intermediate tools, and implicit instructions slipped into that environment. In other words, MosaicLeaks positions itself exactly where the promises of agents meet operational risks.

In the French-speaking context, the angle resonates particularly strongly with the concerns of large companies, public administrations, and regulated sectors. In France as in Europe, the question is not only whether an agent is high-performing, but whether it can be deployed without endangering trade secrets, contractual information, business data, or elements subject to compliance obligations. In that respect, Hugging Face’s announcement goes beyond a simple academic exercise: it puts words and a methodology to a concern already very present in IT and cybersecurity departments.

What MosaicLeaks specifically measures according to Hugging Face

According to Hugging Face’s original publication, MosaicLeaks is designed to evaluate whether research agents disclose sensitive information. The benchmark therefore targets a specific scenario, but one that is increasingly representative: that of an agent tasked with searching, reading, synthesizing, and answering based on a set of resources potentially mixed with confidential content.

The important point is that Hugging Face does not present leakage as a simple isolated bug. The benchmark targets a systemic risk linked to the very architecture of modern agents. These systems constantly handle several types of context:

• conversation context, with user requests and conversation history;
• document context, drawn from files, notes, web pages, knowledge bases, or search results;
• tooling context, coming from APIs, search engines, browsers, or internal tools;
• system or developer instructions, which define behavioral rules, priorities, and limits.

Leaks arise at the intersection of these layers. An agent may, for example, be led to consult a document containing confidential information while also having to answer a question that, on the face of it, does not authorize its disclosure. It may also be confronted with an external source that tries to make it exfiltrate data it has access to. The problem is therefore not only “model security,” but the governance of context in a complete agentic system.

The very wording of the title chosen by Hugging Face — “Can your research agent keep a secret?” — clearly shows the shift in the debate. It is no longer just about measuring an agent’s apparent intelligence, but its ability to respect an informational boundary. That boundary is fundamental in professional uses: a good agent is not one that answers at all costs, it is also one that knows when to stay silent.

The benchmark highlights leakage vectors that speak directly to product and security teams. The publication notably cites the risks of disclosure through context, tools, and indirect instructions. This triad is particularly relevant.

Leakage through context occurs when sensitive information present in the agent’s working memory or in the documents it has consulted ends up being repeated in a response when it should not have been. Leakage through tools can appear when an agent uses a connector or an API that gives it access to broader data than necessary, and then reinjects that data into its output. Finally, leakage through indirect instructions refers to a family of attacks that has become widely discussed since the rise of agents: malicious instructions hidden in documents, pages, or external results, capable of diverting the system’s behavior.

The fact that Hugging Face chose to formalize this last point is notable. The industry has talked a lot about prompt injection at the chatbot level. But in the case of research agents, the stakes rise another notch: the system does not merely answer a user, it traverses sources, extracts signals from them, calls tools, then acts or writes on that basis. A hostile instruction buried in that flow can therefore have broader effects than a simple erroneous response.

The benchmark thus targets a category of systems that is now at the heart of product roadmaps: agents capable of reading, comparing, synthesizing, and navigating. By subjecting them to leakage scenarios, Hugging Face reminds us that a research agent is not just a productivity engine. It is also a concentration point for information risk.

Hugging Face’s original publication also has the merit of framing the subject in simple language, understandable well beyond the circle of security researchers. “Keeping a secret” is an accessible formulation, but it covers a deep technical difficulty: how do you ensure that a system that has seen information does not reuse it outside its authorized scope? This question touches on privilege separation, instruction hierarchy, memory, output filtering, and the design of the tools themselves.

In other words, MosaicLeaks does not describe a single flaw. The benchmark instead serves as a framework for observing a set of undesirable behaviors in realistic configurations. That is what makes it an important signal for the market: agent security can no longer be thought of as a simple module added afterward. It must be evaluated as a fundamental property of the system.

Why this benchmark is arriving at a pivotal moment for the agent industry

The timing of MosaicLeaks is no accident. For about two years now, the generative AI ecosystem has gradually shifted its center of gravity. After the initial fascination with large conversational models, the market turned toward agentic applications: assistants capable of chaining steps together, using tools, traversing sources, automating research, and interacting with software environments.

This evolution has been driven by major labs, software vendors, and open source alike. Demonstrations of web agents, domain-specific copilots, and autonomous workflows have multiplied. In this dominant narrative, the main indicator was often capability: how many steps can the agent execute, how much autonomy can it achieve, what quality of synthesis or planning can it offer?

But the more autonomy agents gain, the more the attack surface expands. A traditional chatbot already raises security questions. An agent connected to emails, HR documents, legal files, financial spreadsheets, or collaborative spaces raises even more. It does not merely generate text from a prompt: it moves through an informational environment where every access can become a disclosure risk.

That is where MosaicLeaks takes on a strategic dimension. Hugging Face is not just trying to publish yet another benchmark. The company is pointing to a change in the criterion of maturity. For a long time, the evaluation of AI systems focused on accuracy, robustness, toxicity, bias, or the ability to follow instructions. With agents, it is now necessary to evaluate the system’s informational discipline.

This concern aligns with discussions already very present in cybersecurity. In a traditional environment, the goal is to limit privileges, compartmentalize access, trace actions, and prevent exfiltration. AI agents are being inserted into this landscape, but with one particularity: they handle natural language, aggregate heterogeneous content, and make decisions based on instructions that are sometimes ambiguous. Traditional security mechanisms remain necessary, but they are not always sufficient to anticipate an agent’s emergent behaviors.

Hugging Face’s benchmark is also arriving as companies actively experiment with internal research agents. This type of system is particularly attractive: it promises to save time by retrieving information scattered across several silos. Yet that is precisely what makes it sensitive. The better an agent is at cross-functional search, the more likely it is to encounter data that should never have been brought together in the same response.

The subject is all the more current because the industry has heavily promoted connectivity to tools and data as a commercial argument. The promises are well known: retrieve the right information at the right time, automate monitoring, produce reliable syntheses, reduce friction between applications. But this logic has a downside: every additional connector potentially increases the volume of exploitable context, and therefore the probability of an unintentional or induced leak.

Hugging Face steps in here with a message that usefully contrasts with the surrounding enthusiasm. Where part of the market emphasizes autonomy, MosaicLeaks reminds us of the need for control. Where many demonstrations show agents capable of “reading everything,” the benchmark asks whether they also know how not to repeat everything. This reversal of perspective matters, because it shifts perceived value: in a professional setting, trust is based not only on competence, but on restraint.

This announcement can also be read as a sign of the open-source ecosystem’s maturation. Hugging Face has often served as a meeting point between academic research, industry, and the developer community. By highlighting a secret-leakage benchmark, the platform is helping spread a security culture at a time when many developers are assembling agents themselves from open components. In this context, the existence of dedicated evaluation tools can weigh heavily on best practices.

Finally, the moment is pivotal because decision-makers are beginning to distinguish more clearly between two questions that were once conflated: “can the agent do the job?” and “can the agent do that job without exposing the organization?” The first dominated experimentation cycles. The second is becoming decisive for scaling. MosaicLeaks fits exactly into this shift.

An approach that stands apart from benchmarks centered on pure performance

The AI benchmark landscape is dense, but most of them have historically measured positive capabilities: answering correctly, reasoning properly, coding, summarizing, translating, planning, or using tools. These are indispensable metrics, but they sometimes leave in the shadows an essential dimension of deployed systems: their ability to resist adversarial uses or booby-trapped contexts.

With MosaicLeaks, Hugging Face emphasizes a negative but decisive property: not disclosing. This orientation is not entirely new in absolute terms, because model security and prompt injection attacks are already the subject of much work. However, the interest of this benchmark is that it explicitly anchors the issue in the case of research agents, that is, systems that combine access to context, navigation, and tooling.

This nuance matters. A model evaluated in the lab on isolated prompts does not always reflect the behavior of an agent embedded in a product. As soon as you add memory, connectors, documents, a browser, or a search engine, the vulnerability space changes. The risks no longer come only from the model’s response, but from the path information takes before reaching that response.

Hugging Face’s benchmark thus draws attention to what could be called the compositional security of agents. A component taken separately may seem reasonably secure. But once combined with tools, external sources, and instruction-prioritization rules, the overall system may behave in unexpected ways. It is precisely this kind of gap that companies often discover late, when a prototype moves to the pilot or production stage.

Compared with competing announcements on the market, MosaicLeaks’ originality lies less in the promise of a stronger agent than in the desire to equip the measurement of a risk. Many players today highlight guardrails, policy layers, filters, or control architectures. But it is rarer to see the question framed as a public benchmark centered on a use case as concrete as the research agent. This gives the initiative methodological value: it invites developers and companies to systematically test what they often assume they intuitively control.

Hugging Face’s approach also has cultural significance. In the AI industry, performance rankings have long served as a compass. They guide technical choices, marketing narratives, and sometimes investments. By introducing a benchmark focused on secret leakage, Hugging Face is helping shift attention toward criteria of operational reliability. For teams building agents, this can change the very definition of a “good” system.

In practice, an agent that achieves excellent results on research tasks but fails to preserve sensitive information can become unusable in critical professional contexts. Conversely, an agent that is slightly less ambitious in raw autonomy but more robust on confidentiality may prove far more deployable. MosaicLeaks therefore provides a framework for objectifying a trade-off that is often implicit.

This orientation toward leakage evaluation is also important for buyers. Innovation departments and business teams may be won over by productivity demonstrations. CISOs, lawyers, and compliance leaders ask another set of questions: what data does the agent see, what does it retain, what can it repeat, how does it react to a malicious document, what happens if an external source tries to divert its instructions? A benchmark like MosaicLeaks brings these two worlds closer together by turning qualitative concerns into a concrete evaluation object.

Finally, it is worth emphasizing what this announcement says about the market’s current stage. When a technology is nascent, benchmarks focus mainly on feasibility. As it matures, they increasingly focus on safety, governance, and robustness. In that sense, MosaicLeaks is a marker of maturity: the agent industry is beginning to accept that performance is only part of the problem.

Direct implications for French and European companies

For French-speaking organizations, Hugging Face’s publication lands on particularly sensitive ground. In France as in the rest of Europe, enterprise generative AI projects often move forward under a double constraint: gaining productivity without weakening control over data. Research agents, precisely because they promise strong value on internal corpora, find themselves at the heart of this tension.

In many companies, the information useful to an agent is scattered across several environments: collaborative suites, intranets, document management systems, CRMs, messaging systems, support tickets, office documents, legal or technical databases. The main value of an agent lies in linking these silos. But that cross-functional capability immediately clashes with the compartmentalization principles that organizations have spent years building.

MosaicLeaks therefore highlights a very concrete question for CIOs and security teams: how can they ensure that an agent with access to several layers of information does not reconstruct, in a single response, a secret that should never have left its perimeter? This risk does not concern only personal data. It also affects trade secrets, commercial information, strategic elements, contractual clauses, R&D content, or internal exchanges.

The issue is particularly acute in regulated sectors or those heavily exposed to confidentiality: banking, insurance, healthcare, defense, industry, energy, consulting, legal, and the public sector. In these environments, the question is not only whether an agent can respond usefully, but whether its architecture respects requirements for minimization, traceability, and separation of access. A benchmark like MosaicLeaks does not replace an audit, but it provides a language and a starting point for testing realistic scenarios.

For the European market, Hugging Face’s announcement also fits into a broader sensitivity around digital sovereignty and AI governance. Without extrapolating beyond the original publication, it can be observed that companies in the region place particular importance on data localization, flow control, and compliance. In this framework, an agent’s ability not to disclose sensitive information becomes a purchasing criterion as important as the quality of its answers.

The benchmark may also interest integrators, consulting firms, and vendors that design agents for their clients. Many current projects rely on assemblies of components: model, orchestration, retrieval, connectors, tools, memory, access policies. This modularity is powerful, but it multiplies friction points. Having an evaluation framework oriented around “secret leakage” can help industrialize security testing from the design phases onward, rather than after an incident or a validation blockage.

For product teams, the lesson is just as direct. If an agent is connected to documents, emails, or internal databases, the product must be thought of not only as a conversation interface, but as an information-access system. Every search feature, every summary, every multi-source synthesis can become a point of unintentional exfiltration. MosaicLeaks reminds us that an agent’s UX quality must not make us forget the discipline of its permissions and outputs.

There is also an internal governance issue. In many organizations, agentic experiments begin in a business team before being taken over or supervised by IT. Hugging Face’s benchmark can serve as an argument for moving these pilots toward more structured practices: security test sets, indirect prompt injection scenarios, connector control, review of access policies, response filtering, logging, and supervision.

For the French economic fabric, made up of a large number of mid-sized companies and innovative SMEs, the message is also important. These companies do not always have the means to build red-teaming teams specialized in AI. A public benchmark focused on an identifiable business risk can offer them a practical basis for evaluating their agents without starting from scratch. As turnkey agent offerings multiply, this ability to ask the right security questions becomes a competitive advantage.

Finally, the impact may be cultural. The French-speaking debate on generative AI has often oscillated between fascination with productivity gains and general concern about data. MosaicLeaks helps make that concern more operational. It is no longer just a matter of saying “watch out for data,” but of testing how, concretely, an agent can be led to betray information through context, through a tool, or through an indirect instruction. This methodological precision is valuable for turning diffuse caution into clear technical requirements.

The coming battle: agents that are less demonstrative, but more reliable with secrets

MosaicLeaks’ significance goes beyond its mere existence as a benchmark. Hugging Face’s initiative suggests a deeper evolution in competition within applied AI. The next battle for agents will probably not be fought only over the ability to accomplish more tasks, but over the ability to do so under constraints: constraints of security, confidentiality, permissions, compliance, and governance.

This perspective matters because it could alter the hierarchy of product priorities. Until now, the dominant image of the agent has valued maximum autonomy: read everywhere, act everywhere, synthesize everything, accelerate everything. Yet an agent that is truly deployable in the enterprise will often have to accept the opposite: see less, ask for confirmation more often, refuse more, trace its accesses, ignore certain instructions, compartmentalize its knowledge. In practice, this may make agents less spectacular in demonstrations, but far more credible in production.

Hugging Face’s benchmark therefore arrives as a salutary reminder. As agentic systems gain depth of integration, the fault line between an impressive prototype and a trustworthy product becomes clearer. An agent that knows how to browse the web or summarize a file is not automatically ready for sensitive internal use. It must prove that it knows how to distinguish what it can use from what it can say.

In the long term, this logic could influence several layers of the market. First, the benchmarks themselves: it is likely that agent evaluation will increasingly incorporate scenarios of leakage, indirect manipulation, and permission management. Next, product architectures: agent designers will be pushed to better separate contexts, reduce privileges, and strengthen control mechanisms around tools. Finally, enterprise purchasing: RFPs and internal validations should give increasing weight to the demonstrated ability to preserve secrets.

For Hugging Face, MosaicLeaks fits into an increasingly important role: that of a player that does not merely distribute models and tools, but helps define the ecosystem’s criteria of maturity. In a sector where the pace of innovation is extreme, this framing function is far from secondary. It helps prevent the notion of progress from being reduced to a simple inflation of capabilities.

For French and European companies, the message is clear without needing to be dramatized. The challenge of AI agents is no longer just to connect a model to internal data. It is to build systems capable of working in contact with that data without letting it escape, including when external documents, tools, or adversarial instructions try to blur the rules. That is where a major part of trust will be decided.

From this perspective, MosaicLeaks acts as a revealer. It shows that agent security can no longer be treated as a simple box to check, nor as a subject reserved for red-teaming experts. It is becoming a central criterion of design, evaluation, and adoption. The most decisive next advances may not be those of the agents that speak best, but those of the agents that know, consistently, when to reveal nothing.