Managing Confidential Data and Transparency Policies in LLM-Empowered Science

April 24th, 2025

The Challenge

AI is transforming science. Large Language Models (LLMs) like GPT-4 and Claude are accelerating research by helping scientists generate hypotheses, analyse data, and even draft papers. The 2024 Nobel Prizes in Biology and Physics—awarded for breakthroughs in machine learning—show just how deeply AI is now embedded in scientific discovery.

But with great power comes great risk.

LLMs operate differently from traditional software. Unlike forms or APIs that request only predefined fields, LLM prompts can obtain unrestricted input—confidential data such as gene sequences, material formulas, proprietary algorithms, and more. Scientists may unknowingly expose confidential or IP-sensitive data in their casual, conversational queries.

And it gets riskier.

Many LLM-based systems integrate with external tools and APIs to perform actions autonomously. While this boosts productivity, it also fragments data control. Information may silently travel through third-party services, often without users realizing it—or knowing the privacy policies governing those services.

Although there’s been a technical push to protect Personally Identifiable Information (PII) using tools such as Presido, Laker AI inspired by GDPR and NIST guidelines, these don’t cover the full spectrum of scientific confidentiality. Proprietary research data falls outside traditional PII definitions, leaving a major privacy gap.

How can we make LLM systems not just powerful, but responsible—especially in high-stakes fields like scientific research? How do we detect and guard against leaks of sensitive data beyond PII?

The Research

To tackle the privacy and ethical challenges in LLM-powered scientific workflows, our research group built DataShield – an explainable framework that helps scientists understand what confidential data they might be exposing, how external tools handle their data, and whether it violates internal policy.

 

DataShield has three core modules, all working together to detect, explain, and visualize potential data risks:

  • Confidential Data Detection Module
    This module scans user prompts—like gene names, protein sequences, material information, proprietary algorithms—and flags anything that might be confidential. For example, when a user asks about a novel gene or submits a protein sequence, the system analyses it using:
    • Rule-based pattern matching for known sequence structures,
    • RAG-powered Local LLMs using knowledge bases like UniProt for gene/protein references,
    • And a human-in-the-loop method to flag organization-specific sensitive data using fuzzy matching.

It even handles indirect exposures—cases where sensitive information is implied, not explicitly stated. A simple color-coded risk system (red = high, yellow = medium, green = low) helps users quickly assess confidentiality levels. Users can provide feedback, and if needed, redact sensitive parts using placeholder tags like [GENE_NAME].

  • Policy Summarization Module
    Next up, DataShield summarizes complex privacy policies from both external tools and your organization’s internal code of conduct.

Here’s how:

  • We identify tools involved to execute the task mentioned in the user prompt.
  • Their privacy policies are scraped and processed in two layers:
    • PoliGraph: Extracts structured policy elements like data use, retention, third-party sharing, etc.
    • LLM + RAG: Refines this into human-readable “Privacy Nutrient Labels” – compact, informative cards with the essentials.

We do the same for internal policies to spot conflicts. The result? Clear, actionable insights about what each tool does with your data—and whether it aligns with your company’s standards.

  • Visualization Dashboard

The dashboard ties it all together. It shows:

  • What confidential data was detected and its sensitivity level.
  • Which tools were used to execute the task in the prompt.
  • Privacy summaries for each tool and their compliance or violations with internal policies.

This gives users a full picture of data flow, risk points, and next steps—without needing to dig through dense legal text.

Our work undergoes a comprehensive evaluation, combining quantitative module-based testing with qualitative insights from a multi-institutional user study involving scientists from diverse disciplines. This study assesses the framework’s usability, effectiveness, and trustworthiness in real-world research environments. Participants reported high satisfaction with the framework’s accuracy (Mean = 8.64, SD = 1.37) and overall usefulness (Median = 9). Notably, understandability (Mean = 8.46, SD = 1.26) and perceived trust (Mean = 8.11, SD = 1.71) were also rated positively.

Related Publications

  1. Shanmugarasa, Shidong Pan, Ming Ding, Dehai Zhao, Thierry Rakotoarivelo “Privacy Meets Explainability: Managing Confidential Data and Transparency Policies in LLM-Empowered Science” CHI 2025 (Accessible at https://dl.acm.org/doi/10.1145/3706599.3720099)

 

People

Yashothara Shanmugarasa

Ming Ding

Shidong Pan

Dehai Zhao

Thierry Rakotoarivelo