{
  "id": "litellm-pypi-compromise",
  "url": "https://ohshit.app/incidents/litellm-pypi-compromise/",
  "title": "LiteLLM 1.107.0 on PyPI and Docker was a credential stealer with a remote-code loader, built from a lookalike repo",
  "level": 4,
  "level_name": "septic-tank-blew",
  "date": "2026-10-21",
  "date_is_approximate": false,
  "updated": null,
  "tags": [
    "supply chain",
    "PyPI",
    "Docker",
    "AI",
    "secrets"
  ],
  "ecosystems": [
    "pypi",
    "docker"
  ],
  "summary": "Attackers used stolen GitHub CLI tokens from three LiteLLM engineers to publish litellm 1.107.0 to PyPI and Docker from a lookalike repo. It steals cloud, AI and LiteLLM proxy keys on every Python start. Live ~2 hours.",
  "splash": "Anyone who installed litellm 1.107.0 or pulled latest / main-latest / main-stable / v1.107.0 images on Oct 21",
  "wipe": "Rebuild affected hosts from scratch on litellm 1.106.2 pinned by hash/digest, then rotate every credential the host could read",
  "in_the_wild": "yes",
  "source_exception": null,
  "cves": [],
  "fixed": [
    {
      "product": "LiteLLM (PyPI)",
      "version": "1.106.2",
      "status": "Last clean release; 1.107.0 removed"
    },
    {
      "product": "LiteLLM Docker images",
      "version": "v1.106.2 (pin by digest)",
      "status": "Floating tags were repointed"
    },
    {
      "product": "LiteLLM Enterprise (private registry)",
      "version": "not affected",
      "status": "Per LiteLLM"
    }
  ],
  "checklist": [
    {
      "title": "Find every 1.107.0 install",
      "hint": "requirements, uv.lock, poetry.lock, Pipfile.lock, CI caches, wheelhouses, and any container started from latest/main-* on Oct 21. Grep for litellm-1.107.0.dist-info"
    },
    {
      "title": "Check images by digest, not tag",
      "hint": "Bad index is sha256:1bbcb1443bc9…c339e (litellm); litellm-database and litellm-non_root had their own bad digests. The tag looks innocent now"
    },
    {
      "title": "Don't pip install your way out",
      "hint": "The .pth runs on every Python start, pip included. Isolate the host, kill the daemon holding /tmp/remote_code_loader.lock, then rebuild from scratch"
    },
    {
      "title": "Rotate every key the process could read",
      "hint": "All provider keys in the LiteLLM DB, master and virtual keys, the DATABASE_URL password, instance/task IAM roles, env vars, SSH/git/npm/PyPI/kube creds. Yes, all of them"
    },
    {
      "title": "Hunt the IOCs",
      "hint": "Egress to litellm[.]dev, raw.githubusercontent.com/letters9813/…; files cook.pth, cook.py, sitecustomize.py, /tmp/.pc_once_72129118c59f5636c0bb; env PRIVCOLLECT_ACTIVE=1"
    },
    {
      "title": "Check cloud audit logs",
      "hint": "CloudTrail GetCallerIdentity from your instance role near install time, and any use from 168.100.8.57 or 165.227.201.8"
    },
    {
      "title": "Pin to 1.106.2 by hash and stop using floating tags",
      "hint": "Hash-pinned lockfiles and image digests would have made this a non-event. The proxy was supposed to be the only middleman"
    }
  ],
  "sources": [
    {
      "name": "LiteLLM incident report",
      "url": "https://docs.litellm.ai/blog/security-incident-oct-21-2026",
      "says": "Three engineers' GitHub CLI OAuth tokens used to bypass branch protection, approve and publish from Berri-AI/litellm; initial access suspected via an internet-exposed internal deployment (unconfirmed); 1.106.2 safe; enterprise unaffected",
      "primary": true
    },
    {
      "name": "GitHub issue BerriAI/litellm#47826",
      "url": "https://github.com/BerriAI/litellm/issues/47826",
      "says": "Reporter's analysis; all 7 wheels affected, payload not in the v1.107.0 git tag; Docker tags yanked by 17:13 UTC",
      "primary": true
    },
    {
      "name": "Socket",
      "url": "https://socket.dev/blog/litellm-compromise",
      "says": "Full payload breakdown, wheel and file hashes, image digests; flagged wheels ~7 min after publish",
      "primary": false
    },
    {
      "name": "JFrog Security Research",
      "url": "https://research.jfrog.com/post/litellm-hijacked-again/",
      "says": "Exfil to www.litellm[.]dev/healthcheck, hourly dead-drop loader; no evidence linking this to the March TeamPCP compromise",
      "primary": false
    },
    {
      "name": "Wiz",
      "url": "https://www.wiz.io/blog/litellm-compromised-steals-credentials",
      "says": "Observed stolen AWS role credentials validated from 168.100.8.57 and 165.227.201.8; PyPI quarantine at 13:20 UTC",
      "primary": false
    },
    {
      "name": "Datadog Security Labs",
      "url": "https://securitylabs.datadoghq.com/articles/litellm-compromised-pypi-docker/",
      "says": "Live roughly two hours; no second-stage payload observed in the wild",
      "primary": false
    }
  ],
  "headline": "LiteLLM, round two: your LLM gateway now proxies your keys to someone else",
  "one_liner": "A hijacked release workflow built LiteLLM from an attacker's copy of the repo. The wheel drops a .pth file, so every Python process on the box, even pip, runs a stealer that ships your secrets to a fake litellm.dev and starts a daemon that can fetch and run more code.",
  "in_the_wild_note": "Wiz saw stolen AWS role credentials used from attacker IPs hours later",
  "mitigation": null,
  "timeline": [
    {
      "date": "Oct 13, 2026",
      "title": "Lookalike domain litellm.dev registered",
      "body": "Per JFrog's WHOIS data."
    },
    {
      "date": "Oct 21, 2026, 08:56–10:46 UTC",
      "title": "Payload committed to the lookalike repo Berri-AI/litellm",
      "body": "Wiz counts 10 commits; the first three add cook.pth, cook.py and sitecustomize.py."
    },
    {
      "date": "Oct 21, 2026, 10:08 UTC",
      "title": "Branch protection overridden on the private release workflow",
      "body": "An org-owner's stolen GitHub CLI token disables protection and points the workflow at the lookalike repo."
    },
    {
      "date": "Oct 21, 2026, 11:11 UTC",
      "title": "A second stolen credential approves the production release"
    },
    {
      "date": "Oct 21, 2026, 11:39–11:40 UTC",
      "title": "Seven malicious litellm 1.107.0 wheels uploaded to PyPI",
      "body": "Socket's scanner flags them about seven minutes later."
    },
    {
      "date": "Oct 21, 2026, 11:46 UTC",
      "title": "Malicious images published to GHCR and Docker Hub",
      "body": "v1.107.0, latest, main-latest and main-stable all point at the same bad image index."
    },
    {
      "date": "Oct 21, 2026, 11:53 UTC",
      "title": "A third stolen credential deletes the workflow run records"
    },
    {
      "date": "Oct 21, 2026, ~13:20 UTC",
      "title": "PyPI removes 1.107.0",
      "body": "Wiz and Datadog say 13:20; LiteLLM's report says 13:23."
    },
    {
      "date": "Oct 21, 2026, 13:51–14:20 UTC",
      "title": "Docker tags pulled or repointed to clean images",
      "body": "GHCR first, Docker Hub by 14:20 per LiteLLM. Wiz and Datadog put removal at ~13:53."
    },
    {
      "date": "Oct 21, 2026, ~16:20–17:01 UTC",
      "title": "Stolen AWS instance-role credentials used from attacker IPs",
      "body": "Observed by Wiz in customer environments, about 4.5 hours after exfiltration."
    }
  ],
  "body_markdown": "LiteLLM is the Python SDK and proxy that sits between apps and dozens of LLM providers, so it tends to run with\nevery provider key a company has. On **October 21, 2026** a malicious `litellm==1.107.0` landed on PyPI at\n11:39 UTC, and the official Docker images under the `v1.107.0`, `latest`, `main-latest` and `main-stable` tags\npointed at a poisoned build from 11:46 UTC. This is LiteLLM's second release-pipeline compromise this year,\nafter 1.82.7 and 1.82.8 in March. JFrog and Datadog say nothing links the two so far.\n\n### How it got out\n\nAccording to [LiteLLM's own report](https://docs.litellm.ai/blog/security-incident-oct-21-2026), the attacker\nheld GitHub CLI OAuth tokens belonging to three engineers. One belonged to an organization owner and was used to\noverride branch protection on the private release workflow and point it at `Berri-AI/litellm`, a lookalike repo\none hyphen away from the real `BerriAI/litellm`. A second token passed the production approval gate, and a third\ndeleted the workflow run records afterwards. The real `v1.107.0` git tag is clean. The artifacts were built\nfrom the attacker's copy and signed by the real pipeline. LiteLLM suspects the tokens leaked through an\ninternet-exposed internal deployment, possibly via a recently patched bug, but **has not confirmed** the initial\naccess.\n\n### What the payload does\n\nAll seven 1.107.0 wheels carry three extra files: `cook.pth`, `cook.py` and `sitecustomize.py`. The source\ndistribution does not. Because of the `.pth` hook, the stealer runs at **every Python start** in that\nenvironment, even if nothing imports LiteLLM and even when you run `pip`. Per Socket and JFrog it collects:\n\n- cloud credentials (AWS, Azure, GCP), instance-metadata tokens, and an STS `GetCallerIdentity` check\n- Kubernetes, Docker, Vault, Terraform and CI secrets; SSH, GPG and TLS keys; git, npm and PyPI configs\n- AI tool credentials (Claude, Codex, Gemini, Cursor, Copilot, Hugging Face and others)\n- browser databases, shell history, and every readable process environment on Linux\n- if `DATABASE_URL` is set, the LiteLLM proxy's own `LiteLLM_CredentialsTable` and `LiteLLM_ProxyModelTable`,\n  which hold your provider keys\n\nThe data is encrypted (AES-256 with an RSA-4096-wrapped key) and POSTed to `www.litellm[.]dev/healthcheck`.\n`sitecustomize.py` also forks a detached daemon that checks a GitHub dead drop every hour for a domain, then\ndownloads and `exec()`s Python from it with no integrity check. The dead drop was empty when researchers looked\nand has since been removed, but the stealer part does not depend on it.\n\n### Is it being used?\n\nYes. Wiz observed stolen AWS instance-role credentials being validated from `168.100.8.57` and\n`165.227.201.8` roughly 4.5 hours after exfiltration, and IAM enumeration in \"a small number\" of\nenvironments. Datadog reports no second-stage payload seen in the wild. No source gives a download count for\n1.107.0; the only figure cited is LiteLLM's ~23 million weekly downloads overall, which is not the exposure.\n\n### Why L4\n\nA package that by design holds every LLM provider key, a payload that steals cloud and AI credentials on any\nPython start, and confirmed attacker use of stolen credentials: that is active exploitation with mass credential\ntheft. It stays below L5 because it was a single release live for about two hours, not self-propagating, and\nno core runtime or trust anchor broke.\n\n### What's still unclear\n\n- **Exact windows.** LiteLLM says PyPI removal at 13:23 and Docker Hub cleaned by 14:20 UTC; Wiz and Datadog say\n  13:20 and ~13:53; the GitHub issue reported Docker tags still bad at 13:23 and yanked by 17:13. If you pulled a\n  floating tag anywhere in that afternoon, check the digest.\n- **Initial access** to the engineers' tokens is suspected, not confirmed.\n- **No GHSA or CVE** has been issued yet. Wiz tracks it as `wiz-adv-2026-188`, JFrog as `XRAY-955589`.\n- LiteLLM mentions an embargoed enterprise fix due for disclosure on October 27; it is not clear whether it is\n  related."
}