TL;DR
Fireworks AI, a popular inference provider, had two security incidents in 18 days. The second came through an unnamed third-party cloud provider that hosted Fireworks’ own R&D machines, and was disclosed in November. What leaked included billing data, resource metadata and, for some customers, secrets, datasets and “incomplete model weights.” Downstream, Notion confirmed that AI chat transcripts were exposed. The investigation is ongoing, so details may change. Most teams vetted their inference vendor like a SaaS subscription: a procurement form and a credit card. In practice it holds your API keys, your prompts, your fine-tuning data and sometimes your weights. That’s more than any npm package you depend on. Treat it like a dependency, with an inventory, scoped credentials, spend caps and a plan for the day it’s compromised.
Originally published at portfolio.hagzag.com.
Two incidents, 18 days apart
On October 6, Fireworks detected misuse of internal credentials, and said at the time it saw no sign of access to models or datasets. Some outside analysts, such as Aviatrix, tie the activity to a financially motivated group reselling stolen LLM access. That is their attribution, not Fireworks’. The industry calls this LLMjacking: someone else runs inference on your bill.
Then, per Fireworks’ November 9–10 update (BleepingComputer, The Register), a second incident came to light:
- Oct 23: a third-party cloud provider hosting Fireworks’ R&D machines had a security incident. Fireworks hasn’t named the provider.
- Oct 24: the attacker used that access to reach Fireworks development machines and run an unauthorized workload “that led to additional customer data impact.” Fireworks says it detected it the same day.
- Exposed: resource identifiers, names and descriptions; billing addresses, invoices, payment history and credit usage; and “incomplete model weights and datasets for some customers.” Card numbers weren’t stored, so they weren’t exposed. Fireworks hasn’t said what “incomplete” means for the weights.
CEO Lin Qiao has said the attackers don’t appear to be after model weights, so the weights exposure may be collateral rather than the goal. The company says its investigation is ongoing. I’m not writing this to pile on. I’m writing it because the attack path is one most of us haven’t modeled.
The blast radius reached downstream
Notion confirmed that its AI chat transcripts were exposed through its Fireworks integration (SecurityWeek). Per reporting, that covered fewer than 1% of workspaces and only chats from its Insights feature, mostly not attributable to a specific customer. Some chats contained pasted credentials, and Notion told users to rotate any passwords or API keys they had put into Notion AI.
Two details matter for the rest of this post. First, VentureBeat reports that Notion had a zero-data-retention agreement and the chats were still stored somewhere. Fireworks says its zero-data-retention default held, and neither company has said which feature stored them. “We have a ZDR contract” is not the same as “it isn’t stored.” Second, the lesson for your own users: don’t paste secrets into AI tools.
Update, Nov 2026: details are still emerging. Check Fireworks’ security bulletin for the current state.

Your vendor’s vendor is your attack surface
We learned this lesson with npm. In Shai-Hulud one compromised maintainer account reached 444+ packages, and the malicious releases had valid provenance. The fix was to treat install time as a trust boundary.
Inference is the same pattern at a different layer. Your app trusts the provider. The provider trusts its cloud host. The attacker in October went through the host, two hops away from you. You never signed a contract with that company, and you probably couldn’t name it.
There’s an AI-specific twist. Look at what an inference provider holds compared to a typical SaaS tool:
| Asset | Typical SaaS | Inference provider |
|---|---|---|
| Your API keys / tokens | Sometimes | Always |
| Prompts and outputs | Rarely | Every request, often logged |
| Training / fine-tuning datasets | No | If you fine-tune there |
| Model weights | No | Your fine-tunes and LoRAs |
| Spend you can be billed for | Seats | Uncapped tokens, by default |
That last row is why LLMjacking works. A stolen inference key turns directly into money, so attackers go looking for them.
The contrarian part: the gateway won’t save you by itself
The usual advice is “put an AI gateway in front of every provider.” I agree, and I recommend it below. But a gateway is also a dependency. In March, the open-source LiteLLM gateway was caught in a supply-chain attack of its own. The entry point was upstream: a compromised Trivy scanner that LiteLLM’s build pipeline installed unpinned, which led to malicious releases 1.82.7 and 1.82.8 on PyPI. By most accounts the packages were live for about 40 minutes (one outlet says closer to three hours). In August, CloudSEK estimated that the exposure potentially reached more than 2,500 organizations and roughly 434,000 CI/CD pipelines. Those are estimates of reach, not a verified victim list (SecurityWeek). The malicious code could read cloud, SSH and Kubernetes credentials and LLM API keys.
So the gateway concentrates risk as well as controlling it. That’s fine, but only if you treat it as a critical dependency: pinned versions (including the tools in its build), a release cooldown, and credentials it can only use, not read.
What I’d put in place this week
1. Inventory every inference credential. List each provider, each key, which environment uses it and who owns it. If you find keys you can’t explain, you’ve found your first problem. It’s the same weekly hygiene I described in The AI Vendor You Forgot to Offboard, applied to the other direction: credentials you hold at the vendor.
2. One key per app and environment, with a spend cap. Most providers support per-key limits or project budgets. A leaked key should be able to cost you an alert, not a month’s budget.
3. Keep provider keys out of app config. Pull them at runtime from your secrets manager through OIDC, or let the gateway hold them so apps never see the raw key.
4. Minimize what you upload. Before fine-tuning on a hosted platform, ask whether the dataset needs to live there afterwards. Keep the source of truth for datasets and adapters in your own storage, and delete hosted copies you don’t need. You can’t leak what isn’t there.
5. Ask the questions you’d ask a database vendor. Who are your sub-processors and cloud hosts? How long are prompts and outputs retained, and where, even under a zero-data-retention agreement? How fast do you notify customers? Can we bring our own key for encryption at rest? A vendor that can’t answer is giving you an answer.
6. Have a rotation runbook and a second provider. When the advisory lands, you want to rotate every key in minutes and fail over traffic, not spend a day finding where keys are configured.

What actually goes wrong
The honest trade-off is step 4. Hosted fine-tuning exists to make things easy, and “keep everything on our side” adds friction teams will route around. I’d rather accept hosted copies for a defined period than pretend a policy nobody follows is protecting anything.

The second problem is ownership again. Inference keys tend to be created by whoever ran the first prototype. That person’s personal card or project is often still behind a production integration a year later. That’s the key an attacker finds first.
Conclusion
Fireworks didn’t get breached through a model vulnerability or a clever prompt. It got breached through its infrastructure supply chain, like everyone else. The difference is what sits at the end of that chain: your keys, your data and your weights. Your inference provider is now part of your supply chain, whether or not your SBOM shows it. Start by listing every inference key you have. If you can’t list them, an attacker can find them before you do.
Discussion