Most businesses treating this as a philosophical debate are asking the wrong question. Open source AI models — like Meta's Llama family, Mistral, or Qwen — are models whose weights are publicly available. You can run them yourself, modify them, deploy them without paying per-token fees or sending your data to a third party. Closed source models — GPT-4o, Claude, Gemini — are APIs you rent access to. At Nuclear Marmalade, this distinction shapes nearly every AI build we take on. Getting it wrong is expensive.
Here's what most people miss: the model itself is rarely the hard part. The hard part is everything around it — the data pipelines, the eval loops, the deployment environment. The open vs. closed decision affects all of that, not just the invoice line item.
What's the actual difference between open and closed source AI?
Open source AI means you have the weights — the trained parameters that make the model work. You run inference on your own hardware or a cloud VM, fine-tune on your own data, integrate it however you like. Closed source means you're calling an API owned by someone else. Text in, text back, pay per token, no visibility into what happens in between.
Both approaches produce capable models right now. As of mid-2025, Llama 3.3 70B performs comparably to GPT-4-class models on most standard benchmarks — at a fraction of the per-query cost when self-hosted. That gap closed faster than most people expected. It changes the calculus significantly if you're building at any real volume.
Why does data privacy matter more than most people admit?
Sending customer data to a closed API is a policy decision that most businesses are making without realising it. When a healthcare company, a legal firm, or a financial services business sends queries to GPT-4o or Claude's API, that data leaves their environment. OpenAI and Anthropic both have enterprise agreements with stronger privacy terms, but those cost money and require negotiation. The default API terms are not the same thing.
With an open source model on your own infrastructure, the data never leaves. That's not paranoia — it's a compliance posture.
We hit this directly on a client build: a business handling sensitive customer intake data couldn't use a cloud API under their existing contracts. Running a quantised Mistral 7B on a dedicated server solved it in a week. Not elegant. But it worked, and they didn't have to renegotiate their entire legal framework to get there.
When does closed source AI actually win?
Closed source wins when you need top-tier reasoning right now and don't have the infrastructure to self-host. Building a one-off internal tool? A quick prototype? Something where query volume is low enough that per-token costs don't bite? Calling the OpenAI or Anthropic API is genuinely faster and simpler.
You don't need a GPU server. You don't need to manage model updates. And you get capabilities — like Claude's extended context window or GPT-4o's multimodal inputs — that open source models are still catching up on.
There's also a support argument. If something breaks with a closed API, the vendor owns that problem. With a self-hosted open model, you own it. That's fine if you have engineers. It's a hell of a problem if you don't.
For most businesses we talk to at Nuclear Marmalade, the honest answer is often: start with a closed API, then migrate if volume justifies it.
What does self-hosting actually cost?
This is where the numbers get real. A self-hosted setup for Llama 3.3 70B that handles moderate business traffic without grinding needs at least an A100 80GB GPU, or a pair of smaller cards. Renting that on AWS or GCP runs roughly $3–$5 per hour — around $2,000–$3,500 per month for a dedicated instance.
Compare that to GPT-4o at $5 per million input tokens and $15 per million output tokens. If your app sends 50 million tokens a month, you're looking at $750–$2,500 in API costs. Potentially cheaper than self-hosting. Push past 200 million tokens and the math flips hard.
Most small businesses never hit that threshold. Most mid-size businesses building real AI products do. The crossover is usually somewhere between 100 and 200 million tokens per month, depending on model size and infrastructure efficiency. Do the actual arithmetic before assuming one approach is obviously cheaper.
What's the fine-tuning argument for open source?
Fine-tuning is where open source has a structural advantage that closed source genuinely can't match. With an open model, you train on your own proprietary data and bake that knowledge directly into the weights. You build a customer service bot that knows your product catalogue, your return policy, your edge cases — not because you stuffed it all into a system prompt, but because the model was trained on it.
OpenAI offers fine-tuning on some models. But you're still sending your training data to their servers, and you don't own the resulting model. That's a meaningful distinction if your data is a competitive asset.
I've written about this before in the context of vertical AI products — the businesses that win in AI aren't the ones with the best base model, they're the ones with the best training data and the infrastructure to use it. Open source models make that play possible. See more on how we think about this in our product work.
What do most businesses actually get wrong here?
They treat the open vs. closed decision as permanent. It's not.
The smart move is to start with a closed API to validate that the AI feature is worth building at all. Don't spend three weeks standing up infrastructure for a product idea that hasn't been tested. Once you know it works, once you've got real usage data, then you evaluate whether volume and privacy requirements justify moving to open source.
We've done this across several builds. The pattern holds.
The mistake is going the other way — starting with a self-hosted open model because it feels more serious or more technically interesting, and spending two months on infrastructure instead of on the product. I've watched teams do this. It's genuinely painful. The technology should serve the business decision, not the other way around.
If you want to see what that looks like in practice, check out the Telehance project — we cut their call-handling process from a four-hour manual review down to about 12 minutes using a combination of closed and open source approaches.
How should a business actually decide?
Ask four questions in order.
One: does this use case involve data you can't send to a third party? If yes, open source wins by default. Two: will your monthly token volume exceed 150 million? If yes, run the infrastructure cost numbers — self-hosting may be cheaper. Three: do you need to fine-tune on proprietary data? Open source is the only real answer. Four: do you have engineers who can maintain a self-hosted deployment? If not, closed source is more honest about your actual capabilities.
Most businesses sit somewhere in the middle — they want the cost and privacy benefits of open source but don't have the team to support it. That's a real tension, and pretending otherwise doesn't help. The Nuclear Marmalade blog has more on practical AI infrastructure decisions for businesses that aren't running a 20-person ML team.
Key Takeaways
- Open source AI means you own and run the model; closed source means you're renting access via an API — the difference matters most for data privacy and high-volume cost.
- The per-token crossover point where self-hosting beats API costs is roughly 100–200 million tokens per month. Below that, closed APIs are usually simpler and cheaper.
- Fine-tuning on proprietary data is the clearest argument for open source — you can't truly own the resulting model if you're training on someone else's infrastructure.
- Start with a closed API to validate the product idea. Migrate to open source once volume and requirements justify the infrastructure overhead.
- The decision isn't permanent. The teams that get this right treat it as an engineering trade-off, not an ideology.
If your business is weighing an AI build and you're not sure which direction makes sense for your situation, get in touch with Nuclear Marmalade — we've worked through this decision enough times that we can usually give you a straight answer in one conversation.