What We Learned Building On-Device AI Into Twelve iOS Apps

Several of the apps we have shipped process user data entirely on the phone. OpusCam analyses an entire photo library locally. B-Roll AI tags a creator's camera roll on-device. Veyl keeps travel history on the phone with no account at all.

None of those decisions were made for ideological reasons. They were made because for those specific products, on-device turned out to be the option that worked better — and in a couple of cases, the only one that was viable at all. Here is what that actually involves, including the parts that are genuinely worse.

The economics are the part people miss

Most discussion of on-device AI focuses on privacy. That is real, but the argument that usually changes a founder's mind is the cost curve.

A server-based app that analyses photos pays for every analysis, forever. Every user, every photo, every re-run. Your infrastructure cost scales linearly with usage — which means your most engaged users are your most expensive, and a burst of success is a bill. For a consumer app at a low price point or on a free tier, this can invert your unit economics precisely when things start going well.

For a photo-cleaning app, this is decisive. A user with 40,000 photos is exactly the user who most needs the product and would be most expensive to serve remotely. On-device, that user costs the same as any other: nothing.

What actually gets better

It works with no connection

Underrated until you watch someone use your app on a plane, on a train, or abroad with data roaming off. This is not an edge case for a travel app — it is the primary case. Veyl exists to be used while travelling, which is exactly when connectivity is least reliable and most expensive. An app that needs a server to answer "how many Schengen days do I have left" fails at the moment it is needed.

Latency stops being a design constraint

A round trip to a server is a few hundred milliseconds at best, and that budget shapes your interface whether you want it to or not. It pushes you towards batch operations, progress spinners, and asking users to wait.

On-device inference on modern Apple silicon is fast enough that results can be immediate, which lets you build interfaces that respond as you scroll rather than after you submit. That is a different feel, and users register it as quality even when they cannot name why.

Permission conversations become much easier

"This app reads your entire photo library" is a hard ask. "This app reads your entire photo library, and nothing leaves your phone" is a much easier one — and it has the advantage of being verifiably true rather than a promise about a server the user cannot inspect.

This matters commercially, not just ethically. Photo library access is a permission users increasingly decline, and a declined permission is a churned user before onboarding finishes.

What is genuinely harder

It would be dishonest to present this as free. Four things get meaningfully worse.

You cannot fix a model without shipping an app update

This is the big one. With a server, a bad model is a deploy away from being fixed. On-device, a model problem means a build, a review, and waiting for users to update — and some never will. Your quality bar before shipping has to be higher, because your correction loop is measured in days rather than minutes.

The device floor is real

Performance across the supported device range varies far more than a server, where you control the hardware. Work that is instant on a current iPhone can be sluggish on a five-year-old one, and that older device belongs to a real user who paid the same price. You end up testing on old hardware deliberately, and sometimes degrading the experience gracefully rather than shipping something that technically runs.

Battery and thermals become your problem

Concepts you can ignore entirely on a server. Processing a large photo library is sustained work, and sustained work heats a phone and drains a battery. In practice this forces real architectural decisions: chunking work, deferring to when the device is charging, respecting Low Power Mode, and stopping when the system signals thermal pressure. None of this is difficult, but all of it is work that simply does not exist in a server design.

You are capped by what fits

You are not running a frontier-scale model on a phone. You are running something small enough to ship inside an app bundle and fast enough to be worth using — or leaning on what Apple already provides through Vision and the Neural Engine. For classification, detection, similarity, and tagging, that is more than sufficient. For open-ended reasoning over arbitrary input, it is not, and pretending otherwise produces a worse product than admitting it.

When you should still use a server

We do not build everything on-device, and the line is fairly clear.

Use a server when the task genuinely requires a large model — open-ended generation, complex reasoning, anything where quality depends on scale that cannot fit on a phone. VisuCraft generates step-by-step repair guidance from an arbitrary photo of an arbitrary object; that is not an on-device task today, and claiming otherwise would mean shipping something worse.

Use a server when the data is inherently shared — anything social, collaborative, or competitive. A daily game with friends and standings needs a shared source of truth by definition.

Use a server when the content must be current. Trend catalogues, prices, live data: the value is in freshness, which means a network call somewhere regardless.

The useful question is not "on-device or cloud." It is "which parts of this genuinely need to leave the phone." Usually the honest answer is: fewer than you assumed, but not none.

The hybrid shape is common and works well — sensitive analysis stays local, and only the minimum required goes out. B-Roll AI does exactly this: your footage is tagged on your device, and only the trend catalogue comes down from the network. Your camera roll never leaves; the thing that genuinely has to be fresh does.

The practical advice

If you are deciding this for a product right now, three things are worth doing early.

• Work out your cost per active user under the server design, at 10× your current usage. If that number frightens you, on-device deserves a serious look before you build the expensive version.
• Prototype on your oldest supported device, not your newest. The feasibility question is answered there, not on the phone in your pocket.
• Decide the split before you build, not after. Retrofitting on-device processing into an architecture that assumed a server is substantially more work than designing for it from the start — and it is the retrofit, not the original decision, that makes teams conclude on-device is impractical.

The default has drifted towards the cloud because it is the path of least resistance during development. But the cost, latency, and permission consequences all land later, on the product and on the business. For a whole category of consumer app, the harder path at the start is the cheaper one to live with.