Building Offline AI on Android: 5 LiteRT-LM Pitfalls and Fixes

H

HandyDude

Guest
We ship an offline-first LLM chat app on Google Play. Everything runs on-device with Google's LiteRT-LM runtime (com.google.ai.edge.litertlm:litertlm-android) — no server, no account, no data leaving the phone. Here's what actually broke in production and how we fixed it.

1. The GPU delegate will get your app killed​


The obvious move is "enable GPU, it's faster". In LiteRT-LM ≤0.17 the GPU delegate allocates RSS an order of magnitude above the model size — we measured ~11 GB resident for a 0.8B int4 model on a Galaxy S24 Ultra. lmkd kills the app in the foreground.

Fix: CPU is the default for every device. GPU is an opt-in "experimental" switch, plus two fallback layers:

  • init-time: if Engine creation with GPU throws, retry with CPU;
  • generation-time: if inference fails with known GPU markers, flip a permanent force_cpu flag and retry.

2. dlopen("libOpenCL.so") fails on Android 12+ even when the driver exists​


On Samsung/One UI (Vulkan-only GPUs) the GPU path dies at runtime with Can not find OpenCL library. The device has the library — it's in /vendor/etc/public.libraries.txt — but the driver has private dependencies unavailable to the app namespace.

Fix in AndroidManifest.xml:

Code:
<uses-native-library android:name="libOpenCL.so" android:required="false"/>
<uses-native-library android:name="libOpenCL-car.so" android:required="false"/>
<uses-native-library android:name="libOpenCL-pixel.so" android:required="false"/>

Without the required="false" declarations the app won't even install on devices that lack the libs.

3. Naive chat templates serialize JSON into your context​


LiteRT-LM SDK passes message content as a parts array, and the engine stores model replies as multi-part content. Templates that do without a content is string branch (huihui-family finetunes are the usual suspects) dump raw JSON straight into the conversation:

Code:
[{"type":"text""text":"…"}]

The next turn then echoes the prompt back because the context is poisoned.

Fix, two layers:

  • at session creation, build a content-safe copy of the embedded template with a parse_content macro that iterates parts and joins text fields — substitute it into ConversationConfig.chatTemplate;
  • at the sink, run a sanitizer that unwraps [{"type":"text",…}] envelopes in already-generated text before persisting to Room.

Verify with a real multi-turn test on the affected model — the bug only appears on turn 3+ when the stored message re-enters the context.

4. .litertlm bundles are version-fragile​


Bundles exported for SDK 0.13.1 refused to run on 0.16.1: prefill work group size exceeds available state entries, on both GPU and CPU. The model catalog had to be re-converted wholesale for the new runtime. Keep a minRuntimeVersion field in your model JSON and treat bundle regeneration as part of the SDK bump, not an afterthought.

The good news: xnnpack weight caches are portable — same model file (mtime+size identity), different device, cache still valid. We ship no cache, but users' second device inits are faster than the first download+compile.

5. Engine/Conversation are not thread-safe — serialize everything​


Any concurrent native call during streaming risks use-after-free → SIGSEGV. We wrapped every native interaction — initializeModel, prepareConversation, generateResponseStream, closeModel, even title generation — under a single Mutex. An AtomicInteger of in-flight native ops gates the idle auto-unload timer so the engine is never closed mid-stream.

Bonus: reuse the live Conversation across turns. We verify append-only history growth (seeded params + native turn count + suffix check on the wrapped prompt) and skip re-creating the session — prefill then runs only on the new tokens, not the whole history.

Numbers worth knowing (measured, device + runtime noted)​

  • 0.8B warm init: ~40s on Snapdragon S20-class, ~98s on a $200 Nord N200
  • 4B warm init: ~168s
  • The native LITERTLM container load phase — not your Kotlin — is where init time lives

If you're building on-device AI on Android, I'd love to compare notes. I'm also building LocalClaw (offline agent runtime) and LocalAI (the chat client described above) — both free, both no-account.

All measurements device-verified; runtime versions: litertlm-android 0.16.1–0.18.0, minSdk 26, targetSdk 36.
 

Thread statistics

Created
HandyDude,
Replies
0
Views
16
Back
Top