Creating LLM apps from components

I have a colleague who is trying to study her mother's native language (she is second generation non-resident Taiwanese) and wants to have a tool for transliterating her parents' dialog into English with learning highlights for study as flash cards. This is beyond what Duolingo, Apple Translate and Google Translate were designed to do. It seemed this could be built using several existing tools. We could ask Duolingo, Apple or Google to build it for us. But our use case here might be too niche to make sense to the general population of language learners who use those other tools. 

Right now she uses a recorder, then ends one step in the process. Then she would look up different words in a dictionary phonetically to find the Chinese character she was hoping to learn. Then she saves those words in a list and practices those terms in her own context. I’ve done this too. I also study Chinese characters for fun. (Han characters, called 漢字 (Kanji) in Japanese, are foundational concepts of language that comprise the basis of several languages across China, Taiwan, Korea and Japan. The characters are like Latin is to French, Italian, Spanish and English. If you know the symbols, they offer a basis to comprehend multiple languages.) 

I learned kanji through a slow process over the years of seeing the characters in context during my travels. I started with a book called “Read Japanese Today” by Len Walsh that gave me the “cave drawing” image of what various ideas in the world were drawn as thousands of years ago. Len shows ~300 pen scrawled drawings, then he shows how these morphed to become modern characters that billions of people read everyday. When I moved to Japan, one of my friends gave me a treasure. It was a book that is given to grade schoolers in the first 5 years of their study that shows the derivation of the pictographs that make up over 1000 Chinese characters. Another friend gave me a whole dictionary of archaic Chinese, which was introduced into the islands of Japan during the feudal era. After that experience, I started looking up every character I saw daily on the streets. After learning to recognize several thousand, I took the Japanese language proficiency test, passed it, and went to work at a Japanese company where I’d have the joy of reading these characters every day at work.

10 years ago I moved to Berlin to work on business development for my company in the region. At that time I was reading a rather complex book about physics. I had a process of keeping a dictionary on hand when I read German in case I came across an unfamiliar term. (I’d studied German in high school and college. But my level at the time was only intermediate.) 10 years before that, Lab 126 (an Amazon subsidiary) had launched the Kindle e-reader. After the iPhone and Android platforms emerged, Kindle transitioned to supporting a digital app. This was what I would download my German books to. I also decided to buy a Harper Collins college German dictionary on the Kindle app. I was delighted to see that I could synchronize my German dictionary on the app, to be used as single-word lookups while I was reading the German text without having to hop back and forth between the two books. This sped up my process considerably. 

Ten years after the Kindle dictionary linking, now broadly available across devices, Apple has expanded their "Continuity" platforms for tools like Share-play and Airdrop. These functions allow any two devices that a user owned, logged as the same account, to share a "clipboard" capability to copy and paste between them. Typically this is used to copy an address from a webpage or contacts app and then paste it into the Apple Maps application or other kinds of quick context-shifting tasks. But I found it could also be used to sample text in my kanji study and then paste into an advanced dictionary app like Imiwa. (pictured at right with the phone app showing a non-wired lookup of the term on my computer) This is one step beyond the Kindle case mentioned previously where the two books were on one device. Here the dictionary resides on a different device, but the intent to look up a word in the Japanese dictionary is recognized and the user is invited to complete the process on one device that was started on another, with the "highlight" and "copy" intent starting the continuity flow between MacOS and iOS. The intent and action is abstracted to the level of the person logged into both devices. Apple software here recognized that I was using two different machines and allowed them to share a memory space that would allow me to select on one device and take action on that selection on another. This in turn sped my process of cross-context language research.  

What my friend needs for Mandarin study is a similar kind of fused product leveraging different component parts that work in isolation at present. She wants a transcribing tool, plus a dictionary, plus flash cards, plus a language synthesis layer that builds a learning dialog off of previously saved terms in her local repository of vocabulary. Because the text and audio of study is going to be self-generated and private, this needed to operate in the sandbox of her phone. So it can't use a cloud service. Earlier this year I attended a Google I/O event where the Android Studio team had mentioned a future product that would allow users to assemble modular parts of an app together to achieve things they needed in their daily lives. In that context speaking the commands into an Android phone would build a Humpty-Dumpty string of components that could be wrapped into an apk file and published on the Android Play Store. They say it’s coming soon to Android devices.

However, if you’re a developer, you can still cobble together components on an iPhone or Android phone to create this structure as a downloadable piece of software. While we’re waiting for our own Jarvis-like computer assistant to self assemble the jetpacks of the future, we just have to roll up our sleeves and built it in Xcode for now. If you’re interested in building this tool for yourself, feel free to fork this code from Github and tailor it to your own language of study. I’ll be using this and improving on it over coming months for my own study and for my friends. 

 https://ncubeeight.github.io/deja-entendu/

 Technical stuff: 

Note this is a very rough design of an app that imports from your recordings, uses a local LLM on your device pre-bundled by Apple, not distributed in the code. All your translations happen locally and no information is sent to the cloud. The app interface was designed in Claude Code, the back-end files are your own, processed locally on your action on your device with no export capability. Meaning your data stays your own and is not transmitted elsewhere. 

Functionality: 

The app runs a parsing process on audio samples you select in the Audio Recordings tab. Only iPhone 15 and later have robust LLMs on them that are capable of parsing with high accuracy. (Pixel 10 in the Android ecosystem) Apple will likely expand the supported language of the LLM over time. So I've included languages in the settings tab that are either already supported or are likely to be supported by 2027. 

I made a design decision that the range of languages that could be selected on upload should be toggles in the settings tab, not the upload tab. This way, your upload tab only shows the languages that you regularly translate. You can limit this down to one that you use frequently or several that you want to use. I didn't include language auto-detect because the user already knows the language they are studying in this use case. Language disambiguation AI tools are available at a cost from web based LLMs. But using those requires outbound transmission of data to a cloud server, that I didn't want to include for privacy reasons. I also wanted to be able to offer this app to users for free. So integration of external APIs would have meant that the app (or re-compiled forks of it) would have an external dependency. So the app will not be able to identify the language sample pre-upload. If you try to parse a sample with the wrong language selected, the results will not be particularly pretty. And they will not be able to create an on-device flashcard because the local LLM doesn't know what the transcribed text represents. 

 

Also, just like the Apple AirPods, the on-device LLM has trouble interpreting with background noise. So make sure to use sound samples that are clear, close to the speaker and use good diction. Samples that include strong dialects or slurred speech obviously will not parse when run through the dictionary. So your speech-to-text outputs would appear jumbled or imprecise if the original recording was poor. 

Once you select a term on the transcription (at left in a French example) your term can be added to a Vocabulary list which will populate in the next tab to the right and also a preview on the Home screen. Selecting the term from those tabs will launch a brief flashcard generation process with the LLM locally. 

 

 

For developers

If you want to fork this code, you can add feature-rich utilities on top of it such as porting in the paid-LLM tools that a subscriber already has on their phone. Derived audio from the recording could thereby be fused into other applications such as poem generators or full lesson plans. But those features would incur cost to the user, so they are not in the bare-bones open source version. Developers could introduce an audio pre-processor with LLM tools such as Whisper or Adobe Voice Enhance that might clean up recordings for the transcription stage. If you were to add a downloaded LLM such as Gemma to your version of this app, that would bloat the memory footprint on the user device. So think about bandwidth cost on the user's phone connectivity, memory impact to their runtime and battery life before doing that.

Why does this exist? Why is it free?

Learning a language is one of the most important things that can improve the world. By studying German and Japanese through the years, I've been able to discover many aspects of those cultures in-depth. I've had the fortunate opportunity to meet amazing people, learn new ideas from them in their language and negotiate business deals that increase global inter-dependency and build cultural bridges between nations. Many teachers through the years have taught me for free or given me tools to learn for free. So learning apps like these may inspire the next generation of language learners.  

It's not really free beyond the software itself. A user has to have acquired a phone in the first place. But that phone has a capability that developers can unlock atop it. Because no external LLMs are used in this app, it can function just on the electricity the device itself uses to process the memory chips locally. 

Environmental impact?   

No external cloud services are initiated by this app running on your device. So it doesn't generate calls to a network through your carrier and doesn't require a wifi signal to operate. So there are no machines outside your phone expending energy. Distributed locally-operating open weight or open source LLMs are energy efficient in contrast to using a cloud based LLM. So using a computer to do local inference is not an energy wasteful process. Apple and Google (the model partner for Apple's LLM initiatives) do have to run servers to build and enhance the foundation models like Gemma that they distribute for download later. But the usage of those models by computers once they are built is more energy efficient than running a cloud based LLM from your phone to the cloud.  

LLM caveats:  

It is important to note if you develop your own app is that the LLM dictionary's accuracy may be limited. LLMs create and interpret approximate language based on statistical stochastic models of things others have said or written in the past. They aren't literal dictionaries. So they often make errors in interpretation. Introducing rough speech-to-text process from another LLM tool at the start of this process magnifies the problem as there are two steps where the on-device LLM is making guesses and approximations, first the listening to rough audio and creating the transcription, then the layer of doing the interpretation. This may be improved in the future by LLM vendors expanding the size or granularity of foundation models being developed and embedded in all mobile phones. 

 

 

Follow ncubeeight on LinkedIn or connect with Christopher Arnold