I have a colleague who is trying to study her mother's native language (she is second generation non-resident Taiwanese) and wants to have a tool for transliterating her parents' dialog into English with learning highlights for study as flash cards. This is beyond what Duolingo, Apple Translate and Google Translate were designed to do. It seemed this could be built using several existing tools. We could ask Duolingo, Apple or Google to build it for us. But our use case here might be too niche to make sense to the general population of language learners who use those other tools.
Right now she uses a recorder, then ends one step in the process. Then she would look up different words in a dictionary phonetically to find the Chinese character she was hoping to learn. Then she saves those words in a list and practices those terms in her own context. I’ve done this too. I also study Chinese characters for fun. (Han characters, called 漢字 (Kanji) in Japanese, are foundational concepts of language that comprise the basis of several languages across China, Taiwan, Korea and Japan. The characters are like Latin is to French, Italian, Spanish and English. If you know the symbols, they offer a basis to comprehend multiple languages.)
I learned kanji through a slow process over the years of seeing the characters in context during my travels. I started with a book called “Read Japanese Today” by Len Walsh that gave me the “cave drawing” image of what various ideas in the world were drawn as thousands of years ago. Len shows ~300 pen scrawled drawings, then he shows how these morphed to become modern characters that billions of people read everyday. When I moved to Japan, one of my friends gave me a treasure. It was a book that is given to grade schoolers in the first 5 years of their study that shows the derivation of the pictographs that make up over 1000 Chinese characters. Another friend gave me a whole dictionary or archaic Chinese, which was introduced into the islands of Japan during the feudal era. After that experience, I started looking up every character I saw daily on the streets. After learning to recognize several thousand, I took the Japanese language proficiency test, passed it, and went to work at a Japanese company where I’d have the joy of reading these characters every day at work.
Learning can become a life-long obsession and passion. So when my friend said she wanted to be able to understand her mother’s native tongue and learn it, I thought I should be able to jump in to help her. What she needed was a good mash-up of several different tools into one. 10 years ago I moved to Berlin to work on business development for my company in the region. At that time I was reading a rather complex book about physics. I had a process of keeping a dictionary on hand when I read German in case I came across an unfamiliar term. (I’d studied German in high school and college. But my level at this language was only intermediate.) 10 years before that, Lab 126 (an Amazon subsidiary) had launched the Kindle e-reader. After the iPhone and Android platforms emerged, Kindle transitioned to supporting a digital app. This was what I would download my German books to. I also decided to buy a Harper Collins college German dictionary on the Kindle app. I was delighted to see that I could synchronize my German dictionary on the app, to be used as single-word lookups while I was reading the German text without having to hop back and forth between the two books. This sped up my process considerably.
Ten years after that, Apple had expanded their "Continuity" platforms for tools like Shareplay and Airdrop. These functions allow any two devices that a user owned, logged as the same account, to share a "clipboard" capability to copy and paste between them. Typically this is used to copy an address from a webpage or contacts app and then paste it into the Apple Maps application or other kinds of quick context-shifting. But I found it could also be used to sample text in my kanji study and then paste it into an advanced dictionary app like
Imiwa. This is one step beyond the Kindle case mentioned above where the two books were on one device. Here the dictionary resides on a different device, but the intent to look up a word in the Japanese dictionary is recognized and the user is invited to complete the process on one device that was started on another, with the "highlight" and "copy" intent. This in turn sped my process of cross-context language research.
What my friend needed was that kind of a fused product. Her mom’s conversation, plus a dictionary, plus flash cards. She wanted to build study guides for herself off of private family dialogs. Because the text and audio of study is going to be self-generated and private, this needed to operate in the sandbox of her phone. So it couldn’t use a cloud service. Fortunately, I’d just attended a Google I/O event where the Android Studio team had mentioned a future product that would allow users to assemble modular parts of an app together to achieve things they needed in their daily lives. Speaking the commands into an Android phone and building a Humpty Dumpty string of components isn’t yet ready for prime time. But they say it’s coming soon.
However, if you’re a developer, you can still cobble together components
on an iPhone or Android phone to create this structure as a
downloadable piece of software. While we’re waiting for our own
Jarvis-like computer assistant to self assemble the jetpacks of the
future, we just have to roll up our sleeves and built it in Xcode for
now. If you’re interested in building this tool for yourself, feel free
to fork this code from Github and tailor it to your own language of
study. I’ll be using this and improving on it over coming months for my
own study and for my friends.
https://ncubeeight.github.io/deja-entendu/
(Note this is a very rough design of an app that imports from your recordings, uses a local LLM on your device pre-bundled by Apple, not distributed in the code. All your translations happen locally and no information is sent to the cloud. The app interface was designed in Claude Code, the back-end files are your own, processed locally on your action on your device with no export capability. Meaning your data stays your own and is not transmitted elsewhere. Important to note if you develop your own app is that the LLM dictionary's accuracy may be limited. LLMs approximate language based on statistical stochastic models. They aren't dictionaries per se. So they often make errors in interpretation. Introducing rough speech-to-text process from another LLM tool magnifies the problem as there are two steps where the on-device LLM is making guesses and approximations. This may be enhanced in the future by advanced foundation models being developed by the LLM vendors. But adding a downloaded LLM to this app will bloat its footprint on the user device. So think about bandwidth cost, memory impact and battery life before doing that.)
Comments
Post a Comment