Timeline
Apr 2022 (Presented at VRTO 2022)
Libraries
TFLite (Object Recognition, Translation, and Text-to-Speech conversion)
Overview
Since I arrived in Canada about 6 months ago, I’ve been trying to learn French through different techniques. I’ve tried some of the most popular apps such as Duolingo, completed certain French grammar exercises, and watched a bunch of French movies as well. Although I’ve caught some bits of grammar and vocabulary along the way, I feel that these techniques are not efficient. After a bit of research, I realized that conventional technological tools to learn languages are inefficient as they lack a real context. Immersion can significantly accelerate the learning process and I was curious to build an app that could make learning languages fun and highly efficient for me and all the other language learners.
How does it work?
Things is an app built for Android smart glasses with a camera and a microphone. The app features real-time intelligence to help you learn languages in an immersive context. When users wearing smart glasses look at any object around them, they see what’s that object called in the target language on the HUD(heads up display) screen. It currently supports 3 languages (French, Italian, and Japanese) and allows the users can switch between languages by tapping on the capacitive touch panel on the glasses. The smart glasses display the translation of the object’s name and speak out the pronunciation as well.
Further Development in progress
The app is being ported to Hololens 2 and would use its RGB cameras and depth sensors for spatial anchoring. Google Cloud Vision APIs will also be used to support many more object categories and other language APIs to support sentence construction are being added.
