Microsoft Patent Targets Virtual Nutrition Labels

A concept video shows how a Microsoft Hololens wearer could call up a virtual recipe in the kitchen. Credit: Microsoft A Microsoft patent idea sees the future of eating for smart glasses wearers going something like this. You're about to chow down on a juicy hamburger when a virtual nutrition label pops up that says: "Typically a hamburger of this size may have 800 - 1200 calories." Ignoring the information, you pick up the greasy burger and take a large bite. As you chew, another floating warning pops up in your vision: "This is your second big meal today — are you sure you want to do that?" Having your smart glasses scold you for disregarding your personal diet goals may not sound like a lot of fun. But someone at Microsoft thinks an augmented reality app focused on health and nutrition could translate into big business down the line. A Microsoft patent application titled "Wearable Food Nutrition Feedback System" was recently discovered through research provided by legal technology firm ClientSide. The patent application covers the concept of a wearable head-mounted device—basically smart glasses—that can recognize a wide variety of food and display virtual information such as virtual nutrition labels, food allergy warnings floating in midair or even menu recommendations posted by friends on social media. It's always worth keeping in mind that a patent application or grant does not mean Microsoft will actually develop such a nutrition-focused product for augmented reality. But Microsoft has been hard at work on unveiling an augmented reality device: the Hololens wearable that can display interactive holograms overlaid in the real world. A food nutrition app seems like a no-brainer for the Hololens or a next-generation version of the smart glasses. https://www.youtube.com/watch?v=AaTyeDtht-8 The Microsoft patent application includes an example of the smart glasses wearer looking at a veal special dish on a restaurant menu. In one case, the wearable device might display an "avoid" allergy warning about the veal dish containing nuts. That assumes the wearable system already has information about the human wearer having specific food allergies. In a second example, the patent application envisions the wearable device telling the wearer: "You didn’t like the Veal Special last time…" That case assumes the system can store data about specific restaurant dishes a person had tried before and recorded their reactions or ratings for each dish. A third example involves the wearable device displaying the following: "Marsha was here last week and had the Veal Special and gave it four stars!" In that case the wearable device would presumably pull the friend's food rating and recommendation from something like an online social media service. The biggest technical issue for the nutrition label app may come from creating the software capable of recognizing different types of food and deducing the appropriate nutritional information. A piece of fruit such as an apple or a packaged product may not present the greatest challenge for visual recognition algorithms, but a casserole might prove otherwise. Still, Microsoft seems to have anticipated that problem. Its patent application envisions the wearable device asking for the human wearer's help in labeling foods that it can't identify. Many computer algorithms and programs have improved their capability to identify certain images based on that time-honored tactic of learning from human feedback and input. In that sense, it's likely just a matter of time before a virtual nutrition label app similar to what Microsoft envisions becomes a reality. This particular Microsoft patent application, published on Jan. 28, 2016, was first filed on Oct. 8, 2015. It builds on a previous patent application that was originally filed in 2012 and approved in 2015.