Love a Dish? Take a Photo and MIT’s Pic2Recipe Artificial Intelligence Will Guess the Recipe.

Attention all Instagram Food Bloggers: Do not just snap beautiful food photos. Run the MIT Pic2Recipe Artificial Intelligence system that will not only predict the ingredients, but also suggest similar recipes.

Given a photo of a food item, Pic2Recipe can identify ingredients like flour, eggs, and butter — and then suggest several recipes that are similar to images from the “Recipe1M” database, a database of over 1 million recipes annotated with the ingredients used in a wide range of dishes.

How accurate is it all? Go ahead and try it out: upload the photo of a dish you’ve cooked and see if Pic2Recipe can correctly guess the ingredients you used and how you cooked the dish.

Yep, as you noticed, there’s still lots of room for improvement. For example, the system does particularly well with desserts like cookies or muffins, but has difficulty determining ingredients for more ambiguous foods, like sushi rolls and smoothies. It also cannot quite determine whether the food was prepared stewed or diced, nor distinguish between different variations of foods, like mushrooms or onions.

Spicy Braised Beef Noddle Soup: You haven't lived if you haven't tried this at least once. But watch out: it's spicy!

Spicy Braised Beef Noddle Soup: You haven’t lived if you haven’t tried this at least once. But watch out: it’s spicy!

I uploaded an image of a Spicy Braised Beef Noddle Soup (similar to the one above) and it returned: No matches. 🙁 I then tried some more images until I got some not-quite-accurate matches, e.g., it mistook beef for pork and thought there were peaches or potatoes in the soup (there aren’t).

It’s obvious that, unless a similar picture of the dish resides in the Recipe1M database, you will only get rough approximations — when what you really want to know is how to reproduce that exact dish that you so enjoyed at the restaurant. Close enough ain’t good enough.

I believe this MIT team is on the right track but may have bitten off more than they can chew. Their neural network, as it stands, may be too simple and, if you pardon the pun, lacks a few ingredients: It is not enough to match pictures to a database. Consider that many people who cook can intelligently guess at the ingredients in a dish (familiar or foreign) they get to taste. What is more difficult — and hence the value that Pic2Recipe could potentially bring to the table — is to come up with a recipe that is either exact or comes very very very close. That means a neural network that can “place” the dish as Vietnamese, Korean, British, French, or, in this case, Chinese. And then have enough smarts to know that a Chinese dish usually uses these certain ingredients and is prepared a certain way. That means the research team need to know quite a bit about food (send them off to sample the food at restaurants, chief — it’s in the name of research). In my example above, none of the recipes offered come even a little close to being that of a Spicy Braised Beef Noodle Soup.

And here’s a way to monetize it: I would gladly settle for a list of restaurants close to me that serve that dish. Hint: there aren’t many that does it really well. And that’s why I’m looking for a good restaurant or the recipe so I can try cooking it myself. That Recipe1M database could be augmented with people’s suggestions and recommendations (similar to Google’s Translate asking if its translation was good enough and can you provide a better one). A learning Pic2Recipe system would then organically get better each time someone queries it, tries out the recipe and comes back to rate it or suggest improvements.

Once the researchers iron the problems out, they plan to add dietary preference as a feature (so certain foods are substituted for others) and a “dinner aide” feature that takes into consideration the items you currently have in your fridge and makes the appropriate ingredient substitutions and recipe modifications. I’ll drink to that!

No, you didn’t! That is, try to get it to predict the recipe for Classic Coke or KFC?


/* ]]> */