Throughout 2016, Facebook boss Mark Zuckerberg has been quietly working on his own AI system. He set a “challenge” for himself earlier this year to build “a simple AI” to handle various smart home functions like controlling lights, temperature, or playing music. Now that 2016 is almost over, he’s posted a lengthy note on Facebook describing how he built the AI, which he calls “Jarvis” in honor of Iron Man, and explains everything it can do. It is, in short, ridiculously awesome.
Not to dive too deep into the coding and actual function of the bot — partly because it’s extremely technical and partly because I don’t understand most of it myself —Zuckerberg utilized Python, PHP and Objective C in its creation, and it currently works via his home computer and Facebook Messenger.
Zuck’s home is equipped with a smart home network that manages lights, the thermostat, and doors, and there’s a Sonos sound system with Spotify, smart TVs, and IP webcams. The basic tasks he set out to accomplish, like giving the bot voice recognition and the power to manage his home’s basic functions, pale in comparison to some of the more impressive features like, well, its ability to actually learn.
“Once I wrote the code so my computer could control my home, the next step was making it so I could talk to my computer and home the way I’d talk to anyone else,” Zuckerberg writes, adding that Jarvis can accept text inputs as well as translate speech into text and then read it itself. Things got more complicated when he sought to make Jarvis understand natural language.
“The more context an AI has, the better it can handle open-ended requests,” says Zuckerberg. “At this point, I mostly just ask Jarvis to “play me some music” and by looking at my past listening patterns, it mostly nails something I’d want to hear. If it gets the mood wrong, I can just tell it, for example, “that’s not light, play something light”, and it can both learn the classification for that song and adjust immediately.”
By adding cameras to his home’s entrances, he was able to integrate the video capture with the door lock system, so Jarvis knows who is coming and going, and whether or not they should be let in. “I built a simple server that continuously watches the cameras and runs a two step process: first, it runs face detection to see if any person has come into view, and second, if it finds a face, then it runs face recognition to identify who the person is. Once it identifies the person, it checks a list to confirm I’m expecting that person, and if I am then it will let them in and tell me they’re here.”
As for the future, Zuckerberg plans to build an Android app for Jarvis soon, to go along with the iOS version he already built. He also says he’s interested in exploring “ways to make [Jarvis] available to the world.” He said he considered making it open source, but that the bot is catered so specifically to his own needs that it wouldn’t be practical.
