PORTFOLIO > J.A.R.V.I.S.

Categories , , ,
When? August 2013 - Present

J.A.R.V.I.S. stands for “Jeremy’s Astute Residential Virtual Intelligent System.” Is that a backronym? Yes. I couldn’t help myself. After seeing Tony Stark’s awe-inspiring AI from the Iron Man movies, I knew I had to have one of my own. So, I set to work to build one. J.A.R.V.I.S. is an ongoing side-project for me. I fiddle with it on nights and weekends, and I am not developing it to commercialize it – I’m purely interested in automating my home, and learning new things along the way. I do hope to open source all the mechanical components, electrical designs, block diagram, and software…but only once I’ve gotten it to be a bit more polished.

For a quick run-through of some of JARVIS’s capabilities, check out the video below:

What Can J.A.R.V.I.S. Do?

I’ve designed the J.A.R.V.I.S. system to be highly modularized and distributed, so that features can be added with relative easily explain what the platform is capable of, let me break it up into two categories: interfaces and controls. Interfaces are ways for J.A.R.V.I.S. and I to exchange information (screens, voice, buttons, etc.). Controls are things that J.A.R.V.I.S. is capable of doing. For the most part, any of the interfaces can be used to trigger any of the controls (i.e. I can turn the lights on via voice or via wall switches – the lights are the control and the voice/switches are interfaces).

Interfaces

These are the various ways that J.A.R.V.I.S. and I interact.

  • Natural Language Voice Interface
    Voice is my primary means of interacting with the system. Currently, the only microphone resides on the Command Center, but it’s a conference room mic that can easily hear me from anywhere in my room. Wit.ai, a machine learning natural language processing platform, enables me to handle natural voice requests (i.e. “Please turn on the lights” is understood just as easily as “lights on” even if the former phrase has not been directly trained into the system).
  • Natural Language Instant Messaging Interface
    If you can speak it, you can type it. I use the same message parsing system to handle instant messages. A simple web app enables me to have a two-way conversation with J.A.R.V.I.S. I can ask the system to do something, like opening the shades, and it will reply to confirm that it has done it. It also asyncronously notifies of system state changes over the IM interfaces. So, if somebody else uses the light switches to turn on my lights, for example, I’ll get an instant message saying that.
  • Command Center
    I built a control interface from scratch. It includes a 10 inch screen that shows system state (lights, weather, room conditions, alarm settings, etc). It has a big button to initiate voice requests, and manual dials and buttons so I can control my lights and music from my nightstand without having to use voice if I don’t want. Also includes temperature and humidity sensing, a gesture sensor (wave to silence alarm clock), and a room occupancy sensor (Automatically turns off display when room is vacant to save power, and automatically turns screen back on when somebody enters the room).
  • Light Switches
    An important goal of J.A.R.V.I.S. is to not disable existing mechanisms for home control. Although I can control my lights with voice, IM, macros, or other means, I can also still simply tap the light switches to control my lights. The light switches are connected to the Phillips Hue Hub, so I’m able to maintain state across the system when the light switches are used.
  • Macros
    Macros are automatic actions taken by the system to control various things. For example, when the alarm goes off in the morning, it can open the shades, turn the lights on, announce the weather, and start playing music.
  • Phone Gestures
    This is probably my most used interface – it’s also the most simple. Shake your phone to turn the lamp on or off. There is no need to open an app, or to even unlock the phone first. The shaking motion can easily be mapped to any function in the system.

Controls

These are the various things that J.A.R.V.I.S. can do.

  • Overhead Lighting Control (Phillips Hue Bulbs)
    J.A.R.V.I.S. can control of all the overhead lights in my room. On/Off, Toggle, and arbitrary brightness and fade settings are supported. The system is synced with wall switches, so you can still use those too, and state is properly maintained. The lights can also be controlled by natural voice (“Turn down the lights, please”) and macros (like the Alarm going off), as well as via my control center or via IM.
  • Desk Lighting strip Color and Brightness Control (Hue lighting interface)
    My desk has a Phillips Hue RGB Lightstrip to illuminate my workspace. It can also be controlled with natural voice or via instant messages or macros – “Please set my desk light strip to Red at 50% brightness.”
  • General Knowledge Voice interface (Wikipedia interface)
    I can ask J.A.R.V.I.S.: “What is the Machine Learning?” (or any other query) and it will tell me.
  • Weather with arbitrary date and location (forecast.io interface)
    “J.A.R.V.I.S., What’s the weather going to be like in New York in two days?” It will tell me estimated conditions. including the temperature range and odds of precipitation.
  • Music (Google Play Music interface)
    I have it connected to my Google Play music unlimited access account. So I can give it any arbitrary music request and it will play it: “Please play music by Billy Joel” and it will generate a playlist with Billy Joel’s most popular songs and play that.
  • Lamp Control (120VAC hardware control interface)
    I can control the lamp in my room with voice, via macros, or by simply shaking my phone. I use this when I have to get up in the middle of the night, because it’s dim and doesn’t blind me. Simply give the phone on the nightstand a quick shake and the lamp will turn on. Shake again to turn it off.
  • Shades control
    I can control the shades with voice, or as part of a macro (shades open when alarm goes off, and shades close when I say “goodnight”).
  • A bunch of other smaller features.

How Does J.A.R.V.I.S. Work?

J.A.R.V.I.S. is designed as a hub and spoke network that lives on a Local Area Network (LAN). A single “brain” machine runs a RESTful, secured API, as well as a websocket interface that client components can subscribe to. Simple interfaces can simply send authenticated commands to the RESTful API (i.e. shaking my phone to turn the lamp on). Whenever an API request is triggered (whether by an external component like my phone, or via an internal macro like the alarm clock turning the lights on), the brain updates its master understanding of the system state, and broadcasts a state update to any connected websocket clients. The Command Center is an example of a subscribed websocket client, since its dials and screen show real-time system state. The brain only sends out a state update broadcast once it has received affirmation from the control interface that the action has completed successfully. Similarly, websocket clients must confirm when they receive and acknowledge a state update.

Supporting Code and Designs

I haven’t open-sourced J.A.R.V.I.S. yet, but I do plan to. I’m not an expert software engineer (I’m an electrical engineer), and I want to get the software more cleaned up before I share it with the world.

J.A.R.V.I.S. in the News and on my Blog