Skip to content

Building My Own Siri / Jarvis part 2

June 28, 2012

I noticed there was quite a lot of interest in that first blog post. That’s enough to warrant a follow-up.

So a few things changed. In the original post, I primarily relied on Wolfram Alpha’s AI engine to power Jarvis. Thanks to Shiv Kokroo, I discovered True Knowledge (which is now called Evi). I signed up for a free account. They are far more generous with the number of queries you can make, and I like the results better. The only problem is that it doesn’t return concise responses like Wolfram… which would be better suited for Jarvis.
What I ended up doing was to use both. If True Knowledge doesn’t return with a response, it asks Wolfram.

The source code is available here on github.
… and here it is in action.

Yes. The biggest improvement is Jarvis’ ability to control appliances. First, I set up a “mini server” using an arduino + ethernet shield. I set up the miniature arduino server to accept GET requests such as:
http://192.168.1.177/?dev=light&cmd=on to turn the light on. Eventually, I would issue this request:
http://192.168.1.177/?dev=tv&cmd=off to turn the tv off (though I didn’t implement the IR module just yet). To change the channel on the set top box, I had plans to use
http://192.168.1.177/?dev=stb&cmd=618. I was actually confident I could do this one until I tried hacking my verizon fios box. It turns out it’s not very easy and I just don’t have that kind of time.

Here is the arduino web server code:


#include <SPI.h>
#include <Ethernet.h>

int ledPin = 2;

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(192,168,1, 177);

// Initialize the Ethernet server library
// with the IP address and port you want to use 
// (port 80 is default for HTTP):
EthernetServer server(80);

void setup()
{
  pinMode(ledPin, OUTPUT);
  // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);
  server.begin();
}

void loop()
{
  String readstring = ""; 
  // listen for incoming clients
  EthernetClient client = server.available();
  if (client) {
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        // if you've gotten to the end of the line (received a newline
        // character) and the line is blank, the http request has ended,
        // so you can send a reply
        readstring += c;
        if (c == '\n' && currentLineIsBlank && readstring.indexOf("dev=garage")>0){
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          if (readstring.indexOf("cmd=open")>0){
            client.print("opened garage");
          }
          else{
            client.print("closed garage");
          }
          readstring = "";
          break;
        }
        else if (c == '\n' && currentLineIsBlank && readstring.indexOf("dev=tv")>0){
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          if (readstring.indexOf("cmd=on")>0){
            client.print("turned on tv");
          }
          else{
            client.print("turned off tv");
          }
          readstring = "";
          break;
        }
        else if (c == '\n' && currentLineIsBlank && readstring.indexOf("dev=light")>0){
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          if (readstring.indexOf("cmd=on")>0){
            switchlight(1);
            client.print("turned on light");
          }
          else{
            switchlight(0);
            client.print("turned off light");
          }
          readstring = "";
          break;
        }
        else if (c == '\n' && currentLineIsBlank) {
          // send a standard http response header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();
          readstring = "";
          break;
        }
        if (c == '\n') {
          // you're starting a new line
          currentLineIsBlank = true;
        }
        else if (c != '\r') {
          // you've gotten a character on the current line
          currentLineIsBlank = false;
        }
      }
    }
    // give the web browser time to receive the data
    delay(1);
    // close the connection:
    client.stop();
  }
}

int switchlight(int onoroff){
  if(onoroff==1){
    digitalWrite(ledPin, HIGH);
  }
  else{
    digitalWrite(ledPin, LOW);
  }
  return onoroff;
}

You’ll notice the only section that’s actually functional is the “light”. Now, the 3.3v – 5v microcontroller power is clearly insufficient to power anything significant. We need a relay. The way a relay works, the small current charges up a coil/electromagnet which moves the switch carrying the larger current.

On the relay that I’m using, there are 5 contacts. The NC stands for “normally closed” and the NO stands for “normally open”.
I took an extension cord, used my multimeter to confirm which side was +, and I spliced it.

If you want to see the schematics, you can find it on the Arduino website.

I don’t own any standalone lamps… so the best I could do is to plug in Christmas Lights…. that is, until I get a better relay and plug it into main. (If you attempt this at home, please be cautious.)

Now that I have 1 or more miniature arduino webservers placed strategically around my house, my internal network configuration looks something like this:

None of the arduino webservers are public facing… Jarvis now runs on a spare 1u rackmount server that’s laying around my house. I communicate with Jarvis, and Jarvis communicates with each of the minions.

Now, the real fun begins. I can look for different components around the house that I wish to control via Jarvis. 🙂

All of the necessary source code can be found on github.

From → Hacks

47 Comments
  1. Mind putting up an index.php file as you did the last time ?

  2. Cunctator permalink

    Great!
    I’m working on the same project, but instead of Jarvis I tought to HAL 😉
    By now I’m writing the Arduino core to dump/repeat IR codes, still in development but working.
    Thank u for sharing those info!
    Btw why are u using an eth. shield on Arduino? Wasn’t the COM enough for this purpose?

    • Hey Cuncator,
      HAL is a great name!
      IR to control IR devices such as TV, etc? That’s awesome. I have an IR transmitter and receiver waiting to be integrated into Jarvis as well!
      I use the eth shield because I have several arduino webservers placed throughout the house. Jarvis can simply make HTTP requests to each of them without being physically connected to the Jarvis server. On top of that, if I run Jarvis locally from say… my laptop, I still have access to all of them. 🙂

      • Cunctator permalink

        Damn man we got almost the same idea! My approach about comunicating with far arduino boards was trough radio signals based on a custom simple protocol, say, a kind of 30Mhz tx/rx that should give a reasonable range at very low power.
        BTW thank u again for ure great work, i’m writing a C version of ure speech-to-text PHP script wich should be simple enough to use it in already existing projects without messing things up ;), for windows and linux, if u think it could be useful i’ll share it here.

      • using a radio frequency is a great idea!
        I’d love to follow the progress of HAL. That would be awesome and definitely useful.

  3. Justin permalink

    Were you ever able to figure out the voice input piece? I spent this weekend toying with a new raspberry pi and was able to implement most of the features you incorporated into Jarvis. Thanks for the awesome documentation!

    • Hey Justin. In the source code, there is a chunk of code that I commented out. That handles the voice to text without relying on chrome’s non-standard HTML.
      I am still waiting for my raspberry pi to arrive in the mail. I can’t wait to get it and experiment with it.
      Did you run into any problems running Jarvis on the raspberry pi? Man, I can’t wait to get mine! I’m also curious how well my face-tracking robot will run on the raspberry pi.

      • Justin permalink

        I didnt use Jarvis. I ended writing everything perl and using your notes and code as a jumping off point. I wrote a “listener” the just runs sox, when it detects silence, it stops recording and encodes that file from wav to flac and sends that to the google stt. As far as speed is concerned, it takes about 3 – 4 seconds for the above process to take place. To speed things up a little i wrote a server/client in perl and the server is running on a “real” computer. My plan is for that guy to do all of the heavy lifting.

      • That’s great! The “listener” is definitely something I need to implement as well. I’m getting tired of clicking on the microphone button each time.

  4. Jarvis permalink

    what operating system is this tutorial used for?

    • So sorry about the late response. I have been swamped with work. I am using Ubuntu Linux 10.04

  5. jarvis permalink

    can send me an email on how to set up my computer for this?

  6. Hey! I’m very interested in this project, though I have to admit my comp sci ends at Java and BASIC, and I’m currently working towards making my own version of Jarvis. I would appreciate any tips you could throw my way through the development process. I am planning on incorporating the speech to text element and plan on more or less creating my own instance of an AI rather than depending on wolfram and such. I feel it would give me more control and flexibility over what I want done.; though I feel I will utilize Wolfram as a fall back to implement general information at most of the general knowledge implementation has already been done on that front. Again, anything you could provide me with would be greatly appreciate! I admire your work! 🙂

    • That’s great! When working on your own AI, you may want to search for key words that help you(the software) determine whether it’s a request for an action or a question, etc. You’ll notice that my version of Jarvis searches for key words such as “turn on” or “turn off” and that forks into another set of instructions that search for “lights, tv, etc”. I notice wolfram and other online AI engines do something similar.

  7. Also, I’d love to know how you input your coding format into wordpress, I have yet to format my own code in such a way…

    • Hi! I’m so sorry for the late reply. I know there are better ways to display code in wordpress… but I’ve simply been using <pre><code>…..</code></pre> and replacing all the < with &lt; and all the > with &gt;

  8. Given the facility/eagerness with which people are replicating the functions of wolfram/apple siri
    implementations, maybe we can dream of one day where they will open the source code.
    When I say dream maybe the should be firmly “persuaded” first…

    jp

  9. glassejt permalink

    Just wanted to say that this is an amazing idea. By far one of the coolest projects I have ever followed. Please keep the posts coming!

  10. chefwear permalink

    Hey Crank, you seen these Ubiquiti mPower devices? It sort of takes the need for a relay out of the equation, at the cost of customization. I’m trying to research hacking the ssh connection. I’ve found a few leads, but so far things look doable but obfuscated.

    http://www.ubnt.com/mfi#m-Power
    http://blog.linitx.com/ubiquiti-mfi-monitoring-control/#comment-57

    • Hey chefwear, I haven’t checked out those devices.. Have you tried using SSLStrip or using ettercap’s built in SSL hacking tools? Neither of them actually “hack” the encryption per say, but allows the man in the middle to either strip the encryption completely or spoof it with its own certificate.

      • chefwear permalink

        Actually, I don’t own one of these devices yet. I was awaiting a response from their support staff regarding whether the device required an mPort device along with it to connect to an ethernet network. It does not need this device, for it can connect to your network using the internal 802.11 adapter. I swear I read somewhere that it required an mPort device, so I wanted to be sure. Anywho…

        In another thread I spoke to you about getting ettercap to work in Ubuntu 12.04 LTS. I keep getting these ‘message too long L3 errors’. I could only get the MITM you described to function correctly using backtrack 5 r3, and it only worked on one site (opensource.com). You mentioned modifying the etter.conf which I attempted to no avail. I’m still trying to get that to work on my Latitude E6420 but it’s getting replaced soon anyways.

        From what the aforementioned blog post about hacking the mPower states, if you ssh to the device before setting it up with the mFI software, I believe you can just login with ubnt/ubnt. The author doesn’t mention the underlying OS, but I suspect it’s busybox or something similar. If you had one, would you try to use SSLStrip or similar to communicate with the device? I was thinking a start would be to use something like Expect to login and do whatever I wanted it to (after changing the default creds, of course). Maybe fire an Expect script off from a webserver or write an app to trigger the commands remotely from a phone or tablet.

  11. Great article! Would love to set this up. I tried using the files provided. Unfortunately Evi is no longer accepting new API users 😦 … I commented out the else statement that tries to use Evi first. Created a Wolfram account and added my correct details. Couldn’t get it to work though unfortunately. The other 2 tests work for google and wolfram but not the jarvis test file. Anyone else having issues?

    • Hey Nick, I noticed the google translate API doesn’t seem to be operating anymore. My own implementation of jarvis uses a standalone text-to-speech engine which doesn’t sound as good, but works faster.

  12. Hey! You are awesome dude! I have just started programming, and know HTML, CSS, and PHP, although not up to a very high level.

    I was just wondering if you had any advise for me, as to how to continue progressing with my programming? Any other languages you would recommend? And how I would get up to a level of programming like you (I mean, I can do PHP, but I don’t understand most of your code. It seems a bit out of my league)?

    • Thanks Oliver! That’s great to hear. The deeper you get into coding, the more you’ll realize it’s not about the language syntax but how you solve problems. Anybody can learn a new language syntax in 24 hrs, but it just takes lots of time and experience to really become a great programmer. Just remember the language you use is nothing but a tool. There is an excellent book I recommend. “Introduction to Algorithms”. If I can keep just 2 books off my shelf it would be this and a KJV Bible. It’s language agnostic and I guarantee it will make you a better engineer no matter what stage you are currently in.

      • Thanks for the reply! I will be sure to buy the “Introduction to Algorithms” book. (Is this it? http://www.amazon.co.uk/dp/0262531968).

        I have only just discovered the “CURL” part of PHP. I don’t really know what it does yet, but it looks cool. I think it’s to talk to other websites, right?

        Anyway, I have been looking at the code and Google’ing different functions, etc. I was just wondering how you take the speech and send it off to the Google server, because some of the code is so foreign to me that I don’t even know what it codes for. Could you expand a bit on that process please?

      • Oliver, check out Codeacademy.com. Great place to learn all kinds of scripting/coding languages.

  13. Hello, do you run the code on an wamp server or similar? because php is serverside, so i don’t know how you host it, for the arduino to connect, doesn’t they both have to be on the same network? Or do you forward the requests to the arduino from a website? Victor

    • Hey Victor. I run the PHP off of a linux box which is on my local network. You can run it remote as well, although I would take some security precautions.

      • Thanks for the very fast reply 🙂 So you host it on a liux server or on a linux based pc (ubuntu?) And how do you connect to it from your laptop? do you open a port? or something?

      • sorry for the late reply. I have been so busy lately. This version of Jarvis has a web interface so you connect to it through Chrome

      • Also when uploaded to my website i can use the google api tts, but when hosting a local server i can’t?

      • There have been some changes to the Google TTS API. I will have an update soon.

      • Also (again) which version of Arduino IDE did you use for the webserver script?

      • I believe I was using the 1.0 IDE

  14. Hank permalink

    This is one of the most amazing things I have ever seen on the Internet. You are truly awesome! Is all the source code on git hub? I am interested in trying to make one of these for myself.

    Thanks!

    -Hank

  15. Andy Rod permalink

    Hi i have been wondering about your JARVIS and thought that this is really cool, but i do not know the programs you used so if you can please tell me them i will be most pleased to make my own JARVIS! P.S. Ive been looking for this for a while and this seems perfect! (this is just in case you didnt see my first reply on the 1st post)

  16. I found a way to get Google Voice Recognition working outside of a browser if you’re interested. I’d love to maybe package with a universal hot key. Reply if you are interested in maybe some collaboration.

    • Andy Rod permalink

      Ya wat is it?!

    • Hi Aaron. If you look at part 1 of this post, I demonstrate how to use Google Voice Recognition outside of a browser. However, I use the browser’s webkit version since it is easier to deal with.

  17. kev1n permalink

    isn`t that mSpeach app for Arduino ? ;d

  18. brandon permalink

    Where did you speak into? Did you have a mic and speaker in with the arduino? And can you post a list of materials? Thanks

Trackbacks & Pingbacks

  1. Building My Own Siri / Jarvis « cranklin.com

Leave a reply to chefwear Cancel reply