Rockhopper wins first place in Expert division!

Needless to say I am rather happy with the result! The contest consisted of 3 runs. On each run there are a total of 4 tasks worth one point each. The tasks are: put out the candle in the lower level of the maze, put out the candle on the upper level, find and mark the "baby" and go up and down the ramp between floors. I was not able to find the baby on any of the runs (the first run it actually did find the baby but dropped the beacon in the hallway outside of the room instead of in the room) put out the candle and made it up and down the ramp (total of 2 points). On my 2nd and 3rd runs it put out both candles and made it up and down the ramp on each for 3 points on each run. My total came to 8 points, the 2nd place robot only scored 2, 3rd place put out one candle for 1 point.

The thing that gave me the edge was reliability. I kept my navigation code very simple (in fact so simple I hesitate to call it "navigation"). I had done a _lot_ of testing ahead of time and tried to make sure it was able to handle any situation without getting stuck. At one point it did manage to get partially wedged against a wall but it _knew_ it was stuck and was able to get free (after 15 or 20 seconds of spinning it's wheels). Other than that my biggest issue was since it doesn't have any real sort of navigation or knowledge of where it is located in the maze it often went into the same room multiple times while missing other rooms entirely. I made some small tweaks (ok, hacks really) to the code between runs and improved it a bit.

The competition was a lot of fun. I enjoyed meeting several of the other robot builders. I had several people ask me a lot of questions about the robot which I really enjoyed. Hopefully people learned some new things from talking to me, I know I learned a few new things from talking to the other builders as well.

I plan on still making a few more updated to the blog with what I have learned over the past couple of months and also some thoughts for the future. I do however need to get back to working on our kitchen before Jane shoots me! So no more robot projects for a little while but I will get back to it! I am thinking I will likely do the competition at Trinity next year but after that I am thinking robot soccer would be a new and interesting challenge.

One last note. For those of you how have followed along and are getting the robot building bug I highly recommend the competition at Trinity. They have entry level divisions for younger builders as well as 2 different high school divisions and a "senior" division open to college students or individual hobbyists. And of course the expert division if you want to run your robot against Rockhopper II next year!

Greetings from Hartford

The big weekend is finally here. Jane and I are in Hartford for the weekend for the Trinity College robot competition. So the robot is officially "done". Of course by "done" I mean I am out of time! There are still a lot of things I would like to do, mostly on the software end but I am still fairly happy with what I have. The robot is able to get around the maze without getting stuck. It is able to put the candle out nearly all of the time, the only "glitch" here is sometimes it bumps into the candle before it sees it. I can find the "baby" somewhat reliably although that hasn't been tested quite as much.

The biggest thing I wasn't able to get to is some sort of mapping software so that it has some concept of where it is at. It is pretty simple navigation code at this point. It simply follows the left wall then after a certain amount of time (which can be configured at compile time) it switches to following the opposite wall. Hopefully this will be enough to prevent it getting stuck in a loop. Unlike previous years they are not going to reconfigure the maze between runs so I hopefully can look at the layout early in the day and make any needed changes.

I am told there are a total of 10 robots entered in the "expert" division. I talked with 3 or 4 of the other "expert" division builders today. It sounds like most of them made the typical mistake of making things too complex and leaving too much to do toward the end. My robot was the only one I saw actually moving about reliably in the maze so it is looking good as long as nothing breaks (knocking on wood as I type of course).

They usually post results on the web page later in the day after the competition if any of you are curious to see how I did. I will certainly add a new post here but likely not until Monday and since that is my last day before going back to work I will probably sleep late!

The final week!

It is crunch time now. I am down to less than a week to go before the contest. Sorry I have not been good about posting lately. Rockhopper is completely built now (or as built as it is going to be). Building the bot itself and getting the low-level programming done and working took longer than expected (i.e. see my last post about the i2c performance issue). So, it looks like once again I don't have much time left to do the interesting high-level AI programming I wanted to do. I do however have it working fairly well now. It can detect and put out the candle without much trouble. It can also do a fairly good job of not getting stuck in the maze. I was having some trouble where it would run into the corner of a wall if it was in the blind spot of my ir range sensors and sometimes the bumper switches would not trigger so it would just keep pushing against the wall (very bad). I added some code to check the wheel speed based on the wheel encoders so now when the rpm of the wheels drops it knows it is stuck. This seems to be working nicely now.

I also built a beeper used to mark where the "baby" is in the maze. The "baby" is a 15 watt light bulb painted black. When the robot finds this it is supposed to drop a beeper that beeps 2 times per second. I am really happy with my solution for how I drop the beeper. I will try to take pictures of it if I get a chance. Basically it is just a motor on a hinge with a small rubber wheel on it. The wight of the motor holds the beeper in place then when the motor is turned on, the wheel spins the beeper off. This also releases the switch on the beeper which turns it on. This all works better than expected. I now need to work on my code for finding the baby itself. I plan on doing that later today.

What I have been working on the last couple of days is trying to use a neural network to interpret the sonar data. I hope that this will help filter out the false pings you commonly get with sonars. So far I am not having much luck but I have found a few critical errors in my code. I wrote the original code late at night long after I should have gone to bed so it isn't as clean as I would like. If I don't get it working somewhat well by lunchtime today I am going to scrap the neural net idea and write some simple code that uses the raw data. It will be more likely to do strange things because of false pings but at this point I am running out of time!

So, looks like I can certainly do the task of the competition but not as well as I would like. Probably the biggest thing that I need that I have done no work on is somewhat to know if it is just doing circles in the maze and not actually exploring new areas. If I get my sonars and the "baby" detection stuff working that will be my next task.

Next Tuesday I go back to work. I really am enjoying my time off but it will be nice to get back to a 5 day work week! I have been averaging about 10 hours a day 7 days a week on this.

The joy of performance bugs

I just realized it has been over a week since my last post. It has been a very busy week. I was able to get a lot of the old code ported (well more like re-written using the original code as a rough guide). It was able to navigate the maze and do decent obstacle avoidance. This all still needs a bit of work but it usable for now. I want to get all the critical bits working at an acceptable level before I work on perfecting all of the code.

I then built my flame sensors which took a bit of time but no real surprises. I took some pics of the sensors but my wife borrowed the camera and I have not gotten around to downloading the photos off of it.

The "fun" part started when I added code to talk to the flame sensors. The sensors are connected directly to a couple of AVR ATMEGA32 microcontrollers (the ones I designed the boards for which I posted about a few weeks back). Then the ARM (which is my main processors running Linux) talks to the AVR using the i2c protocol. When I added this code I ran into some serious latency issues which slowed things down to the point where the robot would constantly hit walls and not be able to navigate at all. I had done some calculations based on the speed of the i2c bus and had figured that it would have been able to read the sensor data many times over with no problems. In reality this obviously wasn't the case.

I was able to get the "oprofile" tool cross compiled for the ARM so I could gather some data to see where it was spending the most time. The data showed that over 20% of the time was being spent inside the i2c driver. Keep in mind that this ARM board does not actually have an i2c device so I am using the i2c "bit bang" driver which implements the driver in software using a couple of general purpose i/o pins. Still it should not have been eating up as much time as it did.

After a _lot_ of debugging and experimenting I came to the realization that the real bottleneck was the AVR was holding the clock line low (which is used in the protocol as a way to let the master device know that the slave device is not yet ready). I also realized that the Linux bit-bang i2c driver doesn't block in this situation (at least not when running with kernel preemption disabled as is the default for ARM). So, the ARM was wasting time spinning while waiting for the clock line to get released.

So I got a chance to learn some more Linux kernel internals! This of course was one of my goals for this project. I was able to modify the code so that it would block if the clock line was being held low and I also added an interrupt handler so that it could wake up when the line was then released.

This didn't _quite_ fix it. My code is made up of multiple threads, each of which implements a different behavior of the robot. Since the thread that was looking at the flame sensors was constantly getting blocked on interrupts the Linux scheduler essentially increased it's priority since it is then an "interactive" thread. This is the same functionality that allows your text editor to remain responsive even when someone else is running another process that is compute bound (i.e. a database). This part I knew how to fix using some pthreads scheduler magic. In the process of fixing this I implemented some other thread related optimizations I had been considering.

I now have the robot being able to navigate the maze with no critical issues (it bumps walls on occasion but not excessively) and is able to find and blow out the candle. None of this is perfect yet as I mentioned but it does work.

My next goal is to get the audio start working. For the "expert" division of the contest you are required to start the robot using a tone that is supposed to simulate a smoke detector going off. I have a few schematics for circuits to do this and I have all the parts but this is one of the analog electronics things I am not all that comfortable with. Hopefully I can get this working without any more big delays. According to the schedule I put together before I started I am 2 weeks behind. Obviously I won't be able to do everything I wanted to do. My schedule did have a full 2 weeks at the end for nothing but debug time so it sounds like that is what I will loose. I guess I just can't allow myself to have any bugs in my code ;)

halfway point report

I am now at the halfway point through my sabbatical and Rockhopper is starting to look somewhat complete as far as the building part of it goes. Now I can start getting serious with writing code. I have spent the last few days with getting the bumper build and installed (along with switches to tell the CPU if it has hit something), fixing up and organizing the wiring and writing a Linux device driver to send the PWM signal to to the motors to control speed.

The first real usable user-space program I wrote was a simple bit of code that allowed me to log in remotely and drive the robot around with keyboard commands. I was surprised at how fast it moves with 12V batteries. I won't be driving the motors at full speed much. I considered just running them at 6V but I want the full 12V to power the fan to blow out the candle. I might be able to run it at full speeds down hallways when the sonars and IR rangefinders both say all is clear but at the speeds it moves it rolls so far after a STOP command that it will likely just crash a lot. I can do active breaking with the motors however I am concerned about how much stress that puts on the motor's gears.

I then wrote a short program to wander around and turn if it sees something close using the ir range finding sensors. This is Rockhopper's first bit of autonomous programming! I set up some boards as boundaries and let it roam about for a while. This pic below shows it happily roaming about.
My next plan is to take the code for roaming about the maze I wrote for my last robot (Fawkes) and port it over. It was written in Java and Rockhopper's code is in C but I am mostly interested in the algorithms I used. Fawkes was able to follow the maze completely flawlessly so that should be a good starting point. Once I get it working farily well I will incorporate the sonar sensor inputs for redundancy.

Rockhopper takes it's first baby steps!

I feel like a proud papa. At around 11:00 AM today Rockhopper took it's first steps. I don't have any sensors hooked up yet but I wrote a quickie program to just turn the motors on for 5 seconds and go straight and it moved across the floor for a few feet completely under battery power and while I was logged in via ssh over the wireless net. I do have one minor bug where it doesn't turn both motors on at the same time. I have some "optimization" code in the motor routine that prevents it from doing anything if the speed has not changed, I think I have a bug there. Shouldn't be too tricky. Once I have that fixed I am going to write a program to drive it around via keyboard commands.

So far I am very happy with how long it runs on the battery. I had it booted with wireless ethernet for over an hour last night on a battery pack that wasn't even fully charged. Granted that was without any sensors and I expect the IR rangefinders will pull a bit of current but this is very promising.

[technical stuff] packed full of electronics goodness!

I spend the day doing more wiring and creating a couple of simple boards "the old fashioned way". One is a breakout board that gives me easy connection to power, ground and signal wires for analog sensors. The other is a board that allows me to connect 3.3V (i.e. my ARM board) and 5V hardware on the same i2c bus. The documentation for the ARM had many very serious warnings about not driving any of the pins above 3.3V. Since this wasn't cheap I figured I should listen!

I now have nearly everything that will go inside the base installed. My planning paid off. I really don't think I would have been able to fit everything in if I had not spend the time planning out the base.

The plan for tomorrow is to write a Linux device driver for the A/D converter board on the ARM and if I get that done I want to put together a library of routines so I can start writing some actual robot code.

I am still way behind schedule but moving forward.

My AVR boards are in

The FedEx guy just delivered my custom designed AVR boards! They look good so far. Hopefully I won't find any surprises when I assemble them and test them out. This was kind of a rush job because I realized I needed them just before we went on the cruise. I needed to get the design sent out before we left so I wouldn't get delayed.

I plan on using 3 of these. Mostly they will be used to add additional analog inputs (each has 8 inputs). Two of them will be connected to 8 IR phototransistors which I will use to find the candle. My last robot used the same sensors however I had only 5 free inputs. I will have 2 sets of 8 of these, each in a radial pattern. I will have one set on the left side and one on the right. This should allow me to determine the distance as well as direction to the flame.

I have spent the last couple of days working on wiring and getting more electronics installed. This has been time consuming and tedious but I know from past experience that if I don't do a neat and tidy job here it will cause lots of issues later. I really thought I would have something functional by now but everything seems to take longer than I expected. I have just under 5 weeks left. I know I will have something working by then but I may not be able to do as much as I dreamed of.

Starting to look like something!

Assembly of the base didn't go quite as smoothly as I had hoped for. I ended up having to re-make a few of the parts. I am however happy with the results (but not happy with the amount of time it has taken to get this far). I got all of the parts for the base assembled yesterday. Today I have been working on getting the electronics and wiring installed.

So far I have the motors, motor driver, 5V regulator and the ARM board installed. I added an external RJ45 ethernet jack for the network (the wired net will be just used for development and debugging, will use wireless for the actually running) and an external DB9 port so I can get to the console. I also added 2 switches to the top. One will control the power to the 5V regulator so I can shut off all the electronics and the other will just kill the power to the motors. I plan on running 3 battery packs. A 9.6V RC car battery pack will power the electronics and 2 - 6V packs wired in serial to give 12V will power the motors. I should be able to have plenty of runtime. The time limit on each round of the competition is something like 5 minutes however I am more concerned about long runtimes when doing debugging.

I discovered today that I ordered the wrong connectors to connect to the batteries. Radio Shack sells them (however they are a bit overpriced) so I will pick up a couple tomorrow. I did have one of the right type on hand so I used that to connect the 5V regulator to the battery. None of my batteries are charged right now but I hope to be able to power the ARM up on battery power in the morning. I think I am done for today. Been a long day and I should probably spend some time with my wife before she forgets who I am!

Getting closer to having something that can roam about under it's own power. I thought I would be there by now but keep hitting snags and having to redesign parts. I still think time spent now will pay off in the end.

Parts Parts Parts!

I realize I have been bad this week about posting updates. I spent the week making parts out of PVC. Honestly much more time was spent designing, cutting pieces and then redesigning. This took longer than I expected (which puts me even farther behind schedule) but I wanted to be sure I have all of this working they way I want. Otherwise it will likely cause more delays later. My goal it to have interchangeable parts that I can easily make duplicates of. So far I have made enough parts for 2 bots. I plan on building 2 both so I have a backup and also because the expert division of the competition allows for cooperating robot swarms so I do plan on using both at the same time if all goes well.

I made the wheels out of 1/2 PVC sheet and silicone. The silicone is sold as mold making material for casting flexible molds which can be used to cast plastic parts. It comes as a liquid with a hardener. This is the first I have used it but it was easy to work with and turned out great. For those interested it is HobbyMold 150. I made a form out of larger PVC pipe which I cut into 3 sections (to allow it to be removed from the final part) and drilled shallow holes with a unibit to create the tread pattern. I am very happy with the result. So far I have only made 1 pair of wheels but most of the time was spend designing the form so making another pair won't take long. It takes about 24 hours for the silicone to cure but not much effort needed at this point.

I have all but 2 minor pieces needed to make the base of the robot. This is where most of my week was spent. Each robot is made up of 18 PVC pieces (not counting the wheels). I designed it so that it should go together like a kit. We will see tomorrow when I actually start assembling the first robot.

Once I was satisfied with the design of each piece I used xfig (a nice simple UNIX drawing program) to draw each part up and print out a template. I then stuck the template onto the PVC sheet using an elmers glue stick. This made drilling holes go fairly quickly. I bought a new drill press a while back that has a laser sight which puts an X right where the hole will go. This made life much nicer. My old drill press didn't have that and it also had a bit of a wobble making accurate holes nearly impossible.

I will put it all together tomorrow. I will take a pic an post it. I should have enough of the electronics mounted to make it move about on its own by the end of the weekend!

Back in cold New Hampshire

New Hampshire is cold this time of year. It seems even colder when you spend the past week on a ship in 85 degree weather! The vacation was very nice. I didn't take a lot of pictures but I am posting a few here. The first two here are Jane and myself each standing in front of the ship in Freeport Bahamas. We didn't do much in Freeport, just wandered around and walked on the beach for a bit.

The second stop for the ship was Grand Caymen. I forgot the camera on the ship so no pics from there. We found a nice outdoor restaurant/bar right on the water where Jane went snorkeling while I had a couple of beers. Fun for the whole family!

The last stop was _supposed_ to be Costa Maya Mexico however the above picture is as close as we got. High winds and 8ft waves made it too dangerous to dock there. The ship you see in the picture got in earlier in the day before the winds got too high. We heard they were stuck there for a couple of days.

So, what did we spend most of our time doing on the trip? The picture below sums it up pretty well. Yes, we are wild and crazy party people!

To the caribbean!

You won't see any updates for a week because Jane and I are heading out on a caribbean cruise! When you live in New Hampshire it is critical to go someplace warm every other winter or so. I will take pictures and post them here when we get back.

[technical stuff] my atmega32 PCB design

Been a very busy day! I plan on using some AVR ATMega32 microcontrollers as a cheap and easy way to add some A/D converters which can do their own averaging / filtering of analog signals. Last night as I was experimenting with the smaller ATMega8 I quickly came to the realization that things would go MUCH more smoothly if I had some PCBs fabed for these (so I don't have wires running everywhere). Since we are leaving Saturday for our cruise I need to get this sent out for manufacturing TODAY!

Now I am far from a hardware expert. I have only designed 1 board before and that was several years ago (and was a much simpler board). Luckily there is some good software for this. Eagle CAD from CadSoft runs on Linux and has a free version which has all the capabilities I need. It took me about 1/2 the day just to learn the software, 1/2 the day to design the board, and 1/2 the day to figure out how to generate the files needed by the manufacturer (yes, that is 3/2, like I said it was a busy day!).

Tomorrow I plan on putting together a digikey order so the last of my components should be here when I get back. The boards have a 10 day lead time (which is why it was critical to get that in now). That will give me a week when we get back for working on the mechanical parts of the bot. So for I _think_ I have things planned out so that I won't waste any time waiting on parts!

[technical stuff] The hard drive saga

I ran into a few snags over this weekend. My goal on Saturday was to get the sonar hooked up so I could experiment with it. I put together a distribution board that I can use to connect multiple i2c devices onto the bus and added a couple resisters to pull the data lines high. Then I went to my workbench computer (an old G3 mac running ubuntu linux) and it crashed! After attempting to reboot I found that it was complaining about bad sectors on the hard drive. Oh well, it is an old computer and all of my critical data is on my newer desktop system. Since it appeared to be just the drive I figured it would be worthwhile to just get a new drive for it. After booting from CD I was able to do an fsck (after several tries) and mount the disk read-only. I backed the data up over the network and called it a day (had a party to go to that night).

I picked up a new 320GB drive and installed it sunday. I made several unsuccessful attempts to install ubuntu to it, fiddled with connections, tried various methods of partitioning the drive, none of which worked. I was starting to suspect the new drive was bad but then did some research. Turns out the IDE controller on the old G3 mac has a maximum of somewhere around 130GB!!!! I was frustrated but at least I understood the problem. I had been wanting to add another drive to my newer system to allow me to do backups. The drive worked just fine there.

So, after nearly 2 days of fiddling (and yes a little slacking off I admit, it was the weekend!) my workbench system was still down. Since the drive seemed happy after the fsck and the data was then backed up I went ahead and put it back in. It seems to be working ok at the moment but I plan on buying a new (under 130GB of course) drive and replacing it soon.

Today (monday) I finally got back to experimenting with the sonar. I was using the linux kernel's i2c "bit bang" code along with my own driver for the digital I/O board from technologic systems for this. I was not able to get the sonar device to send the ack's when it is addressed on the bus. I decided to back up and write some user-space code to implement at least a minimal subset of i2c. With a lot of technical specs, example code, and most importantly my logic analyzer I was able to get it working. The code is a bit of an ugly hack for now but I can use what I learned from this to make the kernel driver work. I imagine it is something simple.

Tomorrow I plan on experimenting with my AVR microcontrollers and and make sure I can communicate with them over the i2c bus as well.

My sabbatical has begun!

You may have noticed that I have not done many updates lately. That should change now that I am officially on sabbatical for the next 8 weeks! It sort of feels like summer vacation when I was a kid (except for the weather). It sounds like a lot of time but I will be busy getting rockhopper going. But don't worry, it won't be all hard work. Jane and I are taking a week long cruise leaving a week from today.

Ok, now more technical stuff...

I received a few more parts in the mail the other day. Most importantly my sonar units. I found some really nice units made by Devantech. I chose the SRF08 sonar because it should be easy to interface using an i2c bus and also because it has the ability to listen for multiple pings. I have not had the chance to do any work with sonar since college but what we found from experimenting back then was the first ping received is often not not always what you want. The plan is to take the data from 3 sonar units and feed that into a neural network. I will write up more on this later on. The plan for today is to just try to interface the sonar to my ARM board to make sure I can get readings.

I also ordered a digital compass, also from devantech. My first robot for the Trinity robot contest also used a compass. I found that while in a building such as the gymnasium at Trinity it will at first appear to give random readings. You can move in a straight line and have massive compass deflection. While this may appear random it _is_ consistent in a single location. I hope to be able to use this to help recognize locations inside the maze. The compass also interfaces using i2c.

[technical stuff] rockhopper gets a brain!

This is _not_ battlebots. While I enjoy robotic mayhem of two robots fighting to the death as much as anyone I find the challenge of autonomous robotics much more interesting. For the robot to be autonomous it of course needs some sort of CPU. My first Trinity competition robot from 2001 used 2 PIC microcontrollers for low level functions connected to a Palm for higher level processing. That worked fairly well however since it was all implemented on solderless breadboards it was less than reliable. It did however show me just how much you could do with a $5 PIC. My second entry from 2005 (Fawkes as shown in my first blog post) used a Motorola 68HC11 based "Handyboard" robot controller.

The most difficult part of these earlier robots was debugging the code. Since these run embedded code with no operating system there is little that can be done for debugging beyond printing short messages on an LCD or flashing an LED. I found myself spending most of my time making minor changes to the code, recompiling, downloading to the robot with a serial cable and trying again. So much time was spent on this that it did not leave time to write any complex AI code.

I needed something that could run a real OS. In my line of work there is only ONE choice here! There are several companies that make single board computers (SBC) using the ARM cpu that run Linux quite well. I made the choice to go with the TS7260 from Technologic Systems.

Basic specs:

• 200 MHz ARM cpu
• on board ethernet
• on board USB
• PC104 bus to allow for expansion options
• 64MB RAM
• 32MB Flash memory

Running Linux means development and debugging should go much more smoothly. Code can be compiled using a cross compiler on my desktop Linux system and copied over to the ARM for testing. For the early development work I am running the ARM booted with the root filesystem over NFS but I will move over the the flash filesystem when the initial kernel and driver work is complete. Having onboard USB will allow me to use wirless networking so I can log in and debug code _while_ the robot is moving about. I also plan on using a USB memory stick for additional filesystem space since the 32MB flash will be tight.

The one downside of the Technologic Systems boards is they only support a 2.4 Linux kernel. With the help of others on the discussion group and a lot of digging I was able to cobble together a set of patches for the 2.6.20 kernel.

Why the name "rockhopper"?

So, I am sure people will ask why "rockhopper"? The robot will be running the linux operating system. The official mascot of linux is "tux" the pengin. I went looking for penguin related names and came up with "rockhopper" which is a penguin breed.

Initial post - Welcome!

Welcome to my blog! My plan is to use this to keep people up to date and to document the building of my entry for this years Trinity College Fire Fighting Home Robot contest. I plan on having a mix of non technical stuff for my "non geek" (and I use the word geek proudly) friends and family as well as some interesting technical details about the robot and it's design for my work friends at HP and Red Hat as well as other Linux and robotics enthusiasts.

Pictured above is my previous Trinity competition entry named "Fawkes" from 2005 (I did not find the time to enter in 2006). I added that picture because at this point there isn't much to photograph of my latest robot since I am still building it. I figured I needed something robot related for my first post.

I have high hopes for this years contest. This will be the first time I have been able to spend a large amount of time on a robotics project because I will be taking a 2 month sabbatical from HP (thanks Suzanne!). So from Feb 17th up until the contest on April 15th I will be out of work and focusing on "Rockhopper" (more on the significance of the name later). Before my sabbatical I plan on spending quite a bit of night and weekend time on it as well.

More to come....