The joy of performance bugs

I just realized it has been over a week since my last post. It has been a very busy week. I was able to get a lot of the old code ported (well more like re-written using the original code as a rough guide). It was able to navigate the maze and do decent obstacle avoidance. This all still needs a bit of work but it usable for now. I want to get all the critical bits working at an acceptable level before I work on perfecting all of the code.

I then built my flame sensors which took a bit of time but no real surprises. I took some pics of the sensors but my wife borrowed the camera and I have not gotten around to downloading the photos off of it.

The "fun" part started when I added code to talk to the flame sensors. The sensors are connected directly to a couple of AVR ATMEGA32 microcontrollers (the ones I designed the boards for which I posted about a few weeks back). Then the ARM (which is my main processors running Linux) talks to the AVR using the i2c protocol. When I added this code I ran into some serious latency issues which slowed things down to the point where the robot would constantly hit walls and not be able to navigate at all. I had done some calculations based on the speed of the i2c bus and had figured that it would have been able to read the sensor data many times over with no problems. In reality this obviously wasn't the case.

I was able to get the "oprofile" tool cross compiled for the ARM so I could gather some data to see where it was spending the most time. The data showed that over 20% of the time was being spent inside the i2c driver. Keep in mind that this ARM board does not actually have an i2c device so I am using the i2c "bit bang" driver which implements the driver in software using a couple of general purpose i/o pins. Still it should not have been eating up as much time as it did.

After a _lot_ of debugging and experimenting I came to the realization that the real bottleneck was the AVR was holding the clock line low (which is used in the protocol as a way to let the master device know that the slave device is not yet ready). I also realized that the Linux bit-bang i2c driver doesn't block in this situation (at least not when running with kernel preemption disabled as is the default for ARM). So, the ARM was wasting time spinning while waiting for the clock line to get released.

So I got a chance to learn some more Linux kernel internals! This of course was one of my goals for this project. I was able to modify the code so that it would block if the clock line was being held low and I also added an interrupt handler so that it could wake up when the line was then released.

This didn't _quite_ fix it. My code is made up of multiple threads, each of which implements a different behavior of the robot. Since the thread that was looking at the flame sensors was constantly getting blocked on interrupts the Linux scheduler essentially increased it's priority since it is then an "interactive" thread. This is the same functionality that allows your text editor to remain responsive even when someone else is running another process that is compute bound (i.e. a database). This part I knew how to fix using some pthreads scheduler magic. In the process of fixing this I implemented some other thread related optimizations I had been considering.

I now have the robot being able to navigate the maze with no critical issues (it bumps walls on occasion but not excessively) and is able to find and blow out the candle. None of this is perfect yet as I mentioned but it does work.

My next goal is to get the audio start working. For the "expert" division of the contest you are required to start the robot using a tone that is supposed to simulate a smoke detector going off. I have a few schematics for circuits to do this and I have all the parts but this is one of the analog electronics things I am not all that comfortable with. Hopefully I can get this working without any more big delays. According to the schedule I put together before I started I am 2 weeks behind. Obviously I won't be able to do everything I wanted to do. My schedule did have a full 2 weeks at the end for nothing but debug time so it sounds like that is what I will loose. I guess I just can't allow myself to have any bugs in my code ;)

halfway point report

I am now at the halfway point through my sabbatical and Rockhopper is starting to look somewhat complete as far as the building part of it goes. Now I can start getting serious with writing code. I have spent the last few days with getting the bumper build and installed (along with switches to tell the CPU if it has hit something), fixing up and organizing the wiring and writing a Linux device driver to send the PWM signal to to the motors to control speed.

The first real usable user-space program I wrote was a simple bit of code that allowed me to log in remotely and drive the robot around with keyboard commands. I was surprised at how fast it moves with 12V batteries. I won't be driving the motors at full speed much. I considered just running them at 6V but I want the full 12V to power the fan to blow out the candle. I might be able to run it at full speeds down hallways when the sonars and IR rangefinders both say all is clear but at the speeds it moves it rolls so far after a STOP command that it will likely just crash a lot. I can do active breaking with the motors however I am concerned about how much stress that puts on the motor's gears.

I then wrote a short program to wander around and turn if it sees something close using the ir range finding sensors. This is Rockhopper's first bit of autonomous programming! I set up some boards as boundaries and let it roam about for a while. This pic below shows it happily roaming about.
My next plan is to take the code for roaming about the maze I wrote for my last robot (Fawkes) and port it over. It was written in Java and Rockhopper's code is in C but I am mostly interested in the algorithms I used. Fawkes was able to follow the maze completely flawlessly so that should be a good starting point. Once I get it working farily well I will incorporate the sonar sensor inputs for redundancy.

Rockhopper takes it's first baby steps!

I feel like a proud papa. At around 11:00 AM today Rockhopper took it's first steps. I don't have any sensors hooked up yet but I wrote a quickie program to just turn the motors on for 5 seconds and go straight and it moved across the floor for a few feet completely under battery power and while I was logged in via ssh over the wireless net. I do have one minor bug where it doesn't turn both motors on at the same time. I have some "optimization" code in the motor routine that prevents it from doing anything if the speed has not changed, I think I have a bug there. Shouldn't be too tricky. Once I have that fixed I am going to write a program to drive it around via keyboard commands.

So far I am very happy with how long it runs on the battery. I had it booted with wireless ethernet for over an hour last night on a battery pack that wasn't even fully charged. Granted that was without any sensors and I expect the IR rangefinders will pull a bit of current but this is very promising.

[technical stuff] packed full of electronics goodness!

I spend the day doing more wiring and creating a couple of simple boards "the old fashioned way". One is a breakout board that gives me easy connection to power, ground and signal wires for analog sensors. The other is a board that allows me to connect 3.3V (i.e. my ARM board) and 5V hardware on the same i2c bus. The documentation for the ARM had many very serious warnings about not driving any of the pins above 3.3V. Since this wasn't cheap I figured I should listen!

I now have nearly everything that will go inside the base installed. My planning paid off. I really don't think I would have been able to fit everything in if I had not spend the time planning out the base.

The plan for tomorrow is to write a Linux device driver for the A/D converter board on the ARM and if I get that done I want to put together a library of routines so I can start writing some actual robot code.

I am still way behind schedule but moving forward.

My AVR boards are in

The FedEx guy just delivered my custom designed AVR boards! They look good so far. Hopefully I won't find any surprises when I assemble them and test them out. This was kind of a rush job because I realized I needed them just before we went on the cruise. I needed to get the design sent out before we left so I wouldn't get delayed.

I plan on using 3 of these. Mostly they will be used to add additional analog inputs (each has 8 inputs). Two of them will be connected to 8 IR phototransistors which I will use to find the candle. My last robot used the same sensors however I had only 5 free inputs. I will have 2 sets of 8 of these, each in a radial pattern. I will have one set on the left side and one on the right. This should allow me to determine the distance as well as direction to the flame.

I have spent the last couple of days working on wiring and getting more electronics installed. This has been time consuming and tedious but I know from past experience that if I don't do a neat and tidy job here it will cause lots of issues later. I really thought I would have something functional by now but everything seems to take longer than I expected. I have just under 5 weeks left. I know I will have something working by then but I may not be able to do as much as I dreamed of.

Starting to look like something!

Assembly of the base didn't go quite as smoothly as I had hoped for. I ended up having to re-make a few of the parts. I am however happy with the results (but not happy with the amount of time it has taken to get this far). I got all of the parts for the base assembled yesterday. Today I have been working on getting the electronics and wiring installed.

So far I have the motors, motor driver, 5V regulator and the ARM board installed. I added an external RJ45 ethernet jack for the network (the wired net will be just used for development and debugging, will use wireless for the actually running) and an external DB9 port so I can get to the console. I also added 2 switches to the top. One will control the power to the 5V regulator so I can shut off all the electronics and the other will just kill the power to the motors. I plan on running 3 battery packs. A 9.6V RC car battery pack will power the electronics and 2 - 6V packs wired in serial to give 12V will power the motors. I should be able to have plenty of runtime. The time limit on each round of the competition is something like 5 minutes however I am more concerned about long runtimes when doing debugging.

I discovered today that I ordered the wrong connectors to connect to the batteries. Radio Shack sells them (however they are a bit overpriced) so I will pick up a couple tomorrow. I did have one of the right type on hand so I used that to connect the 5V regulator to the battery. None of my batteries are charged right now but I hope to be able to power the ARM up on battery power in the morning. I think I am done for today. Been a long day and I should probably spend some time with my wife before she forgets who I am!

Getting closer to having something that can roam about under it's own power. I thought I would be there by now but keep hitting snags and having to redesign parts. I still think time spent now will pay off in the end.

Parts Parts Parts!

I realize I have been bad this week about posting updates. I spent the week making parts out of PVC. Honestly much more time was spent designing, cutting pieces and then redesigning. This took longer than I expected (which puts me even farther behind schedule) but I wanted to be sure I have all of this working they way I want. Otherwise it will likely cause more delays later. My goal it to have interchangeable parts that I can easily make duplicates of. So far I have made enough parts for 2 bots. I plan on building 2 both so I have a backup and also because the expert division of the competition allows for cooperating robot swarms so I do plan on using both at the same time if all goes well.

I made the wheels out of 1/2 PVC sheet and silicone. The silicone is sold as mold making material for casting flexible molds which can be used to cast plastic parts. It comes as a liquid with a hardener. This is the first I have used it but it was easy to work with and turned out great. For those interested it is HobbyMold 150. I made a form out of larger PVC pipe which I cut into 3 sections (to allow it to be removed from the final part) and drilled shallow holes with a unibit to create the tread pattern. I am very happy with the result. So far I have only made 1 pair of wheels but most of the time was spend designing the form so making another pair won't take long. It takes about 24 hours for the silicone to cure but not much effort needed at this point.

I have all but 2 minor pieces needed to make the base of the robot. This is where most of my week was spent. Each robot is made up of 18 PVC pieces (not counting the wheels). I designed it so that it should go together like a kit. We will see tomorrow when I actually start assembling the first robot.

Once I was satisfied with the design of each piece I used xfig (a nice simple UNIX drawing program) to draw each part up and print out a template. I then stuck the template onto the PVC sheet using an elmers glue stick. This made drilling holes go fairly quickly. I bought a new drill press a while back that has a laser sight which puts an X right where the hole will go. This made life much nicer. My old drill press didn't have that and it also had a bit of a wobble making accurate holes nearly impossible.

I will put it all together tomorrow. I will take a pic an post it. I should have enough of the electronics mounted to make it move about on its own by the end of the weekend!

Back in cold New Hampshire

New Hampshire is cold this time of year. It seems even colder when you spend the past week on a ship in 85 degree weather! The vacation was very nice. I didn't take a lot of pictures but I am posting a few here. The first two here are Jane and myself each standing in front of the ship in Freeport Bahamas. We didn't do much in Freeport, just wandered around and walked on the beach for a bit.

The second stop for the ship was Grand Caymen. I forgot the camera on the ship so no pics from there. We found a nice outdoor restaurant/bar right on the water where Jane went snorkeling while I had a couple of beers. Fun for the whole family!

The last stop was _supposed_ to be Costa Maya Mexico however the above picture is as close as we got. High winds and 8ft waves made it too dangerous to dock there. The ship you see in the picture got in earlier in the day before the winds got too high. We heard they were stuck there for a couple of days.

So, what did we spend most of our time doing on the trip? The picture below sums it up pretty well. Yes, we are wild and crazy party people!