Ramblings From Radiology

As I have mentioned to many of you, I had an excess of protein in my urine and have had to have a number of medical tests. Today I am at Huntsville Hospital and just got back from a kidney biopsy. I have to get another scan in a few hours to ensure I am not bleeding internally. What does this have to do with RTEMS you ask? Well, nothing directly. But being bored and surrounded by medical devices made me remember some of the RTEMS based medical devices people have told me about over the years.

The first device I heard of was an HP (now Philips) Criticare heart monitor. Philips/HP Criticare equipment is common in our area but I don't know if I have actually ever seen one of these.

I assisted in the development of a BSP for what was going to be a blood analyser for Roche. I recall it be m683xx-based but do not know exactly it tested for.

The AMV Technic I is a certified syringe pump for sale in a number of European countries. It uses RTEMS on an ARM CPU and we have information in the RTEMS Wiki. It uses Microwindows to provide the graphical user interface.

There are also RTEMS based intercom systems which are used by patients to summon or talk to a floor nurse. This product is ARM based and is on its second generation of hardware using RTEMS. Seeing a user build multiple product generations on RTEMS is personally very satisfying. It makes me know they are pleased with RTEMS.

There may well be other medical devices out there, but I don't know about them. Unfortunately, many people do not tell us about their application. I don't know how to change this but will continue to beg for marketing literature from RTEMS based products. Please save me from begging! Just send me the information.

NASA Solar Dynamic Observatory Launched

The NASA Solar Dynamic Observatory (SDO) was successfully launched this morning. The SDO satellite includes 5 Radiation Hardened Coldfire CPUs running RTEMS. The Coldfire/RTEMS computers run many of the I/O functions and one of the instruments. A very nice graphic of the mission is here. The following is the description of the project from the SDO Project Site:

SDO: The Solar Dynamics Observatory is the first mission to be launched for NASA's Living With a Star (LWS) Program, a program designed to understand the causes of solar variability and its impacts on Earth. SDO is designed to help us understand the Sun's influence on Earth and Near-Earth space by studying the solar atmosphere on small scales of space and time and in many wavelengths simultaneously.

SDO's goal is to understand, driving towards a predictive capability, the solar variations that influence life on Earth and humanity's technological systems by determining

• how the Sun's magnetic field is generated and structured
• how this stored magnetic energy is converted and released into the heliosphere and geospace in the form of solar wind, energetic particles, and variations in the solar irradiance.

Congratulations to the folks at NASA Goddard on a successful launch and good luck in meeting their science goals. I look forward to the new insights we will gain into the Sun.

RTEMS Graphics Toolkit

I haven't posted since before the holidays but that doesn't mean that nothing has been going on in the land of RTEMS. I have been trying to work with Roxana Leontie to get her work merged into the Microwindows/Nano-X repository. Roxana was an RTEMS Summer of Code student. The RTEMS Project took the Google donation from the Google Summer of Code and money from OAR Corporation to fund Roxana as the inaugural RTEMS Summer of Code Student. Roxana worked to update and clean up the ancient port of Nano-X to RTEMS. Roxana did all of her work targeting the RTEMS pc386 BSP. The screenshot in the upper right is the Microwindows minesweeper demo program.

As Roxana wrapped up her work, she cooperated with a Google Summer of Student (Xi Yang) who had developed a Gumstix Connex BSP for RTEMS which ran on the Skyeye simulator. Xi implemented a frame buffer device driver for this board which worked with Nano-X. All of Xi's work was merged into the RTEMS Source Code Repository when completed.

As part of preparing her work to be merged into the Nano-X Source Code Repository, I worked with Roxana to clean up the build infrastructure and ensure that the code built easily on multiple BSPs. As part of this, we worked to port some supporting libraries to RTEMS. Before Greg Haerr had merged our work, we had created the RTEMS Graphics Toolkit (RTEMS CVS module rtems-graphics-toolkit) which includes the following libraries:

• libjpeg - (v7) http://www.ijg.org/
• libtiff - (v3.6.2) http://www.libtiff.org/
• libpng - (1.2.42) http://www.libpng.org/pub/png/libpng.html
• libfreetype - (2.3.11) http://freetype.sourceforge.net/index2.html
• libt1 - (5.1.2) http://www.t1lib.org/

This means you can decode and display JPEG, TIFF, PNG, GIF, and BMP graphics on an RTEMS system. It also means that you may have support for Adobe Type 1 TrueType fonts. The screen shot with the "Embedded with RTEMS" logo is the output of nxview running on the arm/gumstix BSP. The logo is a jpeg loaded from the RTEMS In-Memory File System (IMFS). The jpeg support library has no patches for RTEMS.


Using a build script in the top directory of this kit, you can easily build all of these packages and Nano-X (from their git repository). This includes building all demos for a specific BSP. The RTEMS Graphics Toolkit also includes the following libraries which are not currently ported but we would like to see ported.

• svgalib - (1.4.3) http://svgalib.org/
• fltk - (1.1.10) http://www.fltk.org/

Roxana is pursuing porting svgalib to bring its collection of more advanced graphics drivers to RTEMS. Interestingly, this ties in with the work of another Google Summer of Code Student, Aanjhan Ranganathan. Aanjhan's project consisted of adding MMU support to RTEMS. This includes support on some targets for the mmap() family of calls from POSIX. The svgalib package includes approximately 100 calls to members of this call family so we now have incentive for Aanjhan to get his work tidied up and merged.

FLTK is an important piece of the puzzle because it includes a GUI builder and numerous widgets. This will provide a robust foundation to build GUIs for RTEMS systems.

At this point, all Nano-X demo programs that are expected to run on RTEMS do so on both the pc386 and arm/gumstix platforms. There is obviously more to do. But it is important to emphasize that this is a major step forward for RTEMS. It makes it much easier to build GUIs for RTEMS based embedded systems.

Thanks to all the folks at Google who work so hard on the Summer of Code and the students they sponsor. This would not have happened without them. Thanks Leslie, Ellen, and Kat. :)

RTEMS Turns 21

The RTEMS ticker test (a.k.a. sp01) was the first test created during the development of RTEMS. It sets the date and time to 9:00 pm on 31 December 1988. This coincides with the earliest part of the RTEMS Project. Since there is no birth certificate, we are treating this as RTEMS birth date.

Please toast RTEMS turning 21 where ever you are.

Please don't be online -- share the New Year's Eve with loved ones. :)

Violating the Rule of Least Surprise

When you mention user interfaces, people usually think of a graphical interface with mouse and keyboard or a touchscreen. But user interfaces do not exist solely in the world of computer displays. In the world of embedded systems, a user interface can take on any number of variations. One guiding principle that particularly applies in this domain is the "Rule of Least Surprise." An example I use is that if you put someone behind the steering wheel of a vehicle of any type, they would expect that turning the wheel to the left directs the vehicle in that direction. To do anything else, would surprise even a three year old who has ridden a tricycle.

Even though I love my current car, it has two areas which violate this rule. As shown in the picture shown here (source), the instrument panel is in the center -- not directly in front of the driver. Both interface surprises stem from this placement.

The first violation is that when the lights are on at night, there is no light at all directly visible in the direct field of vision of the driver. It is very easy to forget to turn the lights on at night when in a well-lit area because you have to look to the middle to realize that the instruments are unlit. This has actually resulted in me getting pulled over by a friendly police officer to remind me to turn my lights on.

The second violation is when you turn on the turn signal to turn left. The blinking light indicating you have done so is in the right hand side of the driver's field of vision. In other vehicles, this would be in the left hand side
of your field of vision.

Neither of these is a major issue and I have grown accustomed to both. But both show how humans have expectations about how common devices operate. When your clever design changes this, users are surprised. In the end there may be a benefit, but often it is a difference just to be different. Is there a surprise in a system you are familiar with? Is it an improvement or just a difference? Share.

Fedora 12 Upgrade Experience

A few weeks ago, I upgraded the GNU/Linux computers I am personally responsible for from Fedora 10. This includes a work computer (Dell Latitude D830), a home computer (Dell Inspiron 2400), and the home server (upgraded Dell Dimension 2100). The home server is the host of an Elvis Costello Fan Forum phpBB site as well as the Elvis Costello Wiki so it is not just my family that is impacted by down time. I upgraded from Fedora 10 to 11, ran for about a week to make sure things were OK, and then upgraded to Fedora 12. In the end, all of the computers are up and stable performing their assigned tasks but there were issues along the way which I will describe here.

I upgraded via DVD and all of the computers handled that OK once I remembered the magic required to boot from DVD. But afterward the upgrade was complete, two of the computers did not boot. As not so infrequently during GNU/Linux upgrades, GRUB was not updated correctly on two of them and I had to go into rescue mode to address that. This just required mounting the root partition, changing root and running grub-install. Scary the first time it is required, but not after a few times.

One of the laptops booted nicely to the GUI login prompt but was clearly at the wrong resolution. I did a little net research and learned that deleting the /etc/X11/xorg.conf file and restarting X11 would result in a probe and very likely the correct video settings. This worked.

The only issue shared by all of the machines was an issue with yum getting upgraded between Fedora 10 and 11. yum was broken after the upgrade. Before I could load further updates from then using yum upgrade, I had to manually fetch a current yum rpm file and install it manually using rpm.

The server had three very strange issues during the upgrade.

First, I couldn't log in. The GUI login screen was flashing. I remembered the magic key sequence to switch to a console display, logged in, and checked out the logs. I couldn't tell exactly what was wrong but I had a guess and tried it. Apparently the graphical login program GDM had trouble parsing my email address in the name field of /etc/passwd. I think something was trying to use the user name field in an XML file because my entry was something like "Joel Sherrill ". I suspect that the presence of my email address looking like an XML tag confused it. I deleted that and moved along. GDM was happy.

Next on the list was that something changed in MySQL and the user name table was reported as being corrupt. I found reports on the web where others had suffered this and decided the easiest solution was to stop MySQL, delete the old table, and restart MySQL. When restarted, MySQL would create the table with default contents. From there, I could easily add back in the handful of accounts that were needed. The actual database tables were not impacted. So I felt lucky and moved on to the next problem.

At this point, I started looking at the actual web content served from the server. I first looked at the pbpbb-based Elvis Costello Fan Forum, A bit of the page header was displayed along with an error message. I don't remember why but somehow, I couldn't see the error completely on my side. I enlisted some help on the #rtems IRC channel to see if others could see what the message said. Someone cut and pasted me the error message. I Google'd the message and learned that the new php 5.3 now requires the /etc/php.ini file to explicitly set the timezone. It does not trust the operating system's setting. Without the timezone being set for php, php programs reported nasty error messages. This meant that phpbb3 would not run correctly and the Fan Forum was not available. Adding the following to /etc/php.ini resolved this.

[Date]
  date.timezone = America/Chicago

I inspected the pages served from the Elvis Costello Wiki and there were some minor error messages in the logs. Apparently, Mediawiki had some minor mistakes in its coding which php 5.3 detected. These resulted in a plethora of errors in the web logs.

I am a daily GNU/Linux user who rarely works in MS-Windows. As a long time user of various UNIX desktop environments, I prefer mousing over a window to move focus. I do not like to click on a window to move the focus. In previous Fedora distributions, GNOME included an applet to switch the preference under "System > Preferences > Windows". This applet is no longer included by default. I don't know why it isn't part of the default install anymore. Is 59K really that critical? Anyway, the following installed it.

  yum install control-center-extra

At this point, I have been using Fedora 12 daily on two desktops and a server. I use the two desktops for development, email, and word processing using Open Office. I have encountered a few issues which I will share in a future post.

Debugging Data Corruption #1

I recently helped someone debug a data corruption problem on an embedded system that was not RTEMS based and that I had not written any code for. Just to be sure this is perfectly clear, I knew nothing about the source code to this application or its design when I arrived. They had a global variable whose value was becoming invalid. For the purposes of this discussion, the corrupted variable will be referred to as CorruptedVariable.. They had a way to reproduce the corrupted and could tell based upon the behaviour of the program that at least CorruptedVariable was being clobbered.

When I arrived, I listened a few minutes and watched them reproduce the problem. I then asked they knew how to produce a numerically sorted symbol table for their application. They did not and I explained that with any GNU toolset using binutils, there was included a utility named nm which is used to produce a symbol table from an executable. I instructed them to use a command similar to the following:

  avr-nm -g -n application >application.num

The -g option requests that the symbol table include only global symbols and the -n option requests that the symbol table be sorted by address. We then looked at application.num in a text editor and I had them search for CorruptedVariable.

I then looked at the variables immediately before (e.g. lower addresses immediately preceding) CorruptedVariable in memory. The variable immediate preceding CorruptedVariable was a 32-bit integer (RandomVariable so it was unlikely that accesses to it were the culprit. But the variable before it had buffer in the name. That was a big hint and I asked to see the corresponding source code. The C source code where the CorruptedVariable was declared was similar to the following:

    float  TooShortBuffer[8];
    int    RandomVariable;
    int    CorruptedVariable;

So TooShortBuffer was indeed an array and not a structure. For the purposes of this bug, that was awesome. It meant that using an index that was greater than 8 would overwrite the variables immediately after it in memory. And we had confirmed what the variables which would be overwritten were.

I asked what TooShortBuffer was used for and one of the team mentioned that they had recently added something and might have forgotten to extend all the arrays in the program. As a quick test, I asked them to add 100 elements to that array. Not surprisingly the test scenario now worked.

I recommended to the team that the array now be sized with a hard number but with a macro defined to the maximum number of elements. In addition, I recommended that when using an index into this array to access it, validate that the index is within range. Self-checking programs are so much easier to debug.

I spent less than an hour on the customer site and left then with a smile.

In my next post, I will show how gdb can be used to help locate the source of stray writes.