RTEMS Shell as a Debug Aid

Until the 4.9 release series, the RTEMS Shell was very primitive and only a few commands existed. Virtually no one used it. But with 4.9, the RTEMS Shell took a major leap forward with Chris Johns and I doing a lot of work on it. We added commands, command line history, command line editing, and simple scripting. You can use the shell from a serial port or via telnet. There are now approximately 100 standard commands with some such as ls, mv, cp, ln, and dd ported over from NetBSD. In addition, there are RTEMS specific commands to look at CPU usage per thread, stack usage, and rate monotonic period statistics. There are commands to examine the state of most RTEMS OS objects.

Chris Johns used the standard file related commands to great benefit when developing and debugging the RTEMS File System (RFS). He would mount remote NFS volumes and copy great quantities of data to an IDE hard disk. This allowed him to place stress on his new file system and even turned up a bug in the NFS client code.

But the most useful capability for developing and debugging user applications is probably the capability to include custom commands. These allow you to write commands which are specific to your hardware configuration or application. I have used this to capability to write a set of commands for the Winsystems PCM-MIO-G multi-function I/O PC-104 module. (Kudos to Winsystems for relicensing their GNU/Linux driver to be compatible with RTEMS licensing requirements.) This board has the following features:

• Two 8-channel, 16-bit Analog-to-Digital (A/D)
• Two, 4-channel, 12-bit Digital-to-Analog (D/A)
• 48 Bidirectional I/O lines with interrupt support

For testing the driver for this board, I decided to write a series of custom RTEMS Shell commands to configure the analog inputs and perform various IO operations. These commands allowed direct interaction with the hardware and can be used during development as well as hardware checkout and integration. I implemented the following commands as well as providing shorter aliases for each command without the "pcmmio_" prefix:

• pcmmio_din - Read PCMMIO Discrete Inputs
• pcmmio_dout - Write PCMMIO Discrete Outputs
• pcmmio_adc - Read PCMMIO Analog Inputs
• pcmmio_adc_mode - Set PCMMIO Analog Input Modes
• pcmmio_dac - Write PCMMIO Analog Outputs
• pcmmio_irq - Wait for PCMMIO Interrupts
• pcmmio_bench - Benchmark PCMMIO Interrupts

All the test hardware I had available was a multimeter and a push button. Although minimal, this was sufficient for me to test nearly all of capabilities on this board. For example, I could use the pcmmio_din command to poll for the discrete inputs. When I pressed the button, the command reported the state changed.

[/] # pcmmio_din -i 10
Polling discrete inputs for 10 iterations with 1000 msec period
665:159912852 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
669:238111788 0000 0000 1000 0000 0000 0000 0000 0000 0000 0000 0000 0000
671:250111878 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

The board cam be configured such that when a discrete input changes an interrupt is generated. The device driver's interrupt handler determines the pin which changed, its current value and timestamps it using the Time Stamp Counter (TSC) register. This information is placed in a message buffer and send via an RTEMS Classic API Message Queue to an application task which is blocked waiting. This allows the application to know as precisely as possible when an input has changed and process that change at the task level. An example output of the pcmmio_irq command when using the push button (which bounces) is below:

[/] # pcmmio_irq -d -i 5
Polling for DIN IRQ for 5 iterations with 1000 msec period
1000 DIN irq pin 8 @ b9eba88c0e (0 usecs since last)
2000 DIN irq pin 8 @ b9eba932b2 (42 usecs since last)
3000 DIN irq pin 8 @ b9eba9c52e (37 usecs since last)
4000 DIN irq pin 8 @ b9ebadec4e (272 usecs since last)
4 total interrupts from DIN in 5000 milliseconds

In the above example, the command looked for interrupts for 5 iterations of a loop with a delay of 1000 milliseconds between iterations. But the interrupts from pushing the button and it bouncing occurred over a 272 microsecond period. In real application code, you would not put a long delay in between each check but block forever or with a reasonable timeout.

I could test analog output (DAC) by simply entering a command to write a value to a particular DAC channel using the pcmmio_dac command and verifying that the proper voltage was written using my multimeter.

[/] # pcmmio_dac 0 5
Write 5.0000 to to dac 0

The pcmmio_dac command has an interesting feature where you can http://pc104.winsystems.comwrite a "step" pattern. This steps from a low voltage to a high voltage using the specified step voltage and time between steps. When it reaches the high voltage, the command begins to step down. The following example illustrates using the pcmmio_dac command to write a step pattern to DAC 0. The pattern ranges from -2.5V to 2.5V with a .5V change every 250 milliseconds for a total of 10,000 milliseconds. When the voltage reaches 2.5V, the step will change to -.5V.

[/] # pcmmio_dac 0 -2.5 2.5 .5 250 10000
Write -2.5000-2.5000 step=0.5000 stepTime=250 msecs dac=0 max=10000 msecs


When testing analog input (ADC), I attached one DAC output to one ADC input. Then I used the command pcmmio_dac to write a voltage and pcmmio_adc to read a voltage. Just as pcmmio_din can monitor the discrete inputs for changes, the pcmmio_adc command can monitor the ADCs for changes in input. The following commands illustrate using this command to read all ADCs or just a single ADC a single time.

[/] # pcmmio_adc
1117:232053 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
[/] # pcmmio_adc 0
1120:495519 0.0000

Note that in the above, the voltage read isn't the voltage that was written. There are a few potential reasons for this. First, I could have hooked things up wrong (but I checked and I didn't mess that up). Second, I could have gotten confused on the DAC and ADC channels I used. Yes, I did that a couple of times. But the final reason was that I forgot to initialize the channels for the input configuration I was using. This lead to the need for a command to configure an ADC channel.

Each ADC channel could be individually programmer for either single-ended or differential input, unipolar or bipolar voltage ranges and for 5V or 10V as the upper voltage in the range. I didn't want to enter sixteen commands by hand, so I added the feature where pcmmio_adc_mode can configure a contiguous range of ADC's to a particular setting. But this still could require multiple commands. With a flash of insight, I remembered that I could write a shell script to do this for me. This led to me writing the the following very simple shell script to configure the ADCs.

#! joel
echo "Setting all ADCs to +/-10V (bipolar) and single ended"
pcmmio_adc_mode 0 15

As mentioned earlier, to test the ADCs I had to write a voltage using the pcmmio_dac command and then read the voltage using the pcmmio_adc command. I wanted to run a series of voltages through the ADC but the step command didn't let me see the input between steps. I could have figured out a way to get access to the lines but I had used a pre-made jumper wire and didn't want to destroy it. So I wrote another simple shell script which repeated pcmmio_dac, sleep, pcmmio_adc commands. This allowed me to verify that a range of voltage could be written and read. The following is one set of the three commands. There were a lot more than this to have a script that ran for fifteen seconds.

pcmmio_dac 4 -2.0
sleep 1
pcmmio_adc 0

You might wonder how I got the shell scripts onto the target. Well I used another interesting feature of RTEMS. RTEMS has long has the In-Memory File System (IMFS) and the capability to load initial contents from a tar file image linked with the application. I simply wrote the scripts on my development machine and included them in the initial file system contents.

In using the RTEMS Shell and its ability to add custom commands, I was able to refactor the original GNU/Linux device driver, adapt it to RTEMS, add capabilities such as timeouts and timestamping input, and debug this device driver with very little difficulty. Plus these commands are now available for hardware integration and testing for this project and any other project that might use this device driver in the future.

Running SLOCount on RTEMS

SLOC is everyone's favorite metric to hate but it is fun to discuss anyway. I recently ran David Wheeler's SLOCCount [1] on RTEMS and thought I would pass along some of the results. There is an RPM available for Fedora so I started with that. But I found a bug in which it counted our assembly include files as Pascal. Needless to say there is no Pascal in RTEMS so I fixed that. If anyone wants the patch, I am happy to provide it.

The first part of the report from this program is a breakdown of the source code by language. This showed the following breakdown based upon programming language:

ansic:       644676 (87.49%)
asm:          39569 (5.37%)
ada:          27563 (3.74%)
sh:           18204 (2.47%)
cpp:           5236 (0.71%)
perl:          1623 (0.22%)

No real surprise there. RTEMS is mostly in C and high level languages with assembly language primarily for context switch and interrupt dispatching. With over a dozen active ports on the CVS head, it is no shock there is 39.5K of well commented assembly language.
The Shell and Perl is primarily for configuration and build infrastructure.

The second part of the output from sloccount is related to the estimated cost of producing the software and how many man years it would take. Using a rough estimate of 100K USD salary for each software developer implementing RTEMS, sloccount estimates RTEMS would require about 205 person-years to recreate at an estimated cost of 49.2M USD.

Total Physical Source Lines of Code (SLOC)                = 736,871
Development Effort Estimate, Person-Years (Person-Months) = 205.02 (2,460.21)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 4.05 (48.59)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 50.64
Total Estimated Cost to Develop                           = $ 49,204,211
(average salary = $100,000/year, overhead = 2.40).

That makes RTEMS a great bargain! You are getting very high quality software that would cost a sizeable fortune to reinvent. IUnfortunately, the RTEMS team doesn't have $49,000,000 so please consider using the services of the core developers. That's how we pay the bills and keep the kids from going hungry.

Enjoy the gift of RTEMS!

--joel


[1] Per David's request, this data was "generated using David A. Wheeler's 'SLOCCount'."

When Does a GSOC Project End?

Today is the deadline for turning in student evaluations for the 2010 edition of the Google Summer of Code. The RTEMS Project was fortunate enough to have eight students this year and they all did great work. Some of it is still not merged and some needs some polish and love before being production ready. But it looks like all the students successfully met their goals. Personally, I am proud of all the students this year.

So the GSOC 2010 program is nearing its official end and it is tempting for the students to consider their work as over and end their involvement with RTEMS. But that's not the goal of GSOC nor is it desirable from an RTEMS perspective. All projects participating in GSOC want code written during the summer but they also really want long-term contributors. So this is not an end, it is a transition. What do you do next?

As a minimum, you need to work with the mentors to get your code merged into the various projects, functional, documented, and tested. We want your code to be part of the RTEMS experience moving forward. This is not a classroom exercise where the code is thrown away.

Last year, I told someone that you really should make your goal that you should be proud of what is merged and available. You should be able to cite this work as an example of what you are capable of to future employers and graduate schools. Just as all FOSS developers take pride in their work, you should also. Work with us to help get it to that point.

We would like to see you all continue your involvement. This would give you a chance to see YOUR work being used by real world applications. Your involvement does not have to be limited to the area of your GSOC project. RTEMS is a broad project and you can work on something else if that is what you want to do. It is also perfectly acceptable to simply continue to work to perfect your submission. We strive for 100% test coverage, great documentation and exceptional performance. Those are hard to achieve in a first implementation of anything.

So students, do not disappear. Stay involved.

May -- A TIme of Beginnings and Endings

For still in school or who have children in school, May is always an interesting and busy month. It was the end of another academic year for my four children and the end of high school for two of them. It is the anniversary month for my high school, Bachelors and PhD graduations. At the high school graduation commencement, we were once again reminded by the speaker that commencement is a ceremony to celebrate the end of one thing and the beginning of another. It is more a recognition of a milepost on a journey than a goal in itself. At this point, you are probably asking yourself what this has to do with software and the answer is nothing and everything.

A successful software project lives on -- it is not something that is ever complete. Each release announcement is comparable to a graduation commencement. It captures what we have done and is just a milepost. It is a useful milepost in the software world because it represents a completion point. It is a recognizable measure like a diploma that means that the software has passed some measure of completeness and quality.

RTEMS is now over twenty years old. It is the same age as my daughter who is entering her senior year in college. It has matured just as she has. It has gotten more capabilities just as my children have learned more. It is smarter and more efficient than it was. My children are also somewhat more self-sufficient (not efficient yet) but I can hope. RTEMS is in many ways my first child. I was there at the birth. I held its hand as it moved from a research project to an independent free software project.

Today I cut the 4.10 release branch. This represents a major milestone in the life of RTEMS. I can say with certainty and the pride of a parent that this is the best release branch so far. With both coverage testing and the daily builder, the testing has improvement markedly over the past few years. We have automated testing for all of the GNU tools we use. We have overhauled the main web site to be more modern looking and hopefully friendlier. The core technical content was largely unchanged from the older site so this is a lot like a girl getting a make-over before a school dance.

But just as with a graduation commencement, there is a past and a future. The future for RTEMS includes both maturation and additional capabilities. I want to see the RTEMS Project mature its processes by further expand our automated testing and improve the patch review and merge process. I want to see RTEMS grow in capability and I know this is going to happen because there are multiple interesting efforts waiting to be reviewed and merged. There is a SPARC64 port with multiple BSPs, a USB stack, a port of the LWIP stack, and my work on symmetric multiprocesssing support for RTEMS.

So remember that life is full of milestones but those are also points of reflection and landings from which one can climb to the next level. So reflect a bit on 4.10 and look forward to what RTEMS 4.11 or 5.0 will look like. I am excited and hope you are too.

RTEMS SMP Exploration #1

Many of you know that I have been working on an SMP design for RTEMS for a couple of years now as a background activity. This has been a slow project and I have been wanting to work on it more but it has been almost entirely volunteer activity on my part. But recently, the activity level has picked up. Gedare Bloom's GSOC proposal Modular SuperCore Scheduler Manager will make a significant dent in the work required to support SMP. Currently, there is no discrete Scheduler component in the RTEMS SuperCore and a big part of supporting SMP is to have a single processor and an SMP-aware scheduler. To properly support this, there has to be a discrete Scheduler component in the SuperCore and a mechanism to select alternate schedulers. Gedare's project is to refactor the existing code base and provide a SuperCore Scheduler Handler along with the configuration required. Gedare has previously implemented an Earliest Deadline First (EDF) Scheduler for RTEMS and we are planning for this to be available as one of the alternative schedulers. Tiny/RTEMS may even benefit because it may make sense to have a light-weight scheduler algorithm for lower end target processors.

With Gedare focusing on refactoring the Scheduler, I have felt free to focus on the process of initializing a multicore processor, interprocessor interrupts, spinlocks, and transferring control to the first thread on a secondary core. I have succeeded in demonstrating these on pc386 with up to 32 cores and on SPARC/LEON3 with four cores. This work has let me work through the definition of a per CPU OS structure as well as the BSP/RTEMS SMP Interface. If you are interested in experimenting with this on another architecture, let me know.

I am looking forward to working this summer with Gedare and seeing how close we get to SMP RTEMS support before the 2010 Summer of Code is over.

Google Summer of Code 2010

It has been a while since I posted but life and RTEMS have been busy. The past few weeks have been a blur as GSoC 2010 approached. The UT Chattanooga IEEE-CS student chapter visited NASA's Marshall Space Flight Center here in Huntsville and OAR hosted them for lunch. Some Asterisk developers joined us and we presented GSoC, RTEMS and Asterisk to them. Earlier in the year, I had put together a cool double-sided flyer with RTEMS GSOC on one side and the general GSoC information on the other. Both sides had tear-offs which aligned close enough where you get the general and RTEMS specific GSoC URLs. For this visit, I put together a new flyer with the list of all participating organizations. I think it is important to let students know about all the cool organizations that participate.

The past couple of weeks have been hectic. I had work deadlines looming, one of my sons had H1N1, and RTEMS had more students than ever asking questions and submitting proposals for review. I had to update the VirtualBox image of Fedora 12 with RTEMS tools for the i386 and sparc pre-installed and make the torrent available. We had more students complete the hello world requirement and have proposals to review before the web app opened than we did all of last year!! We ask them to put their proposal into Google Docs and let mentors insert comments into that. We have so many students, I created a Google Docs Spreadsheet to track when each mentor had read the proposal and whether the student had submitted the official application. I spent this weekend ensuring I had read all of the proposals in hand.

Overall, I am thrilled with this year's crop of proposals and the number of cross-project proposals. We have students interested in Go on RTEMS, Parrot on RTEMS, and RTEMS support for the MilkyMist board. On top of that, we have students interested in adding a pluggable scheduler capability and EDF scheduler, coverage improvements, adding POSIX asynchronous and list IO, and much more. And two students from last year are back in a mentoring capacity. With a week to go before the student application deadline, I expect we will see more proposals and hope we are able to do a good job reviewing those. I know we will try to do so and be fair to all students making proposals. I am excited and hope we get enough slots to take on the students with quality, well thought out projects.

Ramblings From Radiology

As I have mentioned to many of you, I had an excess of protein in my urine and have had to have a number of medical tests. Today I am at Huntsville Hospital and just got back from a kidney biopsy. I have to get another scan in a few hours to ensure I am not bleeding internally. What does this have to do with RTEMS you ask? Well, nothing directly. But being bored and surrounded by medical devices made me remember some of the RTEMS based medical devices people have told me about over the years.

The first device I heard of was an HP (now Philips) Criticare heart monitor. Philips/HP Criticare equipment is common in our area but I don't know if I have actually ever seen one of these.

I assisted in the development of a BSP for what was going to be a blood analyser for Roche. I recall it be m683xx-based but do not know exactly it tested for.

The AMV Technic I is a certified syringe pump for sale in a number of European countries. It uses RTEMS on an ARM CPU and we have information in the RTEMS Wiki. It uses Microwindows to provide the graphical user interface.

There are also RTEMS based intercom systems which are used by patients to summon or talk to a floor nurse. This product is ARM based and is on its second generation of hardware using RTEMS. Seeing a user build multiple product generations on RTEMS is personally very satisfying. It makes me know they are pleased with RTEMS.

There may well be other medical devices out there, but I don't know about them. Unfortunately, many people do not tell us about their application. I don't know how to change this but will continue to beg for marketing literature from RTEMS based products. Please save me from begging! Just send me the information.