Open Source and Generational Differences

It is time again for another entry from guest blogger Chris Johns. Chris and I have chatted and emailed a lot over the past few months about the issues in this post. They are tough because it is always hard to question your decisions and embrace change. But it is critical to do so on anything that is long-term in your life. RTEMS is a long-term software projects and we need to embrace self-examination and change.

Developers start projects to scratch itches or to bring about change. They join projects as users because they need to use a piece of software. They get involved because they to need to fix bugs or develop new features. The reasons are many, well documented and understood by those who work in or around open source software. What happens when a project becomes old enough that generational change is needed and those who start a project reach an age where they do not have the energy, mental capacity or desire is not well understood. As a project and its leadership age do they move from being intensive productive developers to mentors and governors of the project. Understanding this change is difficult as the interests and focuses of the newer generations are different and sometimes clash with the original developers yet both are right and neither are wrong. The primary function of the project maybe the same, the way it is developed and maintained can be different. Open source is starting to reach this point and some projects have such a long life cycles in user projects it is starting to become an issue. RTEMS is such a project. It is used in space flight and some new projects do not take flight until 2018. Being open source each user has the code and can make changes long past the life of the project, but it is the project and community this discussion is about. 

RTEMS is now 22 years old. It is able to drink, vote and hold a drivers license in most countries. It has experimented with a few things it should not have and so far has not been in trouble with the law. You could say it has had a stable and happy up bring. RTEMS is now looking to the future and life without the current custodians.

RTEMS at its core is a collection of C source files that are built into a C library and linked with user application code to provide single executable image often embedded into a custom piece of hardware. The key factors for the user of this device is performance, resources and stability. The key factors for the developer of this device is availability of source code, easy to use software interfaces, easy to integrate into a team environment, and stability of the project. The key factors for the maintainers of RTEMS is the ability to effectively integrate changes, respond to hardware changes, stable infrastructure and the ability to attract new developers. Developers are the food source that feeds, refreshes and sustains a project.

RTEMS in its post toddler years moved to a new version control tool called CVS that allowed concurrent development of the code. It was liberating because a single set of code did not have to be maintained. Before CVS patches were emailed to the maintainer, merged and then released back to developers as tar files. With CVS this task could be spread among a number of trusted developers. RTEMS also moved from custom makefiles to autoconf and automake. This improved the productivity of the developers allowing the code to be configured and built on a range of host operating systems. RTEMS still uses these same tools 10 to 15 years later and they still work. The developers are comfortable with their work flow and know the problems or issues they have. Why the need to change? There are problems and over time these have grown in size as the project has grown. What were problems are now distance memories and all we have left is the new problems that came with the tools. 

We have files in places that have long since lost there meaning. The board support packages is an example. They are located under 'c/src/lib/libbsp' when they could be located in 'libbsp' or even 'bsps'. This path does not effect the build time or the disk space used and the developers know this path very well so why is this a problem. It makes no sense. Any new users of RTEMS, and by new I mean anyone who has joined in the last 10 years, would have no idea why this structure exists. RTEMS use to have an Ada version and all code was under 'c' or 'ada' and the source was under 'c/src'. Why not move the files? We cannot because CVS does not have a rename command and repository hacks are something we discourage. 

Would we move them if CVS allowed it? Maybe, however this effects the build system. Why is that a problem, is there something wrong with it? Building RTEMS is complex. As a user of RTEMS a release comes with all the autotool's generated files in place ready to work. You can configure RTEMS with a few options passed to configure, plus provide a few more on the command line to the build a range of BSP specific options and then at runtime you have a large array of configurations and runtime options. Are these documented? Only a small number are. The user needs to look into the source to find the full set and even for a seasoned developer this can be complex and not accurate or complete. As a user you just build RTEMS and that does happen and it does it well. By well I mean you get a library of code that is stable and will perform the task asked. As a developer you need to work with the build system and this is where problems start to appear. Performance is an issue. A clean check out from CVS requires a bootstrap to generate all the autoconf and automake files as they are not held in the repository and this can take a lengthy period of time even on large hosts and fast disks. Fortunately this is not often needed as the maintainer mode helps how-ever it makes build-bot type support on check in difficult if not impossible. Also contributing to this is the repeated installing of header files. If you build all 120+ BSPs you will install over 50,000+ header files. This is just building RTEMS and does not include installation of the build output. When installing the 50,000+ files are copied to the install paths. Does this seem normal or ok? Maybe there really needs to be this many headers, or maybe header files have been added to RTEMS following a common template with little regard to the consequence and over the years this has grown to this figure. Most users are only interested in one or two BSPs so this is not a major issue. For a maintainer is it a problem because they need to make sure everything builds and works. 

 I suppose the important questions regarding the build system are "Is it efficient given the new generation of build tools?" and "Does it aid or inhibit the development process?". These are debatable questions which span the boundaries of technical merits, broad range support, supported hosts, and personal preferences. This last one being the most contentious.

The question the current developers and maintainers of RTEMS need to ask is not "Are these tools working and doing the job they are suppose to?", rather if we handed the project to a new group of developers and maintainers "What would new maintainers think of the state of the project?". While we may be comfortable and able to release and maintain RTEMS it may look to a new generation as something from a time past. 

Change is never easy. There needs to be leadership, desire and willingness to refresh to bring about change. It is easy to be negative and to find fault in any new change, then offer no path forward. Leading is not always about "What I think is right", it is about being honest and openly critical of how we work and approach problem solving, and it is about providing paths to new ways of solving problems we face in the project. Not all paths will succeed how-ever being open to change means a new path can be taken until a solution found. Inviting new and young talent to follow these paths and find solutions involves them in the project. They become responsible for various parts and that builds pride and commitment. The hope being someday they will be managing and leading the project.

Flight Software Workshop and Goddard Space Center Visit

OAR was a sponsor of 2011 Workshop on Spacecraft Flight Software at the John Hopkins University Applied Physics Laboratory held October 20 - 22 2011. This was the first time OAR had been an event sponsor and the first time RTEMS stickers have ever been given away. We had spent weeks preparing a new table top display for RTEMS, updating flyers for the project and our services, and having stickers made. The preparation was a lot of work that was outside our primary skill sets. Jennifer Averett is a core RTEMS developer who did the artwork. We are technical folks and not marketing or sales types. If you have ever worked with us to get a quote, you would know. Hence this was a challenge. If you have any suggestions on our display, flyers, etc., please share them.

Chris is much more special.

Mark Johannes, Chris Johns, and I were fortunate enough to have Alan Cudmore take us on a tour of Goddard Space Center. We first visited their SpaceCube lab and saw the smaller current generation and a larger newer one which had been flown on a sounding rocket.

MMS "Flat Sat" Work Area

Alan showed us the MMS laboratory where we chatted and learned about their workflow. The lab included a “flat sat” version with the electronics of a single node from the constellation. Their laboratory also included hardware required to test the system driving all inputs. Very impressive.


The laboratory setup was very nice but we were all shocked when we saw the MMS assembly area and how large each of the four satellites in the constellation was. We expected birthday cake size and they are the diameter of a large rocket body. Each satellite is running a hardened Coldfire CPU with RTEMS and application software built on top of the Core Flight Executive. 

Although not running RTEMS, seeing the James Webb Telescope and Hubble Service Mission assembly areas was a real treat. It is ultimately about the science and I am proud RTEMS is a part of it.

The Workshop on Spacecraft Flight Software  was a real treat. Gedare Bloom and his wife joined Chris, Mark, and I there. I have been to the last three FSW's and have been blown away repeatedly at the incredible work being done by this community. As might be expected, we collectively were delightfully pleased to see so many people using or considering RTEMS. It is quite humbling. Many attendees shared their RTEMS experiences and plans with us. The presentations were video recorded and should be available with slides in the near future.  Presentations and video from previous years is also available online.

All in all, it was a fabulously rewarding experience getting to meet so many people, tour Goddard Space Center, and hear about so many projects.

Johnson Space Center Visit

I recently was invited to teach a week long RTEMS class to 14 people at Johnson Space Center (JSC) . Multiple projects are considering using RTEMS and two are being the trailblazers which bring RTEMS to JSC. Before saying anything else, I want to thank the folks who invited me and were such good hosts. In addition to a self-guided tour of the Rocket Park, I got a field trip where I got to see a few of the interesting things at JSC.

A complete set of photos from my field trips are in a Facebook album. My field trips included seeing the Saturn V rocket, space suits that had been to the moon, Morpheus lunar lander, and the infamous Building 9 where the ISS and space shuttle training modules, CanadARM, and zero-gravity practice facilities. It also includes a small area of cool projects that didn't make the cut including Robonaut and the movie-scary Spidernaut. Everything I saw was impressive but much of it leaves you with an unsettling feeling of sadness when you realize it has been almost forty years since man went to the moon and, with the end of the shuttle program, we have no ability to put a person in orbit. Big science is not cheap and takes years of effort, but without it, we quit learning more about our universe, getting insight into basic physics, and solving the hard problems.

The Morpheus lunar lander is one of the two projects taking the leap and switching to RTEMS. It has already had multiple successful test flights and at least one “interesting” one (videos). Morpheus uses a PowerPC based computer system and is built using the Core Flight Executive from Goddard Space Center. CFE has long supported RTEMS and as more applications are based upon it, I am sure we will see at least a few of those applications use RTEMS.

DownMASS is a small automated capsule that can be filled with contents that need to be shipped from the International Space Station (ISS). It is being designed to hold approximately 100 pounds (43.5 kg) of cargo. When filled and released from the ISS< it will reenter the Earth's atmosphere and eventually deploy its parachutes. The hardware configuration for prototyping is using a ruggedized embedded PC-104 system.

I am thrilled to see more space applications using RTEMS – especially since have applications at Johnson opens a potential path to having man-rated applications using RTEMS. Thanks to Morpheus and DownMASS for giving RTEMS a chance.

Google Code In 2011 Announced

The 2011 edition of Google Code-In has been announced.  Google Code-In is a unique opportunity for up and coming hackers.  It is a competition, an open source development contest for 13-17 year old students around the world. The purpose of the Google Code-in competition is to give students everywhere an opportunity to explore the world of open source development. We not only run open source software throughout our business, we also value the way the open source model encourages people to work together on shared goals over the Internet.

Each participating project identifies a variety of tasks which students can choose to perform.  Tasks are not just coding - they can involve documentation, testing, or outreach.  These are the categories:

1. Code: Writing or refactoring code
2. Documentation: Creating and editing documents
3. Outreach: Community management and outreach, as well as marketing
4. Quality Assurance: Testing and ensuring code is of high quality
5. Research: Studying a problem and recommending solutions
6. Training: Helping others learn more
7. Translation: Localization (adapting code to your region and language)
8. User interface: User experience research or user interface design and interaction

Black stickers are friom Google Code-In task

The RTEMS Project was fortunate to be one of the twenty organization that participated in the 2010 edition of Google Code-In. We identified about 150 potential tasks had almost 100 tasks done by a variety of students.  One of the highlights was a new RTEMS logo which we are using on stickers and project calling cards. Another student modified the shell scripts which generate our test coverage reports to add a timeline capability.  Some students created Wiki pages for the Board Support Packages that did not have one.  And still other students created question sets for the RTEMS Moodle.

The RTEMS Project is planning on applying again this year.  We have some new tasks in mind for those students who volunteer.  It was an interesting and challenging holiday season for the RTEMS mentors.  I know that I personally did a lot of Google Code-In mentoring and task approval using my phone while visiting family.  We are looking forward to the opportunity be a part of Google Code-In 2011 and to be challenged by the students.

RTEMS Configuration and Resource Limits

This post started as an answer to a question from an ESA Summer of Code In Space student. She had hit one of the things that every person new to RTEMS hits at one point. She attempted to create a pthread mutex via pthread_mutex_init() during the package's initialization. It failed and returned EAGAIN. This was an especially surprising error since it happened in a C++ global constructor which was executed before the first task was entered. It appeared to her that RTEMS was not initialized or something even weirder was wrong. RTEMS was, in fact, initialized and since C++ global constructors are supposed to run before main(). On RTEMS, we run them in the context of the first user task which executes. Let me start with an obvious and decidedly unhelpful assertion:

RTEMS != GNU/ Linux

I am sure that didn't help at all to understand why she got an out of resources error. But it is a lead-in to try to explain the philosophical difference between RTEMS and GNU/Linux that leads us to this. The error she encountered was is in fact an out of resources error. Most people start programming on operating systems that do not discourage you from using dynamic allocation and do not put restrictive limits on the number of instances of an object class you can create. For example, on a GNU/Linux system, you don't worry about how many instances of an OS object you create. There are often limits but these are so high as to not present problems. But the Linux kernel does have some behavioural and limits configuration parameters available to the end user.  If you are curious about these, see /etc/sysctl.conf and sysctl(8).

In contrast, RTEMS is a member of a class of real-time operating systems that was designed for to target systems with safety and hard real-time requirements. There are often limited computing resources. In this design view, it is better to pre-allocate as much as possible so you don't have to deal with running out of resources at run-time. This makes the resulting system safer and less likely to have a weird failure mode in this situation. It is not uncommon for the entire set of tasks, semaphores, etc. to be well known and listed in the application design documentation.  Configuring the resources required is common in this environment. Moreover, it is not uncommon for malloc() to be forbidden after system initialization. There is nothing inherently right or wrong with either of these contrasting philosophies. They are just different approaches given different system requirements.

In RTEMS you configure the maximum number of each type of object you want. The defaults tend to be 0. Memory is reserved for RTEMS separate from the C Program Heap based upon your cofniguration requests. The sample in testsuites/samples/ticker has the following configuration in the file system.h:

#include /* for device driver prototypes */

#define CONFIGURE_APPLICATION_NEEDS_CLOCK_DRIVER

#define CONFIGURE_APPLICATION_NEEDS_CONSOLE_DRIVER

#define CONFIGURE_MAXIMUM_TASKS 4

#define CONFIGURE_RTEMS_INIT_TASKS_TABLE

#define CONFIGURE_EXTRA_TASK_STACKS (3 * RTEMS_MINIMUM_STACK_SIZE)

#include

The ticker application says it needs a console (used for stdio) and clock tick (time passage) device drivers. It may have a maximum of four concurrently instantiated Classic API tasks. It is using a Classic API style initialization task -- the alternative is a POSIX Threads initialization thread. And each of the tasks it is creating has a stack larger than the minimum required. It is assumed that each task requires only the minimum amount of stack space so we have to tell the RTEMS configuration to reserve some extra memory for those that are larger than minimum. You can look at the calls to rtems_task_create() in init.c for the calling parameters that indicate the desired stack size.

The hello world sample application is simpler. It doesn't need a Clock device driver and would only have one task. Your application will likely have a more complicated configuration than hello world but it doesn't need to specify any configuration parameters unless it requires that class of object to be supported.

Our young programmer simply missed defining CONFIGURE_MAXIMUM_POSIX_MUTEXES to however many is required. This is an RTEMS specific issue that is not done on non-embedded operating systems.

With all that background on the hard limit focus on RTEMS configuration, it is easy to forget that RTEMS also has "unlimited object creation mode" and "unified workspace" options. This is probably more useful for our intrepid programmer at this stage. These configuration options lets you specify that you want a potentially unlimited number of a class of objects and that you want the RTEMS Workspace and the C Program Heap to be the same pool of memory.

#define CONFIGURE_MAXIMUM_POSIX_MUTEXES \
    rtems_resource_unlimited( 5 )

#define CONFIGURE_MAXIMUM_POSIX_CONDITION_VARIABLES \

    rtems_resource_unlimited( 10 )
#define CONFIGURE_UNIFIED_WORK_AREAS

The above specifies that POSIX mutexes and condition variables are "unlimited" but the set is extended by five (5) instances of mutexes and ten (10) instances of condition variables at a time. When you create the sixth mutex instance, instances 6-10 will be added to the inactive pool for that object class.

The full set configuration macros are (hopefully) well documented in the Configuring a System chapter of the RTEMS User's Manual. This is a link to the appropriate section for RTEMS 4.10.1:

http://www.rtems.org/onlinedocs/releases/rtemsdocs-4.10.1/share/rtems/html/c_user/c_user00414.html

For normal application development, that's really about all there is to this issue. If you get an out of resources error, you will need to raise the limit. When looking inside the code, any time you see a NULL returned from _Objects_Allocate() fail, it is a maximum objects issue. If you see a task, thread or message queue create fail, then you are not accounting for variable memory like stack space or message buffers which must be allocated when the object instance is created.

Since our intrepid young lady is actually porting a software package, let me throw out another thought  which impacts this situation. There is likely a fixed base set of objects the package creates such as for global mutexes or message queues. A user of package X will create instances of objects that it defines. So if the package X has a macaroni object that requires a mutex, condition variable, and a message queue, then you can let the end user of package X know that for each macaroni instance, they need to reserve A, B, and C. For certain cases, like the tasking in Ada and Go, RTEMS provides higher level configuration helpers like CONFIGURE_MAXIMUM_ADA_TASKS to encapsulate this configuration information.

I know my answer to her was a over the top but I realized that she had run into something that many others hit when using RTEMS for the first time. I really wanted her, and now you, to understand this part of RTEMS. Figuring out how many of each kind of object when developing an application can be tough but figuring that same information out when porting a software package is can really be a challenge. Embedded developers focused on safe, reliable systems don't like surprises and using various techniques to avoid running out of resources at run-time is a big part of it.

Elvis Costello and RTEMS.info History

Over the years, I have told many RTEMS users that I provide hosting and system administration for an Elvis Costello fan site (PHPBB Forum and Wiki).  I have had the fan site for about 9 years now.  What many of you probably don't realize is that I have also hosted the RTEMS.info Mirror Site since 2006.

This is a personal effort and I receive no subsidy from OAR for doing this.  In order to have a static IP address and host services, I have to have a business class account which is more than a residential class account.  All of the sites I host plus the family Internet activities share a 7Mbps/1Mbps connection.

I had been helping on the technical side of administrating the Elvis Costello Fan Forum for a while when the hosting service got hacked and our site trashed.  We were unable to get anyone to contact us via email or phone for over a week.  I realized that I had a computer running GNU/Linux Fedora that was largely unused since I had upgraded.  Even though it was only a 350 Mhz Pentium II with 384MB RAM, it was perfectly suitable to host a small (~100K hits a day) website.  I made a phone call to get a static IP address, moved the domains and within about a week we were back up.

A couple of years after that, the person who ran the Elvis Costello Wiki asked if I could host it.  I had already planned to upgrade to a 2.4 Ghz Pentium 4 with 2GB RAM.  We decided to wait to move the Wiki until after the server upgrade. The hardware upgrade went easily and we moved the Wiki.  What surprised us both was that the performance on the site went to hell.  The server logs showed nothing, load looked low and no amount of tuning or probing helped. I begged my ISP for help and got an unlocked, uncapped cable modem to test with.  After more research and fighting, I learned that my router could not hand the number of simultaneous connections and was dropping them randomly.  I upgraded routers and the performance issues were settled.

The site has always used external hard disks for backup.  There is a script which runs every night and dumps all user directories, databases, etc to a special directory on an internal disk.  Then another which runs later and "rsync's" the internal disk copy with an external one.  Backups are placed in dated directories and a few a month are saved.

When I set up the RTEMS.info Mirror, I was more concerned with disk space than bandwidth consumption.  I don't have the fastest connection and I am sure users would appreciate a faster uplink. But I foot the bill and until there is funding, this is what there is.  The RTEMS.info site mirrors at least 4 times a day.  Ralf Corsepius has set up automated checks which let us know when a mirror site is down or out of sync.

In late 2010, I became worried that the 2.4 Ghz Pentium 4 was getting very old.  It was not new when it became the server and I was sure it had seen at least 6 years as server.  The Elvis Costello Fan community rallied around my request for a new server and within a month or so, the fund raising goal was met.  The new server is from AS Labs who specialize in building custom GNU/Linux systems.  It has a quad-core 3.0 Ghz CPU and 8 GB RAM.  It runs very cool and is far from overloaded.

Over the years we have had period power outages with the worst being the tornadoes of April 2011. But overall, I believe our uptime is very good.  I don't track it but thanks to the Elvis Costello fan community, it can't be down over 20 minutes without me getting an email. Thanks folks!

My wife and I have learned a lot about system administration over the years of maintaining these sites.  She personally reviews and approves every account request for the PHPBB Fan Forum.  The number of spam account requests is boggling and periodically she begs me to try to find another way to stem an increase.  The Wiki account requests and spam were solved when we instituted a very strict policy on getting an account.  A small group of people review and approve these accounts.

The server also hosts a couple of very low volume sites for friends.  They are more interesting from a content viewpoint and I want to share more about them in a future post.

RTEMS Pair Programming

One of the most interesting and under-utilized RTEMS services that OAR Corporation offers is RTEMS Pair Programming.  This service is a great solution when dealing with a customer who wants to a big head start on some type of development effort. In Agile terms, this is a development sprint with a team consisting of RTEMS and customer supplied experts.  Most of the time, we do this for BSPs and device drivers.

Left to Right: Walter Nakano, Wendell Pereira da Silva,
Joel Sherrill and Jennifer Averett

The key to pair programming success is that we know RTEMS and the customer knows their hardware and test equipment.  OAR folks can concentrate on quickly providing the framework for the BSP and needed devices drivers. Then we work together to author the device drivers. This provides them with specialized training on the details of the BSPs and device drivers that are critical to the success of their application.  Usually the initial testing is performed as joint effort with subsequent detailed testing performed by the customer engineers.

Recently, OAR got to host Wendell Pereira da Silva and Walter Nakano from COMPSIS  for two weeks of intense development activity.  Their system consisted of an embedded PC plus some add-on boards which added up to a lot of individual pins and ports to test.  They brought a LabView test right which allowed us to test every input and output on the Multi-I/O board.The hardware list was:

• RTD CME137686LX Embedded PC
• 4 COM ports and i82551 NIC of particular interest to their project
• RTD 17320HR Octal UART PC-104 board (PCI interface)
• Exar PCI Vendor Id with 8 NS16550 compatible serial ports
• NOTE: We only had one of these boards but they will have 4 in the real configuration!!
• RTD 316HR Dual Synchronous Serial Port PC-104 board
• single Zilog Z85230
• RTD 6425HR Multi-I/O PC-104 board
• 16 differential or 32 single-ended analog input channels
• 4 analog output channels
• 32-bit discrete I/O with 16 bit programmable for interrupt on input change

One thing should quickly stand out with viewing that hardware list.  They have a LOT of serial ports.  Four asynchronous on the embedded PC, 32 on 4 PCI-104 boards, and 2 synchronous on the Z85230 board for a total of 38 serial ports.  This is actually a classic example of a case where using the libchip serial driver framework would be very useful.  But the PC386 console driver was not designed this way.  Before the guys arrived, Jennifer reworked this driver to be libchip style.  At the same time, I factored out the mouse input stream parsing code.  It really wasn't BSP dependent and by moving it to cpukit/libmisc/mouse, I made it potentially available to every BSP.  Jennifer and I had tested COM1 and COM2 on qemu before they arrived but waited for their hardware to test COM3 and COM4.

COM1 and COM2 worked as soon as the cabling was correct. COM3 and COM4 proved more difficult.  After struggling to find a software problem, it occurred to me that it could be as simple as RTS/CTS not being wired together in the shell since we were using a 3-wire connection.  That was indeed the problem.  Next came the octal serial port board.

After realizing that we would end up with a libchip configuration table with 38 entries and most of them would be disabled for "normal" configurations, I had the idea to allow for dynamic registration of new "ports" in the libchip configuration table.   The idea is that if you probe for a bank of 8 serial ports and find them, then you can dynamically add 8 more entries to the libchip configuration table.  Currently this allows probes to insert entries prior to console_initialize() being called.  It is possible to allow them to be registered after this point but they would not be available to be /dev/console or used for printk().

While I was implementing dynamic registration, Jennifer and our guests worked to get the PCI probe to find the card and the first serial port working.  We were surprised to learn that it didn't have a vendor Id of RTD but Exar.  This explained the sparse programming documentation from RTD.  As soon as the probe and one serial port worked, we switched to my dynamic registration code.  Soon all eight ports on the board we had were working.  Plus I added code to detect the 2, 4, and 8 port variants of the Exar chip.

Next was the dual port synchronous board.  Unfortunately, RTEMS does not have a Z8530 synchronous driver but does have a standard libchip asynchronous driver.  After fiddling to figure out the baud rate clock divisor math, we ended up with both ports working.  There was one issue in the driver we did not resolve in the two weeks they were here.  The two ports on a Z8530 share a single interrupt status register which when read, clears the source.  You have to be extremely careful to touch it one time and process all interrupt sources on both ports.  The ports worked individually but not when both were installed.  Jennifer and I had a solution but not enough time to implement it. Hopefully Wendell and Walter can implement it and we can get this resolved in the main tree.

Next was the Multi-IO board.  If you have been following my blog a while, then you might remember the entry RTEMS Shell as Debug Aid where I discussed adding commands to the RTEMS Shell to aid in debugging a Winsystems Multi-IO board similar in capability to this board.   One of the last things Jennifer and I had done to the existing multiio was to define a board independent interface between the shell commands and the actual driver.  My plan was to let this interface evolve and grow as we learned more about user application requirements.  This was the first opportunity we had to write a driver to this interface and reuse the commands.  As might be expected, there were places where 0/1 based numbering of inputs still reflected the Winsystems board.  And there were places in the RTD documentation that were unclear.  But after a while of fighting these and the normal cabling issues, we were able to use the existing commands to debug the driver and verify that all discrete I/Os to work polled and interrupt driven and that all analog inputs and outputs work polled.  We ran out of time before we were able to attempt analog input interrupts.

The final thing we attempted was getting the RTEMS TCP/IP stack to run on this board.  It had an Intel i82551ER NIC which required using the drivers in the libbsdport kit of late model FreeBSD drivers.  This driver works on qemu when you configure qemu for the i82559 simulation.  We verified the basics were OK on qemu.  Then we moved on to the real hardware.  After the normal hunt for an extra cable and battle of the network settings, we were able to run the telnetd application from the network-demos module.

Walter and Wendell drove their hardware.  Jennifer and I were the main forces driving the code but they reviewed every line of code and we all verified that each line of code programmed the hardware as we all agreed it should be.  Along the way, if something was unclear, we took a break from coding and testing to focus on a portion of the RTEMS Open Class that was very specific to what we were working on. The goal was not only to have as much functional code as possible; it was also to ensure that the code was high quality and they left understanding it and capable of modifying it should the need arise.

At the end of two weeks, we all were thrilled.  Walter and Wendell had been sending home progress reports and every day I continued to be amazed at the progress we -- as a team -- had made.  This amount of progress was possible because each of us brought unique skills and knowledge to the table.  Jennifer and I knew where to reuse code from in RTEMS and how to create an elegant solution the RTEMS way.  Walter and Wendell were intimately familiar with their hardware and test equipment and ensured we tested well.  Together we reviewed all code to ensure they left understanding it.

I really enjoy teaching the RTEMS Classes but RTEMS Pair Programming is one of the most fun and personally rewarding services we offer.  I always come away amazed at how much is working at the end of an intense 2-3 week development sprint.  By bringing together engineers with complementary skills and knowledge, solutions are found quicker.  And solutions are ultimately what we all want.

Thanks to Walter, Wendell and Daniel who couldn't make the trip for two weeks of fun and productive work.