Friday, October 28, 2011

Flight Software Workshop and Goddard Space Center Visit

OAR was a sponsor of 2011 Workshop on Spacecraft Flight Software at the John Hopkins University Applied Physics Laboratory held October 20 - 22 2011. This was the first time OAR had been an event sponsor and the first time RTEMS stickers have ever been given away. We had spent weeks preparing a new table top display for RTEMS, updating flyers for the project and our services, and having stickers made. The preparation was a lot of work that was outside our primary skill sets. Jennifer Averett is a core RTEMS developer who did the artwork. We are technical folks and not marketing or sales types. If you have ever worked with us to get a quote, you would know. Hence this was a challenge. If you have any suggestions on our display, flyers, etc., please share them.

Chris is much more special.
Mark Johannes, Chris Johns, and I were fortunate enough to have Alan Cudmore take us on a tour of Goddard Space Center. We first visited their SpaceCube lab and saw the smaller current generation and a larger newer one which had been flown on a sounding rocket.

MMS "Flat Sat" Work Area
Alan showed us the MMS laboratory where we chatted and learned about their workflow. The lab included a “flat sat” version with the electronics of a single node from the constellation. Their laboratory also included hardware required to test the system driving all inputs. Very impressive.

The laboratory setup was very nice but we were all shocked when we saw the MMS assembly area and how large each of the four satellites in the constellation was. We expected birthday cake size and they are the diameter of a large rocket body. Each satellite is running a hardened Coldfire CPU with RTEMS and application software built on top of the Core Flight Executive. 

Although not running RTEMS, seeing the James Webb Telescope and Hubble Service Mission assembly areas was a real treat. It is ultimately about the science and I am proud RTEMS is a part of it.

The Workshop on Spacecraft Flight Software  was a real treat. Gedare Bloom and his wife joined Chris, Mark, and I there. I have been to the last three FSW's and have been blown away repeatedly at the incredible work being done by this community. As might be expected, we collectively were delightfully pleased to see so many people using or considering RTEMS. It is quite humbling. Many attendees shared their RTEMS experiences and plans with us. The presentations were video recorded and should be available with slides in the near future.  Presentations and video from previous years is also available online.

All in all, it was a fabulously rewarding experience getting to meet so many people, tour Goddard Space Center, and hear about so many projects.

Thursday, October 20, 2011

Johnson Space Center Visit

I recently was invited to teach a week long RTEMS class to 14 people at Johnson Space Center (JSC) . Multiple projects are considering using RTEMS and two are being the trailblazers which bring RTEMS to JSC. Before saying anything else, I want to thank the folks who invited me and were such good hosts. In addition to a self-guided tour of the Rocket Park, I got a field trip where I got to see a few of the interesting things at JSC.

A complete set of photos from my field trips are in a Facebook album. My field trips included seeing the Saturn V rocket, space suits that had been to the moon, Morpheus lunar lander, and the infamous Building 9 where the ISS and space shuttle training modules, CanadARM, and zero-gravity practice facilities. It also includes a small area of cool projects that didn't make the cut including Robonaut and the movie-scary Spidernaut. Everything I saw was impressive but much of it leaves you with an unsettling feeling of sadness when you realize it has been almost forty years since man went to the moon and, with the end of the shuttle program, we have no ability to put a person in orbit. Big science is not cheap and takes years of effort, but without it, we quit learning more about our universe, getting insight into basic physics, and solving the hard problems.

The Morpheus lunar lander is one of the two projects taking the leap and switching to RTEMS. It has already had multiple successful test flights and at least one “interesting” one (videos). Morpheus uses a PowerPC based computer system and is built using the Core Flight Executive from Goddard Space Center. CFE has long supported RTEMS and as more applications are based upon it, I am sure we will see at least a few of those applications use RTEMS.

DownMASS is a small automated capsule that can be filled with contents that need to be shipped from the International Space Station (ISS). It is being designed to hold approximately 100 pounds (43.5 kg) of cargo. When filled and released from the ISS< it will reenter the Earth's atmosphere and eventually deploy its parachutes. The hardware configuration for prototyping is using a ruggedized embedded PC-104 system.

I am thrilled to see more space applications using RTEMS – especially since have applications at Johnson opens a potential path to having man-rated applications using RTEMS. Thanks to Morpheus and DownMASS for giving RTEMS a chance.

Sunday, October 16, 2011

Google Code In 2011 Announced

The 2011 edition of Google Code-In has been announced.  Google Code-In is a unique opportunity for up and coming hackers.  It is a competition, an open source development contest for 13-17 year old students around the world. The purpose of the Google Code-in competition is to give students everywhere an opportunity to explore the world of open source development. We not only run open source software throughout our business, we also value the way the open source model encourages people to work together on shared goals over the Internet.

Each participating project identifies a variety of tasks which students can choose to perform.  Tasks are not just coding - they can involve documentation, testing, or outreach.  These are the categories:
  1. Code: Writing or refactoring code
  2. Documentation: Creating and editing documents
  3. Outreach: Community management and outreach, as well as marketing
  4. Quality Assurance: Testing and ensuring code is of high quality
  5. Research: Studying a problem and recommending solutions
  6. Training: Helping others learn more
  7. Translation: Localization (adapting code to your region and language)
  8. User interface: User experience research or user interface design and interaction
Black stickers are friom Google Code-In task
The RTEMS Project was fortunate to be one of the twenty organization that participated in the 2010 edition of Google Code-In. We identified about 150 potential tasks had almost 100 tasks done by a variety of students.  One of the highlights was a new RTEMS logo which we are using on stickers and project calling cards. Another student modified the shell scripts which generate our test coverage reports to add a timeline capability.  Some students created Wiki pages for the Board Support Packages that did not have one.  And still other students created question sets for the RTEMS Moodle.

The RTEMS Project is planning on applying again this year.  We have some new tasks in mind for those students who volunteer.  It was an interesting and challenging holiday season for the RTEMS mentors.  I know that I personally did a lot of Google Code-In mentoring and task approval using my phone while visiting family.  We are looking forward to the opportunity be a part of Google Code-In 2011 and to be challenged by the students.

Friday, October 14, 2011

RTEMS Configuration and Resource Limits

This post started as an answer to a question from an ESA Summer of Code In Space student. She had hit one of the things that every person new to RTEMS hits at one point. She attempted to create a pthread mutex via pthread_mutex_init() during the package's initialization. It failed and returned EAGAIN. This was an especially surprising error since it happened in a C++ global constructor which was executed before the first task was entered. It appeared to her that RTEMS was not initialized or something even weirder was wrong. RTEMS was, in fact, initialized and since C++ global constructors are supposed to run before main(). On RTEMS, we run them in the context of the first user task which executes. Let me start with an obvious and decidedly unhelpful assertion:

RTEMS != GNU/ Linux

I am sure that didn't help at all to understand why she got an out of resources error. But it is a lead-in to try to explain the philosophical difference between RTEMS and GNU/Linux that leads us to this. The error she encountered was is in fact an out of resources error. Most people start programming on operating systems that do not discourage you from using dynamic allocation and do not put restrictive limits on the number of instances of an object class you can create. For example, on a GNU/Linux system, you don't worry about how many instances of an OS object you create. There are often limits but these are so high as to not present problems. But the Linux kernel does have some behavioural and limits configuration parameters available to the end user.  If you are curious about these, see /etc/sysctl.conf and sysctl(8).

In contrast, RTEMS is a member of a class of real-time operating systems that was designed for to target systems with safety and hard real-time requirements. There are often limited computing resources. In this design view, it is better to pre-allocate as much as possible so you don't have to deal with running out of resources at run-time. This makes the resulting system safer and less likely to have a weird failure mode in this situation. It is not uncommon for the entire set of tasks, semaphores, etc. to be well known and listed in the application design documentation.  Configuring the resources required is common in this environment. Moreover, it is not uncommon for malloc() to be forbidden after system initialization. There is nothing inherently right or wrong with either of these contrasting philosophies. They are just different approaches given different system requirements.

In RTEMS you configure the maximum number of each type of object you want. The defaults tend to be 0. Memory is reserved for RTEMS separate from the C Program Heap based upon your cofniguration requests. The sample in testsuites/samples/ticker has the following configuration in the file system.h:

#include /* for device driver prototypes */




The ticker application says it needs a console (used for stdio) and clock tick (time passage) device drivers. It may have a maximum of four concurrently instantiated Classic API tasks. It is using a Classic API style initialization task -- the alternative is a POSIX Threads initialization thread. And each of the tasks it is creating has a stack larger than the minimum required. It is assumed that each task requires only the minimum amount of stack space so we have to tell the RTEMS configuration to reserve some extra memory for those that are larger than minimum. You can look at the calls to rtems_task_create() in init.c for the calling parameters that indicate the desired stack size.

The hello world sample application is simpler. It doesn't need a Clock device driver and would only have one task. Your application will likely have a more complicated configuration than hello world but it doesn't need to specify any configuration parameters unless it requires that class of object to be supported.

Our young programmer simply missed defining CONFIGURE_MAXIMUM_POSIX_MUTEXES to however many is required. This is an RTEMS specific issue that is not done on non-embedded operating systems.

With all that background on the hard limit focus on RTEMS configuration, it is easy to forget that RTEMS also has "unlimited object creation mode" and "unified workspace" options. This is probably more useful for our intrepid programmer at this stage. These configuration options lets you specify that you want a potentially unlimited number of a class of objects and that you want the RTEMS Workspace and the C Program Heap to be the same pool of memory.

    rtems_resource_unlimited( 5 )
    rtems_resource_unlimited( 10 )

The above specifies that POSIX mutexes and condition variables are "unlimited" but the set is extended by five (5) instances of mutexes and ten (10) instances of condition variables at a time. When you create the sixth mutex instance, instances 6-10 will be added to the inactive pool for that object class.

The full set configuration macros are (hopefully) well documented in the Configuring a System chapter of the RTEMS User's Manual. This is a link to the appropriate section for RTEMS 4.10.1:

For normal application development, that's really about all there is to this issue. If you get an out of resources error, you will need to raise the limit. When looking inside the code, any time you see a NULL returned from _Objects_Allocate() fail, it is a maximum objects issue. If you see a task, thread or message queue create fail, then you are not accounting for variable memory like stack space or message buffers which must be allocated when the object instance is created.

Since our intrepid young lady is actually porting a software package, let me throw out another thought  which impacts this situation. There is likely a fixed base set of objects the package creates such as for global mutexes or message queues. A user of package X will create instances of objects that it defines. So if the package X has a macaroni object that requires a mutex, condition variable, and a message queue, then you can let the end user of package X know that for each macaroni instance, they need to reserve A, B, and C. For certain cases, like the tasking in Ada and Go, RTEMS provides higher level configuration helpers like CONFIGURE_MAXIMUM_ADA_TASKS to encapsulate this configuration information.

I know my answer to her was a over the top but I realized that she had run into something that many others hit when using RTEMS for the first time. I really wanted her, and now you, to understand this part of RTEMS. Figuring out how many of each kind of object when developing an application can be tough but figuring that same information out when porting a software package is can really be a challenge. Embedded developers focused on safe, reliable systems don't like surprises and using various techniques to avoid running out of resources at run-time is a big part of it.