RTEMS Ramblings by Dr Joel

MINIX versus Linux versus BSD

2011-11-20T14:31:00.001-06:00

This morning an article was posted to Slashdot in which Andrew Tanenbaum is interviewed. One question and answer from the interview seemed to draw the most reaction on Slashdot. LinuxFr.org asked: "If you could return in the past to change the MINIX original proprietary licence to the GPL licence, do you think your system might have become the dominant free OS today?". Andrew Tanenbaum answered:

Never. The reason MINIX 3 didn't dominate the world has to do with one mistake I made about 1992. At that time I thought BSD was going to take over the world. It was a mature and stable system. I didn't see any point in competing with it, so I focused MINIX on education. Four of the BSD guys had just formed a company to sell BSD commercially. They even had a nice phone number: 1-800-ITS-UNIX. That phone number did them and me in. AT&T sued them over the phone number and the lawsuit took 3 years to settle. That was precisely the period Linux was launched and BSD was frozen due to the lawsuit. By the time it was settled, Linux had taken off. My mistake was not to realize the lawsuit would take so long and cripple BSD. If AT&T had not brought suit (or better yet, bought BSDI), Linux would never have become popular at all and BSD would dominate the world.

Now as we are starting to go commercial, we are realizing the value of the BSD license. Many companies refuse to make major investments in modifying Linux to suit their needs if they have to give the code to their competitors. We think that the BSD license alone will be a great help to us, as well as the small size, reliability, and modularity.

My first UNIX experience was in the 1984-85 timeframe and my first job out of college was developing software for intelligent I/O controllers for a UNIX System V mini-computer. I remember what commercial UNIX was like in those days. You may have wanted it but the options were limited and expensive. On the 80286, you had XENIX. When the i386 came out, there were a number of nice options including Interactive 386/ix and even SCO UNIX. Yes SCO had a decent product back in the day.

When I could finally afford a computer capable of running some UNIX-ish system, I found myself in on the consumer side of what the question is about.

From my perspective, I didn't use MINIX because I viewed it as an educational and teaching OS. Its desired user base was not "real" users doing non-academic work. We had experimented with it in our labs at work and found it quite primitive in comparison to the "real UNIX" we were used to. Personally, I found the acquisition process painful as well. But all UNIXy systems were painful to get back then. The focus on educational users was it for me. I don't ever want to be the "odd user" of anything who is not in the desired target audience for a product.

Why did I choose a Linux distribution over a BSD? I honestly don't remember. My vaguest recollection is that I preferred System V more than BSD systems and Linux leaned to System V. I doubt this was a factor for most others though. If I had to guess, I would go back and look at how you had to obtain it, community responses to newbies, etc.. Was the AT&T lawsuit have factor? Maybe. Linux was certainly perceived to be immune from that lawsuit among those I knew. It did not suffer from that heritage.

Finally, I want to look back with the free software community building experience I have. When viewed from this perspective and the prism of time, I think the answer has a lot to do with what we should have learned from Google Summer of Code. A project has to be easy to obtain, get started with, contribute to, have a vibrant and friendly community, etc.. The license is important but as long as it is imposes legal impediments or obligations, that won't stop most people. In the old days, Minix was not really easy to obtain and was not focused on general use. It was not available as an impulse download. That was enough of a hurdle to stop a lot of folks.

When one examines the choices faced by someone who wanted a UNIX-like system on their personal computer in the early 1990's, it is easy to see how Linux was the default choice. It simply did not have the "targeted to teaching operating systems" stigma, was easy to obtain, and didn't have a lawsuit looming over its head.

But one of the nice things about free software is that if there is interest, a project will continue on. MINIX 3 is a great OS that has a BSD-style license, is easy to obtain, and they are clearly interested in MINIX 3 being used for more than teaching operating systems design. Variety is the spice of life. I would recommend that you give it a try and tell them I sent you.

Open Source and Generational Differences

2011-11-10T16:41:00.001-06:00

It is time again for another entry from guest blogger Chris Johns. Chris and I have chatted and emailed a lot over the past few months about the issues in this post. They are tough because it is always hard to question your decisions and embrace change. But it is critical to do so on anything that is long-term in your life. RTEMS is a long-term software projects and we need to embrace self-examination and change.

Developers start projects to scratch itches or to bring about change. They join projects as users because they need to use a piece of software. They get involved because they to need to fix bugs or develop new features. The reasons are many, well documented and understood by those who work in or around open source software. What happens when a project becomes old enough that generational change is needed and those who start a project reach an age where they do not have the energy, mental capacity or desire is not well understood. As a project and its leadership age do they move from being intensive productive developers to mentors and governors of the project. Understanding this change is difficult as the interests and focuses of the newer generations are different and sometimes clash with the original developers yet both are right and neither are wrong. The primary function of the project maybe the same, the way it is developed and maintained can be different. Open source is starting to reach this point and some projects have such a long life cycles in user projects it is starting to become an issue. RTEMS is such a project. It is used in space flight and some new projects do not take flight until 2018. Being open source each user has the code and can make changes long past the life of the project, but it is the project and community this discussion is about.

RTEMS is now 22 years old. It is able to drink, vote and hold a drivers license in most countries. It has experimented with a few things it should not have and so far has not been in trouble with the law. You could say it has had a stable and happy up bring. RTEMS is now looking to the future and life without the current custodians.

RTEMS at its core is a collection of C source files that are built into a C library and linked with user application code to provide single executable image often embedded into a custom piece of hardware. The key factors for the user of this device is performance, resources and stability. The key factors for the developer of this device is availability of source code, easy to use software interfaces, easy to integrate into a team environment, and stability of the project. The key factors for the maintainers of RTEMS is the ability to effectively integrate changes, respond to hardware changes, stable infrastructure and the ability to attract new developers. Developers are the food source that feeds, refreshes and sustains a project.

RTEMS in its post toddler years moved to a new version control tool called CVS that allowed concurrent development of the code. It was liberating because a single set of code did not have to be maintained. Before CVS patches were emailed to the maintainer, merged and then released back to developers as tar files. With CVS this task could be spread among a number of trusted developers. RTEMS also moved from custom makefiles to autoconf and automake. This improved the productivity of the developers allowing the code to be configured and built on a range of host operating systems. RTEMS still uses these same tools 10 to 15 years later and they still work. The developers are comfortable with their work flow and know the problems or issues they have. Why the need to change? There are problems and over time these have grown in size as the project has grown. What were problems are now distance memories and all we have left is the new problems that came with the tools.

We have files in places that have long since lost there meaning. The board support packages is an example. They are located under 'c/src/lib/libbsp' when they could be located in 'libbsp' or even 'bsps'. This path does not effect the build time or the disk space used and the developers know this path very well so why is this a problem. It makes no sense. Any new users of RTEMS, and by new I mean anyone who has joined in the last 10 years, would have no idea why this structure exists. RTEMS use to have an Ada version and all code was under 'c' or 'ada' and the source was under 'c/src'. Why not move the files? We cannot because CVS does not have a rename command and repository hacks are something we discourage.

Would we move them if CVS allowed it? Maybe, however this effects the build system. Why is that a problem, is there something wrong with it? Building RTEMS is complex. As a user of RTEMS a release comes with all the autotool's generated files in place ready to work. You can configure RTEMS with a few options passed to configure, plus provide a few more on the command line to the build a range of BSP specific options and then at runtime you have a large array of configurations and runtime options. Are these documented? Only a small number are. The user needs to look into the source to find the full set and even for a seasoned developer this can be complex and not accurate or complete. As a user you just build RTEMS and that does happen and it does it well. By well I mean you get a library of code that is stable and will perform the task asked. As a developer you need to work with the build system and this is where problems start to appear. Performance is an issue. A clean check out from CVS requires a bootstrap to generate all the autoconf and automake files as they are not held in the repository and this can take a lengthy period of time even on large hosts and fast disks. Fortunately this is not often needed as the maintainer mode helps how-ever it makes build-bot type support on check in difficult if not impossible. Also contributing to this is the repeated installing of header files. If you build all 120+ BSPs you will install over 50,000+ header files. This is just building RTEMS and does not include installation of the build output. When installing the 50,000+ files are copied to the install paths. Does this seem normal or ok? Maybe there really needs to be this many headers, or maybe header files have been added to RTEMS following a common template with little regard to the consequence and over the years this has grown to this figure. Most users are only interested in one or two BSPs so this is not a major issue. For a maintainer is it a problem because they need to make sure everything builds and works.

I suppose the important questions regarding the build system are "Is it efficient given the new generation of build tools?" and "Does it aid or inhibit the development process?". These are debatable questions which span the boundaries of technical merits, broad range support, supported hosts, and personal preferences. This last one being the most contentious.

The question the current developers and maintainers of RTEMS need to ask is not "Are these tools working and doing the job they are suppose to?", rather if we handed the project to a new group of developers and maintainers "What would new maintainers think of the state of the project?". While we may be comfortable and able to release and maintain RTEMS it may look to a new generation as something from a time past.

Change is never easy. There needs to be leadership, desire and willingness to refresh to bring about change. It is easy to be negative and to find fault in any new change, then offer no path forward. Leading is not always about "What I think is right", it is about being honest and openly critical of how we work and approach problem solving, and it is about providing paths to new ways of solving problems we face in the project. Not all paths will succeed how-ever being open to change means a new path can be taken until a solution found. Inviting new and young talent to follow these paths and find solutions involves them in the project. They become responsible for various parts and that builds pride and commitment. The hope being someday they will be managing and leading the project.

Flight Software Workshop and Goddard Space Center Visit

2011-10-28T11:09:00.003-05:00

OAR was a sponsor of 2011 Workshop on Spacecraft Flight Software at the John Hopkins University Applied Physics Laboratory held October 20 - 22 2011. This was the first time OAR had been an event sponsor and the first time RTEMS stickers have ever been given away. We had spent weeks preparing a new table top display for RTEMS, updating flyers for the project and our services, and having stickers made. The preparation was a lot of work that was outside our primary skill sets. Jennifer Averett is a core RTEMS developer who did the artwork. We are technical folks and not marketing or sales types. If you have ever worked with us to get a quote, you would know. Hence this was a challenge. If you have any suggestions on our display, flyers, etc., please share them.

Chris is much more special.

Mark Johannes, Chris Johns, and I were fortunate enough to have Alan Cudmore take us on a tour of Goddard Space Center. We first visited their SpaceCube lab and saw the smaller current generation and a larger newer one which had been flown on a sounding rocket.

MMS "Flat Sat" Work Area

Alan showed us the MMS laboratory where we chatted and learned about their workflow. The lab included a “flat sat” version with the electronics of a single node from the constellation. Their laboratory also included hardware required to test the system driving all inputs. Very impressive.

The laboratory setup was very nice but we were all shocked when we saw the MMS assembly area and how large each of the four satellites in the constellation was. We expected birthday cake size and they are the diameter of a large rocket body. Each satellite is running a hardened Coldfire CPU with RTEMS and application software built on top of the Core Flight Executive.

Although not running RTEMS, seeing the James Webb Telescope and Hubble Service Mission assembly areas was a real treat. It is ultimately about the science and I am proud RTEMS is a part of it.

The Workshop on Spacecraft Flight Software was a real treat. Gedare Bloom and his wife joined Chris, Mark, and I there. I have been to the last three FSW's and have been blown away repeatedly at the incredible work being done by this community. As might be expected, we collectively were delightfully pleased to see so many people using or considering RTEMS. It is quite humbling. Many attendees shared their RTEMS experiences and plans with us. The presentations were video recorded and should be available with slides in the near future. Presentations and video from previous years is also available online.

All in all, it was a fabulously rewarding experience getting to meet so many people, tour Goddard Space Center, and hear about so many projects.

Johnson Space Center Visit

2011-10-20T08:30:00.002-05:00

I recently was invited to teach a weeklong RTEMS class to 14 people at Johnson Space Center (JSC) . Multiple projects are considering using RTEMS and two arebeing the trailblazers which bring RTEMS to JSC. Before sayinganything else, I want to thank the folks who invited me and were suchgood hosts. In addition to a self-guided tour of the Rocket Park, Igot a field trip where I got to see a few of the interesting thingsat JSC.

A complete set of photos from my fieldtrips are in a Facebook album. My field trips included seeing theSaturn V rocket, space suits that had been to the moon, Morpheuslunar lander, and the infamous Building 9 where the ISS and spaceshuttle training modules, CanadARM, and zero-gravity practicefacilities. It also includes a small area of cool projects thatdidn't make the cut including Robonaut and the movie-scarySpidernaut. Everything I saw was impressive but much of it leavesyou with an unsettling feeling of sadness when you realize it hasbeen almost forty years since man went to the moon and, with the endof the shuttle program, we have no ability to put a person in orbit. Big science is not cheap and takes years of effort, but without it,we quit learning more about our universe, getting insight into basicphysics, and solving the hard problems.

The Morpheus lunar lander isone of the two projects taking the leap and switching to RTEMS. Ithas already had multiple successful test flights and at least one“interesting” one (videos). Morpheus uses a PowerPC basedcomputer system and is built using the Core Flight Executive fromGoddard Space Center. CFE has long supported RTEMS and as moreapplications are based upon it, I am sure we will see at least a fewof those applications use RTEMS.

DownMASS is a small automatedcapsule that can be filled with contents that need to be shipped fromthe International Space Station (ISS). It is being designed to holdapproximately 100 pounds (43.5 kg) of cargo. When filled and releasedfrom the ISS< it will reenter the Earth's atmosphere andeventually deploy its parachutes. The hardware configuration forprototyping is using a ruggedized embedded PC-104 system.

I am thrilled to see more spaceapplications using RTEMS – especially since have applications atJohnson opens a potential path to having man-rated applications usingRTEMS. Thanks to Morpheus and DownMASS for giving RTEMS a chance.

Google Code In 2011 Announced

2011-10-16T13:28:00.003-05:00

The 2011 edition of Google Code-In has been announced. Google Code-In is a unique opportunity for up and coming hackers. It is a competition, an open source development contest for 13-17 year old students around the world. The purpose of the Google Code-in competition is to give students everywhere an opportunity to explore the world of open source development. We not only run open source software throughout our business, we also value the way the open source model encourages people to work together on shared goals over the Internet.

Each participating project identifies a variety of tasks which students can choose to perform. Tasks are not just coding - they can involve documentation, testing, or outreach. These are the categories:

Code: Writing or refactoring code
Documentation: Creating and editing documents
Outreach: Community management and outreach, as well as marketing
Quality Assurance: Testing and ensuring code is of high quality
Research: Studying a problem and recommending solutions
Training: Helping others learn more
Translation: Localization (adapting code to your region and language)
User interface: User experience research or user interface design and interaction

Black stickers are friom Google Code-In task

The RTEMS Project was fortunate to be one of the twenty organization that participated in the 2010 edition of Google Code-In. We identified about 150 potential tasks had almost 100 tasks done by a variety of students. One of the highlights was a new RTEMS logo which we are using on stickers and project calling cards. Another student modified the shell scripts which generate our test coverage reports to add a timeline capability. Some students created Wiki pages for the Board Support Packages that did not have one. And still other students created question sets for the RTEMS Moodle.

The RTEMS Project is planning on applying again this year. We have some new tasks in mind for those students who volunteer. It was an interesting and challenging holiday season for the RTEMS mentors. I know that I personally did a lot of Google Code-In mentoring and task approval using my phone while visiting family. We are looking forward to the opportunity be a part of Google Code-In 2011 and to be challenged by the students.

RTEMS Configuration and Resource Limits

2011-10-14T12:22:00.000-05:00

This post started as an answer to a question from an ESA Summer of Code In Space student. She had hit one of the things that every person new to RTEMS hits at one point. She attempted to create a pthread mutex via pthread_mutex_init() during the package's initialization. It failed and returned EAGAIN. This was an especially surprising error since it happened in a C++ global constructor which was executed before the first task was entered. It appeared to her that RTEMS was not initialized or something even weirder was wrong. RTEMS was, in fact, initialized and since C++ global constructors are supposed to run before main(). On RTEMS, we run them in the context of the first user task which executes. Let me start with an obvious and decidedly unhelpful assertion:

RTEMS != GNU/ Linux

I am sure that didn't help at all to understand why she got an out of resources error. But it is a lead-in to try to explain the philosophical difference between RTEMS and GNU/Linux that leads us to this. The error she encountered was is in fact an out of resources error. Most people start programming on operating systems that do not discourage you from using dynamic allocation and do not put restrictive limits on the number of instances of an object class you can create. For example, on a GNU/Linux system, you don't worry about how many instances of an OS object you create. There are often limits but these are so high as to not present problems. But the Linux kernel does have some behavioural and limits configuration parameters available to the end user. If you are curious about these, see /etc/sysctl.conf and sysctl(8).

In contrast, RTEMS is a member of a class of real-time operating systems that was designed for to target systems with safety and hard real-time requirements. There are often limited computing resources. In this design view, it is better to pre-allocate as much as possible so you don't have to deal with running out of resources at run-time. This makes the resulting system safer and less likely to have a weird failure mode in this situation. It is not uncommon for the entire set of tasks, semaphores, etc. to be well known and listed in the application design documentation. Configuring the resources required is common in this environment. Moreover, it is not uncommon for malloc() to be forbidden after system initialization. There is nothing inherently right or wrong with either of these contrasting philosophies. They are just different approaches given different system requirements.

In RTEMS you configure the maximum number of each type of object you want. The defaults tend to be 0. Memory is reserved for RTEMS separate from the C Program Heap based upon your cofniguration requests. The sample in testsuites/samples/ticker has the following configuration in the file system.h:

#include /* for device driver prototypes */

#define CONFIGURE_APPLICATION_NEEDS_CLOCK_DRIVER

#define CONFIGURE_APPLICATION_NEEDS_CONSOLE_DRIVER

#define CONFIGURE_MAXIMUM_TASKS 4

#define CONFIGURE_RTEMS_INIT_TASKS_TABLE

#define CONFIGURE_EXTRA_TASK_STACKS (3 * RTEMS_MINIMUM_STACK_SIZE)

#include

The ticker application says it needs a console (used for stdio) and clock tick (time passage) device drivers. It may have a maximum of four concurrently instantiated Classic API tasks. It is using a Classic API style initialization task -- the alternative is a POSIX Threads initialization thread. And each of the tasks it is creating has a stack larger than the minimum required. It is assumed that each task requires only the minimum amount of stack space so we have to tell the RTEMS configuration to reserve some extra memory for those that are larger than minimum. You can look at the calls to rtems_task_create() in init.c for the calling parameters that indicate the desired stack size.

The hello world sample application is simpler. It doesn't need a Clock device driver and would only have one task. Your application will likely have a more complicated configuration than hello world but it doesn't need to specify any configuration parameters unless it requires that class of object to be supported.

Our young programmer simply missed defining CONFIGURE_MAXIMUM_POSIX_MUTEXES to however many is required. This is an RTEMS specific issue that is not done on non-embedded operating systems.

With all that background on the hard limit focus on RTEMS configuration, it is easy to forget that RTEMS also has "unlimited object creation mode" and "unified workspace" options. This is probably more useful for our intrepid programmer at this stage. These configuration options lets you specify that you want a potentially unlimited number of a class of objects and that you want the RTEMS Workspace and the C Program Heap to be the same pool of memory.

#define CONFIGURE_MAXIMUM_POSIX_MUTEXES \
rtems_resource_unlimited( 5 )

#define CONFIGURE_MAXIMUM_POSIX_CONDITION_VARIABLES \

rtems_resource_unlimited( 10 )
#define CONFIGURE_UNIFIED_WORK_AREAS

The above specifies that POSIX mutexes and condition variables are "unlimited" but the set is extended by five (5) instances of mutexes and ten (10) instances of condition variables at a time. When you create the sixth mutex instance, instances 6-10 will be added to the inactive pool for that object class.

The full set configuration macros are (hopefully) well documented in the Configuring a System chapter of the RTEMS User's Manual. This is a link to the appropriate section for RTEMS 4.10.1:

http://www.rtems.org/onlinedocs/releases/rtemsdocs-4.10.1/share/rtems/html/c_user/c_user00414.html

For normal application development, that's really about all there is to this issue. If you get an out of resources error, you will need to raise the limit. When looking inside the code, any time you see a NULL returned from _Objects_Allocate() fail, it is a maximum objects issue. If you see a task, thread or message queue create fail, then you are not accounting for variable memory like stack space or message buffers which must be allocated when the object instance is created.

Since our intrepid young lady is actually porting a software package, let me throw out another thought which impacts this situation. There is likely a fixed base set of objects the package creates such as for global mutexes or message queues. A user of package X will create instances of objects that it defines. So if the package X has a macaroni object that requires a mutex, condition variable, and a message queue, then you can let the end user of package X know that for each macaroni instance, they need to reserve A, B, and C. For certain cases, like the tasking in Ada and Go, RTEMS provides higher level configuration helpers like CONFIGURE_MAXIMUM_ADA_TASKS to encapsulate this configuration information.

I know my answer to her was a over the top but I realized that she had run into something that many others hit when using RTEMS for the first time. I really wanted her, and now you, to understand this part of RTEMS. Figuring out how many of each kind of object when developing an application can be tough but figuring that same information out when porting a software package is can really be a challenge. Embedded developers focused on safe, reliable systems don't like surprises and using various techniques to avoid running out of resources at run-time is a big part of it.

Elvis Costello and RTEMS.info History

2011-09-16T15:50:00.000-05:00

Over the years, I have told many RTEMS users that I provide hosting and system administration for an Elvis Costello fan site (PHPBB Forum and Wiki). I have had the fan site for about 9 years now. What many of you probably don't realize is that I have also hosted the RTEMS.info Mirror Site since 2006.

This is a personal effort and I receive no subsidy from OAR for doing this. In order to have a static IP address and host services, I have to have a business class account which is more than a residential class account. All of the sites I host plus the family Internet activities share a 7Mbps/1Mbps connection.

I had been helping on the technical side of administrating the Elvis Costello Fan Forum for a while when the hosting service got hacked and our site trashed. We were unable to get anyone to contact us via email or phone for over a week. I realized that I had a computer running GNU/Linux Fedora that was largely unused since I had upgraded. Even though it was only a 350 Mhz Pentium II with 384MB RAM, it was perfectly suitable to host a small (~100K hits a day) website. I made a phone call to get a static IP address, moved the domains and within about a week we were back up.

A couple of years after that, the person who ran the Elvis Costello Wiki asked if I could host it. I had already planned to upgrade to a 2.4 Ghz Pentium 4 with 2GB RAM. We decided to wait to move the Wiki until after the server upgrade. The hardware upgrade went easily and we moved the Wiki. What surprised us both was that the performance on the site went to hell. The server logs showed nothing, load looked low and no amount of tuning or probing helped. I begged my ISP for help and got an unlocked, uncapped cable modem to test with. After more research and fighting, I learned that my router could not hand the number of simultaneous connections and was dropping them randomly. I upgraded routers and the performance issues were settled.

The site has always used external hard disks for backup. There is a script which runs every night and dumps all user directories, databases, etc to a special directory on an internal disk. Then another which runs later and "rsync's" the internal disk copy with an external one. Backups are placed in dated directories and a few a month are saved.

When I set up the RTEMS.info Mirror, I was more concerned with disk space than bandwidth consumption. I don't have the fastest connection and I am sure users would appreciate a faster uplink. But I foot the bill and until there is funding, this is what there is. The RTEMS.info site mirrors at least 4 times a day. Ralf Corsepius has set up automated checks which let us know when a mirror site is down or out of sync.

In late 2010, I became worried that the 2.4 Ghz Pentium 4 was getting very old. It was not new when it became the server and I was sure it had seen at least 6 years as server. The Elvis Costello Fan community rallied around my request for a new server and within a month or so, the fund raising goal was met. The new server is from AS Labs who specialize in building custom GNU/Linux systems. It has a quad-core 3.0 Ghz CPU and 8 GB RAM. It runs very cool and is far from overloaded.

Over the years we have had period power outages with the worst being the tornadoes of April 2011. But overall, I believe our uptime is very good. I don't track it but thanks to the Elvis Costello fan community, it can't be down over 20 minutes without me getting an email. Thanks folks!

My wife and I have learned a lot about system administration over the years of maintaining these sites. She personally reviews and approves every account request for the PHPBB Fan Forum. The number of spam account requests is boggling and periodically she begs me to try to find another way to stem an increase. The Wiki account requests and spam were solved when we instituted a very strict policy on getting an account. A small group of people review and approve these accounts.

The server also hosts a couple of very low volume sites for friends. They are more interesting from a content viewpoint and I want to share more about them in a future post.

RTEMS Pair Programming

2011-09-08T18:22:00.002-05:00

One of the most interesting and under-utilized RTEMS services that OAR Corporation offers is RTEMS Pair Programming. This service is a great solution when dealing with a customer who wants to a big head start on some type of development effort. In Agile terms, this is a development sprint with a team consisting of RTEMS and customer supplied experts. Most of the time, we do this for BSPs and device drivers.

Left to Right: Walter Nakano, Wendell Pereira da Silva,
Joel Sherrill and Jennifer Averett

The key to pair programming success is that we know RTEMS and the customer knows their hardware and test equipment. OAR folks can concentrate on quickly providing the framework for the BSP and needed devices drivers. Then we work together to author the device drivers. This provides them with specialized training on the details of the BSPs and device drivers that are critical to the success of their application. Usually the initial testing is performed as joint effort with subsequent detailed testing performed by the customer engineers.

Recently, OAR got to host Wendell Pereira da Silva and Walter Nakano from COMPSIS for two weeks of intense development activity. Their system consisted of an embedded PC plus some add-on boards which added up to a lot of individual pins and ports to test. They brought a LabView test right which allowed us to test every input and output on the Multi-I/O board.The hardware list was:

RTD CME137686LX Embedded PC

4 COM ports and i82551 NIC of particular interest to their project

RTD 17320HR Octal UART PC-104 board (PCI interface)

Exar PCI Vendor Id with 8 NS16550 compatible serial ports
NOTE: We only had one of these boards but they will have 4 in the real configuration!!

RTD 316HR Dual Synchronous Serial Port PC-104 board

single Zilog Z85230

RTD 6425HR Multi-I/O PC-104 board

16 differential or 32 single-ended analog input channels
4 analog output channels
32-bit discrete I/O with 16 bit programmable for interrupt on input change

One thing should quickly stand out with viewing that hardware list. They have a LOT of serial ports. Four asynchronous on the embedded PC, 32 on 4 PCI-104 boards, and 2 synchronous on the Z85230 board for a total of 38 serial ports. This is actually a classic example of a case where using the libchip serial driver framework would be very useful. But the PC386 console driver was not designed this way. Before the guys arrived, Jennifer reworked this driver to be libchip style. At the same time, I factored out the mouse input stream parsing code. It really wasn't BSP dependent and by moving it to cpukit/libmisc/mouse, I made it potentially available to every BSP. Jennifer and I had tested COM1 and COM2 on qemu before they arrived but waited for their hardware to test COM3 and COM4.

COM1 and COM2 worked as soon as the cabling was correct. COM3 and COM4 proved more difficult. After struggling to find a software problem, it occurred to me that it could be as simple as RTS/CTS not being wired together in the shell since we were using a 3-wire connection. That was indeed the problem. Next came the octal serial port board.

After realizing that we would end up with a libchip configuration table with 38 entries and most of them would be disabled for "normal" configurations, I had the idea to allow for dynamic registration of new "ports" in the libchip configuration table. The idea is that if you probe for a bank of 8 serial ports and find them, then you can dynamically add 8 more entries to the libchip configuration table. Currently this allows probes to insert entries prior to console_initialize() being called. It is possible to allow them to be registered after this point but they would not be available to be /dev/console or used for printk().

While I was implementing dynamic registration, Jennifer and our guests worked to get the PCI probe to find the card and the first serial port working. We were surprised to learn that it didn't have a vendor Id of RTD but Exar. This explained the sparse programming documentation from RTD. As soon as the probe and one serial port worked, we switched to my dynamic registration code. Soon all eight ports on the board we had were working. Plus I added code to detect the 2, 4, and 8 port variants of the Exar chip.

Next was the dual port synchronous board. Unfortunately, RTEMS does not have a Z8530 synchronous driver but does have a standard libchip asynchronous driver. After fiddling to figure out the baud rate clock divisor math, we ended up with both ports working. There was one issue in the driver we did not resolve in the two weeks they were here. The two ports on a Z8530 share a single interrupt status register which when read, clears the source. You have to be extremely careful to touch it one time and process all interrupt sources on both ports. The ports worked individually but not when both were installed. Jennifer and I had a solution but not enough time to implement it. Hopefully Wendell and Walter can implement it and we can get this resolved in the main tree.

Next was the Multi-IO board. If you have been following my blog a while, then you might remember the entry RTEMS Shell as Debug Aid where I discussed adding commands to the RTEMS Shell to aid in debugging a Winsystems Multi-IO board similar in capability to this board. One of the last things Jennifer and I had done to the existing multiio was to define a board independent interface between the shell commands and the actual driver. My plan was to let this interface evolve and grow as we learned more about user application requirements. This was the first opportunity we had to write a driver to this interface and reuse the commands. As might be expected, there were places where 0/1 based numbering of inputs still reflected the Winsystems board. And there were places in the RTD documentation that were unclear. But after a while of fighting these and the normal cabling issues, we were able to use the existing commands to debug the driver and verify that all discrete I/Os to work polled and interrupt driven and that all analog inputs and outputs work polled. We ran out of time before we were able to attempt analog input interrupts.

The final thing we attempted was getting the RTEMS TCP/IP stack to run on this board. It had an Intel i82551ER NIC which required using the drivers in the libbsdport kit of late model FreeBSD drivers. This driver works on qemu when you configure qemu for the i82559 simulation. We verified the basics were OK on qemu. Then we moved on to the real hardware. After the normal hunt for an extra cable and battle of the network settings, we were able to run the telnetd application from the network-demos module.

Walter and Wendell drove their hardware. Jennifer and I were the main forces driving the code but they reviewed every line of code and we all verified that each line of code programmed the hardware as we all agreed it should be. Along the way, if something was unclear, we took a break from coding and testing to focus on a portion of the RTEMS Open Class that was very specific to what we were working on. The goal was not only to have as much functional code as possible; it was also to ensure that the code was high quality and they left understanding it and capable of modifying it should the need arise.

At the end of two weeks, we all were thrilled. Walter and Wendell had been sending home progress reports and every day I continued to be amazed at the progress we -- as a team -- had made. This amount of progress was possible because each of us brought unique skills and knowledge to the table. Jennifer and I knew where to reuse code from in RTEMS and how to create an elegant solution the RTEMS way. Walter and Wendell were intimately familiar with their hardware and test equipment and ensured we tested well. Together we reviewed all code to ensure they left understanding it.

I really enjoy teaching the RTEMS Classes but RTEMS Pair Programming is one of the most fun and personally rewarding services we offer. I always come away amazed at how much is working at the end of an intense 2-3 week development sprint. By bringing together engineers with complementary skills and knowledge, solutions are found quicker. And solutions are ultimately what we all want.

Thanks to Walter, Wendell and Daniel who couldn't make the trip for two weeks of fun and productive work.

Spell Checking Not Working in Firefox

2011-08-28T10:10:00.002-05:00

This is not RTEMS related at all but everyone in the field has done free technical support before. I got asked

Why isn't spell checking working on my new Firefox install?

It turns out there can be a variety of reasons and from my searches, the possible answers are not all in one place. This article is an attempt to put all the things to check in one place.

WARNING: I checked this on Fedora (GNU/Linux) and menu options may vary based upon your host operating system. Hopefully they are close enough to keep you on track.

The first thing to check is whether spell check is enabled at all. Select "Edit > Preferences" from the menu. Navigate to the "Advanced" tab and select the "General" subtab. Make sure the "Check my spelling as I type" option is enabled as shown in this figure:

If this option is selected, then we will have to check two more things which might not be correct. In order to do these you will have to navigate to a website which has a text entry form. Composing mail on a web mail client like Google or Yahoo mail will work. When you are there, right click in the text entry area and verify that "Check Spelling" is checked.

If it is, there is still one more thing to check. Apparently some Firefox distributions were shipped without dictionaries. If you installed one of these as a clean install, rather than as an upgrade, then you didn't get a dictionary. I don't know why and it doesn't really matter. If you have installed one of these versions, then you need to install a dictionary. Again, you need to right click but this time, select "Languages > Add Dictionaries". This will guide you through installing the dictionary you need.

As I was investigating this to help the person who asked, I learned I was using the "English / Zimbabwe" dictionary which I found surprising. It definitely explains why I use UK spellings frequently. It is necessary to make that language variant. Colour me surprised. Or color me surprised not that I am using "English / United States".

If there are any other things which might explain spell check not working in Firefox, please leave a comment.

Passing of Dr James Johannes

2011-05-18T17:38:00.004-05:00

My normal blog entries cover technical issues. This post is going to be very different. It is an homage to the man who referred to himself as my academic grandfather because my Ph.D. advisor had been his first Ph.D. student. A man I have had the pleasure of knowing and working for for almost twenty-five years.

Dr. James D. Johannes

Dr James Johannes (Dr J) passed away Tuesday May 17 2011 at the age of 76. This came as a shock to those who knew him because he was a vibrant person. He was the type of person that one just expected to always be there. Until a few months ago, he came to the office nearly every day. Within the past few years, he earned his pilot's license for the first time.

Dr Johannes earned his Ph.D. in his early 40's from Vanderbilt University. At the time, he lived in Huntsville and had two children (Mark and Michele) and commuted about 100 miles each way to take classes. His dear wife Aurelia -- who was the epitome of a classy and tough southern lady -- supported him and the children through this. This was clearly a factor in them being understanding and supportive of my finishing my Ph.D. with four small children in the house. Aurelia appreciated what my wife Michele (not his daughter) was doing.

Dr Johannes founded On-Line Applications Research (OAR) Corporation in 1978. He was an Emeritus Professor of the Computer Science department faculty at University of Alabama in Huntsville and was the first head of the department. Based upon the number of dissertations on the shelves at OAR, he advised over twenty-five successful Ph.D. students. When he retired, he was serving as the Graduate Dean of the UAH College of Science. He wrote the Thesis and Dissertation Style Guide required by the university that I followed on my own dissertation. It certainly made it easier to get clarifications.

I first encountered Dr J as a student in Spring 1988. I was taking the Ph.D. level Operating Systems class he was teaching. I must have made a good impression because after the class was over, I received a job offer. I asked if they could hold the offer for a few months. My daughter Jessica (now 22) had just been born and I wanted to make sure she passed her six-week check up before switching jobs. In July 1988, I started work at OAR and my first project was RTEMS. You should know the RTEMS story.

Dr J also could show pride in those around him. I was the lucky recipient of his special events twice. The first was a company wide lunch at the Huntsville Country Club when I passed my Ph.D. defense! He knew the system and didn't wait for graduation.

Twentieth Anniversary

The other special event was in 2009 when we celebrated my twentieth anniversary at OAR.

There are many Dr J stories but I will only share a few. Long ago, he called me at home on a Saturday completely unexpectedly. I assumed I was about to be fired and went to a quiet place in the house to take the call. Dr J had that professorial demeanour that makes professional students always a bit leery. It turned out he needed some sysadmin help with a Solaris computer he had at home. Why he had a Solaris computer at home I don't know.

He didn't like to dispose of old computers. OAR still has a CP/M computer with 8" floppy drives in storage. The HP1000 was taken to his house after the OAR folks refused to move it to our third office location in the mid-1990s. I think Aurelia finally made him dispose of it.

Dr J was loved and respected by a wide circle of people. He will be missed.

Power Restored But Issues Remain

2011-05-03T17:43:00.001-05:00

Power was restored to OAR at ~5pm CST Tuesday May after being off since about the same time May 27. The area is still under a dawn to dusk curfew so no hands on to fix things until tomorrow morning. If a machine came up, we are running checks remotely. The known status is

rtems.org rebooted cleanly and appears to have all services running correctly.
OAR VOIP phone service rebooted cleanly.
mail.oarcorp.com did not automatically come up and needs hands on help. I was not the one checking this machine and that's all I know.
None of the RTEMS lab machines are up. I smelled something acrid in
the lab after the outage so don't know. They are on different UPS's so unless it is the circuit the machines are on, it has to be a single piece of equipment. That has to be investigated tomorrow.

Tomorrow's work is focused on testing batteries in UPS's, running checks on machines, and cleaning the refrigerator. Everyone is really chomping at the bit to get back to work.

rtems.info and Power Update

2011-05-03T14:45:00.000-05:00

rtems.info is my personal server. It is a purely volunteer effort and unfunded. It is in my home. We lost power Wednesday April 27 about 6pm CST. Power was restored overnight Sunday. Monday we returned home and I started to get the server back online. There was some damage to various MySQL databases so I did a check like this after stopping mysqld:

service mysqld stop
cd /var/lib/mysql
for i in */*.MYI; do myisamchk --max-record-length=1048576 -r -f $i; done

That seemed to resolve all of those issues.

The outstanding issue now is that it appears my ISP had some damage to their network operations center. All appears OK but in the process of recovering, they have deleted the DNS entry for rtems.info. This was filed with them last night.

Huntsville is still under a dusk to dawn curfew and power is NOT restored to Research Park. City schools are scheduled to start again on Thursday May 5. So we are hoping for power tomorrow or Thursday.

More as it becomes available.

Terrible Storm and RTEMS Outage

2011-05-03T08:52:00.001-05:00

Surely by now, you have noticed that the RTEMS Project appeared to drop completely off the face of the planet about 6pm CST April 27. It was at this time that the third storm system moved through north Alabama and knocked out all major power transmission lines. This blog is a first in a series to let you all know what happened and what is happening now.

The primary servers and lab machines for the RTEMS Project (.org and .com) are in Huntsville Alabama which was in the path of the storm April 27th 2011. This storm killed 340+ across multiple states and left a huge trail of destruction. I have heard reports of business signs being found 100 miles (160km) away. The following is a nice weather summary without the heart wrenching photos of the death and devastation.

http://www.washingtonpost.com/blogs/capital-weather-gang/post/alabama-tornado-outbreak-visuals-jaw-dropping-radar-and-satellite-imagery/2011/04/29/AFg1C5YF_blog.html

Huntsville had three systems pass over it that day. The first was nasty but no issues impacting the server or our home. The second system resulted in water getting into the rtems.info server area but we still had power. This allowed us to clean up and get the server back online until the third system hit. The third system was the killer. It was the devastating one that wiped out communities from Mississippi through Alabama and Georgia to points further north. It destroyed the major power transmission lines into north Alabama and Mississippi. Local utilities get power from the Tennessee Valley Authority (TVA) and TVA could not supply power to them. Their blog is here with details:

http://www.tva.com/news/releases/aprjun11/storm.htm

I was teaching an RTEMS class during this with the sole attendee being a wonderful fellow from the UK. We spent much of Wednesday in a safe area inside OAR. And once the storm had passed and we realized we were the lucky ones, we finished the class without power. His hotel room was wet and without power but his bed was dry. He could charge his laptop from the car. We took a table and a couple of chairs from OAR and sat outside the door. When the sun moved and we got hot, we moved the furniture. At one point, we were on the other side of the parking lot. We had nothing else to do and a dusk to dawn curfew, so we followed the class material, chatted, drank soda, etc.. Just chilled and did RTEMS stuff. Phillip deserves a big thank you for helping me make sure all was turned off Thursday and cleaning the fridge and freezer Friday. I sent him to Chattanooga for the weekend and I hope it was some nice relaxing site-seeing.

Friday, my family went to a hotel in a neighbouring city and waited for power to be restored. We came home Monday afternoon since our home had no apparent damage and power was restored. We are near the main hospital so we usually get power early. I started ensuring the rtems.info and elviscostellofans.com server came back up OK. Michele is cleaning the fridge and freezer out. If we didn't lose any electronics due to power spikes, then that's all I think we have. That makes us very lucky. Michele and I know people with deaths in their families or homes destroyed. Cleaning the fridge looks pretty tame in comparison.

We tried to stay in touch using our cell phones for email but the towers died about 12 hours in. Plus if we didn't know you on Facebook or via our private emails, it looked like we disappeared. I apologize for not remembering linkedin and the RTEMS facebook group. I didn't even remember IRC until Friday when we got to the hotel.

Huntsville Utilities announced yesterday (Monday night) that they have done all they can. Everywhere TVA has given them power has been passed on to residents. Only 30% have power. I believe that Redstone Arsenal, Marshall Space Flight Center, and Research Park will be the last in the area to be restored. They consume a LOT of power between them and it is more important to get power back to houses.

We really appreciate the good karma that was sent our way. We are both tired and frazzled but that's no biggie.

More RTEMS SMP Patches Coming

2011-04-22T09:08:00.002-05:00

Some of you may be aware that SMP for RTEMS has been underway for about a year now. The goal of the SMP effort is to have a simple, working, and correct implementation. The first incarnation will have the following characteristics.

BSP SMP Interface definition with implementations for

PC386
LEON3

Simple SMP Aware Priority Based Scheduler
Faithful SMP safe version of RTEMS OS Critical Sections

Dispatch Disable
Interrupt Disable

Scheduler Simulator

Test scenarios for new Simple SMP Scheduler

Features Not Present

Processor affinity
Deferred Floating Point context switch
Taking a core offline

The SMP implementation plan broken the effort into as many small steps as possible so it could be incrementally reviewed and merged. This plan also allows for intermittent work. This was critical due to the fact that all initial work was completely volunteer. In addition to being planned as a series of small steps, the initial SMP implementation is focused on simplicity and correctness. We can improve a simple working implementation.

Gedare Bloom and I made the first steps last summer when implemented the Pluggable Scheduler Framework for RTEMS and I added a "per cpu" data structure. Together these allow us to provide an alternative scheduler that is SMP aware and to have the data required by RTEMS SuperCore to manage each core encapsulated and allocated properly.

Jennifer Averett and I have been working the past couple of months on completing the SMP support to RTEMS. Jennifer and I are approaching a milestone of having a basic SMP system functional with the only major missing item being SMP safe interrupt disable sections. We are about to file a set of PRs and merge our current work and it made sense to post a blog entry with status that the PRs could reference.

Test code - our test code is hacky since it has to force interprocessor interrupts. We need to integrate where these are generated inside RTEMS. Tests will be submitted once the code is in shape to work without "user-level" intervention.
Simple SMP Scheduler - implemented, tested with schedsim and our hacky test
Scheduler Simulator - multiple changes to improve its use during Scheduler development and to track changes in code base
PC386 BSP - SMP BSP support seems complete.
LEON3 BSP - SMP BSP support seems complete.
Context Switch Disable Critical Section - Working

Overall, (today) SMP RTEMS can bring an SMP system out of reset, schedule across multiple cores, and command the first dispatch on the secondary cores. The "disable dispatch" should be SMP safe now. It can do this on pc386 and leon3.

The next major tasks are to integrate the generation of interprocessor interrupts for subsequent dispatch requests and system shutdown. We also have to address interrupt disable SMP safety.

We are doing our best to break this into as many small incremental pieces as possible so the review and integration into the main tree is easier.

Behind the Scenes of the RTEMS Tool Binaries

2011-04-20T15:09:00.000-05:00

I recently posted a long email to the RTEMS Users mailing list about what went into the building and distribution of the pre-built RTEMS Cross Development Tools. I thought it would be interesting to clean that post up and turn it into a blog entry for posterity.

I don't think most people in the community realize what goes on quietly behind the scenes for the tools. When someone installs a pre-built toolset, it is the result of Ralf Corsepius' ongoing effort.

OAR hosts the RTEMS Build Farm and Ralf uses these machines to build the tools. When there is a change in a patch or tool revision, he very quickly responds and kicks off tool builds. I have no idea how long it takes for them to finish building but the number of individual toolset combinations is staggering when you consider the multipliers:

number of target architectures (~10-12 depending on RTEMS version)
number of host OS distributions and version

SUSE
CentOS/RHEL
Fedora
mingw32
Cygwin

32 and 64 bit hosts

Today there are 15 unique host variations for 4.11. This results in approximately 25GB of tool content for 4.11 on ftp.rtems.org. In addition, there are binary toolsets on the ftp site for release branches back to 4.6. So the main ftp site has a LOT of stuff on it.

Ralf is very quick about getting new tool binaries out. Because of this, RTEMS is typically the first project to release binary tools after a binutils, gcc, or gdb release. For gcc 4.6.0, he tracked the final release candidates so we were using the release image before the announcement. :-D

After the tools land on the RTEMS FTP site, there are two yum mirrors of the rtems.org site:

rtems.eu [1]
rtems.info [2]

It can take hours for the mirror process to complete. Ralf has a script that checks the mirrors each hour. This script emails those interested when things get out of sync. The Yum repository for each RTEMS branch, distribution, OS version, and 32/64-bit variation is checked individually. When a mirror out of sync for a variation, that single mirror is taken out of the yum mirror list for that variation until it has time to resynchronize. When a tool build is under way, I might get email for 8+ hours showing the progress of the synchronization.

Check out the Munin performance graphs for rtems.info to see how long a recent tool mirroring took.

This is what goes on behind the scenes to make the tool binaries available. There is a different process for building the various tool chains, running the tests on them and reporting them to both the RTEMS Tool Test Results and GCC Test Results mailing lists.

If you would be interested in DVD distributions of the pre-built tools, let me know.

--joel

[1] rtems.eu is sponsored by Embedded Brains. I don't know its speed.

[2] rtems.info is my personal server and is sponsored by love and donations. It is an 8/1 Mbps connection which could be upgraded.

Merging Multiple PDF Files

2011-04-19T16:02:00.000-05:00

If you have taken one of the RTEMS Classes from me, you will remember that the material for the Open Class comprises over 1000 PowerPoint slides. [1] These slides are broken down into sections and within each section, there is a unit of 20-100 slides. Each unit is an individual file. Getting from 50+ PowerPoint files to printed material is a tedious and error prone process by hand. The class and this process have evolved over the past ten years. In this post, I will provide some insight into how this is done.

The first piece of magic is an MS-Office macro written by someone here are OAR. It reads in a list of files from a text file. The files are in the order they are to be printed. This macro automates either generating PDFs or directly printing the files in the various handout formats (1 per page, 3 per page, 6 per page, etc.). The PDFs are generated using PDFCreator which makes it possible to specify a unique file name for each PDF file. The PDF files are prepended with a number so they sort and print in the correct order when wild-carded. This produces files like this:

001-OpenClass.pdf
002-IntroToRTEMS.pdf
003-ProfilesAndRTEMS.pdf
...

Once the PDF files are generated, they can be printed easily. However, I sometimes teach the class in Munich and have to send the PDFs to the nice folks embedded brains GmbH to print there. For the first few classes, there I sent them a large number of PDFs. When someone dropped the master copy, we learned it didn't have page numbers. This taught us to add page numbers. :-D

But this still leaves us with a large number of PDFs. The solution to this was a custom shell script that merges them into proper double-sided "units". Each unit is then a single PDF file which goes between divider tabs in a binder. Now there are seven PDF files for the Open Class and each page is numbered. Much safer and easier.

The script to merge the PDF files was developed and executes on GNU/Linux (no surprise, right?). The key to this program is this shell function:

merge_them()
{
outf=$1
shift
inf=$*
gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=${outf} -dBATCH ${inf}
}

This function takes the name of output file as the first argument and the set of PDF files to merge as the rest of the arguments. When invoked, the command looks something like this in my shell script:

merge_them ${mergedir}/01-Intro.pdf 00[1-5]*.pdf

That takes the first five "section" PDF files and merges them to produce the PDF file named 01-Intro.pdf for the Introduction to RTEMS "unit". This file is placed in the output directory ${mergedir}. This is repeated for each of the units in the class.

But remember -- I want to produce a double-sided master copy. Sometimes, the merged PDF files for a unit will have an odd number of pages. The script has another section to detect merged PDFs with odd number of pages and add a page the says "Intentionally Blank" [2] The following fragment of the shell script determines how many pages are in the PDF file. If the number of pages is odd, it them adds the Intentionally Blank PDF file.

pages=`pdfinfo $1 | grep Pages | cut -d':' -f2`
remainder=`expr ${pages} % 2`
if [ ${remainder} = 1 ] ; then
   mv $1 XXX.pdf
   merge_them $1 XXX.pdf ${BLANKPDF}
   rm -f XXX.pdf
fi

And that's it. It only takes about a minute to run and produces double-sided files that are very easy to send to a printer. We have a nice duplex printer and by using paper that is already 3-hole punched, constructing the material for the RTEMS classes is much simpler than it was 10 years ago.

--joel

[1] OpenOffice did not exist when the slides were created. I have tried to use OpenOffice with them, but it butchers the slides and destroys. them. If this is ever resolved, I will happily use OpenOffice for the class.

[2] The "Intentionally Blank" page was generated in OpenOffice. :D

A Close Look at a Funny Spam

2011-04-07T08:45:00.000-05:00

I get a lot of spam. And when I say a lot, I really mean it. Most of it is caught by the OAR spam filter but I have to have them keep my settings down a bit to ensure I always get random sales inquiries. I have had the same email address for over 15 years and am very open about it. I post to free software development mailing lists and sometimes those get archived with email addresses. When people get viruses, they have my email from those lists or personal correspondence. So I have received dating and penis enlargement spam that is supposedly from people who would die if they knew. I generally just delete it quickly but sometimes read it.

This morning, I received this gem. I changed the return address and the phone numbers.

Received: from [32.11.46.109] (helo=tdyoireznpl.tmxmykph.ua)
Subject: Free heroin shipping!

FREE HEROIN SHIPPING!

1. Heroin, in liquid and crystal form.
2. Rocket fuel and Tomohawk rockets (serious enquiries only).
4. New shipment of cocaine has arrived, buy 9 grams and get 10th for free.

Everebody welcome, but not US citizens, sorry.

ATTENTION. Clearance offer. Buy 30 grams of heroin, get 5 free.

Please contact: SPAMMER_EMAIL@gmail.com

PHONE 0093(0)1234567
FAX 0093(0)1234567

Afghanistan

There are so many things to notice. The incorrect spellings and numbering (1, 2, 4) are how they arrived. The interesting mix of drug spam and weapon spam.

On the technical side, they claim to be from Afghanistan and the 0093 country code is for Afghanistan but the email address listed is gmail (frowny face to them), the "helo" exchange with the mail service indicates it came from .ua which is the Ukraine. But the IP address listed is neither of those. The IP address is allocated to .... are you ready...

OrgName:        AT&T Global Network Services, LLC

OrgId:          ATGS

Address:        3200 Lake Emma Road

City:           Lake Mary

StateProv:      FL

PostalCode:     32746

 Country:        US 

So we have spam from Florida, claiming to be from the Ukraine, and wanting us to contact some drug/weapon runner in Afghanistan. My guess is that this is likely spam from one of those infamous Russian botnets driven by the Russian mob.

The marketing and salesmanship is brilliant in a warped way. You have to smile at the offer of free shipping and buying 9 grams and get one free. And who could resist the Clearance offer? All we are missing is an order by midnight tonight and the more you buy, the more you save.

The pinnacle of the marketing here is that this offer is not available for American citizens. Wow! What a great use of reverse psychology. Now all of the Americans reading this spam want the drugs with free shipping and a six-pack of "Tomohawk" missiles. Hold me back.

This is nothing I will reply and it is already deleted but very entertaining to read. This is even funnier than the offer I got recently which was supposed to be from an RTEMS Steering Committee member and slipped through the the RTEMS Users mailing list this week. It wanted us all to look at their photo set at a Russian "sex flirt girls" site. I really hope they were not pictures of him. LOL

Coverage Testing Finds Unexpected Bug

2011-03-08T16:17:00.003-06:00

Recent posts have been about debugging experiences. Jennifer and I have been working on a simple implementation of the RTEMS priority based scheduler to illustrate how to write an alternative scheduler. The primary RTEMS Scheduler is deterministic (e.g. constant, predictable) in performance because it uses a FIFO per priority and a two level bit map to assist in look ups. The simple scheduler uses a single list for all tasks and searches down the list to perform inserts. Its worst case is thus O(n) where n is the number of ready tasks. But this post is above debugging, not about the new scheduler. We will discuss that another time.

Jennifer and I want the simple scheduler to have 100% test coverage when it is merged. This has required us to run coverage analysis on RTEMS and check the results. We haven't yet merged this code so it isn't showing up in published runs. She noticed that thee method _Scheduler_priority_Enqueue_first was reported as never executed. We both found this incredulous since this is called as part of inheriting a priority. We knew this was well tested. She was perplexed and asked me what I thought. It turned out to be a recently introduced typo where _Scheduler_priority_Enqueue was configured as the handler for both the enqueue and enqueue entries points in the Scheduler Priority table. This also explained a couple of test failures Jennifer had noticed but hadn't investigated yet. This is the pertinent part of patch:

- _Scheduler_priority_Enqueue, /* enqueue_first entry point */ \
+ _Scheduler_priority_Enqueue_first, /* enqueue_first entry point */ \

Again, this problem was highlighted by an unexpected drop in the coverage results. RTEMS has very good test coverage and an unexpected change can indicate a problem. The fact that this method was never called was a huge hint to the cause. The answer popped almost instantly in my head. Debugging the test failures would almost certainly have taken much longer.

Debugging an Invalid Memory Access

2011-03-08T10:09:00.014-06:00

This time we have a guest at RTEMS Ramblings -- Chris Johns. Chris is a long time RTEMS contributor and member of the Steering Committee. He was the second person outside the core team to submit code to RTEMS. He has fielded some pretty impressive RTEMS applications. This is his discussion of he and I trying to track down an invalid memory access on the SPARC/SIS BSP. From here on.. "I" == Chris. :D

I have a SIS BSP failure with RTEMS head (+rbtree patch) and the latest tools on a MacOS host running the "Hello" sample application in the erc32 simulator. Joel is not seeing this so I have to dig into what has been a stable platform for me to developer the RTL code on. This is an account of the debugging session as I work though the problem.

I run the hello test and all I get is:

$ sparc-rtems4.11-run sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe

Memory exception at ffffffe0 (illegal address)

Not much help from the tool. The same happens if I run the same application in gdb.

Setting a break point on boot_card and entering 'r' to run the application shows me the target is loading, booting then reaching the boot_card function. Nice because this means the tools are not broken and RTEMS and the BSP are sort of sane. I build the MacOS tools from source using the SpecBuilder tool and track Ralf's changes closely. This means there is always a chance something went wrong in the build. I have never seen this but there can always be a first time. I can step the code up to the API_Mutex_initialization call then something goes wrong but I am not sure where. Maybe I should look from the bottom up.

I decide to track down who is printing the error message "Memory exception ...". Running strings and grep on the executable shows it is not in RTEMS. I see it is generated by the simulator. I also see the simulator generates a trap 9 exception. This exception is not handled in the target so the simulator terminates. Fair enough but the stopped simulator destroys the state information of where the fault is which means I cannot see what happens.

A 'sparc-rtems4.11-objdump -D --source hello.exe | less' shows me trap 9 is address 0x2000090 so I set a break point with b *0x2000090 and run again. I break at the trap 9 entry point and stop the simulator exiting:

(gdb) c
Memory exception at ffffffe0 (illegal address)

Breakpoint 4, trap_table () at c/src/lib/libbsp/sparc/erc32/../../sparc/shared/start.S:68
68 BAD_TRAP; ! 09 data access exception

I can now inspect the state of the processor and try and find the source of the problem. First stop is a back trace:

(gdb) bt
#0 trap_table () at c/src /lib/libbsp/sparc/erc32/../../sparc/shared/start.S:68

Nothing helpful here. We know we are at this location and what we want is what happened before this.

I do not know SPARC processors very well and a dump of the registers gives me little information. I do know the stack works down and most processors save the return address on the stack. GDB is nice by providing me with the stack pointer as $sp. I dump the stack:

(gdb) x /32xw $sp
0x23ffcd0: 0x00000034 0x02007378 0x0200737c 0x00000008
0x23ffce0: 0x00002000 0x02014928 0x00000001 0x00000000
0x23ffcf0: 0x02012a30 0x02014924 0x02012a30 0x00006054
0x23ffd00: 0xffffffff 0x00000004 0x023ffd38 0x0200792c
0x23ffd10: 0x00000000 0x00000000 0x00000000 0x00000000
0x23ffd20: 0x00000000 0x00000000 0x00000000 0x00000000
0x23ffd30: 0x00000000 0x00000000 0x02012a30 0x00006089
0x23ffd40: 0x0201a9ac 0x00000008 0x020148b0 0x02014928

I have a separate window open with the sparc-rtems4.11-objdump output in less so I can search around with ease. With a few simple 'less' commands I can find the code at a specific address. The first address is '0x02007378' which must be the last one pushed. In less a '1G' takes me to the start of the dump then entering '/2007378' and enter brings up some code to do with the heap:

02007374 Heap_Protection_block_check_default:
static void _Heap_Protection_block_check_default(
Heap_Control *heap,
Heap_Block *block
)
{
2007374: 9d e3 bf a0 save %sp, -96, %sp
if (
2007378: c2 06 60 04 ld [ %i1 + 4 ], %g1

Dumping '$l1' gives:

(gdb) p /x $i1
$14 = 0xffffffdc

Dumping '$l1' gives:

(gdb) p /x $i1
$14 = 0xffffffdc

This is very close to the address in question. Time to run again this time with a break point on this address and a couple of displays to help me see what is happening:

(gdb) b *0x2007378
(gdb) display /i $pc
(gdb) display /x $i1
(gdb) r

The break point gets hit a number of times and the arguments all look ok so just continue. On the 6th hit of the breakpoint we get something that does not look ok:

(gdb) c
Breakpoint 8, _Heap_Protection_block_check_default (heap=0x2012a30, block=0xffffffdc) at c/src/../../cpukit/score/src/heap.c:149
149 if (
2: /x $i1 = 0xffffffdc
1: x/i $pc
=> 0x2007378 &#60_heap_protection_block_check>default+4&#62: ld [ %i1 + 4 ], %g1

A back trace this time is much better:

(gdb) bt
#0 _Heap_Protection_block_check_default (heap=0x2012a30, block=0xffffffdc) at c/src/../../cpukit/score/src/heap.c:149
#1 0x0200c82c in _Heap_Protection_block_check (heap=0x2012a30, alloc_begin_ptr=) at ../../cpukit/../../../sis/lib/include/rtems/score/heap.h:625
#2 _Heap_Free (heap=0x2012a30, alloc_begin_ptr=) at c/src/../../cpukit/score/src/heapfree.c:119
#3 0x02007d08 in _Objects_Extend_information (information=0x2012b18) at c/src/../../cpukit/score/src/objectextendinformation.c:224
#4 0x02006cc8 in _API_Mutex_Initialization (maximum_mutexes=1) at c/src/../../cpukit/score/src/apimutex.c:23
#5 0x0200672c in rtems_initialize_data_structures () at c/src/../../cpukit/sapi/src/exinit.c:125
#6 0x0200137c in boot_card (cmdline=) at c/src/lib/libbsp/sparc/erc32/../../shared/bootcard.c:163
#7 0x02001158 in zerobss () at c/src/lib/libbsp/sparc/erc32/../../sparc/shared/start.S:334
#8 0x02001158 in zerobss () at c/src/lib/libbsp/sparc/erc32/../../sparc/shared/start.S:334
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Walking up the stack with the 'up' command until we end up in the _Object_Extend_information call:

(gdb) up
#3 0x02007d08 in _Objects_Extend_information (information=0x2012b18) at c/src/../../cpukit/score/src/objectextendinformation.c:224
224 _Workspace_Free( old_tables );
(gdb) p old_tables
$16 = (void *) 0x0

It would seem _Object_Extend_information is calling the workspace with a NULL which should be ok or it use to be ok. I chat with Joel and he informs me the code in the interface to the workspace heap has changed and this has exposed some bugs. A check of the code in the heap free call shows the heap protection check is being called before the block pointer has been validated. This also explains why Joel does not see the problem. I built RTEMS with the debug configure option. There are other cases so I will need to perform a careful check of all the heap code to make sure we are correct.

It looks like I am not the only one who has the problem. Peter Dufault has just posted to the RTEMS user list:

http://www.rtems.org/ml/rtems-users/2011/february/msg00142.html

The PR Peter has kindly raised is:

https://www.rtems.org/bugzilla/show_bug.cgi?id=1746

I was talking with Joel about the changes when I noticed the heap extend now allows discontinuous memory regions. I did not know this was allowed and I had been assuming the memory had to be continuous because of the code in _Heap_Is_block_in_heap. A check of rtems_region_extend shows it uses heap extend and its documentation states memory must be continuous. I have raised a PR to handle this:

https://www.rtems.org/bugzilla/show_bug.cgi?id=1747

Back to Joel.. Chris' discussion should provide some insight into how an free software project and community work. I made a modification to move some scattered NULL checks before calls to _Workspace_Free into that routine. Sebastian Huber noted that since NULL pointers shouldn't be processed by _Heap_Free, so the check was technically redundant so he removed it. This exposed a latent bug in _Heap_Free when passed an invalid address and debug checks were enabled. At the same time the Chris and I were tracking this down, a user tripped the same bug and filed a PR. In reviewing the code, Chris and I found other cases which could cause the same fault and a disconnect between extending the heap and checking whether a block was in the heap. This was a side-effect of recent enhancements and had never been caught by a user. So we see a community coding together, reviewing each other's code, and working together to resolve an issue.

It is important to note that this bug was only present in the RTEMS Development Head. It cannot occur in released versions.

RTEMS Shell as a Debug Aid

2011-02-17T09:49:00.013-06:00

Until the 4.9 release series, the RTEMS Shell was very primitive and only a few commands existed. Virtually no one used it. But with 4.9, the RTEMS Shell took a major leap forward with Chris Johns and I doing a lot of work on it. We added commands, command line history, command line editing, and simple scripting. You can use the shell from a serial port or via telnet. There are now approximately 100 standard commands with some such as ls, mv, cp, ln, and dd ported over from NetBSD. In addition, there are RTEMS specific commands to look at CPU usage per thread, stack usage, and rate monotonic period statistics. There are commands to examine the state of most RTEMS OS objects.

Chris Johns used the standard file related commands to great benefit when developing and debugging the RTEMS File System (RFS). He would mount remote NFS volumes and copy great quantities of data to an IDE hard disk. This allowed him to place stress on his new file system and even turned up a bug in the NFS client code.

But the most useful capability for developing and debugging user applications is probably the capability to include custom commands. These allow you to write commands which are specific to your hardware configuration or application. I have used this to capability to write a set of commands for the Winsystems PCM-MIO-G multi-function I/O PC-104 module. (Kudos to Winsystems for relicensing their GNU/Linux driver to be compatible with RTEMS licensing requirements.) This board has the following features:

Two 8-channel, 16-bit Analog-to-Digital (A/D)
Two, 4-channel, 12-bit Digital-to-Analog (D/A)
48 Bidirectional I/O lines with interrupt support

For testing the driver for this board, I decided to write a series of custom RTEMS Shell commands to configure the analog inputs and perform various IO operations. These commands allowed direct interaction with the hardware and can be used during development as well as hardware checkout and integration. I implemented the following commands as well as providing shorter aliases for each command without the "pcmmio_" prefix:

pcmmio_din - Read PCMMIO Discrete Inputs
pcmmio_dout - Write PCMMIO Discrete Outputs
pcmmio_adc - Read PCMMIO Analog Inputs
pcmmio_adc_mode - Set PCMMIO Analog Input Modes
pcmmio_dac - Write PCMMIO Analog Outputs
pcmmio_irq - Wait for PCMMIO Interrupts
pcmmio_bench - Benchmark PCMMIO Interrupts

All the test hardware I had available was a multimeter and a push button. Although minimal, this was sufficient for me to test nearly all of capabilities on this board. For example, I could use the pcmmio_din command to poll for the discrete inputs. When I pressed the button, the command reported the state changed.

[/] # pcmmio_din -i 10
Polling discrete inputs for 10 iterations with 1000 msec period
665:159912852 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
669:238111788 0000 0000 1000 0000 0000 0000 0000 0000 0000 0000 0000 0000
671:250111878 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000

The board cam be configured such that when a discrete input changes an interrupt is generated. The device driver's interrupt handler determines the pin which changed, its current value and timestamps it using the Time Stamp Counter (TSC) register. This information is placed in a message buffer and send via an RTEMS Classic API Message Queue to an application task which is blocked waiting. This allows the application to know as precisely as possible when an input has changed and process that change at the task level. An example output of the pcmmio_irq command when using the push button (which bounces) is below:

[/] # pcmmio_irq -d -i 5
Polling for DIN IRQ for 5 iterations with 1000 msec period
1000 DIN irq pin 8 @ b9eba88c0e (0 usecs since last)
2000 DIN irq pin 8 @ b9eba932b2 (42 usecs since last)
3000 DIN irq pin 8 @ b9eba9c52e (37 usecs since last)
4000 DIN irq pin 8 @ b9ebadec4e (272 usecs since last)
4 total interrupts from DIN in 5000 milliseconds

In the above example, the command looked for interrupts for 5 iterations of a loop with a delay of 1000 milliseconds between iterations. But the interrupts from pushing the button and it bouncing occurred over a 272 microsecond period. In real application code, you would not put a long delay in between each check but block forever or with a reasonable timeout.

I could test analog output (DAC) by simply entering a command to write a value to a particular DAC channel using the pcmmio_dac command and verifying that the proper voltage was written using my multimeter.

[/] # pcmmio_dac 0 5
Write 5.0000 to to dac 0

The pcmmio_dac command has an interesting feature where you can http://pc104.winsystems.comwrite a "step" pattern. This steps from a low voltage to a high voltage using the specified step voltage and time between steps. When it reaches the high voltage, the command begins to step down. The following example illustrates using the pcmmio_dac command to write a step pattern to DAC 0. The pattern ranges from -2.5V to 2.5V with a .5V change every 250 milliseconds for a total of 10,000 milliseconds. When the voltage reaches 2.5V, the step will change to -.5V.

[/] # pcmmio_dac 0 -2.5 2.5 .5 250 10000
Write -2.5000-2.5000 step=0.5000 stepTime=250 msecs dac=0 max=10000 msecs

When testing analog input (ADC), I attached one DAC output to one ADC input. Then I used the command pcmmio_dac to write a voltage and pcmmio_adc to read a voltage. Just as pcmmio_din can monitor the discrete inputs for changes, the pcmmio_adc command can monitor the ADCs for changes in input. The following commands illustrate using this command to read all ADCs or just a single ADC a single time.

[/] # pcmmio_adc
1117:232053 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
[/] # pcmmio_adc 0
1120:495519 0.0000

Note that in the above, the voltage read isn't the voltage that was written. There are a few potential reasons for this. First, I could have hooked things up wrong (but I checked and I didn't mess that up). Second, I could have gotten confused on the DAC and ADC channels I used. Yes, I did that a couple of times. But the final reason was that I forgot to initialize the channels for the input configuration I was using. This lead to the need for a command to configure an ADC channel.

Each ADC channel could be individually programmer for either single-ended or differential input, unipolar or bipolar voltage ranges and for 5V or 10V as the upper voltage in the range. I didn't want to enter sixteen commands by hand, so I added the feature where pcmmio_adc_mode can configure a contiguous range of ADC's to a particular setting. But this still could require multiple commands. With a flash of insight, I remembered that I could write a shell script to do this for me. This led to me writing the the following very simple shell script to configure the ADCs.

#! joel
echo "Setting all ADCs to +/-10V (bipolar) and single ended"
pcmmio_adc_mode 0 15

As mentioned earlier, to test the ADCs I had to write a voltage using the pcmmio_dac command and then read the voltage using the pcmmio_adc command. I wanted to run a series of voltages through the ADC but the step command didn't let me see the input between steps. I could have figured out a way to get access to the lines but I had used a pre-made jumper wire and didn't want to destroy it. So I wrote another simple shell script which repeated pcmmio_dac, sleep, pcmmio_adc commands. This allowed me to verify that a range of voltage could be written and read. The following is one set of the three commands. There were a lot more than this to have a script that ran for fifteen seconds.

pcmmio_dac 4 -2.0
sleep 1
pcmmio_adc 0

You might wonder how I got the shell scripts onto the target. Well I used another interesting feature of RTEMS. RTEMS has long has the In-Memory File System (IMFS) and the capability to load initial contents from a tar file image linked with the application. I simply wrote the scripts on my development machine and included them in the initial file system contents.

In using the RTEMS Shell and its ability to add custom commands, I was able to refactor the original GNU/Linux device driver, adapt it to RTEMS, add capabilities such as timeouts and timestamping input, and debug this device driver with very little difficulty. Plus these commands are now available for hardware integration and testing for this project and any other project that might use this device driver in the future.

Running SLOCount on RTEMS

2010-10-20T12:27:00.003-05:00

SLOC is everyone's favorite metric to hate but it is fun to discuss anyway. I recently ran David Wheeler's SLOCCount [1] on RTEMS and thought I would pass along some of the results. There is an RPM available for Fedora so I started with that. But I found a bug in which it counted our assembly include files as Pascal. Needless to say there is no Pascal in RTEMS so I fixed that. If anyone wants the patch, I am happy to provide it.

The first part of the report from this program is a breakdown of the source code by language. This showed the following breakdown based upon programming language:


ansic:       644676 (87.49%)
asm:          39569 (5.37%)
ada:          27563 (3.74%)
sh:           18204 (2.47%)
cpp:           5236 (0.71%)
perl:          1623 (0.22%)

No real surprise there. RTEMS is mostly in C and high level languages with assembly language primarily for context switch and interrupt dispatching. With over a dozen active ports on the CVS head, it is no shock there is 39.5K of well commented assembly language.
The Shell and Perl is primarily for configuration and build infrastructure.

The second part of the output from sloccount is related to the estimated cost of producing the software and how many man years it would take. Using a rough estimate of 100K USD salary for each software developer implementing RTEMS, sloccount estimates RTEMS would require about 205 person-years to recreate at an estimated cost of 49.2M USD.


Total Physical Source Lines of Code (SLOC)                = 736,871
Development Effort Estimate, Person-Years (Person-Months) = 205.02 (2,460.21)
(Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05))
Schedule Estimate, Years (Months)                         = 4.05 (48.59)
(Basic COCOMO model, Months = 2.5 * (person-months**0.38))
Estimated Average Number of Developers (Effort/Schedule)  = 50.64
Total Estimated Cost to Develop                           = $ 49,204,211
(average salary = $100,000/year, overhead = 2.40).

That makes RTEMS a great bargain! You are getting very high quality software that would cost a sizeable fortune to reinvent. IUnfortunately, the RTEMS team doesn't have $49,000,000 so please consider using the services of the core developers. That's how we pay the bills and keep the kids from going hungry.

Enjoy the gift of RTEMS!

--joel

[1] Per David's request, this data was "generated using David A. Wheeler's 'SLOCCount'."

When Does a GSOC Project End?

2010-08-20T09:30:00.002-05:00

Today is the deadline for turning in student evaluations for the 2010 edition of the Google Summer of Code. The RTEMS Project was fortunate enough to have eight students this year and they all did great work. Some of it is still not merged and some needs some polish and love before being production ready. But it looks like all the students successfully met their goals. Personally, I am proud of all the students this year.

So the GSOC 2010 program is nearing its official end and it is tempting for the students to consider their work as over and end their involvement with RTEMS. But that's not the goal of GSOC nor is it desirable from an RTEMS perspective. All projects participating in GSOC want code written during the summer but they also really want long-term contributors. So this is not an end, it is a transition. What do you do next?

As a minimum, you need to work with the mentors to get your code merged into the various projects, functional, documented, and tested. We want your code to be part of the RTEMS experience moving forward. This is not a classroom exercise where the code is thrown away.

Last year, I told someone that you really should make your goal that you should be proud of what is merged and available. You should be able to cite this work as an example of what you are capable of to future employers and graduate schools. Just as all FOSS developers take pride in their work, you should also. Work with us to help get it to that point.

We would like to see you all continue your involvement. This would give you a chance to see YOUR work being used by real world applications. Your involvement does not have to be limited to the area of your GSOC project. RTEMS is a broad project and you can work on something else if that is what you want to do. It is also perfectly acceptable to simply continue to work to perfect your submission. We strive for 100% test coverage, great documentation and exceptional performance. Those are hard to achieve in a first implementation of anything.

So students, do not disappear. Stay involved.

May -- A TIme of Beginnings and Endings

2010-06-04T21:24:00.002-05:00

For still in school or who have children in school, May is always an interesting and busy month. It was the end of another academic year for my four children and the end of high school for two of them. It is the anniversary month for my high school, Bachelors and PhD graduations. At the high school graduation commencement, we were once again reminded by the speaker that commencement is a ceremony to celebrate the end of one thing and the beginning of another. It is more a recognition of a milepost on a journey than a goal in itself. At this point, you are probably asking yourself what this has to do with software and the answer is nothing and everything.

A successful software project lives on -- it is not something that is ever complete. Each release announcement is comparable to a graduation commencement. It captures what we have done and is just a milepost. It is a useful milepost in the software world because it represents a completion point. It is a recognizable measure like a diploma that means that the software has passed some measure of completeness and quality.

RTEMS is now over twenty years old. It is the same age as my daughter who is entering her senior year in college. It has matured just as she has. It has gotten more capabilities just as my children have learned more. It is smarter and more efficient than it was. My children are also somewhat more self-sufficient (not efficient yet) but I can hope. RTEMS is in many ways my first child. I was there at the birth. I held its hand as it moved from a research project to an independent free software project.

Today I cut the 4.10 release branch. This represents a major milestone in the life of RTEMS. I can say with certainty and the pride of a parent that this is the best release branch so far. With both coverage testing and the daily builder, the testing has improvement markedly over the past few years. We have automated testing for all of the GNU tools we use. We have overhauled the main web site to be more modern looking and hopefully friendlier. The core technical content was largely unchanged from the older site so this is a lot like a girl getting a make-over before a school dance.

But just as with a graduation commencement, there is a past and a future. The future for RTEMS includes both maturation and additional capabilities. I want to see the RTEMS Project mature its processes by further expand our automated testing and improve the patch review and merge process. I want to see RTEMS grow in capability and I know this is going to happen because there are multiple interesting efforts waiting to be reviewed and merged. There is a SPARC64 port with multiple BSPs, a USB stack, a port of the LWIP stack, and my work on symmetric multiprocesssing support for RTEMS.

So remember that life is full of milestones but those are also points of reflection and landings from which one can climb to the next level. So reflect a bit on 4.10 and look forward to what RTEMS 4.11 or 5.0 will look like. I am excited and hope you are too.

RTEMS SMP Exploration #1

2010-05-04T14:22:00.004-05:00

Many of you know that I have been working on an SMP design for RTEMS for a couple of years now as a background activity. This has been a slow project and I have been wanting to work on it more but it has been almost entirely volunteer activity on my part. But recently, the activity level has picked up. Gedare Bloom's GSOC proposal Modular SuperCore Scheduler Manager will make a significant dent in the work required to support SMP. Currently, there is no discrete Scheduler component in the RTEMS SuperCore and a big part of supporting SMP is to have a single processor and an SMP-aware scheduler. To properly support this, there has to be a discrete Scheduler component in the SuperCore and a mechanism to select alternate schedulers. Gedare's project is to refactor the existing code base and provide a SuperCore Scheduler Handler along with the configuration required. Gedare has previously implemented an Earliest Deadline First (EDF) Scheduler for RTEMS and we are planning for this to be available as one of the alternative schedulers. Tiny/RTEMS may even benefit because it may make sense to have a light-weight scheduler algorithm for lower end target processors.

With Gedare focusing on refactoring the Scheduler, I have felt free to focus on the process of initializing a multicore processor, interprocessor interrupts, spinlocks, and transferring control to the first thread on a secondary core. I have succeeded in demonstrating these on pc386 with up to 32 cores and on SPARC/LEON3 with four cores. This work has let me work through the definition of a per CPU OS structure as well as the BSP/RTEMS SMP Interface. If you are interested in experimenting with this on another architecture, let me know.

I am looking forward to working this summer with Gedare and seeing how close we get to SMP RTEMS support before the 2010 Summer of Code is over.

Google Summer of Code 2010

2010-04-04T11:03:00.003-05:00

It has been a while since I posted but life and RTEMS have been busy. The past few weeks have been a blur as GSoC 2010 approached. The UT Chattanooga IEEE-CS student chapter visited NASA's Marshall Space Flight Center here in Huntsville and OAR hosted them for lunch. Some Asterisk developers joined us and we presented GSoC, RTEMS and Asterisk to them. Earlier in the year, I had put together a cool double-sided flyer with RTEMS GSOC on one side and the general GSoC information on the other. Both sides had tear-offs which aligned close enough where you get the general and RTEMS specific GSoC URLs. For this visit, I put together a new flyer with the list of all participating organizations. I think it is important to let students know about all the cool organizations that participate.

The past couple of weeks have been hectic. I had work deadlines looming, one of my sons had H1N1, and RTEMS had more students than ever asking questions and submitting proposals for review. I had to update the VirtualBox image of Fedora 12 with RTEMS tools for the i386 and sparc pre-installed and make the torrent available. We had more students complete the hello world requirement and have proposals to review before the web app opened than we did all of last year!! We ask them to put their proposal into Google Docs and let mentors insert comments into that. We have so many students, I created a Google Docs Spreadsheet to track when each mentor had read the proposal and whether the student had submitted the official application. I spent this weekend ensuring I had read all of the proposals in hand.

Overall, I am thrilled with this year's crop of proposals and the number of cross-project proposals. We have students interested in Go on RTEMS, Parrot on RTEMS, and RTEMS support for the MilkyMist board. On top of that, we have students interested in adding a pluggable scheduler capability and EDF scheduler, coverage improvements, adding POSIX asynchronous and list IO, and much more. And two students from last year are back in a mentoring capacity. With a week to go before the student application deadline, I expect we will see more proposals and hope we are able to do a good job reviewing those. I know we will try to do so and be fair to all students making proposals. I am excited and hope we get enough slots to take on the students with quality, well thought out projects.

Ramblings From Radiology

2010-02-22T14:03:00.002-06:00

As I have mentioned to many of you, I had an excess of protein in my urine and have had to have a number of medical tests. Today I am at Huntsville Hospital and just got back from a kidney biopsy. I have to get another scan in a few hours to ensure I am not bleeding internally. What does this have to do with RTEMS you ask? Well, nothing directly. But being bored and surrounded by medical devices made me remember some of the RTEMS based medical devices people have told me about over the years.

The first device I heard of was an HP (now Philips) Criticare heart monitor. Philips/HP Criticare equipment is common in our area but I don't know if I have actually ever seen one of these.

I assisted in the development of a BSP for what was going to be a blood analyser for Roche. I recall it be m683xx-based but do not know exactly it tested for.

The AMV Technic I is a certified syringe pump for sale in a number of European countries. It uses RTEMS on an ARM CPU and we have information in the RTEMS Wiki. It uses Microwindows to provide the graphical user interface.

There are also RTEMS based intercom systems which are used by patients to summon or talk to a floor nurse. This product is ARM based and is on its second generation of hardware using RTEMS. Seeing a user build multiple product generations on RTEMS is personally very satisfying. It makes me know they are pleased with RTEMS.

There may well be other medical devices out there, but I don't know about them. Unfortunately, many people do not tell us about their application. I don't know how to change this but will continue to beg for marketing literature from RTEMS based products. Please save me from begging! Just send me the information.

NASA Solar Dynamic Observatory Launched

2010-02-11T10:00:00.003-06:00

The NASA Solar Dynamic Observatory (SDO) was successfully launched this morning. The SDO satellite includes 5 Radiation Hardened Coldfire CPUs running RTEMS. The Coldfire/RTEMS computers run many of the I/O functions and one of the instruments. A very nice graphic of the mission is here. The following is the description of the project from the SDO Project Site:

SDO: The Solar Dynamics Observatory is the first mission to be launched for NASA's Living With a Star (LWS) Program, a program designed to understand the causes of solar variability and its impacts on Earth. SDO is designed to help us understand the Sun's influence on Earth and Near-Earth space by studying the solar atmosphere on small scales of space and time and in many wavelengths simultaneously.
SDO's goal is to understand, driving towards a predictive capability, the solar variations that influence life on Earth and humanity's technological systems by determining

how the Sun's magnetic field is generated and structured
how this stored magnetic energy is converted and released into the heliosphere and geospace in the form of solar wind, energetic particles, and variations in the solar irradiance.

Congratulations to the folks at NASA Goddard on a successful launch and good luck in meeting their science goals. I look forward to the new insights we will gain into the Sun.

RTEMS Graphics Toolkit

2010-01-20T18:56:00.004-06:00

I haven't posted since before the holidays but that doesn't mean that nothing has been going on in the land of RTEMS. I have been trying to work with Roxana Leontie to get her work merged into the Microwindows/Nano-X repository. Roxana was an RTEMS Summer of Code student. The RTEMS Project took the Google donation from the Google Summer of Code and money from OAR Corporation to fund Roxana as the inaugural RTEMS Summer of Code Student. Roxana worked to update and clean up the ancient port of Nano-X to RTEMS. Roxana did all of her work targeting the RTEMS pc386 BSP. The screenshot in the upper right is the Microwindows minesweeper demo program.

As Roxana wrapped up her work, she cooperated with a Google Summer of Student (Xi Yang) who had developed a Gumstix Connex BSP for RTEMS which ran on the Skyeye simulator. Xi implemented a frame buffer device driver for this board which worked with Nano-X. All of Xi's work was merged into the RTEMS Source Code Repository when completed.

As part of preparing her work to be merged into the Nano-X Source Code Repository, I worked with Roxana to clean up the build infrastructure and ensure that the code built easily on multiple BSPs. As part of this, we worked to port some supporting libraries to RTEMS. Before Greg Haerr had merged our work, we had created the RTEMS Graphics Toolkit (RTEMS CVS module rtems-graphics-toolkit) which includes the following libraries:

libjpeg - (v7) http://www.ijg.org/
libtiff - (v3.6.2) http://www.libtiff.org/
libpng - (1.2.42) http://www.libpng.org/pub/png/libpng.html
libfreetype - (2.3.11) http://freetype.sourceforge.net/index2.html
libt1 - (5.1.2) http://www.t1lib.org/

This means you can decode and display JPEG, TIFF, PNG, GIF, and BMP graphics on an RTEMS system. It also means that you may have support for Adobe Type 1 TrueType fonts. The screen shot with the "Embedded with RTEMS" logo is the output of nxview running on the arm/gumstix BSP. The logo is a jpeg loaded from the RTEMS In-Memory File System (IMFS). The jpeg support library has no patches for RTEMS.

Using a build script in the top directory of this kit, you can easily build all of these packages and Nano-X (from their git repository). This includes building all demos for a specific BSP. The RTEMS Graphics Toolkit also includes the following libraries which are not currently ported but we would like to see ported.

svgalib - (1.4.3) http://svgalib.org/
fltk - (1.1.10) http://www.fltk.org/

Roxana is pursuing porting svgalib to bring its collection of more advanced graphics drivers to RTEMS. Interestingly, this ties in with the work of another Google Summer of Code Student, Aanjhan Ranganathan. Aanjhan's project consisted of adding MMU support to RTEMS. This includes support on some targets for the mmap() family of calls from POSIX. The svgalib package includes approximately 100 calls to members of this call family so we now have incentive for Aanjhan to get his work tidied up and merged.

FLTK is an important piece of the puzzle because it includes a GUI builder and numerous widgets. This will provide a robust foundation to build GUIs for RTEMS systems.

At this point, all Nano-X demo programs that are expected to run on RTEMS do so on both the pc386 and arm/gumstix platforms. There is obviously more to do. But it is important to emphasize that this is a major step forward for RTEMS. It makes it much easier to build GUIs for RTEMS based embedded systems.

Thanks to all the folks at Google who work so hard on the Summer of Code and the students they sponsor. This would not have happened without them. Thanks Leslie, Ellen, and Kat. :)

RTEMS Turns 21

2009-12-28T11:19:00.002-06:00

The RTEMS ticker test (a.k.a. sp01) was the first test created during the development of RTEMS. It sets the date and time to 9:00 pm on 31 December 1988. This coincides with the earliest part of the RTEMS Project. Since there is no birth certificate, we are treating this as RTEMS birth date.

Please toast RTEMS turning 21 where ever you are.

Please don't be online -- share the New Year's Eve with loved ones. :)

Violating the Rule of Least Surprise

2009-12-13T17:00:00.007-06:00

When you mention user interfaces, people usually think of a graphical interface with mouse and keyboard or a touchscreen. But user interfaces do not exist solely in the world of computer displays. In the world of embedded systems, a user interface can take on any number of variations. One guiding principle that particularly applies in this domain is the "Rule of Least Surprise." An example I use is that if you put someone behind the steering wheel of a vehicle of any type, they would expect that turning the wheel to the left directs the vehicle in that direction. To do anything else, would surprise even a three year old who has ridden a tricycle.

Even though I love my current car, it has two areas which violate this rule. As shown in the picture shown here (source), the instrument panel is in the center -- not directly in front of the driver. Both interface surprises stem from this placement.

The first violation is that when the lights are on at night, there is no light at all directly visible in the direct field of vision of the driver. It is very easy to forget to turn the lights on at night when in a well-lit area because you have to look to the middle to realize that the instruments are unlit. This has actually resulted in me getting pulled over by a friendly police officer to remind me to turn my lights on.

The second violation is when you turn on the turn signal to turn left. The blinking light indicating you have done so is in the right hand side of the driver's field of vision. In other vehicles, this would be in the left hand side
of your field of vision.

Neither of these is a major issue and I have grown accustomed to both. But both show how humans have expectations about how common devices operate. When your clever design changes this, users are surprised. In the end there may be a benefit, but often it is a difference just to be different. Is there a surprise in a system you are familiar with? Is it an improvement or just a difference? Share.

Fedora 12 Upgrade Experience

2009-12-12T13:31:00.005-06:00

A few weeks ago, I upgraded the GNU/Linux computers I am personally responsible for from Fedora 10. This includes a work computer (Dell Latitude D830), a home computer (Dell Inspiron 2400), and the home server (upgraded Dell Dimension 2100). The home server is the host of an Elvis Costello Fan Forum phpBB site as well as the Elvis Costello Wiki so it is not just my family that is impacted by down time. I upgraded from Fedora 10 to 11, ran for about a week to make sure things were OK, and then upgraded to Fedora 12. In the end, all of the computers are up and stable performing their assigned tasks but there were issues along the way which I will describe here.

I upgraded via DVD and all of the computers handled that OK once I remembered the magic required to boot from DVD. But afterward the upgrade was complete, two of the computers did not boot. As not so infrequently during GNU/Linux upgrades, GRUB was not updated correctly on two of them and I had to go into rescue mode to address that. This just required mounting the root partition, changing root and running grub-install. Scary the first time it is required, but not after a few times.

One of the laptops booted nicely to the GUI login prompt but was clearly at the wrong resolution. I did a little net research and learned that deleting the /etc/X11/xorg.conf file and restarting X11 would result in a probe and very likely the correct video settings. This worked.

The only issue shared by all of the machines was an issue with yum getting upgraded between Fedora 10 and 11. yum was broken after the upgrade. Before I could load further updates from then using yum upgrade, I had to manually fetch a current yum rpm file and install it manually using rpm.

The server had three very strange issues during the upgrade.

First, I couldn't log in. The GUI login screen was flashing. I remembered the magic key sequence to switch to a console display, logged in, and checked out the logs. I couldn't tell exactly what was wrong but I had a guess and tried it. Apparently the graphical login program GDM had trouble parsing my email address in the name field of /etc/passwd. I think something was trying to use the user name field in an XML file because my entry was something like "Joel Sherrill ". I suspect that the presence of my email address looking like an XML tag confused it. I deleted that and moved along. GDM was happy.

Next on the list was that something changed in MySQL and the user name table was reported as being corrupt. I found reports on the web where others had suffered this and decided the easiest solution was to stop MySQL, delete the old table, and restart MySQL. When restarted, MySQL would create the table with default contents. From there, I could easily add back in the handful of accounts that were needed. The actual database tables were not impacted. So I felt lucky and moved on to the next problem.

At this point, I started looking at the actual web content served from the server. I first looked at the pbpbb-based Elvis Costello Fan Forum, A bit of the page header was displayed along with an error message. I don't remember why but somehow, I couldn't see the error completely on my side. I enlisted some help on the #rtems IRC channel to see if others could see what the message said. Someone cut and pasted me the error message. I Google'd the message and learned that the new php 5.3 now requires the /etc/php.ini file to explicitly set the timezone. It does not trust the operating system's setting. Without the timezone being set for php, php programs reported nasty error messages. This meant that phpbb3 would not run correctly and the Fan Forum was not available. Adding the following to /etc/php.ini resolved this.


[Date]
  date.timezone = America/Chicago

I inspected the pages served from the Elvis Costello Wiki and there were some minor error messages in the logs. Apparently, Mediawiki had some minor mistakes in its coding which php 5.3 detected. These resulted in a plethora of errors in the web logs.

I am a daily GNU/Linux user who rarely works in MS-Windows. As a long time user of various UNIX desktop environments, I prefer mousing over a window to move focus. I do not like to click on a window to move the focus. In previous Fedora distributions, GNOME included an applet to switch the preference under "System > Preferences > Windows". This applet is no longer included by default. I don't know why it isn't part of the default install anymore. Is 59K really that critical? Anyway, the following installed it.


  yum install control-center-extra

At this point, I have been using Fedora 12 daily on two desktops and a server. I use the two desktops for development, email, and word processing using Open Office. I have encountered a few issues which I will share in a future post.

Debugging Data Corruption #1

2009-12-10T18:56:00.006-06:00

I recently helped someone debug a data corruption problem on an embedded system that was not RTEMS based and that I had not written any code for. Just to be sure this is perfectly clear, I knew nothing about the source code to this application or its design when I arrived. They had a global variable whose value was becoming invalid. For the purposes of this discussion, the corrupted variable will be referred to as CorruptedVariable.. They had a way to reproduce the corrupted and could tell based upon the behaviour of the program that at least CorruptedVariable was being clobbered.

When I arrived, I listened a few minutes and watched them reproduce the problem. I then asked they knew how to produce a numerically sorted symbol table for their application. They did not and I explained that with any GNU toolset using binutils, there was included a utility named nm which is used to produce a symbol table from an executable. I instructed them to use a command similar to the following:


  avr-nm -g -n application >application.num

The -g option requests that the symbol table include only global symbols and the -n option requests that the symbol table be sorted by address. We then looked at application.num in a text editor and I had them search for CorruptedVariable.

I then looked at the variables immediately before (e.g. lower addresses immediately preceding) CorruptedVariable in memory. The variable immediate preceding CorruptedVariable was a 32-bit integer (RandomVariable so it was unlikely that accesses to it were the culprit. But the variable before it had buffer in the name. That was a big hint and I asked to see the corresponding source code. The C source code where the CorruptedVariable was declared was similar to the following:


    float  TooShortBuffer[8];
    int    RandomVariable;
    int    CorruptedVariable;

So TooShortBuffer was indeed an array and not a structure. For the purposes of this bug, that was awesome. It meant that using an index that was greater than 8 would overwrite the variables immediately after it in memory. And we had confirmed what the variables which would be overwritten were.

I asked what TooShortBuffer was used for and one of the team mentioned that they had recently added something and might have forgotten to extend all the arrays in the program. As a quick test, I asked them to add 100 elements to that array. Not surprisingly the test scenario now worked.

I recommended to the team that the array now be sized with a hard number but with a macro defined to the maximum number of elements. In addition, I recommended that when using an index into this array to access it, validate that the index is within range. Self-checking programs are so much easier to debug.

I spent less than an hour on the customer site and left then with a smile.

In my next post, I will show how gdb can be used to help locate the source of stray writes.

PC386 Memory Size Bug

2009-12-10T14:57:00.001-06:00

Another day, another bug. At least this one was interesting. This issue was initially reported as Ada programs compiled for the pc386 BSP would run on the Qemu simulator but would not run on the real embedded PC we were using. Since GNAT/RTEMS toolsets are built from source and users do not get the luxury of using the RPMs, my first suspicion was an Ada specific issue. But after a few days of thinking on this, it occurred to me that the code was locking up long before Ada code ran. I stripped the test case down to C only and added a dummy method for gnat_main(). This used the exact RTEMS application configuration and initialization as the broken Ada program did. This program failed in exactly the same way. Feeling the rush of some success, I did a binary search to determine what configuration setting was making this fail. After a few attempts, I discovered that when the application configued zeroing out the RTEMS Work Area, the board locked up. Eureka!

With that piece of insight, I began to focus on the code that zeroed memory. I looked at the starting address and length. The BSP initialization claimed there was 1007MB of RAM available on the board. It looked fine given the board's configuration. I decided to try deliberately lowering the amount of RAM RTEMS knew about. Various low values worked and I tried 1004MB. That worked. I suddenly realized that some memory was reserved for some purpose. I went away to think since I had a work around.

I looked up the manual for the motherboard's chip set and low and behold, I had forgotten that the video controller shared RAM from the main RAM. We needed to stay away from that area. I posted to the RTEMS mailing list and it was suggested that we should trust the RAM size provided by the multiboot information available NOT the amount determined by our dynamic sizing probe. Looking at the 4.9 release branch, it turned out that that code did give priority to the multiboot information.

I compared the two pieces of code and quickly came to the conclusion that updates had broken this code in a subtle way. A little restructuring and clean up and the problem was resolved. But it had been a long and circuitous path from initial report to problem resolved.

What are the lesson to learn?

In this case, there are two lessons to take away. First, a problem may not be what it first seems to be. My first impression was that this was an Ada related issue not a very low BSP specific issue.

The second lesson is that divide and conquer is a very good strategy. You need to narrow down the problem space so it is possible to track down the specific issue.

First Blog Post

2009-12-10T14:46:00.000-06:00

Every blog has to have a first port and this blog is no different.

I have considered having a blog for a while but never got around to it. I don't know if anyone cares to read it or not. I am the maintainer of the free real-time operating system RTEMS (http://www.rtems.org) and a member of the GNU Compiler Collection (GCC) Steering Committee (http://gcc.gnu.org). I plan to ramble on free software activities that I am involved in, interesting bugs, RTEMS improvements, etc.. I may also ramble about design patterns for use in embedded systems, performance analysis, and GNU/Linux.

This is a modest beginning to what I hope will be a blog containing interesting posts.