Resignation From Network Time Foundation

I don't often quote Bible verses but this seems like an appropriate time for "to every thing there is a season, and a time to every purpose under the heaven." I have served on the Network Time Foundation (NTF) board since its inception in 2012. It has been interesting being involved in the start of the foundation. But as the verse says, everything has a season and a purpose.

My primary responsibilities should always be to my family, career, and to my own free software project RTEMS. All of these need more of my attention. I want to do right for everyone and everything I am involved with.

After years in the free software community, I am always happy to share experiences and provide advice to other projects. I have seen a lot happen since the inception of RTEMS and learned a lot of lessons the hard way. If I can help someone avoid those pains, I am happy to. That offer applies equally to projects under the NTF umbrella as well as any others.

I will close with one piece of advice for all free software projects. Remember that it is all about the code and community. Get those right and the rest will fall into place.

What Android Apps Do I Use?

There are so many Android applications out there but I suspect my choice of apps is probably very mundane in comparison to others. My usage has evolved some as the Android devices are much more powerful and capable than the G1 I first had, although I miss the physical keyboard of the G1 and G2. In fact, my wife used a G1 until it physically broke. She cursed the HTC One S that replaced it until recently moving to a Moto G. The Moto G is a very nice phone for the price and she is pleased with it. But this is not about how my family feels about their phones, it is about the apps I use so here are the apps I use:

• Google apps - I am just going to lump the ones I use into one bullet item. I use Gmail, Calendar, Messenger for SMS/MMS, Chrome, News, Maps, Drive for documents (not backup), Voice, Translate, YouTube occasionally, and am experimenting with Keep.
• K-9 Mail - I probably spend more time in this application than any other. I used the Google Email client for years until I started getting complaints that sometimes messages from me did not thread properly in the RTEMS mailing list discussions. Chris Johns had me send email from every computer and client I used. Turned out that the Google Email client does not send the proper headers. This means that the threading that is relied on by open source community mailing lists is broken, has had a long outstanding bug, and doesn't seem like it will ever get fixed. I tried K-9 and never looked back. It is a very capable Email client for "work" accounts. I used Gmail for my personal mail.
• Apps which help when traveling:
• TVFoodMaps - This is one I used when traveling to see which restaurants have been featured on television shows on the FoodTV network. It sounds kitschy but it is a neat way to find a local place I would never think of trying otherwise. Review sites often recommend the same places but often travelers don't find the truly historic, unique places, or know what a local specialty that shouldn't be missed.
• TripIt Travel Organizer - Another handy app for traveling which provides a mobile interface to the web site. TripIt works by you forwarding email with hotel reservations, car reservations, and plane tickets to it. It then builds an itinerary for you and reminds you of upcoming travel. When on travel, it makes it easy to get destination addresses, confirmation numbers, etc. This is much easier than carrying paper.
• Fly Delta -  This app has improved a lot since I first used it. Everyone hates some airline but Delta has the best options from Huntsville. This gives me notices on gates, arrival time, bar code for paperless boarding, and Sky Club locations. Not essential since I could likely do all this with the website (except paperless boarding) but handy.
• Utilities
• OpenIntents - This project has a collection of apps which include a Notepad, Shopping List, and a File Manager. The shopping list is one of those specialty apps which is very handy.
• Astro File Manager - Just a nice file browser.
• Miscelaneous
• Untappd - Log beers, earn badges, see what friends are drinking and get recommendations. I am a big fan of craft and unusual beers. I like to try different beers as much as I can and this helps me keep track. 
• Vintage Comic Droid - This is a neat application that lets you read comic books that have fallen into the public domain. I primarily use it in off-line mode to kill time. 
• TV Guide - An application with functionality that matches its name. It is nothing more than TV listings but it does know cable systems and highlights new shows as well as let you search, and set alerts. A bit painful to use on small screens but handy.

Probably not that unique a set of applications but it is how I get by. Most of these I have used for years and through multiple Android versions.

In most cases, I refuse to load apps which are only a front end to a website. I can just use Chrome and get there. And I get irritated when a site pushes me to load an app -- LinkedIn comes up high on this list. If anyone is listening, asking me every time if I want to load your stupid site specific app is a turn off for your site in general. If you want to lose me as a user, remember my preference.

Moving to a Nexus 6

Normally I blog about RTEMS and related embedded systems topics. This time is different. I ordered a 64 GB Jedi Blue Nexus 6 on November 25 and it just arrived today (December 22).  When I placed the order, the store person heard me complain about my Nexus 4 thinking a wired headset was randomly plugged in and I had some plan feature which let me get a replacement for USD5. That turned out to be an interesting adventure but I will summarize it in a few bullets:

• November 25 - order replacement Nexus 4 and Nexus 6 which was TBD delivery. You can see the box to the right and not have to wait as long as I did. 
• November 26 - replacement Nexus 4 arrives, migration goes smoothly, updates to Lollipop, and then I discover radio part of phone is bad. It won't connect to cell network. Back to old Nexus 4.
• November 28 - second (or third depending on how you count) Nexus 4 arrives. This time the transition is painful. I ended up spending almost two hours the next day manually loading applications and entering passwords. Luckily, I had a spare screen protector and used the old case. Back in action again.

I have been experiencing Lollilop with all of those Nexus 4's and even got the 5.0.1 upgrade last week. It is a nice experience with my only complaint being that I somehow can't seem to stay on the screen for a call in-session. That means I have to switch apps to hang up. My Nexus 7 upgraded smoothly but it went through another round of slow down and I had to another factory reset. This seems to be a common occurrence with the 2012 Nexus 7 models as they age and the Flash memory ages. I have seen reports on the net of switching apps taking a minute. Mine got that way under KitKat before I factory reset it. If someone has a good way to speed it back up, let me know. 

Today the Nexus 6 arrived while I was blowing leaves. Somehow I resisted abandoning the yard work. The first step was copying the files off my Nexus 4 yet AGAIN. The box has a nifty embossed "6" on it and that's the top side. The first picture is what I thought was the top side. When I opened it using "that" top side, it almost fell out. Glad I had a towel on the table, just in case.

Of course, the phone couldn't stay in the box like that very long. It would take all the fun out of it. I was concerned that the high speed charger was going to be an accessory that was not included with the phone. That would mean another order and I just didn't want to have to do that a few days before Christmas. On average, I travel a week every four to six weeks and keeping a phone charged can be a challenge. Many times I am in meetings in buildings which are very effective Faraday cages. Nothing drains a battery faster than spotty or no service. I have learned to keep it in airplane mode and rely on my old Nexus 7 tablet but still I need to recharge using my external battery or wall charger. The Turbo Charger was a feature I was really looking forward to.

The phone was in a tray so I pulled the tray out. There was more in the box than was included with the Moto G or replacement Nexus 4. I am pretty sure that the Nexus 4 included a similar set of accessories when it was new though.  Inside the box, there is the phone (obviously) along with an instruction packet, what I hope is the quick charger, and a USB cable. Not immediately seeing a tool to open the SIM slot, I am let wondering if it included a tool to eject the SIM slot, I opened the instruction packet and did find a tool to open the SIM slot. Unwrapping the wall charger, I see the magic words Turbo Power Supply and that makes me happy. That makes me two for two on this so far.

Now on to migrating my information to the new phone. I had forgotten to back up my SMS and MMS so ran Backup to Gmail one last time,  When this was done, I was ready to turn off cellular data on the Nexus 4, power it off, and remove the SIM. While this was running, I decided to compare the two phones in size. On paper, the sizes don't sound that different but when you see them side by side, the difference is very clear.

Side by side, the Nexus 6 is clearly quite a bit larger than my old Nexus 4. My concern with a phone this large was (1) would it fit in a dress shirt pocket (it does) and (2) will it tend to fall out of a shirt pocket since it is taller and thus has a higher center of gravity. I tested (1) at the T-Mobile store and only time will tell about (2). I have a screen protector and case for the Nexus 6 and hope this will be enough to protect it if it drops.

Enough with the pictures already. I want to use the phone. I eject the SIM from the Nexus 4 and eject the holder on the Nexus 6. First surprise! The SIM in the Nexus 6 is smaller than that in the Nexus 4. I smile when I notice there is a SIM in the Nexus 6 already so I don't 'have to make an extra affort to acquire one. That was a nice touch. Now to call T-Mobile and get the new SIM activated.

All calls for support to any large company start with an automated system asking you all sorts of questions that eventually lead you to a human if you are lucky. T-Mobile is usually better than most companies for support and this time was no different. It took all of about two minutes before he asked me to boot the new phone. I stayed on the line with them through this. I get to the screen where it asks me if I want to Tap my old device and transfer settings. I turn on NFC on the Nexus 4 and tap the back of the devices. After a few seconds, it went on to set itself up quickly.

It immediately found an LTE tower, asked me for my Google account, and began synchronizing.  By the time I went to check on the status of WiFi, it had connected to the 2.4 Ghz channel of my access point. I switched it to the 5.0 Ghz and apps continued to update. One of the first updates was Android 5.0.1 which required a reboot. The phone had about 40% remaining on the battery when it first powered up and I expect that will be enough to finish the app installations. After clicking accept a few times, I decided to quit watching them download and begin to transfer files from my PC back onto the device. 

Then began the tedious process of entering account and passwords for various applications that were not covered by Google. I had done this a month ago when I went through the Nexus 4 swap and it is just as tedious this time. After a while doing this, I began to put the apps back on the main screen. In doing this, I noticed the Hilton app says it is incompatible with my device and refuses to load. Odd since it worked with the Nexus 4 on Lollipop. After about an hour from first power on, it is mostly setup.

I left it attached to the computer and it charged to 90% while downloading applications and updates. After the downloads stopped, I started playing with it. It is definitely large enough where it is more like a tablet when browsing. It is much faster than my Nexus 5 or Nexus 7. I am finishing this up the next morning and I have no complaints so far. I still need to tweak the notifications so I recognize the apps and go back to the ring tone I had before my phone got stolen last year but those are my problems.

If anyone cares, I will write another blog entry on what apps I have on my phone.

Intel Edison and RTEMS - Road Forward

The Intel Edison is now (11 December 2014) is now running RTEMS but there are issues to resolve and features which need adding. This post is a list of what needs to be done.

In the near term, the current code needs clean up before it can be submitted to the RTEMS Community. The following is a list of some of the activities which need to be performed before the code can be submitted:

• Break conditionals in build system and C code from "if Edison" to more features oriented conditionals. This requires being able to disable at least VGA, IDE, and legacy COM ports at BSP configure time.
• Split the current Edison specific polled console code into its own file. 
• Review dynamic console selection code and see if the current code can be improved so Edison specific conditionals are not needed.
• Generally split the one large patch into smaller discrete patches and try to not make them Edison specific.
• Write instructions.

There may be other clean up but that should be enough to get the current Edison support into the tree.

Longer term, there is more to do and it involves coding. Some of this can be done without access to Edison System On Chip documentation. But other parts will require having access to the details.

• Add support for APIC interrupt controller.
• Figure out why clock tick isn't working. This may require using different hardware for the clock tick driver.
• Provide access routines for the memory mapped PCI configuration space.
• Provide glue so hacked NS16550 console support can use libchip driver like COM[1-4] do. There is support already for PCI cards with multiple NS16550s so this should just be a matter of having the information once PCI configuration space is working.
• Test this driver polled
• Test this driver interrupt driven

That will get a complete basic BSP working with no hackery. Beyond this, RTEMS needs support for the other peripherals on the Edison:

• Add support for the discrete I/O and analog inputs.
• Evaluate supporting the Flash. We may want to be careful to not destroy the Linux installation or we may want to simply take over the entire Flash as long as we avoid conflicts with other uses. Having the Linux and USB file loading is nice. 
• Add WiFi support. This is a more involved project as RTEMS currently does not support TCP/IP over USB or WiFi. 
• Since the new FreeBSD 9.x TCP/IP  stack came with the USB stack, supporting wired Ethernet over USB is a logical first step and is also needed to support Ethernet on the Raspberry Pi.
• Once wired Ethernet over USB works, then the wireless part can be addressed including applications to manage connecting to access points.

There is probably more to do to fully support the Intel Edison board but it does work now and provides a baseline for future enhancements. At this point, interested users can fund core developers to add capabilities or volunteer to add it. That is how open source works.

Intel Edison and RTEMS - Day Three

Overnight, I realized that the error the RTEMS hello world sample was failing with was RTEMS_UNSATISFIED (13) likely indicated a miscalculation in the RTEMS Workspace by . A quick look showed that the primary difference between the pc586-sse BSP variant and all other pc386 variant was that because of SSE, all tasks were forced to be floating point tasks. This results in the allocation of floating point contexts for all tasks and apparently the calculation is a bit too low on the x86 port. I switched the edison BSP variant to be compiled like the non-SSE pc586 and this solved the problem. A short time later, this output was posted to the RTEMS Users mailing list.

## Starting application at 0x0010000c ...
024rtemsWorkAreaStart=12FD40 MemSize=0x08x/0

bspstart.c
work_area_start = 0x12FD40
work_area_size = 1072497344 0x3FED02C0
end = 0x40000000
current stack pointer = 0x12FD18
heap_start = 0x133B72
heap_size = 1072481422
initialize device drivers
probe
Checking on /dev/main
probe
Register /dev/main
Register /dev/main as console
Console initialize complete
post driver hook
rtems start multitasking


*** BEGIN OF TEST HELLO WORLD ***
Hello World
*** END OF TEST HELLO WORLD ***

EXECUTIVE SHUTDOWN! Any key to reboot...

At this point, the next step was simply to remove most of the debug output and get a clean run. Then we tackled seeing if enabling the timer calibration code in the pc386/startup/bspstart.c file would work. The calibration failed and likely indicates some combination of interrupt controller or timer hardware is different from legacy PCs. Fixing this is left for future work.

The failure to have a working clock tick meant that we needed a fall back. RTEMS has a target independent fake clock tick driver which is a special IDLE thread which repeatedly calls rtems_clock_tick() each time it is executed. When a "clock tick" occurs which wakes up another task, the IDLE thread is preempted. This is an effective way to run most RTEMS tests on a simulator with no interrupt sources.

With the IDLE thread clock tick in place, Jeff and I ran the ticker example which has three tasks executing every 5, 10, or 15 seconds. It runs for 35 seconds. Jeff and I tuned a delay in the IDLE thread until time appeared to pass reasonably close to it would on real target hardware if the system were lightly loaded.

After a few iterations, we moved on to the fileio sample test. This demonstrates the shell, IDE file access, and requires user input. On the first run, the test hung and we had to disable the IDE driver because the Edison does not have one. On the second run, it just worked.

That was midday on day three (11 Dec) and that was all that could be done without writing new code. Plus Jeff needed to get the Edison, a BeagleBone Black, and a Raspberry Pi working from a small laptop and then boxed up with cables. OAR has a table and will show them at the Flight Software Workshop in a few days )16-18 December  )

Intel Edison and RTEMS - Day One

It was Monday afternoon when an Intel Edison board showed up in the mail. It is quite small with an expansion board needed to have access to I/O. From an RTEMS perspective, it looks interesting because it has 40 discrete I/O pins and six analog inputs. The challenge was to see

how long it took to get the RTEMS up on this board.

It wasn't long at it was unboxed, that we were looking at a GNU/Linux prompt in a PuTTY window. Poking around, it quickly became apparent that the Edison didn't use Grub so my first hope that we could just add an RTEMS executable as an option on the boot menu was off the table. We had to identify a way to boot an RTEMS executable.

We had already noticed that U-Boot was on the board. Now the question was how to get an executable image into a bootable location and what the magic commands were to boot it. I came across this thread which had a lot of information on the Edison. It quickly became apparent that the Edison did not have much low-level public documentation yet and that support for most (if not all) legacy peripherals was not present. I posted an email to the RTEMS Users mailing list which summarized what we knew so far:

• SoC documentation won't be out until later this month.
• Has U-Boot and doesn't appear to have a 16-bit mode at all.
• Looks like U-Boot will want a binary image to boot.
• This turned out to be wrong. We figured out how to boot an ELF executable.
• There is a comment in the OS Dev forum that the PCI configuration space is memory mapped and not IO mapped like a regular PC. This means the PCI initialization and support code needs to be different for Edison from other PCs.
• All serial ports are on the PCI bus. It looks like the PCI layout is constant so for a start, we are hacking a simple polled driver for one UART and go from there. U-Boot initializes that port so we can skip initialization. Besides the person on the forum couldn't figure out the clock rate for the UART. This should be enough to at least temporarily get around the lack of PCI config space code.
• There is a watchdog on it that needs to be disabled or managed.
• Looks like it doesn't have a legacy i8254 timer and may not have legacy interrupt controller.

A user named mutex on the OS Dev forum had poked and prodded at the Edison board and managed to post a few very useful pieces of information.

• Base address of the UART and the U-Boot command to force a character out:
• mw.b 0xff010180 0x21 1
• Address of the watchdog control register along with commands to disable it:
• mw.l 0xff009000 0x10f8 1
• And to reset the board
• mw.l 0xff009000 0x10f8 0xf8
• Commands to boot a binary image.
• load emmc 0:9 0x100000 /kernel.img
• go 0x100000

I contacted this user. It was clear that he had been the most successful in booting something that wasn't the Linux that came with the board. He quickly replied. Thomas Nilsen has his own kernel and was kind enough to send me a binary which we were able to run. He is owed a big thank you.

This was the end of Monday with the board. 

RTEMS Texinfo Tools Update

A couple of years ago, Chris Johns and I began to discuss that RTEMS has had a long and successful history as a free software project. One aspect of this discussion resulted in the Open Source and Generational Differences. This post reflected on how new developers can have a tendency to avoid projects that use "uncool" tools. They want to be on the leading edge of technology. However, another aspect of using uncool tools is that they are old. RTEMS-based applications often have lifespans measured in decades from development, through fielding, to long term sustainment. This insight lead us to begin to review the tools we depended upon for long-term viability and that they continued to offer a high quality solution. The transition from CVS to git has been the most visible outcome of this effort.

But lurking within the RTEMS source tree was a dependency on a long dead tool - texi2www. RTEMS was the last user of this and had to include the source code in our own tree. This was exactly the type of situation Chris and I had realized could happen and it already had without us noticing. In November 2011, I posted to the GNU Texinfo Help Mailing list asking for advice on converting to the more modern texi2html program. Unfortunately, I learned that texi2html was considered deprecated and being re-implemented in Perl and the new implementation would be known as texi2any. This led to me converting us away from the stone cold dead texi2www to texi2html. With the recent release of texinfo 5.0, I began to ensure that our documentation would build with either texi2html 1,82 or texi2any from texinfo 5.0 and that the build infrastructure could detect which to use.

The goal of this post is to point out how we invoke tools, initialization file differences and the minor changes to our documentation required to support both tools. The other tools in the texinfo package such as makeinfo did require us to make changes to the source but did not change their invocation. I will detail the changes to our texinfo source files after showing the command line differences for the html converters.

The commands executed by our build infrastructure are generated by an autoconf based build infrastructure. This sometimes leads to longer than absolutely necessary command lines. I have made no attempt to shorten or clean these command lines up. These are for the RTEMS C User's Guide whose main texinfo file is c_user.texi.

The following is the invocation of texi2html 1.82:

 texi2html -D use-html --split node --node-files -o ./ --top-file index.html --init-file=../texi2html_init \  
   -I /home/joel/rtems-4.11-work/rtems//doc/user -I /home/joel/rtems-4.11-work/rtems//doc \  
   -I .. -I . --menu /home/joel/rtems-4.11-work/rtems//doc/user/c_user.texi \  
   /home/joel/rtems-4.11-work/rtems//doc/user/c_user.texi  

This is the corresponding invocation of texi2any from texinfo 5.0:

 texi2any --html -D use-html --split node -o ./ --init-file=../texi2any_init \  
   -I /home/joel/rtems-4.11-work/rtems//doc/user -I /home/joel/rtems-4.11-work/rtems//doc \  
   -I .. -I . /home/joel/rtems-4.11-work/rtems//doc/user/c_user.texi  

Notice that both command lines are quite similar. However, texi2html requires the --node-files argument to produce individual html file names which are based on the section or chapter name. By default, they will be named using a pattern line DOC_nnn.html.

The other thing to note is that they both accept initialization files. However, the format of the initialization files is very different between the two implementations. Texinfo supports hierarchically structured documents and allows the author to provide links to the next section, previous section, and the section that contains or is logically above the current one. The RTEMS Project has a tool which automatically constructs the node markups based on chapter, section, and subsection headings. Thus, the RTEMS documentation is fully hierarchically linked with no manual node definition required. The RTEMS documentation build system uses the initialization file to define a custom header, footer and to modify the navigation buttons.

This is the file texi2html_init generated by our build infrastructure:

 my $button_text = '<a href="../index.html">Library</a>';  
 push @SECTION_BUTTONS, \$button_text;  
 push @CHAPTER_BUTTONS, \$button_text;  
 push @MISC_BUTTONS, \$button_text;  
 push @TOP_BUTTONS, \$button_text;  
 $AFTER_BODY_OPEN =  
 '<A HREF="http://www.rtems.com" target="Text Frame">  
 <IMG align=right BORDER=0 SRC="../images/rtems_logo.jpg" ALT="RTEMS  
 Logo"> </A>  
 <H1>RTEMS 4.10.99.0 On-Line Library</H1>  
 ';  
 $PRE_BODY_CLOSE =  
 'Copyright &copy; 1988-2011  
 <A HREF="http://www.oarcorp.com" target="Text Frame">OAR Corporation</A>  
 ';  
 1;  

The initialization files reflects the internal implementation of the two programs and the format used by texi2any is different. We have an initialization file which accomplishes similar things in the generated HTML files but looks different. For example, the texi2html output has navigation icons while the texi2any output has textual links.

This is the file texi2any_init generated by our build infrastructure:

 set_from_init_file ('AFTER_BODY_OPEN',  
 '<A HREF="http://www.rtems.com" target="Text Frame">  
 <IMG align=right BORDER=0 SRC="../images/rtems_logo.jpg" ALT="RTEMS  
 Logo">  </A>  
 <H1>RTEMS 4.10.99.0 On-Line Library</H1>  
 ');  
 texinfo_register_handler('setup', \&add_button);  
 my $button_text = '<a href="../dir.html">Directory</a>';  
 sub add_button($)  
 {  
  my $self = shift;  
  foreach my $button_type ('SECTION_BUTTONS', 'CHAPTER_BUTTONS',   
               'MISC_BUTTONS', 'TOP_BUTTONS') {  
   my $buttons = $self->get_conf($button_type);  
   push @$buttons, \$button_text;  
  }  
  return 1;  
 }  

There were only a couple of issues encountered with our use of texinfo which required modifying the source.

• Missing @item in @itemize lists now results in warnings. 
• The order of menu definition, @top and its @node, and file @include statements in the top level texinfo files had to be reordered. Texinfo 5.0 is not as forgiving on this.

In addition, I spotted mistakes in our documentation when reviewing the various output forms. The entire range of patches can be viewed online here.

NOTE: As of 24 February 2013, these have not been committed. When they are committed, links will be provided to the git repository.

We would like to take advantage of features in the newer tools and are investigating using a print on demand service for RTEMS manuals. I hope there is experience in the texinfo community about this but, if not, I suppose I will pester the maintainers until the results are satisfactory and report on what I had to do.

Thanks to the Texinfo maintainers Patrice Dumas, Karl Berry, and Eli Zaretskii folyzr being incredibly patient and helpful through this process.

GSOC Presentation at University of Tennessee at Chattanooga

Earlier today, I returned to my alma mater, the University of Tennessee at Chattanooga, to give presentations on RTEMS and the Google Summer of Code 2013 (download here). About 25 people were in attendance including two faculty members. Thankfully, my wife Michele had driven and let me do final review on the presentations. Chattanooga is about a two hour drive from Huntsville and in a different time zone (Eastern not Central). We had allowed time for traffic and parking problems but had no traffic. We ending up arriving about forty-five minutes early.   We were met in the parking lot by a student who provided a visitor’s parking pass. This greatly simplified having a car on campus. Parking at any university seems to be a challenge.

After no A/V difficulties, I put up a montage of pictures from some of the projects which use RTEMS.  Those who attended the GSOC 2012 Mentor Summit will remember the slide from the lightning talks. It is memorable because someone from another project presented it. I had forgotten the talks and went to the Google Store. The montage highlights awesome projects based on RTEMS including the BMW Superbike, Curiosity, Herschel, Milkymist, Solar Dynamic Observatory, and MMS. As students came in, there were plenty of questions about the projects.  I created the slide to give at an RTEMS friendly workshop where most knew what RTEMS was and I wanted to highlight users. It turns out this is a great slide to get conversations going. If other FLOSS organizations can brag on where there software is used, then a user montage is a good thing to have.

I presented the official GSOC slides first. I felt it was important to emphasize that all types of FLOSS software was represented and that all of the organizations were interested in student participation. Being effective and appropriate to participate in GSOC requires organizations to provide wish lists, mentors, regular interaction with students, friendly communities, etc..

I then moved to the RTEMS specific presentation which very briefly introduces RTEMS but focuses more on recent activities, ongoing activities, and our wish list. It highlights areas we want improvements to occur even in software development process areas. As the last slide came up, I realized I was finishing on time and had presented thirty minutes, leaving fifteen to twenty minutes for questions. I ended my talk by reminding them that I would love to see them all as RTEMS contributors but would be just as happy to see them involved in the FLOSS community on ANY project. We are a collection of organizations but do have common goals.

There were a lot of questions on GSOC followed by some on RTEMS. One student asked where GSOC work occurred. There were questions on how the mentoring worked and what mechanisms were used to communicate with the mentors. I noticed students were packing up and realized they had ten minutes to get to their next class. There were no more questions but I hung around a while.

The big surprise was when a student came up to me while I was packing up. He asked about real-time and SMP as a potential area for Ph.D. work. I told him that I thought it was an open area of research and with some literature research he should be able to find a good area. Years of research into uniprocessor real-time systems and scheduling have given us practical engineering solutions. But the complexities of modern pipelines, caching, and interactions of multiple cores break some of the underlying assumptions. I am concerned that this same level of maturity has not been reached in SMP embedded systems which require rigorous analysis of predictability.

My wife generously waited in the presentation room while I visited with the only faculty member left from when I was a student there Dr. Jack Thompson. Then my wife and I walked around campus, enjoying a pretty day and reminiscing. After all, it was only one day after Valentines Day and we met one another while students here. 

Transfer Information to Another HTC One S

I am still using my trusty T-Mobile G2 while waiting for the new Nexus 4 to arrive. It has only been on order since Thanksgiving so maybe sometime in January it will arrive. In the meantime, my wife has an HTC One S which has a small crack in the corner of the screen. This isn't a real problem but the USB connector has become quite flaky and it often doesn't get charged. I ordered her a replacement and this is the hopefully short story of how I transferred her data and information to the new phone.

First, I did some research. It quickly become apparent that between Google's sync and backup plus what I could copy from the filesystem, all that was left would be SMS/MMS. Message Sync looked promising so I loaded it. I then performed the following:

• Placed the phone in Airplane mode to prevent new data from showing up.
• Used Message Sync to back up her SMS/MMS to the phone's storage. 
• Mounted the old phone on a computer.
• Used rsync to mirror the phone to an external USB drive. 
• Unplugged the old phone and powered it off.

At this point, I thought I was ready to power on the new phone and begin the restore process. I hooked it to the computer so it would get power and then proceeded to answer the initial questions and connect it to our wireless LAN. Within a few minutes, I could see gmail showing up. I realized that many of the directories probably had content which didn't need to be transferred so I focused on her pictures, downloads and the messagesync directory.

At this point, I loaded the Message Sync application from the the Play Store and then did a "synchronize" operation. My wife had about 7500 SMS/MMS messages and the synchronization took a bit of time. I wrote this paragraph and previewed it in the time it took to synchronize. So not too bad from a time perspective. But it failed to restore anything. :(

We have used Backup to Gamil for automated SMS/MMS backups and it can restore SMS but not MMS. Unfortunately, it wanted to restore all 75000 SMS she has sent and received over the years. This is a fail. No MMS and way too many messages.

Third attempt was SMS Backup and Restore. Seemed simple enough but no hint of MMS support. This doesn't appear to be a very fast application but maybe it will get the SMS from one point to another.

I have given the new phone to Michele. I will wipe the old one once she thinks all is OK.

New vim Trick (to me)

In a recent class, someone mentioned that the reason they liked the editors in IDEs was that they could collapse comments. Collapsing comments was something I never ever even thought about doing. When I look at source code, I look at it in its entirety. I am old school in that I believe programming can be an art form and that source code should be just functional and nice to look at.

Today, I was cleaning up my many browser tabs and noticed that I had Google'd "vim collapsing comments" which quickly got me to an explanation. This was very simple to turn on and I added a total of three lines to my .vimrc file. The first line turns on folding and instructions vim to use syntax as the guide.

:set foldmethod=syntax

In a few minutes of playing around with C and C++ files, it was clear that is collapses at least comment blocks and the code within {...} sections. There are commands to open (e.g. zo) and close (e.g. zc) folded sections. I quickly recognized that if I were going to use folded sections, I would have to bind these to function keys to make them easier to use.

:map <f9> zo
:map <F10> zc

When I open a file, all foldable sections are hidden. Pressing F9 will expand them and pressing F10 anywhere in the section will collapse it again.
Given that I have been using vi since 1986 without this enabled, it will be interesting to see if I like it or not in the long run. Time will tell.

Someone may wonder about what F1 to F8 do. If anyone expresses interest, I will post that.

New TCP/IP Stack Progress Report

It is no secret that OAR has been working furiously on an update of the very old FreeBSD TCP/IP stack in RTEMS. This effort builds upon a port of the USB stack portion of FreeBSD 8.2 implemented by Embedded Brains. Kevin Kirspel did some initial work on incorporating the TCP/IP FreeBSD files and bringing over RTEMS support code from the old port. This blog entry attempts to capture the current state of the project and highlight the challenges still remaining.

The rtems-libbsd code base is currently being debugged using the Intel EtherExpress Pro (e.g. fxp) NIC on the pc386 BSP on qemu. The highlights of the status are as follows:

• Kernel with TCP/IP enabled and FXP NIC completes initialization successfully
• Initialization of the loopback interface IP address and route is currently failing. This appears to be due to RTEMS having a limited subset of proc and ucred (e.g. process and user credential) structures and these not being completely correctly initialized yet.
• User space has 51 of 56 methods in old libnetworking/libc directory compiling cleanly
• Popular PCI NICs compile but no testing
• Target specific code
• Internet packet checksum (e.g. in_cksum) support for architectures supported by both RTEMS and FreeBSD is in place. For targets only supported by RTEMS, the intent is to use one of the implementations that is in 100% C with no assembly support.
• cpufunc.h file in place for all architectures even though on many it is an empty file
• FreeBSD PCI Bus and Legacy Bus drivers are x86 specific. RTEMS will need the equivalent drivers for all architectures. The hope is that this driver can be used across all targets with minor modification. If not, the backup plan is a minimal cross-target RTEMS specific version. Initial indications are that the x86 specific version is target independent.
• BSPs need new sections added to their linkcmds.
• No NICs for ISA or System on Chips.

We feel that we are close to having the fxp driver working on pc386 and qemu. But there is a lot of work left to ensure that the stack is ready to become the preferred TCP/IP stack for RTEMS. The community is encouraged to pitch in and take on some of the following:

• Test FreeBSD PCI NIC drivers currently in tree
• Update NIC drivers in RTEMS not found in FreeBSD
• RTEMS Project may be forced to deprecate some NIC drivers if not updated
• Port other FreeBSD NIC drivers of interest
• no ISA NIC drivers
• Help get rest of libc methods to compile
• ensure set of user APIs is complete
• Select best in_cksum implementation for all architectures
• Bus support methods for all architectures.
• currently assume simple memory access is OK and it is untested
• Help in testing user space code
• includes network demos, servers, clients, etc.
• Need to write API verification tests for each network method
• tests similar to those in psxhdrs which ensure that you can invoke the method using only the header files in the man page.
• Optimizations in code space, resources, and execution time
• effort has focused strictly on getting it to work

In general, OAR has focused on the "depth" part of the project. There has been an enormous amount of effort expended so far. Although some of the work has been sponsored, much of it has been done as OAR overhead and volunteer effort, including Kevin Polulak who is a Google Summer of Code student. Once OAR can successfully ping an RTEMS target with the new stack, it will definitely be time for the RTEMS Community to rally and help speed the transition to the new TCP/IP stack.

Technical Debt and RTEMS

Dr. Dobb's recently had an interview with Ward Cunningham who developed the first wiki among other notable contributions. The interview is interesting and I recommend reading it. But I wanted to pass along some thought on one term that resonated with me.

Technical Debt: Cunningham uses this term to refer to work that needs to be done before a desired change can be implemented or work required to propagate a desired change across a codebase. In the RTEMS world, we have multiple examples of this. 

The most common case of RTEMS technical debt is when a single change must be implemented across all or a set of BSPs at the same time. Recently, Jennifer and I converted the MIPS port from Simple Vectored (SV) to Programmable Interrupt Controller (PIC) interrupt model. We did this because the MIPS/Malta we were developing a BSP for had a more complicated interrupt structure than previous MIPS boards. It was logical to use the PIC rather than the SV model on this BSP. But to ensure that all MIPS BSPs were consistent, we had to implement the same change for six other BSPs.

There are cases with RTEMS where some preparatory work must be done before something else is implemented. A prominent case of this was the CPU Scheduler Plugin Framework work by Gedare Bloom. Before the plugin framework existed, the RTEMS CPU Scheduler was bits and pieces of code embedded in the places where threads changed states and priorities. The plugin framework captured those decision points and made it possible to change the scheduling algorithm at application time configuration (e.g. confdefs.h in RTEMS terms). This was refactoring and clean up needed to be able to implement SMP support for RTEMS.

There are also cases where both types of debt must be paid within a single area. The RTEMS file system infrastructure has evolved over the past few years. Sometimes, there are desirable changes which must be propagated across all file system implementations and for other changes which have required cleaning up an area before refactoring or reworking it.

Not paying your technical debt can lead to long-term pain on a project. The most prominent case of this with RTEMS is when the PowerPC was converted some SV to PIC interrupt model. In contrast to what Jennifer and I did for the MIPS, this time only some BSPs were converted to the PIC model. This meant there were two different BSP interrupt models that co-existed for years. Worse, they were not named at the time and were known as "old" and "new" for years. And if that wasn't bad enough, the method and type names were not the same as the implementation for the x86. It has taken years of paying technical debt to clean up this mess.

It is also something that RTEMS developers would like to avoid repeating. So when you submit a patch and you are asked to modify some code you don't care about to make it consistent or told to be consistent with another BSP, remember that we are just asking you to avoid incurring technical debt. After all, it will probably be someone else who has to pay it. Better to avoid it altogether.

RTEMS Build System Ruminations

This post is a collection of ruminations after a recent post on the RTEMS Users mailing list. The post asked about a few  issues the user was having (italics for quotes):

• my changes not "taking"
• is there any way to limit what bootstrap operates on?  Since it takes quite a while to complete and most of the BSPs are of no interest to me, I would like to avoid bootstrapping them.

Gedare Bloom and Ralf Corsepius replied to the post and I thought it would be a good idea to take the time to write up some issues, workarounds, and benchmarks for various operations using the current build system on a git checkout with no modifications. Any build times quoted in this article were performed on a quad-core computer in the RTEMS Build Farm or my personal laptop with the following specifications:

• RTEMS Build Farm Computer
• Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
• 4 GB RAM
• Seagate 320 GB 7200RPM HDD (ST3320613AS)
• Western Digital Caviar 1 TB 7200RPM HDD (WD1001FALS)
• My Laptop
• Intel(R) Core(TM)2 Duo CPU T7500  @ 2.20GHz
• 4 GB RAM
• Hitachi 160GB 7200RPM HDD (HTS722016K9A300)

The CPUs and disks in these computers are reasonably comparable in performance except that the build farm machine is quad-core. I would expect that the quad-core machine is somewhat faster in single core straight line computation than the clock speed indicates.But I would not expect a 2-3x speed difference in single core performance.

I will be the first to admit that neither of the above computers is the fastest one available today. The fact that one could spend money and get faster computers is important. These are reasonable computers and not obsolete. In fact, when I look at potential laptop upgrades, I am still surprised that my old laptop's CPU is rated much faster than those found in many available today. You have to move to a higher end laptop to beat that. When teaching RTEMS classes, I see attendees with computers that are both faster and much slower than mine. And performance is likely to be much worse in Cygwin or virtual machines than either of the two computers above.

The first issue was my changes not "taking". I personally always configure RTEMS with the --enable-maintainer-mode option which is documented as follows:

--enable-maintainer-mode  enable make rules and dependencies not useful
                          (and sometimes confusing) to the casual installer

This tends to ensure that any changes to configure.ac and Makefile.am files are taken into account in a build tree and the appropriate files regenerated. There are limits to this in that if you changed a compiler flag then it will not result in everything being recompiled. However, if you change the way in which a configure option is interpreted and that is propagated into cpuopts.h or bspopts.h, it should result in impacted files being recompiled.

But what if a .h file changes? Based upon my experiment adding a one line comment to confdefs.h, every test "init file" was recompiled and every test executable was relinked. This is as expected.

Now let's consider changing a C file in cpukit. As an experiment, I added a one line commit to cpukit/sapi/src/exinit.c. This file contains rtems_initialize_data_structures() and thus every RTEMS application is dependent on this object file. The library librtemscpu.a was properly updated but no test was relinked. This untracked dependency is one possible explanation for my changes not "taking".

What is a C file in the BSP changes? As another experiment, I added a one line comment to c/src/lib/libbsp/shared/bootcard.c. Just as in the cpukit experiment, this file is required by every RTEMS application. The file librtemsbsp.a was properly updated but no test was relinked. This untracked dependency is another possible explanation for my changes not "taking".

Gedare makes a point of stating that long-time RTEMS developers know these deficiencies and work around them. Personally, I often find something like this in my command history:

rm -f `find . -name "*.exe"` ; make >b.log 2>&1 ; echo $?

That command ensures that tests are relinked. It covers up the fact that the library dependency is not properly tracked.

The first part of the second issue was is there any way to limit what bootstrap operates on? The solution to this is to only run bootstrap from the lowest level directory that has a configure.ac that you modified or added. Often this is just a single BSP or when initially adding a BSP, the directory c/src/lib/libbsp/ just above your new BSP. Gedare Bloom answered this quite thoroughly and I am just going to quote his answer:

Except when you add new files / modify configure.ac/Makefile.am files you need to re-run bootstrap at the closest level to your modified Makefile.am/configure.ac that contains a configure.ac file. 

So for example if you add a .c file into say cpukit/score/src then the file needs to be added to cpukit/score/Makefile.am and then you need to re-run bootstrap from cpukit because that is the closest parent directory with a configure.ac in it. For BSPs usually you just have to deal with the libbsp/CPU/BSP directory.

In order to run bootstrap from there I use a shell variable that points to my RTEMS root directory ($r) so that I can just ... "cd cpukit ; $r/bootstrap"

The second part of the second issue -- Since it takes quite a while to complete and most of the BSPs are of no interest to me, I would like to avoid bootstrapping them. -- is more complicated. When you initially clone the RTEMS git repository, you have to bootstrap the entire tree. A full bootstrap takes a long time and appears to be very single threaded. On the build farm machine described above, this takes 5m18.331s of user time and 0m48.900s of system time for a total of about 6 minutes to complete.On my laptop, this took 7m56.394s of user time and 1m6.840s of system time for a total of about 9 minutes.  Having a quad-core CPU does not help. The bootstrap process has not significantly improved in time in years. I recall various computers used over the years for RTEMS developing taking from 5 to 12 minutes to execute a complete bootstrap. And this time is much longer on Cygwin due to the inefficiency of the way it must implement POSIX process forking on MS-Windows.

Another thing to note is that bootstrap -p ONLY has to be run when you have modified a Makefile.am and changed the set of header files it installs. This generates the preinstall.am files. It does not need to be run after cloning the RTEMS repository because the preinstall.am files are checked into git. Many people run it more than it needs to be run. 

The need to bootstrap and git branches do not get along as well as one would hope.  As Ralf Corsepius explains in the post in the thread:

One final advise: Do not switch git-branches in git checkouts. As git does not preserve timestamps, while make and the autotools are relying on timesstamps, this will break time-stamps on generated files and eventually result in havoc - You (need) a toplevel bootstrap with each "git branch checkout".

To avoid this, my advise is to use multiple checkouts instead.

I note that even with --enable-maintainer-mode enabled, my experience is that you do often get stuck bootstrapping from the top of the tree when switching branches. The builds will end with a cryptic message in the output. This is a serious hindrance to using git. The typical git usage pattern does not include having multiple clones for different purposes. This is what branches are designed for.

How long does it take to build RTEMS? The answer to this question depends on a lot of factors including the obvious like the computer you are using and the not so obvious such as how you configured RTEMS and did you use the -j option to make to enable parallel jobs.  If you configure RTEMS to include all of the tests, then the build time is significantly longer since there are 399 total tests to compile and link. If you enable only sample tests, then this number drops to 13. The execution of the configure command itself is not a huge factor in build times taking only about 4 seconds on my laptop. It is the actual make that takes so long. The make actually results in a lot of configuration being performed. On my laptop, I got the following times for configure and make when POSIX, TCP/IP, and all tests enabled for sparc/sis (forgive the bad line wrapping):

$ time ../rtems/configure --target=sparc-rtems4.11 --prefix=/home/joel/rtems-4.10-work/bsp-install/ --disable-multiprocessing --enable-cxx --disable-rdbg --enable-maintainer-mode --enable-tests --enable-networking --enable-posix --disable-deprecated --disable-ada --enable-expada --enable-rtemsbsp=sis >c.log 2>&1
real 0m12.511s
user 0m1.970s
sys 0m2.138s
$ time make -j3 >b.log 2>&1
real 10m1.806s
user 8m9.319s
sys 2m8.838s

Building all tests on the quad-core computer at -j7 resulted in a build time of approximately 5 minutes. Given the large number of tests, this indicates that there is opportunity to take advantage of multiple cores during a full build of RTEMS.

Building only the sample tests (e.g. --enable-tests=samples) on my laptop, resulted in a build time of 3m4.420s real time with system and user coming close to adding up to real time. That means 2/3 of the build time is compiling and linking the tests.

How much of the make time is actually configuration? After posting this, I was asked privately how much of the make is spent in configuring versus compiling. To answer this question, I found the first file in RTEMS compiled for the target (e.g. cpukit/score/cpu/sparc/cpu.c in this case) and introduced a compilation error. Then I manually fixed the compilation error and invoked make again. The second make invocation is likely verifying that the configuration didn't change so configuration overhead didn't go to zero but it is close enough.

$ time make -j3 >b1.log 2>&1
real 1m38.264s
user 1m8.027s
sys 0m13.357s
$ time make -j3 >b2.log 2>&1
real 1m29.903s
user 1m56.809s
sys 0m24.112s

I repeated this experiment on the quad-core build farm machine and got the following results:

$ time make -j7 >b1.log 2>&1
real 0m36.652s
user 0m10.918s
sys 0m10.383s
$ time make -j7 >b2.log 2>&1
real 0m50.618s
user 1m37.673s
sys 0m27.296s

Looking at the above, it is pretty clear the the configuration part of make is a significant portion of the entire build time. On my laptop it was slightly over half, while on the quad-core computer, it was about 40%. It  also appears the the configuration stage is unable to take advantage of multiple cores as user plus system time are less than the real time in both cases. On the quad-core, it took nearly three times more real time than CPU time which likely indicates that it is I/O bound. 

In contrast, the build portion of the make command's actions are clearly parallelizable. On the dual core laptop, the approximately 90 seconds spent in the second step used 120 seconds of CPU time. This indicates that there both cores were utilized for about 2/3 of the build time. On the quad-core machine, we see about 51 seconds of real time consuming 135 seconds of CPU time for about 2/3 utilization again. The build time was reduced about 45% by moving from from the dual core to the quad-core computer.

I am personally a proponent of continuous integration and testing. It would be a boon to the RTEMS Project if there were a buildbot and system to get build and test execution feedback on every commit/ Even better would be able to get this feedback before the patch is officially committed. Considering that building all source for one BSP with all tests takes 5 minutes on a reasonable quad-core computer and NO TESTS WERE RUN, one can see the challenge.There are approximately 145 BSPs in the tree currently when one considers variants. On this computer, it would take over 12 hours to build all BSPs and tests. This assumes a fresh checkout and a single bootstrap. If you did that for each BSP, the build would take over 24 hours without executing any tests. Add in a test build of each target in multilib configuration and documentation, and that time goes up even further. That is in a SINGLE CONFIGURATION -- this does not include verifying that BSPs build with and without networking or POSIX enabled. 

This is completely unacceptable for a continuous integration and test effort. According to via.ca, we have had an average of 2.34 hours between commits since moving to git. No single solution will allow us to have a fast enough turn around on build and testing. In order to achieve a turn around under 2.34 hours, we will have to address the speed of the bootstrap, speed of the build process, distribution of building and testing, be smart about only building and testing areas impacted, and ultimately throw more hardware at the problem.

As final food for thought, this is just for RTEMS itself. This does not account for the testing that should be done on the GNU tools we rely upon (e.g. binutils, gcc, gdb, and newlib). A full build and test cycle for all targets  can take up to 4 days on the same quad-core computer. This time can vary based upon the languages being built and tested but GCC simply has a lot of tests. 

Using sed to Remove CVS Ids from RTEMS

Over the Christmas break, the RTEMS Project converted from CVS to git. We have all made mistakes as we transitioned to git and will admit to still be learning. But our workflow is improving and we are making fewer mistakes. However there are still a number of outstanding tasks left over from the transition:

• Remove CVS $Id$ Strings
• Move from ChangeLog files and define new "history" file which lists major changes
• Convert release procedure to git

Since it has been well over four months since we initiated the conversion to git, I decided it was time to push at removing the Ids. Being comfortable with bash and sed, I decided to see how many of these I could remove without editing a file by hand.  This turned out to be harder than I expected.

First, there are a lot of types of files in RTEMS and comment structure varies accordingly. There are C-style comment blocks, C++ one-line comments, "# to end of line", "; to end of line", etc.
Second, even within a single language, there were differing CVS Id string formats.  For example, I expected most Id strings in C code to be at the end of a comment block like this:

 *
 *  $Id$
 */

But sometimes a file would have something like this:

 *  end of a comment paragraph
 *
 *  $Id$
 *
 */

In the above case, removing the $Id$ line and the preceding line would leave an undesired blank line at the end of the comment block.  I implemented this as a set of sed transformations. They were applied in an order which would match the longest sequence I had identified followed by others which matched shorter sequences. On top of that, there were limits on how many transformations could occur inside a single invocation of sed. It was easier conceptually to take the output of a single set of sed transformations and then apply another set of transformations.  The following sed command file dealt with removing the $Id$ and the preceding comment line:

# Remove CVS Ids which are more or less like this (embedded C, preceding)
# ^ *
# ^ *  $Id
/^ \* *$/{
 N
 #
 /^.*\n \* *\$Id.*/d
 /^.*\n \* *@(#) \$Id.*/d
}

Note that sometimes the code had " * @(#) $Id$" and this single sed command file would properly remove both patterns.

The solution turned out to be a set of sed command files and a shell script. The shell script invoked sed multiple times in a pipeline. Each stage in the pipeline performed a different transformation and that was fed into another invocation of sed with a different command file.

In the final step, if the first line of the file had ended up as a blank line because of the transformations, I removed it with this sed command file:

# Remove the first line when it is blank
1{
 /^$/d
}

In the end, I ended up with 28 sed transformations in an 8 stage pipeline. The scripted edits turned out to be a 2.7 megabyte diff which modified 6376 files. I was left with 117 files to edit by hand and only one file mishandled by the script. The changes can be viewed at http://git.rtems.org:

• Scripted changes 
• Fixes after scripted changes
• Manual changes

I hope it is understood that I compiled a lot during this effort. I wanted to make sure that every file I had touched was compiled. In doing this, I learned that a number of odd configurations in RTEMS had not been built recently and were, in fact, broken before I started.

If anyone is interested, I can make the shell script and set of sed scripts available but I must disclaim that they were not written to be classroom sed examples. They are understandable but probably not optimal nor perfect. They were written for a one-time massive edit and will only be reused to remove the CVS Id strings from other modules owned by RTEMS.

This could certainly have been done with other tools including awk and Python. But I knew sed and bash and using what you know is often the most effective way to do the job.

RTEMS GSOC Applications for 2012

Oily Betts of Xapian created a spreadsheet which graphs an organization's GSOC applications over the student application period. While I was fiddling with his spreadsheet, an email popped up from Gedare Bloom who had already plotted the information for RTEMS.

RTEMS had a total of 17 applications which I think is a few more applications this year over the past couple of years. The quality is generally quite good with only a couple being obvious to mark as ignore. These are those that are clearly spam (e.g. cut and paste or not related to RTEMS)..

User Interface Generational Gap

Even though my children are old enough where they are hardly children anymore, we still enjoy going to Chuck E. Cheese. My youngest son is now 17 and we have been visiting "the mouse" since he was barely able to walk. We have always enjoyed the pizza and salad bar but not surprisingly the things we play are different now. They are no longer in whacking a mole or trying to catch ping pong balls in a net. We now play the first person shooters, skee ball, a fire fighting game, or an old-school Pirates of the Carribean pinball machine. I am surprised that my youngest son not only likes to play this but is good enough to get 50,000,000 points and collect 125 tickets.

But this post is not about his proficiency at this game or the two high school couples on their way to the prom in their finest attire dining at Chuck E, it is about me wanting to play Pirates of the Carribean and having to wait on a boy who was probably about 10 years old.

While watching him play, it struck me that he was doing absolutely terrible on this machine. I watched wondering if he just hadn't gotten the timing of the flippers. Then I had a sudden realization. He was tapping on the glass on top of the flippers thinking it would make them move. He thought it was a touch screen and had no idea that there were buttons on the side of the machine you pressed.

After he lost the final ball, I offered to let him watch me play a while so he could see the basics of how it worked. I am not very good but I do know you have to press the buttons to get the flippers to move.

We talk about technology generation gaps and user interface problems but this was a case I had never heard mentioned. Think about this for a moment -- there was a ten year old boy who naturally assumed this had a touch screen. That's how much things have changed. It is like the urban legend about the woman who assumed a mouse was like the foot pedal on her sewing machine and she had to press it to make the computer go. It is a generation gap.

MINIX versus Linux versus BSD

This morning an article was posted to Slashdot in which Andrew Tanenbaum is interviewed.  One question and answer from the interview seemed to draw the most reaction on Slashdot.    LinuxFr.org asked: "If you could return in the past to change the MINIX original proprietary licence to the GPL licence, do you think your system might have become the dominant free OS today?".  Andrew Tanenbaum answered:

Never. The reason MINIX 3 didn't dominate the world has to do with one mistake I made about 1992. At that time I thought BSD was going to take over the world. It was a mature and stable system. I didn't see any point in competing with it, so I focused MINIX on education. Four of the BSD guys had just formed a company to sell BSD commercially. They even had a nice phone number: 1-800-ITS-UNIX. That phone number did them and me in. AT&T sued them over the phone number and the lawsuit took 3 years to settle. That was precisely the period Linux was launched and BSD was frozen due to the lawsuit. By the time it was settled, Linux had taken off. My mistake was not to realize the lawsuit would take so long and cripple BSD. If AT&T had not brought suit (or better yet, bought BSDI), Linux would never have become popular at all and BSD would dominate the world. 

Now as we are starting to go commercial, we are realizing the value of the BSD license. Many companies refuse to make major investments in modifying Linux to suit their needs if they have to give the code to their competitors. We think that the BSD license alone will be a great help to us, as well as the small size, reliability, and modularity.

My first UNIX experience was in the 1984-85 timeframe and my first job out of college was developing software for intelligent I/O controllers for a UNIX System V mini-computer.  I remember what commercial UNIX was like in those days. You may have wanted it but the options were limited and expensive.  On the 80286, you had XENIX.  When the i386 came out, there were a number of nice options including Interactive 386/ix and even SCO UNIX.  Yes SCO had a decent product back in the day.

When I could finally afford a computer capable of running some UNIX-ish system, I found myself in on the consumer side of what the question is about.

From my perspective, I didn't use MINIX because I viewed it as an educational and teaching OS. Its desired user base was not "real" users doing non-academic work.  We had experimented with it in our labs at work and found it quite primitive in comparison to the "real UNIX" we were used to.  Personally, I found the acquisition process painful as well.  But all UNIXy systems were painful to get back then.  The focus on educational users was it for me.  I don't ever want to be the "odd user" of anything who is not in the desired target audience for a product.

Why did I choose a Linux distribution over a BSD?  I honestly don't remember.  My vaguest recollection is that I preferred System V more than BSD systems and Linux leaned to System V.  I doubt this was a factor for most others though.  If I had to guess, I would go back and look at how you had to obtain it, community responses to newbies, etc.. Was the AT&T lawsuit have factor? Maybe. Linux was certainly perceived to be immune from that lawsuit among those I knew.  It did not suffer from that heritage.

Finally, I want to look back with the free software community building experience I have.  When viewed from this perspective and the prism of time, I think the answer has a lot to do with what we should have learned from Google Summer of Code. A project has to be easy to obtain, get started with, contribute to, have a vibrant and friendly community, etc.. The license is important but as long as it is imposes legal impediments or obligations, that won't stop most people.  In the old days, Minix was not really easy to obtain and was not focused on general use. It was not available as an impulse download. That was enough of a hurdle to stop a lot of folks.  

When one examines the choices faced by someone who wanted a UNIX-like system on their personal computer in the early 1990's, it is easy to see how Linux was the default choice.  It simply did not have the "targeted to teaching operating systems" stigma, was easy to obtain, and didn't have a lawsuit looming over its head.

But one of the nice things about free software is that if there is interest, a project will continue on.  MINIX 3 is a great OS that has a BSD-style license, is easy to obtain, and they are clearly interested in MINIX 3 being used for more than teaching operating systems design.  Variety is the spice of life.  I would recommend that you give it a try and tell them I sent you.

Open Source and Generational Differences

It is time again for another entry from guest blogger Chris Johns. Chris and I have chatted and emailed a lot over the past few months about the issues in this post. They are tough because it is always hard to question your decisions and embrace change. But it is critical to do so on anything that is long-term in your life. RTEMS is a long-term software projects and we need to embrace self-examination and change.

Developers start projects to scratch itches or to bring about change. They join projects as users because they need to use a piece of software. They get involved because they to need to fix bugs or develop new features. The reasons are many, well documented and understood by those who work in or around open source software. What happens when a project becomes old enough that generational change is needed and those who start a project reach an age where they do not have the energy, mental capacity or desire is not well understood. As a project and its leadership age do they move from being intensive productive developers to mentors and governors of the project. Understanding this change is difficult as the interests and focuses of the newer generations are different and sometimes clash with the original developers yet both are right and neither are wrong. The primary function of the project maybe the same, the way it is developed and maintained can be different. Open source is starting to reach this point and some projects have such a long life cycles in user projects it is starting to become an issue. RTEMS is such a project. It is used in space flight and some new projects do not take flight until 2018. Being open source each user has the code and can make changes long past the life of the project, but it is the project and community this discussion is about. 

RTEMS is now 22 years old. It is able to drink, vote and hold a drivers license in most countries. It has experimented with a few things it should not have and so far has not been in trouble with the law. You could say it has had a stable and happy up bring. RTEMS is now looking to the future and life without the current custodians.

RTEMS at its core is a collection of C source files that are built into a C library and linked with user application code to provide single executable image often embedded into a custom piece of hardware. The key factors for the user of this device is performance, resources and stability. The key factors for the developer of this device is availability of source code, easy to use software interfaces, easy to integrate into a team environment, and stability of the project. The key factors for the maintainers of RTEMS is the ability to effectively integrate changes, respond to hardware changes, stable infrastructure and the ability to attract new developers. Developers are the food source that feeds, refreshes and sustains a project.

RTEMS in its post toddler years moved to a new version control tool called CVS that allowed concurrent development of the code. It was liberating because a single set of code did not have to be maintained. Before CVS patches were emailed to the maintainer, merged and then released back to developers as tar files. With CVS this task could be spread among a number of trusted developers. RTEMS also moved from custom makefiles to autoconf and automake. This improved the productivity of the developers allowing the code to be configured and built on a range of host operating systems. RTEMS still uses these same tools 10 to 15 years later and they still work. The developers are comfortable with their work flow and know the problems or issues they have. Why the need to change? There are problems and over time these have grown in size as the project has grown. What were problems are now distance memories and all we have left is the new problems that came with the tools. 

We have files in places that have long since lost there meaning. The board support packages is an example. They are located under 'c/src/lib/libbsp' when they could be located in 'libbsp' or even 'bsps'. This path does not effect the build time or the disk space used and the developers know this path very well so why is this a problem. It makes no sense. Any new users of RTEMS, and by new I mean anyone who has joined in the last 10 years, would have no idea why this structure exists. RTEMS use to have an Ada version and all code was under 'c' or 'ada' and the source was under 'c/src'. Why not move the files? We cannot because CVS does not have a rename command and repository hacks are something we discourage. 

Would we move them if CVS allowed it? Maybe, however this effects the build system. Why is that a problem, is there something wrong with it? Building RTEMS is complex. As a user of RTEMS a release comes with all the autotool's generated files in place ready to work. You can configure RTEMS with a few options passed to configure, plus provide a few more on the command line to the build a range of BSP specific options and then at runtime you have a large array of configurations and runtime options. Are these documented? Only a small number are. The user needs to look into the source to find the full set and even for a seasoned developer this can be complex and not accurate or complete. As a user you just build RTEMS and that does happen and it does it well. By well I mean you get a library of code that is stable and will perform the task asked. As a developer you need to work with the build system and this is where problems start to appear. Performance is an issue. A clean check out from CVS requires a bootstrap to generate all the autoconf and automake files as they are not held in the repository and this can take a lengthy period of time even on large hosts and fast disks. Fortunately this is not often needed as the maintainer mode helps how-ever it makes build-bot type support on check in difficult if not impossible. Also contributing to this is the repeated installing of header files. If you build all 120+ BSPs you will install over 50,000+ header files. This is just building RTEMS and does not include installation of the build output. When installing the 50,000+ files are copied to the install paths. Does this seem normal or ok? Maybe there really needs to be this many headers, or maybe header files have been added to RTEMS following a common template with little regard to the consequence and over the years this has grown to this figure. Most users are only interested in one or two BSPs so this is not a major issue. For a maintainer is it a problem because they need to make sure everything builds and works. 

 I suppose the important questions regarding the build system are "Is it efficient given the new generation of build tools?" and "Does it aid or inhibit the development process?". These are debatable questions which span the boundaries of technical merits, broad range support, supported hosts, and personal preferences. This last one being the most contentious.

The question the current developers and maintainers of RTEMS need to ask is not "Are these tools working and doing the job they are suppose to?", rather if we handed the project to a new group of developers and maintainers "What would new maintainers think of the state of the project?". While we may be comfortable and able to release and maintain RTEMS it may look to a new generation as something from a time past. 

Change is never easy. There needs to be leadership, desire and willingness to refresh to bring about change. It is easy to be negative and to find fault in any new change, then offer no path forward. Leading is not always about "What I think is right", it is about being honest and openly critical of how we work and approach problem solving, and it is about providing paths to new ways of solving problems we face in the project. Not all paths will succeed how-ever being open to change means a new path can be taken until a solution found. Inviting new and young talent to follow these paths and find solutions involves them in the project. They become responsible for various parts and that builds pride and commitment. The hope being someday they will be managing and leading the project.