Thursday, December 11, 2014

Intel Edison and RTEMS - Road Forward

The Intel Edison is now (11 December 2014) is now running RTEMS but there are issues to resolve and features which need adding. This post is a list of what needs to be done.

In the near term, the current code needs clean up before it can be submitted to the RTEMS Community. The following is a list of some of the activities which need to be performed before the code can be submitted:

  • Break conditionals in build system and C code from "if Edison" to more features oriented conditionals. This requires being able to disable at least VGA, IDE, and legacy COM ports at BSP configure time.
  • Split the current Edison specific polled console code into its own file. 
  • Review dynamic console selection code and see if the current code can be improved so Edison specific conditionals are not needed.
  • Generally split the one large patch into smaller discrete patches and try to not make them Edison specific.
  • Write instructions.
There may be other clean up but that should be enough to get the current Edison support into the tree.

Longer term, there is more to do and it involves coding. Some of this can be done without access to Edison System On Chip documentation. But other parts will require having access to the details.

  • Add support for APIC interrupt controller.
  • Figure out why clock tick isn't working. This may require using different hardware for the clock tick driver.
  • Provide access routines for the memory mapped PCI configuration space.
  • Provide glue so hacked NS16550 console support can use libchip driver like COM[1-4] do. There is support already for PCI cards with multiple NS16550s so this should just be a matter of having the information once PCI configuration space is working.
    • Test this driver polled
    • Test this driver interrupt driven
That will get a complete basic BSP working with no hackery. Beyond this, RTEMS needs support for the other peripherals on the Edison:

  • Add support for the discrete I/O and analog inputs.
  • Evaluate supporting the Flash. We may want to be careful to not destroy the Linux installation or we may want to simply take over the entire Flash as long as we avoid conflicts with other uses. Having the Linux and USB file loading is nice. 
  • Add WiFi support. This is a more involved project as RTEMS currently does not support TCP/IP over USB or WiFi. 
    • Since the new FreeBSD 9.x TCP/IP  stack came with the USB stack, supporting wired Ethernet over USB is a logical first step and is also needed to support Ethernet on the Raspberry Pi.
    • Once wired Ethernet over USB works, then the wireless part can be addressed including applications to manage connecting to access points.
There is probably more to do to fully support the Intel Edison board but it does work now and provides a baseline for future enhancements. At this point, interested users can fund core developers to add capabilities or volunteer to add it. That is how open source works.

Intel Edison and RTEMS - Day Three

Overnight, I realized that the error the RTEMS hello world sample was failing with was RTEMS_UNSATISFIED (13) likely indicated a miscalculation in the RTEMS Workspace by . A quick look showed that the primary difference between the pc586-sse BSP variant and all other pc386 variant was that because of SSE, all tasks were forced to be floating point tasks. This results in the allocation of floating point contexts for all tasks and apparently the calculation is a bit too low on the x86 port. I switched the edison BSP variant to be compiled like the non-SSE pc586 and this solved the problem. A short time later, this output was posted to the RTEMS Users mailing list.

## Starting application at 0x0010000c ...
024rtemsWorkAreaStart=12FD40 MemSize=0x08x/0

work_area_start = 0x12FD40
work_area_size = 1072497344 0x3FED02C0
end = 0x40000000
current stack pointer = 0x12FD18
heap_start = 0x133B72
heap_size = 1072481422
initialize device drivers
Checking on /dev/main
Register /dev/main
Register /dev/main as console
Console initialize complete
post driver hook
rtems start multitasking

Hello World

EXECUTIVE SHUTDOWN! Any key to reboot...

At this point, the next step was simply to remove most of the debug output and get a clean run. Then we tackled seeing if enabling the timer calibration code in the pc386/startup/bspstart.c file would work. The calibration failed and likely indicates some combination of interrupt controller or timer hardware is different from legacy PCs. Fixing this is left for future work.

The failure to have a working clock tick meant that we needed a fall back. RTEMS has a target independent fake clock tick driver which is a special IDLE thread which repeatedly calls rtems_clock_tick() each time it is executed. When a "clock tick" occurs which wakes up another task, the IDLE thread is preempted. This is an effective way to run most RTEMS tests on a simulator with no interrupt sources.

With the IDLE thread clock tick in place, Jeff and I ran the ticker example which has three tasks executing every 5, 10, or 15 seconds. It runs for 35 seconds. Jeff and I tuned a delay in the IDLE thread until time appeared to pass reasonably close to it would on real target hardware if the system were lightly loaded.

After a few iterations, we moved on to the fileio sample test. This demonstrates the shell, IDE file access, and requires user input. On the first run, the test hung and we had to disable the IDE driver because the Edison does not have one. On the second run, it just worked.

That was midday on day three (11 Dec) and that was all that could be done without writing new code. Plus Jeff needed to get the Edison, a BeagleBone Black, and a Raspberry Pi working from a small laptop and then boxed up with cables. OAR has a table and will show them at the Flight Software Workshop in a few days )16-18 December  )

Intel Edison and RTEMS - Day One

It was Monday afternoon when an Intel Edison board showed up in the mail. It is quite small with an expansion board needed to have access to I/O. From an RTEMS perspective, it looks interesting because it has 40 discrete I/O pins and six analog inputs. The challenge was to see
how long it took to get the RTEMS up on this board.

It wasn't long at it was unboxed, that we were looking at a GNU/Linux prompt in a PuTTY window. Poking around, it quickly became apparent that the Edison didn't use Grub so my first hope that we could just add an RTEMS executable as an option on the boot menu was off the table. We had to identify a way to boot an RTEMS executable.

We had already noticed that U-Boot was on the board. Now the question was how to get an executable image into a bootable location and what the magic commands were to boot it. I came across this thread which had a lot of information on the Edison. It quickly became apparent that the Edison did not have much low-level public documentation yet and that support for most (if not all) legacy peripherals was not present. I posted an email to the RTEMS Users mailing list which summarized what we knew so far:
  • SoC documentation won't be out until later this month.
  • Has U-Boot and doesn't appear to have a 16-bit mode at all.
  • Looks like U-Boot will want a binary image to boot.
    • This turned out to be wrong. We figured out how to boot an ELF executable.
  • There is a comment in the OS Dev forum that the PCI configuration space is memory mapped and not IO mapped like a regular PC. This means the PCI initialization and support code needs to be different for Edison from other PCs.
  • All serial ports are on the PCI bus. It looks like the PCI layout is constant so for a start, we are hacking a simple polled driver for one UART and go from there. U-Boot initializes that port so we can skip initialization. Besides the person on the forum couldn't figure out the clock rate for the UART. This should be enough to at least temporarily get around the lack of PCI config space code.
  • There is a watchdog on it that needs to be disabled or managed.
  • Looks like it doesn't have a legacy i8254 timer and may not have legacy interrupt controller.
A user named mutex on the OS Dev forum had poked and prodded at the Edison board and managed to post a few very useful pieces of information.
  • Base address of the UART and the U-Boot command to force a character out:
    • mw.b 0xff010180 0x21 1
  • Address of the watchdog control register along with commands to disable it:
    • mw.l 0xff009000 0x10f8 1
  • And to reset the board
    • mw.l 0xff009000 0x10f8 0xf8
  • Commands to boot a binary image.
    • load emmc 0:9 0x100000 /kernel.img
    • go 0x100000
I contacted this user. It was clear that he had been the most successful in booting something that wasn't the Linux that came with the board. He quickly replied. Thomas Nilsen has his own kernel and was kind enough to send me a binary which we were able to run. He is owed a big thank you.

This was the end of Monday with the board. 

Sunday, February 24, 2013

RTEMS Texinfo Tools Update

A couple of years ago, Chris Johns and I began to discuss that RTEMS has had a long and successful history as a free software project. One aspect of this discussion resulted in the Open Source and Generational Differences. This post reflected on how new developers can have a tendency to avoid projects that use "uncool" tools. They want to be on the leading edge of technology. However, another aspect of using uncool tools is that they are old. RTEMS-based applications often have lifespans measured in decades from development, through fielding, to long term sustainment. This insight lead us to begin to review the tools we depended upon for long-term viability and that they continued to offer a high quality solution. The transition from CVS to git has been the most visible outcome of this effort.

But lurking within the RTEMS source tree was a dependency on a long dead tool - texi2www. RTEMS was the last user of this and had to include the source code in our own tree. This was exactly the type of situation Chris and I had realized could happen and it already had without us noticing. In November 2011, I posted to the GNU Texinfo Help Mailing list asking for advice on converting to the more modern texi2html program. Unfortunately, I learned that texi2html was considered deprecated and being re-implemented in Perl and the new implementation would be known as texi2any. This led to me converting us away from the stone cold dead texi2www to texi2html. With the recent release of texinfo 5.0, I began to ensure that our documentation would build with either texi2html 1,82 or texi2any from texinfo 5.0 and that the build infrastructure could detect which to use.

The goal of this post is to point out how we invoke tools, initialization file differences and the minor changes to our documentation required to support both tools. The other tools in the texinfo package such as makeinfo did require us to make changes to the source but did not change their invocation. I will detail the changes to our texinfo source files after showing the command line differences for the html converters.

The commands executed by our build infrastructure are generated by an autoconf based build infrastructure. This sometimes leads to longer than absolutely necessary command lines. I have made no attempt to shorten or clean these command lines up. These are for the RTEMS C User's Guide whose main texinfo file is c_user.texi.

The following is the invocation of texi2html 1.82:
 texi2html -D use-html --split node --node-files -o ./ --top-file index.html --init-file=../texi2html_init \  
   -I /home/joel/rtems-4.11-work/rtems//doc/user -I /home/joel/rtems-4.11-work/rtems//doc \  
   -I .. -I . --menu /home/joel/rtems-4.11-work/rtems//doc/user/c_user.texi \  

This is the corresponding invocation of texi2any from texinfo 5.0:
 texi2any --html -D use-html --split node -o ./ --init-file=../texi2any_init \  
   -I /home/joel/rtems-4.11-work/rtems//doc/user -I /home/joel/rtems-4.11-work/rtems//doc \  
   -I .. -I . /home/joel/rtems-4.11-work/rtems//doc/user/c_user.texi  

Notice that both command lines are quite similar. However, texi2html requires the --node-files argument to produce individual html file names which are based on the section or chapter name. By default, they will be named using a pattern line DOC_nnn.html.

The other thing to note is that they both accept initialization files. However, the format of the initialization files is very different between the two implementations. Texinfo supports hierarchically structured documents and allows the author to provide links to the next section, previous section, and the section that contains or is logically above the current one. The RTEMS Project has a tool which automatically constructs the node markups based on chapter, section, and subsection headings. Thus, the RTEMS documentation is fully hierarchically linked with no manual node definition required. The RTEMS documentation build system uses the initialization file to define a custom header, footer and to modify the navigation buttons.

This is the file texi2html_init generated by our build infrastructure:
 my $button_text = '<a href="../index.html">Library</a>';  
 push @SECTION_BUTTONS, \$button_text;  
 push @CHAPTER_BUTTONS, \$button_text;  
 push @MISC_BUTTONS, \$button_text;  
 push @TOP_BUTTONS, \$button_text;  
 '<A HREF="" target="Text Frame">  
 <IMG align=right BORDER=0 SRC="../images/rtems_logo.jpg" ALT="RTEMS  
 Logo"> </A>  
 <H1>RTEMS On-Line Library</H1>  
 'Copyright &copy; 1988-2011  
 <A HREF="" target="Text Frame">OAR Corporation</A>  

The initialization files reflects the internal implementation of the two programs and the format used by texi2any is different. We have an initialization file which accomplishes similar things in the generated HTML files but looks different. For example, the texi2html output has navigation icons while the texi2any output has textual links.

This is the file texi2any_init generated by our build infrastructure:
 set_from_init_file ('AFTER_BODY_OPEN',  
 '<A HREF="" target="Text Frame">  
 <IMG align=right BORDER=0 SRC="../images/rtems_logo.jpg" ALT="RTEMS  
 Logo">  </A>  
 <H1>RTEMS On-Line Library</H1>  
 texinfo_register_handler('setup', \&add_button);  
 my $button_text = '<a href="../dir.html">Directory</a>';  
 sub add_button($)  
  my $self = shift;  
  foreach my $button_type ('SECTION_BUTTONS', 'CHAPTER_BUTTONS',   
               'MISC_BUTTONS', 'TOP_BUTTONS') {  
   my $buttons = $self->get_conf($button_type);  
   push @$buttons, \$button_text;  
  return 1;  

There were only a couple of issues encountered with our use of texinfo which required modifying the source.
  • Missing @item in @itemize lists now results in warnings. 
  • The order of menu definition, @top and its @node, and file @include statements in the top level texinfo files had to be reordered. Texinfo 5.0 is not as forgiving on this.
In addition, I spotted mistakes in our documentation when reviewing the various output forms. The entire range of patches can be viewed online here.
NOTE: As of 24 February 2013, these have not been committed. When they are committed, links will be provided to the git repository.

We would like to take advantage of features in the newer tools and are investigating using a print on demand service for RTEMS manuals. I hope there is experience in the texinfo community about this but, if not, I suppose I will pester the maintainers until the results are satisfactory and report on what I had to do.

Thanks to the Texinfo maintainers Patrice Dumas, Karl Berry, and Eli Zaretskii folyzr being incredibly patient and helpful through this process.

Friday, February 15, 2013

GSOC Presentation at University of Tennessee at Chattanooga

Earlier today, I returned to my alma mater, the University of Tennessee at Chattanooga, to give presentations on RTEMS and the Google Summer of Code 2013 (download here). About 25 people were in attendance including two faculty members. Thankfully, my wife Michele had driven and let me do final review on the presentations. Chattanooga is about a two hour drive from Huntsville and in a different time zone (Eastern not Central). We had allowed time for traffic and parking problems but had no traffic. We ending up arriving about forty-five minutes early.   We were met in the parking lot by a student who provided a visitor’s parking pass. This greatly simplified having a car on campus. Parking at any university seems to be a challenge.

After no A/V difficulties, I put up a montage of pictures from some of the projects which use RTEMS.  Those who attended the GSOC 2012 Mentor Summit will remember the slide from the lightning talks. It is memorable because someone from another project presented it. I had forgotten the talks and went to the Google Store. The montage highlights awesome projects based on RTEMS including the BMW Superbike, Curiosity, Herschel, Milkymist, Solar Dynamic Observatory, and MMS. As students came in, there were plenty of questions about the projects.  I created the slide to give at an RTEMS friendly workshop where most knew what RTEMS was and I wanted to highlight users. It turns out this is a great slide to get conversations going. If other FLOSS organizations can brag on where there software is used, then a user montage is a good thing to have.

I presented the official GSOC slides first. I felt it was important to emphasize that all types of FLOSS software was represented and that all of the organizations were interested in student participation. Being effective and appropriate to participate in GSOC requires organizations to provide wish lists, mentors, regular interaction with students, friendly communities, etc..

I then moved to the RTEMS specific presentation which very briefly introduces RTEMS but focuses more on recent activities, ongoing activities, and our wish list. It highlights areas we want improvements to occur even in software development process areas. As the last slide came up, I realized I was finishing on time and had presented thirty minutes, leaving fifteen to twenty minutes for questions. I ended my talk by reminding them that I would love to see them all as RTEMS contributors but would be just as happy to see them involved in the FLOSS community on ANY project. We are a collection of organizations but do have common goals.

There were a lot of questions on GSOC followed by some on RTEMS. One student asked where GSOC work occurred. There were questions on how the mentoring worked and what mechanisms were used to communicate with the mentors. I noticed students were packing up and realized they had ten minutes to get to their next class. There were no more questions but I hung around a while.

The big surprise was when a student came up to me while I was packing up. He asked about real-time and SMP as a potential area for Ph.D. work. I told him that I thought it was an open area of research and with some literature research he should be able to find a good area. Years of research into uniprocessor real-time systems and scheduling have given us practical engineering solutions. But the complexities of modern pipelines, caching, and interactions of multiple cores break some of the underlying assumptions. I am concerned that this same level of maturity has not been reached in SMP embedded systems which require rigorous analysis of predictability.

My wife generously waited in the presentation room while I visited with the only faculty member left from when I was a student there Dr. Jack Thompson. Then my wife and I walked around campus, enjoying a pretty day and reminiscing. After all, it was only one day after Valentines Day and we met one another while students here. 

Sunday, December 23, 2012

Transfer Information to Another HTC One S

I am still using my trusty T-Mobile G2 while waiting for the new Nexus 4 to arrive. It has only been on order since Thanksgiving so maybe sometime in January it will arrive. In the meantime, my wife has an HTC One S which has a small crack in the corner of the screen. This isn't a real problem but the USB connector has become quite flaky and it often doesn't get charged. I ordered her a replacement and this is the hopefully short story of how I transferred her data and information to the new phone.

First, I did some research. It quickly become apparent that between Google's sync and backup plus what I could copy from the filesystem, all that was left would be SMS/MMS. Message Sync looked promising so I loaded it. I then performed the following:

  • Placed the phone in Airplane mode to prevent new data from showing up.
  • Used Message Sync to back up her SMS/MMS to the phone's storage. 
  • Mounted the old phone on a computer.
  • Used rsync to mirror the phone to an external USB drive. 
  • Unplugged the old phone and powered it off.
At this point, I thought I was ready to power on the new phone and begin the restore process. I hooked it to the computer so it would get power and then proceeded to answer the initial questions and connect it to our wireless LAN. Within a few minutes, I could see gmail showing up. I realized that many of the directories probably had content which didn't need to be transferred so I focused on her pictures, downloads and the messagesync directory.

At this point, I loaded the Message Sync application from the the Play Store and then did a "synchronize" operation. My wife had about 7500 SMS/MMS messages and the synchronization took a bit of time. I wrote this paragraph and previewed it in the time it took to synchronize. So not too bad from a time perspective. But it failed to restore anything. :(

We have used Backup to Gamil for automated SMS/MMS backups and it can restore SMS but not MMS. Unfortunately, it wanted to restore all 75000 SMS she has sent and received over the years. This is a fail. No MMS and way too many messages.

Third attempt was SMS Backup and Restore. Seemed simple enough but no hint of MMS support. This doesn't appear to be a very fast application but maybe it will get the SMS from one point to another.

I have given the new phone to Michele. I will wipe the old one once she thinks all is OK.

Sunday, September 9, 2012

New vim Trick (to me)

In a recent class, someone mentioned that the reason they liked the editors in IDEs was that they could collapse comments. Collapsing comments was something I never ever even thought about doing. When I look at source code, I look at it in its entirety. I am old school in that I believe programming can be an art form and that source code should be just functional and nice to look at.

Today, I was cleaning up my many browser tabs and noticed that I had Google'd "vim collapsing comments" which quickly got me to an explanation. This was very simple to turn on and I added a total of three lines to my .vimrc file. The first line turns on folding and instructions vim to use syntax as the guide.
:set foldmethod=syntax
In a few minutes of playing around with C and C++ files, it was clear that is collapses at least comment blocks and the code within {...} sections. There are commands to open (e.g. zo) and close (e.g. zc) folded sections. I quickly recognized that if I were going to use folded sections, I would have to bind these to function keys to make them easier to use.
:map <f9> zo
:map <F10> zc
When I open a file, all foldable sections are hidden. Pressing F9 will expand them and pressing F10 anywhere in the section will collapse it again.
Given that I have been using vi since 1986 without this enabled, it will be interesting to see if I like it or not in the long run. Time will tell.

Someone may wonder about what F1 to F8 do. If anyone expresses interest, I will post that.