Friday, September 16, 2011

Elvis Costello and RTEMS.info History

Over the years, I have told many RTEMS users that I provide hosting and system administration for an Elvis Costello fan site (PHPBB Forum and Wiki).  I have had the fan site for about 9 years now.  What many of you probably don't realize is that I have also hosted the RTEMS.info Mirror Site since 2006.

This is a personal effort and I receive no subsidy from OAR for doing this.  In order to have a static IP address and host services, I have to have a business class account which is more than a residential class account.  All of the sites I host plus the family Internet activities share a 7Mbps/1Mbps connection.

I had been helping on the technical side of administrating the Elvis Costello Fan Forum for a while when the hosting service got hacked and our site trashed.  We were unable to get anyone to contact us via email or phone for over a week.  I realized that I had a computer running GNU/Linux Fedora that was largely unused since I had upgraded.  Even though it was only a 350 Mhz Pentium II with 384MB RAM, it was perfectly suitable to host a small (~100K hits a day) website.  I made a phone call to get a static IP address, moved the domains and within about a week we were back up.

A couple of years after that, the person who ran the Elvis Costello Wiki asked if I could host it.  I had already planned to upgrade to a 2.4 Ghz Pentium 4 with 2GB RAM.  We decided to wait to move the Wiki until after the server upgrade. The hardware upgrade went easily and we moved the Wiki.  What surprised us both was that the performance on the site went to hell.  The server logs showed nothing, load looked low and no amount of tuning or probing helped. I begged my ISP for help and got an unlocked, uncapped cable modem to test with.  After more research and fighting, I learned that my router could not hand the number of simultaneous connections and was dropping them randomly.  I upgraded routers and the performance issues were settled.

The site has always used external hard disks for backup.  There is a script which runs every night and dumps all user directories, databases, etc to a special directory on an internal disk.  Then another which runs later and "rsync's" the internal disk copy with an external one.  Backups are placed in dated directories and a few a month are saved.

When I set up the RTEMS.info Mirror, I was more concerned with disk space than bandwidth consumption.  I don't have the fastest connection and I am sure users would appreciate a faster uplink. But I foot the bill and until there is funding, this is what there is.  The RTEMS.info site mirrors at least 4 times a day.  Ralf Corsepius has set up automated checks which let us know when a mirror site is down or out of sync.

In late 2010, I became worried that the 2.4 Ghz Pentium 4 was getting very old.  It was not new when it became the server and I was sure it had seen at least 6 years as server.  The Elvis Costello Fan community rallied around my request for a new server and within a month or so, the fund raising goal was met.  The new server is from AS Labs who specialize in building custom GNU/Linux systems.  It has a quad-core 3.0 Ghz CPU and 8 GB RAM.  It runs very cool and is far from overloaded.

Over the years we have had period power outages with the worst being the tornadoes of April 2011. But overall, I believe our uptime is very good.  I don't track it but thanks to the Elvis Costello fan community, it can't be down over 20 minutes without me getting an email. Thanks folks!

My wife and I have learned a lot about system administration over the years of maintaining these sites.  She personally reviews and approves every account request for the PHPBB Fan Forum.  The number of spam account requests is boggling and periodically she begs me to try to find another way to stem an increase.  The Wiki account requests and spam were solved when we instituted a very strict policy on getting an account.  A small group of people review and approve these accounts.

The server also hosts a couple of very low volume sites for friends.  They are more interesting from a content viewpoint and I want to share more about them in a future post.

Thursday, September 8, 2011

RTEMS Pair Programming

One of the most interesting and under-utilized RTEMS services that OAR Corporation offers is RTEMS Pair Programming.  This service is a great solution when dealing with a customer who wants to a big head start on some type of development effort. In Agile terms, this is a development sprint with a team consisting of RTEMS and customer supplied experts.  Most of the time, we do this for BSPs and device drivers.

Left to Right: Walter Nakano, Wendell Pereira da Silva,
Joel Sherrill and Jennifer Averett
The key to pair programming success is that we know RTEMS and the customer knows their hardware and test equipment.  OAR folks can concentrate on quickly providing the framework for the BSP and needed devices drivers. Then we work together to author the device drivers. This provides them with specialized training on the details of the BSPs and device drivers that are critical to the success of their application.  Usually the initial testing is performed as joint effort with subsequent detailed testing performed by the customer engineers.

Recently, OAR got to host Wendell Pereira da Silva and Walter Nakano from COMPSIS  for two weeks of intense development activity.  Their system consisted of an embedded PC plus some add-on boards which added up to a lot of individual pins and ports to test.  They brought a LabView test right which allowed us to test every input and output on the Multi-I/O board.The hardware list was:
  • RTD CME137686LX Embedded PC
    • 4 COM ports and i82551 NIC of particular interest to their project
  • RTD 17320HR Octal UART PC-104 board (PCI interface)
    • Exar PCI Vendor Id with 8 NS16550 compatible serial ports
    • NOTE: We only had one of these boards but they will have 4 in the real configuration!!
  • RTD 316HR Dual Synchronous Serial Port PC-104 board
    • single Zilog Z85230
  • RTD 6425HR Multi-I/O PC-104 board
    • 16 differential or 32 single-ended analog input channels
    • 4 analog output channels
    • 32-bit discrete I/O with 16 bit programmable for interrupt on input change
One thing should quickly stand out with viewing that hardware list.  They have a LOT of serial ports.  Four asynchronous on the embedded PC, 32 on 4 PCI-104 boards, and 2 synchronous on the Z85230 board for a total of 38 serial ports.  This is actually a classic example of a case where using the libchip serial driver framework would be very useful.  But the PC386 console driver was not designed this way.  Before the guys arrived, Jennifer reworked this driver to be libchip style.  At the same time, I factored out the mouse input stream parsing code.  It really wasn't BSP dependent and by moving it to cpukit/libmisc/mouse, I made it potentially available to every BSP.  Jennifer and I had tested COM1 and COM2 on qemu before they arrived but waited for their hardware to test COM3 and COM4.

COM1 and COM2 worked as soon as the cabling was correct. COM3 and COM4 proved more difficult.  After struggling to find a software problem, it occurred to me that it could be as simple as RTS/CTS not being wired together in the shell since we were using a 3-wire connection.  That was indeed the problem.  Next came the octal serial port board.

After realizing that we would end up with a libchip configuration table with 38 entries and most of them would be disabled for "normal" configurations, I had the idea to allow for dynamic registration of new "ports" in the libchip configuration table.   The idea is that if you probe for a bank of 8 serial ports and find them, then you can dynamically add 8 more entries to the libchip configuration table.  Currently this allows probes to insert entries prior to console_initialize() being called.  It is possible to allow them to be registered after this point but they would not be available to be /dev/console or used for printk().

While I was implementing dynamic registration, Jennifer and our guests worked to get the PCI probe to find the card and the first serial port working.  We were surprised to learn that it didn't have a vendor Id of RTD but Exar.  This explained the sparse programming documentation from RTD.  As soon as the probe and one serial port worked, we switched to my dynamic registration code.  Soon all eight ports on the board we had were working.  Plus I added code to detect the 2, 4, and 8 port variants of the Exar chip.

Next was the dual port synchronous board.  Unfortunately, RTEMS does not have a Z8530 synchronous driver but does have a standard libchip asynchronous driver.  After fiddling to figure out the baud rate clock divisor math, we ended up with both ports working.  There was one issue in the driver we did not resolve in the two weeks they were here.  The two ports on a Z8530 share a single interrupt status register which when read, clears the source.  You have to be extremely careful to touch it one time and process all interrupt sources on both ports.  The ports worked individually but not when both were installed.  Jennifer and I had a solution but not enough time to implement it. Hopefully Wendell and Walter can implement it and we can get this resolved in the main tree.

Next was the Multi-IO board.  If you have been following my blog a while, then you might remember the entry RTEMS Shell as Debug Aid where I discussed adding commands to the RTEMS Shell to aid in debugging a Winsystems Multi-IO board similar in capability to this board.   One of the last things Jennifer and I had done to the existing multiio was to define a board independent interface between the shell commands and the actual driver.  My plan was to let this interface evolve and grow as we learned more about user application requirements.  This was the first opportunity we had to write a driver to this interface and reuse the commands.  As might be expected, there were places where 0/1 based numbering of inputs still reflected the Winsystems board.  And there were places in the RTD documentation that were unclear.  But after a while of fighting these and the normal cabling issues, we were able to use the existing commands to debug the driver and verify that all discrete I/Os to work polled and interrupt driven and that all analog inputs and outputs work polled.  We ran out of time before we were able to attempt analog input interrupts.

The final thing we attempted was getting the RTEMS TCP/IP stack to run on this board.  It had an Intel i82551ER NIC which required using the drivers in the libbsdport kit of late model FreeBSD drivers.  This driver works on qemu when you configure qemu for the i82559 simulation.  We verified the basics were OK on qemu.  Then we moved on to the real hardware.  After the normal hunt for an extra cable and battle of the network settings, we were able to run the telnetd application from the network-demos module.

Walter and Wendell drove their hardware.  Jennifer and I were the main forces driving the code but they reviewed every line of code and we all verified that each line of code programmed the hardware as we all agreed it should be.  Along the way, if something was unclear, we took a break from coding and testing to focus on a portion of the RTEMS Open Class that was very specific to what we were working on. The goal was not only to have as much functional code as possible; it was also to ensure that the code was high quality and they left understanding it and capable of modifying it should the need arise.

At the end of two weeks, we all were thrilled.  Walter and Wendell had been sending home progress reports and every day I continued to be amazed at the progress we -- as a team -- had made.  This amount of progress was possible because each of us brought unique skills and knowledge to the table.  Jennifer and I knew where to reuse code from in RTEMS and how to create an elegant solution the RTEMS way.  Walter and Wendell were intimately familiar with their hardware and test equipment and ensured we tested well.  Together we reviewed all code to ensure they left understanding it.

I really enjoy teaching the RTEMS Classes but RTEMS Pair Programming is one of the most fun and personally rewarding services we offer.  I always come away amazed at how much is working at the end of an intense 2-3 week development sprint.  By bringing together engineers with complementary skills and knowledge, solutions are found quicker.  And solutions are ultimately what we all want.

Thanks to Walter, Wendell and Daniel who couldn't make the trip for two weeks of fun and productive work.

Sunday, August 28, 2011

Spell Checking Not Working in Firefox

This is not RTEMS related at all but everyone in the field has done free technical support before.   I got asked

Why isn't spell checking working on my new Firefox install?

It turns out there can be a variety of reasons and from my searches, the possible answers are not all in one place.  This article is an attempt to put all the things to check in one place.

WARNING: I checked this on Fedora (GNU/Linux) and menu options may vary based upon your host operating system.  Hopefully they are close enough to keep you on track.

The first thing to check is whether spell check is enabled at all.  Select "Edit > Preferences" from the menu.  Navigate to the "Advanced" tab and select the "General" subtab.  Make sure the "Check my spelling as I type" option is enabled as shown in this figure:


If this option is selected, then we will have to check two more things which might not be correct.  In order to do these you will have to navigate to a website which has a text entry form.  Composing mail on a web mail client like Google or Yahoo mail will work.  When you are there, right click in the text entry area and verify that "Check Spelling" is checked.

If it is, there is still one more thing to check.  Apparently some Firefox distributions were shipped without dictionaries.  If you installed one of these as a clean install, rather than as an upgrade, then you didn't get a dictionary.  I don't know why and it doesn't really matter.  If you have installed one of these versions, then you need to install a dictionary.  Again, you need to right click but this time, select "Languages > Add Dictionaries".  This will guide you through installing the dictionary you need.

As I was investigating this to help the person who asked, I learned I was using the "English / Zimbabwe" dictionary which I found surprising.  It definitely explains why I use UK spellings frequently.  It is necessary to make that language variant.  Colour me surprised.  Or color me surprised not that I am using "English / United States".

If there are any other things which might explain spell check not working in Firefox, please leave a comment.  

Wednesday, May 18, 2011

Passing of Dr James Johannes

My normal blog entries cover technical issues.  This post is going to be very different.  It is an homage to the man who referred to himself as my academic grandfather because my Ph.D. advisor had been his first Ph.D. student.  A man I have had the pleasure of knowing and working for for almost twenty-five years.

Dr. James D. Johannes
Dr James Johannes (Dr J) passed away Tuesday May 17 2011 at the age of 76.  This came as a shock to those who knew him because he was a vibrant person.  He was the type of person that one just expected to always be there.  Until a few months ago, he came to the office nearly every day. Within the past few years, he earned his pilot's license for the first time.

Dr Johannes earned his Ph.D. in his early 40's from Vanderbilt University.  At the time, he lived in Huntsville and had two children (Mark and Michele) and commuted about 100 miles each way to take classes.  His dear wife Aurelia -- who was the epitome of a classy and tough southern lady -- supported him and the children through this.  This was clearly a factor in them being understanding and supportive of my finishing my Ph.D. with four small children in the house.  Aurelia appreciated what my wife Michele (not his daughter) was doing.

Dr Johannes founded On-Line Applications Research (OAR) Corporation in 1978.  He was an Emeritus Professor of the Computer Science department faculty at University of Alabama in Huntsville and was the first head of the department.  Based upon the number of dissertations on the shelves at OAR, he advised over twenty-five successful Ph.D. students. When he retired, he was serving as the Graduate Dean of the UAH College of Science.  He wrote the Thesis and Dissertation Style Guide required by the university that I followed on my own dissertation.  It certainly made it easier to get clarifications.

I first encountered Dr J as a student in Spring 1988.  I was taking the Ph.D. level Operating Systems class he was teaching. I must have made a good impression because after the class was over, I received a job offer.  I asked if they could hold the offer for a few months.  My daughter Jessica (now 22) had just been born and I wanted to make sure she passed her six-week check up before switching jobs.  In July 1988, I started work at OAR and my first project was RTEMS.  You should know the RTEMS story.

Dr J also could show pride in those around him. I was the lucky recipient of his special events twice.  The first was a company wide lunch at the Huntsville Country Club when I passed my Ph.D. defense!  He knew the system and didn't wait for graduation.

Twentieth Anniversary
The other special event was in 2009 when we celebrated my twentieth anniversary at OAR.

There are many Dr J stories but I will only share a few.  Long ago, he called me at home on a Saturday completely unexpectedly.  I assumed I was about to be fired and went to a quiet place in the house to take the call.  Dr J had that professorial demeanour that makes professional students always a bit leery. It turned out he needed some sysadmin help with a Solaris computer he had at home.  Why he had a Solaris computer at home I don't know.

He didn't like to dispose of old computers. OAR still has a CP/M computer with 8" floppy drives in storage.  The HP1000 was taken to his house after the OAR folks refused to move it to our third office location in the mid-1990s.  I think Aurelia finally made him dispose of it.

Dr J was loved and respected by a wide circle of people.  He will be missed.

Tuesday, May 3, 2011

Power Restored But Issues Remain

Power was restored to OAR at ~5pm CST Tuesday May after being off since about the same time May 27.  The area is still under a dawn to dusk curfew so no hands on to fix things until tomorrow morning.  If a machine came up, we are running checks remotely.  The known status is
  • rtems.org rebooted cleanly and appears to have all services running correctly.
  • OAR VOIP phone service rebooted cleanly.
  • mail.oarcorp.com did not automatically come up and needs hands on help.  I was not the one checking this machine and that's all I know.
  • None of the RTEMS lab machines are up.  I smelled something acrid in
    the lab after the outage so don't know.  They are on different UPS's so unless it is the circuit the machines are on, it has to be a single piece of equipment. That has to be investigated tomorrow.
Tomorrow's work is focused on testing batteries in UPS's, running checks on machines, and cleaning the refrigerator.  Everyone is really chomping at the bit to get back to work.

rtems.info and Power Update

rtems.info is my personal server.  It is a purely volunteer effort and unfunded.  It is in my home.  We lost power Wednesday April 27 about 6pm CST.  Power was restored overnight Sunday.  Monday we returned home and I started to get the server back online.  There was some damage to various MySQL databases so I did a check like this after stopping mysqld:

service mysqld stop
cd /var/lib/mysql
for i in */*.MYI; do myisamchk --max-record-length=1048576 -r -f $i; done
That seemed to resolve all of those issues.

The outstanding issue now is that it appears my ISP had some damage to their network operations center.  All appears OK but in the process of recovering, they have deleted the DNS entry for rtems.info.  This was filed with them last night.

Huntsville is still under a dusk to dawn curfew and power is NOT restored to Research Park.  City schools are scheduled to start again on Thursday May 5.  So we are hoping for power tomorrow or Thursday.

More as it becomes available.

Terrible Storm and RTEMS Outage

Surely by now, you have noticed that the RTEMS Project appeared to drop completely off the face of the planet about 6pm CST April 27.  It was at this time that the third storm system moved through north Alabama and knocked out all major power transmission lines.  This blog is a first in a series to let you all know what happened and what is happening now.

The primary servers and lab machines for the RTEMS Project (.org and .com) are in Huntsville Alabama which was in the path of the storm April 27th 2011. This storm  killed 340+ across multiple states and left a huge trail of destruction.  I have heard reports of business signs being found 100 miles (160km) away.  The following is a nice weather summary without the heart wrenching photos of the death and devastation.

http://www.washingtonpost.com/blogs/capital-weather-gang/post/alabama-tornado-outbreak-visuals-jaw-dropping-radar-and-satellite-imagery/2011/04/29/AFg1C5YF_blog.html

Huntsville had three systems pass over it that day. The first was nasty but no issues impacting the server or our home. The second system resulted in water getting into the rtems.info server area but we still had power. This allowed us to clean up and get the server back online until the third system hit. The third system was the killer. It was the devastating one that wiped out communities from Mississippi through Alabama and Georgia to points further north. It destroyed the major power transmission lines into north Alabama and Mississippi. Local utilities get power from the Tennessee Valley Authority (TVA) and TVA could not supply power to them. Their blog is here with details:

http://www.tva.com/news/releases/aprjun11/storm.htm


I was teaching an RTEMS class during this with the sole attendee being a wonderful fellow from the UK.  We spent much of Wednesday in a safe area inside OAR.  And once the storm had passed and we realized we were the lucky ones, we finished the class without power.  His hotel room was wet and without power but his bed was dry.  He could charge his laptop from the car.  We took a table and a couple of chairs from OAR and sat outside the door.  When the sun moved and we got hot,  we moved the furniture.  At one point, we were on the other side of the parking lot.  We had nothing else to do and a dusk to dawn curfew, so we followed the class material, chatted, drank soda, etc.. Just chilled and did RTEMS stuff.  Phillip deserves a big thank you for helping me make sure all was turned off Thursday and cleaning the fridge and freezer Friday. I sent him to Chattanooga for the weekend and I hope it was some nice relaxing site-seeing.

Friday, my family went to a hotel in a neighbouring city and waited for power to be restored. We came home Monday afternoon since our home had no apparent damage and power was restored.  We are near the main hospital so we usually get power early.  I started ensuring the rtems.info and elviscostellofans.com server came back up OK. Michele is cleaning the fridge and freezer out. If we didn't lose any electronics due to power spikes, then that's all I think we have. That makes us very lucky. Michele and I know people with deaths in their families or homes destroyed. Cleaning the fridge looks pretty tame in comparison.

We tried to stay in touch using our cell phones for email but the towers died about 12 hours in.  Plus if we didn't know you on Facebook or via our private emails, it looked like we disappeared.  I apologize for not remembering linkedin and the RTEMS facebook group.  I didn't even remember IRC until Friday when we got to the hotel.

Huntsville Utilities announced yesterday (Monday night) that they have done all they can.  Everywhere TVA has given them power has been passed on to residents.  Only 30% have power.  I believe that Redstone Arsenal, Marshall Space Flight Center, and Research Park will be the last in the area to be restored.  They consume a LOT of power between them and it is more important to get power back to houses.

We really appreciate the good karma that was sent our way. We are both tired and frazzled but that's no biggie.