Friday, September 16, 2011

Elvis Costello and RTEMS.info History

Over the years, I have told many RTEMS users that I provide hosting and system administration for an Elvis Costello fan site (PHPBB Forum and Wiki).  I have had the fan site for about 9 years now.  What many of you probably don't realize is that I have also hosted the RTEMS.info Mirror Site since 2006.

This is a personal effort and I receive no subsidy from OAR for doing this.  In order to have a static IP address and host services, I have to have a business class account which is more than a residential class account.  All of the sites I host plus the family Internet activities share a 7Mbps/1Mbps connection.

I had been helping on the technical side of administrating the Elvis Costello Fan Forum for a while when the hosting service got hacked and our site trashed.  We were unable to get anyone to contact us via email or phone for over a week.  I realized that I had a computer running GNU/Linux Fedora that was largely unused since I had upgraded.  Even though it was only a 350 Mhz Pentium II with 384MB RAM, it was perfectly suitable to host a small (~100K hits a day) website.  I made a phone call to get a static IP address, moved the domains and within about a week we were back up.

A couple of years after that, the person who ran the Elvis Costello Wiki asked if I could host it.  I had already planned to upgrade to a 2.4 Ghz Pentium 4 with 2GB RAM.  We decided to wait to move the Wiki until after the server upgrade. The hardware upgrade went easily and we moved the Wiki.  What surprised us both was that the performance on the site went to hell.  The server logs showed nothing, load looked low and no amount of tuning or probing helped. I begged my ISP for help and got an unlocked, uncapped cable modem to test with.  After more research and fighting, I learned that my router could not hand the number of simultaneous connections and was dropping them randomly.  I upgraded routers and the performance issues were settled.

The site has always used external hard disks for backup.  There is a script which runs every night and dumps all user directories, databases, etc to a special directory on an internal disk.  Then another which runs later and "rsync's" the internal disk copy with an external one.  Backups are placed in dated directories and a few a month are saved.

When I set up the RTEMS.info Mirror, I was more concerned with disk space than bandwidth consumption.  I don't have the fastest connection and I am sure users would appreciate a faster uplink. But I foot the bill and until there is funding, this is what there is.  The RTEMS.info site mirrors at least 4 times a day.  Ralf Corsepius has set up automated checks which let us know when a mirror site is down or out of sync.

In late 2010, I became worried that the 2.4 Ghz Pentium 4 was getting very old.  It was not new when it became the server and I was sure it had seen at least 6 years as server.  The Elvis Costello Fan community rallied around my request for a new server and within a month or so, the fund raising goal was met.  The new server is from AS Labs who specialize in building custom GNU/Linux systems.  It has a quad-core 3.0 Ghz CPU and 8 GB RAM.  It runs very cool and is far from overloaded.

Over the years we have had period power outages with the worst being the tornadoes of April 2011. But overall, I believe our uptime is very good.  I don't track it but thanks to the Elvis Costello fan community, it can't be down over 20 minutes without me getting an email. Thanks folks!

My wife and I have learned a lot about system administration over the years of maintaining these sites.  She personally reviews and approves every account request for the PHPBB Fan Forum.  The number of spam account requests is boggling and periodically she begs me to try to find another way to stem an increase.  The Wiki account requests and spam were solved when we instituted a very strict policy on getting an account.  A small group of people review and approve these accounts.

The server also hosts a couple of very low volume sites for friends.  They are more interesting from a content viewpoint and I want to share more about them in a future post.

No comments:

Post a Comment