July 30, 2004

freebsd 5.x and fsck

FreeBSD 4.x always annoyed me the way it did filesystem checks when something wasn't unmounted properly. If for some reason you lost power, or patience--it happens even with best of OSes--you'd be dropped to an sh prompt before the boot process went any further. You were then expected to just know what to do. Which was to run fsck. I remember the big freakout I had the first time this happened.

Eventually, I just got used to running fsck manually in such situations. Which was bad for servers that needed to come back up without manual intervention. And only slightly less annoying than that old Windows scandisk feature that would take years to determine that your hardrive was perfectly fine.

With 5.x FreeBSD boots right up & then runs fsck in the background if there is a filesystem problem, after a 60 second delay. Only rarely is there a problem, and these are automatically fixed, the filesystems marked clean, and you hardly notice. Definitely the way to do it.

Posted by Alan at 03:28 PM | Comments (0)

always use (surge) protection

So, check this out: I had a server running here at work off a regular powercord, no surge protector or anything like that. It had only been up a week until a power surge fried the mother board today. Luckily, I dropped the hard drive into another server, made a few small changes to reflect the NIC in the new machine, and got it up & running again.

Posted by Alan at 03:17 PM | Comments (0)

July 23, 2004

scripts to manage ip changes

Very rough sketch of an idea for some scripts that seamlessly handle IP changes & other network interface changes on bsd or linux seamlessly. The problem with an ip change is that it usually involves (1) rebooting to make the settings take effect and/or (2) losing your ssh connection to the machine, so that if something goes wrong, you have to physically visit the server to bring it back up.

The first script will install a one-shot crontab entry for you at a specified time in the future. The second script, run by cron, will make the necessary ifconfig changes. If the interface doesn't come up under the new changes, it will back them out & revert to the original ones. You'll probably lose your ssh connection but you'll have some guarantee the machine will come back up in a few minutes.

Posted by Alan at 02:06 PM | Comments (0)

July 21, 2004

laptop craptop

A little follow-up on my decision to by a Toshiba A45-S150 laptop. Here are some things I really don't like about it:

  1. cpu fans are loud, and bottom of laptop still becomes uncomfortably hot
  2. hard disk is noisy--constant crunchiness
  3. keyboard layout is *weird*, tilde is just left of spacebar; alt, ctl, and delete are small and also off to either side of the spacebar; page up, page down, home, and end are way off to the side in a vertical row; function keys are not the standard size and don't align well--I still have to look everytime I alt+f4; still haven't found scroll lock
  4. keyboard repeat rate is too slow for me even on its fastest setting
Posted by Alan at 05:49 PM | Comments (0)

July 06, 2004

low bandwidth remote diffs

Here's an idea I have for a low bandwidth diff of two remote files. You first compute a checksum for the entire file on each end, and then exchange these checksums. If they are the same, the files are the same (with high probability depending on the checksum algorithm). If, not split the files in half and exchange checksums for the halves. Lather, rinse, repeat: essentially, do a binary search to narrow down those chunks of the file that actually differ, then transfer those differences.

Maybe this idea is kindergarten level and rsync / cvs / whatever else have been using it for years now. Will check on this.

Posted by Alan at 11:27 AM | Comments (1)

spacebar failure = buy new laptop

Tonight my spacebar stopped working when attacked from the right side. I was so frustrated I bought a new laptop, the Toshiba A45-S150. It's not as much of an impulse buy as I would like you to believe...have had the mouse poised over the buy now button for some time. Now that I've finally got this Thinkpad configured to my satisfaction--it only took me two or three weeks--I must needs return it, for it is not actually mine. Sigh.

Here's the deal I got. This will look silly I'm sure in under a year but I document it here for posterity. About it...

  • 512MB, expandable to 1GB. Memory is definitely the most important thing to me since I run a gazillion things at once. Beasts like Firefox, Thunderbird (I will replace you soon with mutt I swear), the cygwin X server, and various local servers running under cygwin are responsible. Right now I use between 230MB and 330MB; on a laptop with 256MB, this spells suck.
  • A Pentium 4-M 2.4 GHz. Yeah it's not a Centrino but at least it's an M. And 2.4 GHz is way, way more than I need or will really notice. Except that compilation will speed up. Now, of course the price is battery life since I'm sure even this M slurps up tons, but I don't really care...when I'm offline, I'm offline. "I went to the woods because I wished to program deliberately?" Nope. When programming I'm always within striking distance of an outlet.
  • 60GB. Nice. I could get by with less. It's only 4200 rpm but at the time of this writing that's pretty standard for a notebook.
  • 7.7 lbs. The Satellite is a clunker but I don't care. The ultraslim models I've seen usually sacrifice way, way too much for compactness' sake, and since I'm not a yuppie hot-rodding around the coffeehouse but a developer I need function, and not form.
  • 15.0 inch screen. Well this is par for the course. Couldn't go any smaller though.
  • Touchpad. I hate these things--especially the distance your thumbs have to stretch to get down there. I will be sad to see the nipple mouse on my Thinkpad go--it's the least intrusive design (from the standpoint of a keyboard zealot) that I've seen. But no one makes these anymore. Down with mouse pointer devices.

Incidentally I want to just mention that I fixed the spacebar on the Thinkpad. Pried off the key cover with a screw driver & sure enough there was a crumb under the right-side spring. My fears about not being able to replace the cover were unfounded: after positioning it over the key and pressing down, it snapped into place again.

Posted by Alan at 02:16 AM | Comments (0)

July 05, 2004

ssl certificates the short way

There is a lot of deliberately obscure, overly complex documentation on creating openssl certificates and signing them. Here's how to generate a self-signed certificate on FreeBSD, the short way:
  cd /usr/local/openssl/certs
  /usr/local/openssl/misc/CA.pl -newcert
  mv newreq.pem yourdomain.crt
  chmod 400 yourdomain.crt
Edit /usr/local/etc/apache2/ssl.conf:
  SSLCertificateFile /usr/local/openssl/certs/yourdomain.crt
Comment out this line:
  # SSLCertificateKeyFile /usr/local/etc/apache2/ssl.key/server.key
Make sure you edit the ServerName and ServerAdmin lines too. Then restart apache2 with ssl:
  apachectl stop
  apachectl startssl
Posted by Alan at 03:31 PM | Comments (0)

July 03, 2004

subversion and apache ports

Notes on getting the FreeBSD subversion (1.0.5) port installed with apache2 (2.0.50). Unfortunately you need to have apache2 configured and compiled against BerkeleyDB. Like so:


   cd /usr/ports/www/apache2
   make install WITH_BERKELEYDB=db4


Now apache2 drops the apr shared libraries into /usr/local/lib/apache2. Subversion will not be able to find them here. Rather than add this to the library path, I decided to create symbolic links to them under /usr/local/lib:


   cd /usr/local/lib
   ln -s apache2/libapr-0.so.9 libapr-0.so.9
   ln -s apache2/libaprutil-0.so.9 libaprutil-0.so.9


Now we're ready to build and install subversion:


   cd /usr/ports/devel/subversion
   make install -DWITH_SVNSERVE_WRAPPER -DWITH_APACHE2_APR


You yourself may need to play with some of subversion's other configure knobs, like -DWITH_PERL or -DWITH_PYTHON or -DWITH_MOD_DAV_SVN. However I keep it restricted to svnserve, since I prefer to tunnel all the traffic I can over ssh.

Posted by Alan at 02:54 PM | Comments (2)

avoid cvsup x11 dependencies

Oops, crap. One of the first ports I install on a new FreeBSD server is cvsup. However by default this pulls in a lot of X11 dependencies, which I don't even want to have around, since this is just a web and project services machine. Next time remember to do "make install -DWITHOUT_X11" for cvsup.

Posted by Alan at 02:01 PM | Comments (0)

July 01, 2004

marshalling and validation of structs

Finally, I have a module distribution on my hands that is possibly CPAN-worthy. We'll have to see, as I'm still writing it. Here's a quick preview though...

Struct::Marshaller is a module that transforms things like this


  my $params = { 'firstname' => 'John',
                 'lastname' => 'Doe',
                 'email' => 'jdoe@fake.com',
                 'account.username' => 'jdoe',
                 'account.password' => 'foobar',
                 'account.settings.enabled' => 1 };


into things like this


  my $user = { firstname => 'John',
               lastname => 'Doe',
               email => 'jdoe@fake.com',
               account => { username => 'jdoe',
                            password => 'foobar',
                            settings => { enabled => 1 } } };


and vice versa. In effect, it just converts a nested structure to and from a flattened form. This is nothing revolutionary.

Neither is the next module, Struct::Validator, which validates a nested structure using a mirror structure filled with compiled regexps:


my $against = { 'firstname' => qr/\w+/,
                  'lastname' => qr/\w+/,
                  'email' => qr/^\s*[\w]+\@[\w\.]+\s*$/,
                  'account' => { 'username' => qr/^[\w]{4,}$/,
                                 'password' => [ qr/^.{8,}$/,
                                                 qr/[a-z]/,
                                                 qr/[A-Z]/,
                                                 qr/\d/,
                                                 qr/[\!-\.]/ ] } };


This structure checks to see that
  • the user provided non-blank first and last names
  • their email address appears to be a valid one
  • their username is at least 4 characters long
  • their password is at least 8 characters long, and contains some uppercase letters, some lowercase letters, some digits, and some symbols
Here's where the coolness comes in. Say you have an html form for editing user information. If you name the input boxes email, account.username, account.settings.enabled etc. you can simply do the following on a post:


  use CGI;
  use Struct::Marshaller;
  use Struct::Validator;
  use strict;

  my $q = CGI->new();
  my $marshaller = Struct::Marshaller->new();
  my $validator = Struct::Validator->new();

  my $user = $marshaller->unmarshal($q->Vars());
  my $results = $validator->validate($user, $against);

  if (!$validator->is_valid($results)) {
    # do something
  }


$results, $against, and $user all have the same layout. If you went the extra mile and specified error messages in your $against structure, they would appear beneath each invalid field in the $results structure. So this results structure could easily be turned back into a page that flags invalid fields, and tells the user beside each invalid field why it was flagged.

What's really nice is that there's nothing web-specific about these modules: you could use the same code to slurp and validate a flat record field file, or validate a web service request if you've already built up a struct from the xml request via XML::Simple.

Posted by Alan at 12:03 AM | Comments (0)