Author: david

All My Stupid Man Pages Are Out Of Date

I got annoyed today when I ls-ed on my Mac OS X 10.5 system and saw this:

drwxr-xr-x@ 3 root  wheel  170 Jun  7 17:35 F9D86DAF-2868-4918-948A-110BB55DAB11

Well, that’s not what got me annoyed. I tried to look up what the ‘@’ after the permissions meant in the man page and it didn’t mention it at all. I searched the web and found someone quote an excerpt of the man page that explained the ‘@’. And it was not in my man page at all! That is when I got annoyed.

Then I remembered an exchange with a commenter: earlier where it seemed like my docs were old.

So I set out to find why my man pages were out of date. After some poking around I discovered this:

$ ls -l /usr/share/man/man1/ls.1*
-r--r--r--    1 root     wheel       15708 Mar  1  2006 /usr/share/man/man1/ls.1
-r--r--r--    1 root     wheel        6220 Apr 20  2008 /usr/share/man/man1/ls.1.gz

So at some point the system installed new, nicely gzipped man pages but somehow failed to remove the old ones. And it turns out the man give the non-gzipped version precedence. Giving me old out of date documentation when I ask for it. Weak.

A quickie perl script seemed in order:

find /usr/share/man -name "*.gz" -or -print \
    | perl -ne 'chomp; print "$_n" if -f "$_.gz" && -M $_ > -M "$_.gz"' \
    | xargsn -n 10 sudo rm

where xargsn is defined in my .bashrc file as:

alias xargsn="tr '\n' '\0' | xargs -0"

Using GNU xargs you can just do xargs -d\\n which is way easier for me to remember but stupid bsd xargs doesn’t appear to have the nice -d option. Weak, but I digress.

Anyway, the script finds non-gzipped files in the system man hierarchy and, if they have an equivalent gzipped version that happens to be newer, deletes them.

I had about 4500 that I was able to delete, and now my ls and chmod man pages have up-to-date info again. Aaaahhhhh.

You might notice that I only deleted the non-gzipped version if it was older. It turns out I had about 10 man pages where the opposite was true. So I tweaked my script a little and did this:

find /usr/share/man -name "*.gz" -or -print \
    | perl -ne 'chomp; print "$_n" if -f "$_.gz" && -M $_ < -M "$_.gz"' \
    | xargsn -n 1 sudo gzip -f

This one gzips the man page and replaces the older gzipped version when they both exist and the gzipped is older.

PC – Perl Calculator

History

For years I was GUI calculator fanatic. Desperately searching for that perfect calculator that did all the things I wanted it to do. I wanted a programmers calculator that did hex and binary operations and didn’t treat them as second class citizens. I used PCalc on the Mac for a while, and I even liked the built in Windows 95 calculator (that was the last time I had a dedicated Windows machine for working). But I was still left wanting.

At some point I decided to write my own. My idea was that I wanted a text entry box where the equation you were typing would go. Then you could edit the text and produce a new result without retyping the whole thing. I only got as far as being able to parse the text and convert it to reverse polish for easy evaluation before I gave up. But I kept the idea in the back of my mind for years.

At some point the idea morphed and I realized I didn’t need any of the GUI buttons to click on when I always had a perfectly good keyboard sitting in front of me. I looked into bc and dc but they were both too complex for me. Finally one day it hit me that Perl’s eval() was really what I wanted. And so “pc”, Perl Calc, was born.

pc started of very simple:

#!/usr/bin/perl

while (<>) {
    $a = eval($_);
    printf("%d 0x%x 0%o %f\n",$a,$a,$a,$a);
}

But it has now grown many features over the years designed to make my life more pleasant.

Getting PC

You can download it here or you can check it out from darcs like this:

darcs get http://porkrind.org/pc

Edit (2013-05-30): I am no longer using darcs for this project. It is now available on github.

You can give it input one of 2 ways: from the command line (this way you get all the editing and history niceties of your shell), or by running with no command line and typing into its stdin (this way you don’t have to quote characters your shell would like to steal).

How to use pc

The basics

$ pc 11+3
14 0xe 016

The first number is the integer result, followed by the hex and octal representations. Simple enough.

Order of operations

This shows that pc uses Perl’s order of operations (operator precedence if you are in a programming mood):

$ pc 1+3*2
7 0x7 07

ASCII

$ pc 1+3*20
61 0x3d 075 '='

Here we see an extra field was printed. In this case pc detected the final integer value was in the ASCII range and printed the character represented by the value 61, an equal sign.

Bitwise Operations

We also get Perl (and C) style bitwise operators:

$ pc '1+3*20<<2 & 0xff'
244 0xf4 0364

Also notice that we had to quote it since (a) I put a space in the expression and (b) I used the ‘<’ and ‘&’ characters which mean something to the shell.

Floating point

Of course it’s not restricted to only integers, it can handle floats too:

$ pc 1+3*20/55
2 0x2 02 2.0909090909090 23/11

You’ll notice it shows the result of the floating point math in addition to the integer math.

Fractions

You might have noticed a fraction in the output of the previous example. pc uses Perl’s “bigrat” library to do fractions:

$ pc 3/4+1/2
0 0x0 01 1.25 5/4

Human readable

$ pc 1000*2000
2,000,000 2000000 0x1e,8480 0x1e8480 07502200 1.90MB

You’ll notice that the integer and hex results are printed twice—one with commas and one without. The one with commas is so that you the human can read the output easily. The one without commas is for copying and pasting. You should also notice that pc helpfully told you the number was 1.90MB. If a number is bigger than 1024 (1KB) it will print it in human readable byte quantity.

Power of 2 magnitude suffixes

It also accepts magnitude suffixes on the input:

$ pc 16m
16,777,216 16777216 0x100,0000 0x1000000 0100000000 16MB

The following suffixes are allowed: kmgtpezy (lowercase, no b). Note that the human readable output “16MB” doesn’t have a decimal point. It will remove the decimal point if the number is exactly that value. So:

$ pc 16m+1
16,777,217 16777217 0x100,0001 0x1000001 0100000001 16.0MB

Since “16.0MB” has a decimal point, we know the value isn’t exactly 16 megabytes.

Large numbers

pc uses Perl’s bigint so that it can handle numbers bigger than 32 bits:

$ pc '<<40'
1,099,511,627,776 1099511627776 0x100,0000,0000 0x10000000000 020000000000000 1TB

Random Perl code

$ pc 'ord("a")'
0x61 0141 97 'a'

Since pc uses Perl’s eval you can do arbitrary perl code too. Though frankly the only thing I’ve ever used is ord().

Conclusion

So there you have it. It’s designed to give you all the output you ever could want without having to memorize any stupid command line switches. There are none, in fact. It also doesn’t overload you with redundant data. If the floating point answer is the same as the integer answer then it doesn’t show it to you. Same with the fractions.

So, if you read this far, go get it already!

Upgrading from Debian 32 bit to 64 bit

I got a new Intel core i7 computer and migrated my Debian server over to it. Here’s how to do it without installing from scratch:

First off, read this article. I followed the instructions and only deviated in a couple places.

I found that during the 64-bit libc install I had to run

dpkg --force-architecture -i libc6-i386_2.7-18_amd64.deb libc6_2.7-18_amd64.deb

instead of:

dpkg --force-architecture -i libc6-i386_2.7-18_amd64.deb

Yeah, even though libc6-i386_2.7-18_amd64.deb had already been installed.

Cheating with a chroot

One thing I did in addition to what the article suggested was to build a chroot for 64 bit stuff so I could download things with ease.

debootstrap --arch=amd64 lenny `pwd`/root64

Then,

chroot root64

From there I could “apt-get install” and look through dependencies and none of it was impacted by the state of my main machine. Apt gets screwed up for a while in the process and it was helpful to go find what a packages real dependencies were.

In particular I did “apt-get install ia32-libs” which got all the compat libraries. From there I could go into root64/var/cache/apt/archives/ and get right to the .deb files.

I also used aptitude to resolve many of the apt-get -f install problems (once I had reinstalled enough by hand to stop the ELFCLASS errors).

apt-get -f install

My biggest “apt-get -f install” problem was some old java stuff that couldn’t run its post-upgrade or post-rm scripts and so couldn’t be upgraded or removed. Paying very close attention to what the script was doing I was able to remove a package the script used, rendering it a NOP (it was only doing something if the other package was installed).

Binary incompatibility

My biggest problem in general was that stupid Sleepycat morons made their %$#@! db’s binary format incompatible between 32 and 64 bit machines. Way to go guys—that’s thinking. This means that any packages you have that rely on libdb4.2 are screwed. For me this was Netatalk and the Cyrus IMAP server.

Netatalk

Netatalk turned out to be pretty easy. Just delete the .AppleDB files in the share directories. Netatalk uses them as caches and so it’s perfectly acceptable for them to go away (though if you have aliases that point into your share then they may break—I didn’t so I didn’t care).

Cyrus

Cyrus turned out to be pretty easy too, once I figured out what was happening. This post on the gentoo bugzilla explained how to recover, and it worked as advertised. Thanks to Rumi Szabolcs for posting that.

I installed Cyrus on a 32 bit linux machine I (luckily) had and converted the mailboxes file. Though it looks from the man page of cyrreconstruct (what it’s called in debian, by the way) that the -f command will reconstruct the mailboxes.db file from the directory layout, which might be handy if you don’t have a 32 bit Debian machine handy.

The Bloop in Real Time

The mp3s:

Here’s my slowed down version of The Bloop. You won’t be able to hear it with tiny little computer speakers. Either find a subwoofer or some nice over the ear headphones and crank the Cthulu Bloop:

Background:

The other day there was an article on Reddit about “The Bloop”–A weird sound captured by underwater hydrophone arrays in the middle of the ocean. On top of that, the sound was caught near the place from which Chtulu is rumored to emerge. Here is the wikipedia article about The Bloop. The sound posted there is a edited down version of The Bloop from the NOAA’s site.

Fine so far. But I noticed that on both wikipedia and the NOAA site they say that the sound is sped up 16x. So I wanted to hear what it sounded like in realtime. I slowed it down with audacity (which turned out to harder than it should have been), and reveled in the deep bass.

Today I saw another link on Reddit to someone who had done the same thing. I checked out their real-time sound and it didn’t sound at all like mine. I realized that they used a pitch shifting style effect to slow the sound down without changing its pitch. When you slow something down that much it introduces a lot of artifacts, which are clearly audible in their file. In order to attempt to fix that they have another version that’s been run through some sort of noise reduction filter which causes even more bloopy (so to speak) artifacts.

If you check the original spectrogram on the NOAA’s site it looks like this:

The Bloop spectrogram
The Bloop spectrogram

Notice that the top frequency there is 50Hz. That is low. We’re talking deep bass. So I’m pretty sure the correct way to slow it down is to actually just slow it down and let the frequency drop all the way down.

Cordoba: The Small Chrysler

I was rearranging my bookcase and happened across a 1976 Readers Digest (when I moved into my Grandma’s old house I left all her books there). I flipped through and read the jokes (they had a “don’t step in the hoya” joke which I had also just seen on reddit in the last couple weeks) but what really caught my eye were the ads. Most had very stilted layout without a lot of font choice or flair. Graphic design must have been annoying back before computers.

Then I saw this ad and just had to laugh!

Cordoba, The "Small" Chrysler

Apparently our vision of what a small car is has changed a lot over the past 40 years. Plus that guy’s suit is just awesome.

“Kharma”

My sister accidentally left her phone in a McDonald’s in LA the other weekend. When she went back, someone had taken it and not given it to the lost and found. When she called her number they would answer and then immediately hang up. So she called Verizon and reported it as stolen. Soon after a woman name Sheila called my mom and said she found my sister’s phone out side a Food For Less store in Santa Clarita. My mom asked if she would mind mailing the phone back down to us so that she could avoid making a 140 mile round trip to pick it up. Sheila balked (it was way too much trouble for her) and so my mom offered to mail a self addressed, stamped shipping envelope to her. She grudgingly decided that would be OK.

Today they got the phone in the mail (yay!) along with a folded up note. Here’s what it said:

Oh yes, Sheila, I think we do.

All my menu extras are gone!

For most of the day today I’ve been missing all the OS X menu extras in the top right of my MacBook screen, including my clock, airport and volume controls. I never knew how much I used that stuff until it suddenly wasn’t there. I knew I could probably restart my system and get it all back up and running, but I’m not going to kill 8 days of uptime for that! So I look around the net and find out that those menus are controlled by a process called SystemUIServer. Luckily they say it will restart itself if you kill it so I force quit it with Activity Monitor. The process comes back (with a new pid so I know it’s new) but my menus are still missing. Hmmph.

I look in the console and don’t see anything that looks like an error. I hit Command-space to invoke spotlight and nothing happens but I do get a console message that says “-[MDMenuWindow _checkTopRight] the window is off the screen topRight point is {-10, 778}”. Interesting. Somehow it pops into my head that I had my MacBook hooked into my HDTV yesterday and that maybe the menu bar is just confused about the width of the screen. So I open System Preferences and change my window size to something random and suddenly all my menu extras show up! Yay! I change my screen back and a few window resizes later everything is back to normal.

All without a reboot. Ahhhhh….

Why is there no process viewer in Firefox?

As I sit here right now my laptop is uncomfortably hot and its fans are making a really loud high pitched whine. Something on my computer is spinning the CPU and it’s making me literally hot. I look at the processes and FireFox is taking between 40% and 80% of the CPU! I’m basically sitting idle too. Well, right now I’m typing this message in Firefox, but 5 minutes ago the computer was sitting unattended and it was still going nuts.

Ok, so Firefox is wigging. But why? I can’t find any way to do some introspection. I suspect it’s one or two errant pages sitting in a tight javascript or flash loop. But how can I find out which pages? I have 26 windows open and most windows have multiple tabs open each (one has 30 tabs). Each of them is open for a reason–some unfinished business I have at each page, so I can’t just close them randomly. Besides, that would admit defeat.

What I need is a plug-in that monitors all the javascript and plugins that are running and keeps track of how long each runs. Then I want a unix top-like view of all the open pages and how many resources each is consuming (how much memory, how much cpu time their javascript is taking, etc). Why doesn’t this already exist? Sadly I just don’t have time right now to write such a thing, considering how much I’d have to first learn about the internal workings of Mozilla threads (I assume they are all green threads, because something is running so much my typing has a significant delay every few seconds).

Please, please, please (let me get what I want) won’t somebody write something like this? How do you Mozilla developers work? Is there a Javascript profiler available already? I’ve searched the add-ons before but I can’t find anything like this.

All I know is my lap is getting hot and I can’t do anything about it because I need Firefox open right now and it’s beginning to really annoy me.

Time Machine and a Linux server

I have a Debian GNU/Linux server with a large raid 5 disk. I recently upgraded my RAID disks and suddenly have ample free space. I put OS X 10.5 on my MacBook back in November but didn’t have enough space at the time to try out Time Machine. Getting Time Machine to work with my Linux server was annoyingly hard–the default Debian server doesn’t support Leopard out of the box and Time Machine itself doesn’t support non-apple file shares. Not to mention Time Machine seems slow and completely bug ridden (I’m generally unimpressed, but when it finally works it’s nice). Anyway, here’s what I had to do to get it to work.

Step 1: rebuild netatalk with ssl support

Apple, starting with Leopard, won’t connect to appleshare servers that don’t support SSL (encryption). This is a good thing, really. What is annoying is that Debian doesn’t ship their netatalk package with encryption enabled (there’s apparently some sort of licensing mismatch and they’re fairly pedantic about those things–witness “iceweasle”).

If you are familiar with Debian packages, the key thing is rebuilding the netatalk package like this:

DEB_BUILD_OPTIONS=ssl debuild

If you are not familiar with Debian packages, first google for “debian ssl netatalk” or “ubuntu ssl netatalk”. There’s a bunch of people out there with instructions. They basically boil down to:

apt-get source netatalk
sudo apt-get build-dep netatalk
sudo apt-get install cracklib2-dev
cd netatalk-2.0.3
DEB_BUILD_OPTIONS=ssl debuild
sudo dpkg -i ../netatalk-*.deb

At this point you should be able to mount an your Linux server on your Mac.

Step 2: Getting The Mac To See The Server

Time Machine, for some stupid reason, doesn’t seem to want to see network shares from non-Apple servers. There’s a well known secret preference (how’s that for an oxymoron) that you have to set from Terminal:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

You can make sure it’s set with this command:

defaults read com.apple.systempreferences TMShowUnsupportedNetworkVolumes

It should print “1″ if it is enabled. With that set, you should now mount the drive you wish to back up to with Finder, then go to Time Machine’s Preferences. If all is well then your network drive will show up when you press the “Change Disk…” button.

If the drive does not show up then you are going to have to mess around. I had a terrible time getting it to show up on a computer with 10.5.2 installed. One thing to try is adding a file to the root of the network share called “.com.apple.timemachine.supported”. Also try deleting the .0016cbcbc4c8 file in the root of the share (that number is for my computer–your computer will have a different number).

Step 3: Try a small backup

Go into Time Machine preferences and hit the “Options…” button. Keep adding excluded directories until only a small part of your disk is set to back up. Right click on Time Machine in the Dock and select “Back Up Now”. If all goes well then Time Machine will chug along for a while “preparing” (going through your whole disk sizing everything up), then doing the actual backup, then “finishing” (Deleting interim backups–if you have a lot of these then finishing can take a really really long time).

If it gets an error then you might need to create the sparse bundle for it.

An aside–sparse bundles and Time Machine

A “sparse bundle” is a new format of disk image added in OS X 10.5. This follows a long line of disk image formats from the old “.img” of System 7 to “.dmg” introduced in OS X.

Sparse bundles are actually pretty neat. They are almost the same as “sparse image” “.dmg”s from previous versions of OS X. Traditional disk images allocate all the room up front. So if you create a 10GB disk image then it would create a 10GB file on your disk.

The sparse image is different–if you create a 10GB sparse disk image then the file that is created on the disk is actually much smaller, basically just the size of the HFS+ filesystem. When you add files to the disk image then the .dmg file itself grows to accommodate new data.

Sparse bundles work the same way except that instead of one gargantuan 10GB file (assuming you fill up the whole disk image), the data is spread out in a large number of small “bands”. Each band is a file that is a specific size (8MB for Time Machine sparse bundles) that holds just one little section of the disk image’s data.

Why is this good? I’m glad I asked! It’s particularly nice when you are syncing the image between 2 computers. Say I mount a normal disk image and change one letter in a TextEdit document. If I try to sync the disk image to another computer I basically have to copy the whole thing over (barring some cool rsync-like behavior). With a sparse bundle, I probably have to only copy one or two bands of data across. According to what I’ve read, Apple create the sparse bundle format so that File Vault was more compatible with Time Machine (it can back up bands instead of entire disk images).

Anyway, when you back up to an external FireWire or USB disk, Time Machine backs up directly to the disk. When you back up to a network disk, Time Machine puts a sparse bundle disk image on the network share and then backs up to the disk image. This theoretically allows them to back up to a network filesystem that by itself doesn’t have the necessary capabilities for Time Machine. NFS, or SMB, for instance should be able to be able to host Time Machine backups with no issues–I haven’t tried this though.

The problem is that OS 10.5.2 seems to have a bug where you can’t always create a sparse bundle on a network share. If you try to do this with Disk Utility you will get an “operation not supported” error.

The solution is to create a sparse bundle on your disk that is the size of your network backup disk and then copy it over. My backup disk is 2.25 TB, so I’d create a 2.25 TB sparse bundle disk image. Once you copy it over to the network drive you can delete it from your local disk. I used this technique to get my dad up and running with his shiny new Ubuntu server.

By the way, for this to work you have to name the sparse bundle disk image correctly, which takes a little trick. Start a Time Machine backup and look in your network disk. Time Machine will have started creating a sparse bundle on the disk. Copy the name of this sparse bundle before it gets the error and deletes it on you. Mine is called “black_0016cbcbc4c8.sparsebundle” (my computer name is “black”). Either create the sparse bundle with that name or rename it once you’ve created it.

Other issues

My computer behaved very, very strangely before I got everything running smoothly. One thing it did was stick in the “preparing” phase for huge periods of time (24 hours, sometimes). I’d just stop the backup and let it try again later. After a couple tries it would mysteriously work.

Another thing it did was get confused with the progress. It would start up a backup and say something reasonable in the progress bar (0GB completed out of 30GB). Everything moved along but when it hit the “end” it would suddenly start incrementing the total size: “32GB completed out of 32GB”. And it would just keep growing and growing. It would reach ludicrous numbers sometimes, saying “2TB completed out of 2TB” while backing up my 250GB disk. Sometimes it would complete on its own and sometimes I just stopped it after 24 hours.

Interestingly, both of these strange symptoms might be related. I captured both of these situations using 10.5’s dtruss and found that in both cases if was seemingly looping through my disk multiple times. I would see the same files and folders being iterated through over and over again, like there was something wrong with my filesystem. I did a fsck with Disk Utility, though, and it reported that everything was fine.

So I think both of those issues are just stupid bugs in Time Machine (or one stupid bug hitting in two places). It’s times like these when I curse Apple’s crappy proprietary software. If this were a linux app I’d just download the source and fix it myself. But the Bastards at Apple think it’s better for Time Machine to be a %$#@! black box. Which is OK when it works, but really sucks when it doesn’t work. </rant>

Step 4: Slowly remove excludes

To get around these issues I would alternately let it run for huge long periods of time and then stop it if it were running for more than 10 minutes. I also excluded almost all my disk in the Preferences (as I mentioned before) and then slowly removed exclusions until my whole disk was being backed up.

I’m at the point now where Time Machine just mounts up my network drive, backs up and only takes a couple minutes to finish. Getting there was slow (it took me about a month to get to this point!!!) and even though it is fairly stable now, I still can’t help feeling like Time Machine is a buggy pile of crap.

Some helpful links:

Leopard Permissions Going Crazy

So a couple days ago I noticed I had no permission to access one of my directories. Since it was a directory that I use in command line mode I naturally checked the permissions that way:

$ ls -ld Downloads
drwxrwxr-x   81 david    david       27370 Feb 23 11:18 Downloads/

Looks ok! So it works, right?

$ cat > Downloads/eat
-bash: Downloads/eat: Permission denied

What?

After puzzling for a few moments I decide to get info in Finder:

Oooo. I didn’t know OS X had ACLs. I don’t really like ACLs in general because they seem too complicated for normal usage.

Well, I’ll just click that nice little minus sign button and delete all the extra ACL things. Except that the minus sign button just straight up doesn’t work. I’ve unlocked it and typed in my password so I should have root permission at that point but the dumb button just doesn’t do anything. Something must be screwed up. Sigh. Back to the command line…

So I do some googling and fine you can check the ACLs at the command line with ls -e.

$ ls -e Downloads/
ls: invalid option -- e
Try `ls --help' for more information.

What? Oh yeah, I put GNU ls on my machine so I could do color ls (turns out Leopard ls can do color with -G). Ok, let’s use the system ls:

$ /bin/ls -ld Downloads/
drwxrwxr-x+ 81 david  david  27370 Feb 23 11:18 Downloads/

Aha!. There’s a + on the end of the permissions to show me ACLs exist.

$ /bin/ls -lde Downloads/
drwxrwxr-x+  81 david  david     27370 Feb 23 11:18 Downloads
 0: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
 1: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
 2: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
 3: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
 4: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
 5: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
 6: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
 7: user:root allow list,add_file,search,delete,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,writesecurity
...

The list goes on exactly like that up to 127! Ok. That looks a bit screwey! So I figure out that you can manipulate ACLs with chmod, but annoyingly chmod doesn’t have an option to just wipe all the ACLs clean. Grrr… Ok, so I get to write a little loop in shell:

$ while /bin/ls -ld Downloads | f 1 | grep -q '+'; do chmod -a# 0 Downloads; done

f is a little program I stole from here.

So… Did it work?

$ /bin/ls -led Downloads/
drwxrwxr-x  81 david  david  27370 Feb 23 11:18 Downloads/

Yes!

Now let’s make that a shell function so I can run it easier:

clear-acls() { while /bin/ls -ld $1 | f 1 | grep -q '+'; do chmod -a# 0 $1; done }

Now I can just do clear-acls Music and fix my music directory, which is also screwed up.

It also occurred to me that fixing the permissions with Disk Utility might work as well, so I am trying that now. The progress bar is not moving and it has been saying it will be done in “less than a 1 minute” for the past 10 minutes. Nice. Does anything work in Leopard?

Half an hour later… Disk Utility fixed some ACLs, but only on system directories. My home directory still has a bunch of folders with weird ACLs on them. I have no idea who put them there (I can only assume some stupid Apple bug, probably in Time Machine–what else scans my whole disk?), but at least I can manually fix it when it happens.

Selling out: How a Free Software advocate ended up releasing a shareware program

I write computer programs for a living and quite often I end up writing code for myself at home. Occasionally I get the software into a releasable state and stick it up on my web site. When I had a steady salary I would always just GPL everything I released. Mostly since I didn’t want to support it–I felt that if I tried to hoard the source code I’d be on the spot if someone needed it fixed for them, and I’d feel bad if I couldn’t do it. Giving away the source cleared my conscience since I figured, if you don’t like it, change it yourself.

When I quit my job and started consulting I lost the security that comes from a steady paycheck. I’m making more money now, but it always feels temporary and fleeting. So when I needed to write a piece of software to scratch my own itch I started thinking about selling it instead of releasing it as free software. I know, that seems antithetical to the Free Software movement and, well, it is. My dad laughed out loud when I told him I was releasing a shareware app since I’ve been know to rail about proprietary software before. It’s amazing how a little financial insecurity can change your outlook. Don’t worry, I still donate to the FSF and believe in Free Software.

Mostly, I was very curious about how much money I would make form a piece of shareware. My program was a pretty simple idea–I wanted to have multiple libraries in iTunes for different versions of my music. I have my music all ripped to a central server that transcodes on the fly so it can serve out different file formats at different encoding rates to my various clients. When I’m away from home I stream .mp3’s to my iTunes and .oggs to my xmms. At home I stream .wavs to my iTunes and .flacs to my xmms. If I’m somewhere with not a lot of bandwidth I can stream at a lower bitrate too.

So I wanted to set up my iTunes with a library full of .mp3 streams and separate library with the same songs as .wav streams. I looked around at the existing freeware and shareware solutions and found them all lacking. They all were separate applications that you had to run which I thought was too cumbersome. At this point Apple hadn’t added the option key and iTunes boot trick (which I still think is more cumbersome than my final solution).

What I really wanted was something that fit inside of iTunes so I could switch without launching a separate program. Something that would just be smooth. Since I started programming on the Mac back in the System 7 days I was familiar with the low level carbon calls and I figured I could abuse iTunes plug-in interface to do what I wanted. So I hacked on it for a weekend and got it working. MultiTunes was born.

When I was done I thought MultiTunes was smooth enough to warrant releasing it. So I downloaded a copy of the GPL and was prepping a release when I thought, “wait, these other guys sell theirs. Why can’t I sell mine? How much would I make anyway? Could I make a living selling dorky little shareware apps?” I looked around for people doing shareware but didn’t really find anyone that would divulge their numbers and so I couldn’t really get a handle on how much it would really pay. So in the spirit of experimentation, I decided to release it as shareware and see what happened.

Releasing a program is hard. It always takes 10 times longer than I expect to get everything right. I think I spent more days on the details of the release than I spent getting the program working in the first place. I had to make some artwork for the disk image, a script to create the disk image so I can just to “make release” and not have any manual steps–the more manual steps you have to make, the more errors will occur in the process causing more delays and more overall time spent resulting in less payoff per hour of work. More importantly, the harder it is to release, the more I dread it and refuse to work on the program. So it’s important for me to have the release process be really smooth. I want to edit a version number in exactly one place, edit the changelog, tag the repo and “make release”.

Releasing my first shareware program was doubly hard. How do I get someone to pay me? I looked up merchant accounts and credit card services and realized they were way too much work. Maybe if I was doing this full time it would make sense, but no, I don’t want that. I looked at some shareware services but ultimately decided to just use Paypal and some custom server software to give out registration codes on receipt of payment. This actually didn’t take too long to write–maybe a day or two–and I made sure I wrote it generically enough that I could easily add more programs to sell when I inevitably came up with more brilliant ideas. It’s been 2 years since then and I’ve come up with approximately zero. But the point is, the infrastructure for selling wasn’t too hard, provided I was willing to give up a percentage of my profits to Paypal (which I was).

The next step was deciding on a price. Too high and no one is going to buy it. Too low and I’m essentially throwing away money that people would be willing to give me. I looked around at MultiTunes‘ competitors, as well as other shareware iTunes plug-ins. I thought my solution was smoother than my competitors’ so I thought the price could stand to be higher. I didn’t think I could charge as much as Octiv’s VolumeLogic, which before it was defunct cost $19.99. I asked myself what I was willing to pay for MultiTunes had I not written it and I decided I was willing to pay $10. But I know I’m a cheapskate when it comes to shareware and so I added 50% to that. I also decided to use the cheesy and cliche $0.99 perception trick and so the price I chose was $15.99.

I thought it might be too high, but then again, it’s easier to lower the price if it’s too high but it’s annoying to try to raise the price if you feel it’s too low. In the end I think the price I chose was right, but it’s more of a feeling than a fact backed by hard evidence. The only real way to tell would be to offer it to different people at a different prices and see what happened, but that seemed unfair and hard to set up.

So, I have a nifty program, a infrastructure for selling it, and a price. How much money am I going to make? Well, I’ll write a separate post about that soon, but in the meantime I’ll answer it this way: Not enough to live off of. Certainly not as much as the Delicious Library guys, but then again my program isn’t nearly as cool, sophisticated or good looking. Then again, I wrote it by myself in a weekend or two! And even factoring the time I’ve spent improving it and the email support I provide (and a random telephone support call that I got while visiting my parents for Christmas from a guy who looked up my phone number (!!!)) I think the profit per hour ratio is not too bad. Sadly I didn’t really keep track of the time I worked on it so it’s hard to calculate that number exactly.

Anyway, that’s all for now. I’ll post something soon with specific numbers–profits, piracy rates, etc. The kind of stuff I wished I’d had an idea about when I first started.

Polling is always wrong

I read this article on reddit and while I thought the author was correct that RSS is really a bad design, I think he missed the real underlying reason why. That reason is that polling is always wrong. Always. Really!

RSS ends up fundamentally being a polling technology. Everyone hits your site saying, “hey, you got something new?”

“How about now?”

“Now?”

“Still nothing, huh”

“Ok, Now?”

Over and over it asks, never taking a break. What is the appropriate speed at which it asks? 1 hour? 1 second? Who knows.

It’s not just internet protocols that poll. Polling as a technique is pervasive, hitting every level from the highest to the lowest. Does your OS go out and hit your CD-ROM drive every so often looking for a disc to finally be inserted? Why isn’t there a SCSI command that makes the drive send a command to the computer when a disc is inserted? If there is, why isn’t your OS using it?

What about something simple like the volume slider in your system tray? My friend Jim was using Intel’s PowerTool and discovered the Gnome volume control kept taking the processor out of C3 sleep. Turns out the volume control was waking up constantly and polling ALSA to see if another program had changed the volume behind its back. ALSA even has an API for being notified of the volume change!

Even at the low level of embedded programming there are the same choices. Does your mouse’s microcontroller poll a GPIO to see if you’ve pressed the mouse button, or does the button trigger an interrupt?

Why do people poll?

There’s only one reason. It’s easy. It’s trivial to wrap your head around—You just loop forever, checking for whatever you’re polling for and then sleeping for a little bit. But to do it the right way is more complicated—you need to have to have some infrastructure. You need a way to register for the event you want, which implies you also need some way of getting called back when your event happens. In some cases (like the mouse button), it’s almost as easy to do it the right way as it is to poll. But others, like the RSS case, it’s much harder.

To do RSS properly you’d need some way of getting events pushed to you. One way would be with email, as that infrastructure already exists. But given the amount of disparate email servers and RSS readers, it might be hard to hook your RSS reader to your email. Or you could simulate a callback by having the server hold the connection open until something happens. But given that the connection will eventually time out, that’s really just a degenerate way of polling.

Ah, forget it. Let’s just poll!

The iPhone is not expensive

Every review I read about the iPhone whines that the price is high. So let’s see how high it really is.

An iPhone can be thought of as an iPod Nano plus a phone. Except it’s not quite the same as a Nano. It’s got that screen that’s even bigger than the disk based iPods. And it’s got that cool visual voice mail.

A 4 GB iPod Nano is $199.
A 8 GB iPod Nano is $250.

The 4GB iPhone is $499.
The 8GB iPhone is $599.

A Palm Treo is $99 to $199, an old (refurb) model for $49.
W/ 4GB Nano: $298 to $398, old (refurb) model $248.
W/ 8GB Nano: $349 to $449, old (refurb) model $299.

A Blackberry is $199 to $299, an old model for $49.
W/ 4GB Nano: $398 to $498, old (refurb) model for $248.
W/ 8GB Nano: $449 to $549, old (refurb) model for $299.

All the phone prices are from ATT’s site and are for a 2 year contract, just like the iPhone. The cost per month with data plans is the same regardless of what phone you have so there’s no point worrying about it.

So, depending on what model you buy, you end up paying between $1 and $200 extra for the iPhone. A refurb phone and iPod Nano bundle is between $200 and $300 cheaper than the iPhone. But lets be real here. The iPod is the top of the line right now so it’s not really comparable to an old refurbished phone.

Right now you’re looking at an average of $150 more for an 4GB iPhone and $200 more for an 8GB iPhone. That effectively is the price for the convenience of having both your iPod Nano and smart phone in the same device. And considering the nice screen and some of the cool features of the phone not found anywhere else, that doesn’t seem that expensive to me.

Tivo Desktop on Linux

I tried out TiVo Desktop on my Mac and it was kind of cool. It lets you play music on your TiVo that’s streamed from your computer. But my mac already streams music from my Linux server and TiVo desktop wanted to serve up real mp3s and not re-broadcast streams–so none of my music would play. I found some articles and discovered the TiVo Desktop is just a http daemon plus some mdns stuff. Sounds easy enough!

I installed mdns-scan (thank you Debian) which reports:

+ David Caldwell's Photos on black._tivo-photos._tcp.local
+ David Caldwell's Music on black._tivo-music._tcp.local

I copied /etc/avahi/services/ssh.service to /etc/avahi/services/tivo.service. I then changed the name to “Lovely Music”, “_ssh._tcp” to “_tivo-music._tcp” and picked a port out of the air: 8000. I reloaded the avahi daemon and wham, the “Lovely Music” shows up on the TiVo. Yay!

The rest is just http on port 8000 that spews XML. That should be fairly easy to do…

…. Time Passes ….

And 8 hours later I say, “Fairly easy?”. Actually it turned out to be pretty easy. The main problem I had was that the %$#@! TiVo had died on me and wouldn’t play any music, not even from the official TiVo Desktop. I had to reset it to make it work again. Of course, I kept thinking it was something I was doing wrong and so I kept implementing more and more of the protocol. For 4 hours. Grrr….

Avahi

Turns out mdns-scan doesn’t give enough info to really get it working. I used avahi-discover which dumped the txt records that were needed. In the end my avahi services file looked like this:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<!-- See avahi.service(5) for more information about this configuration file -->
 <service-group>
  <name>Lovely Music</name>
  <service>
    <type>_tivo-music._tcp</type>
    <port>8001</port>
    <txt-record>protocol=http</txt-record>
    <txt-record>path=/tivo.cgi/music</txt-record>
  </service>
</service-group>

The path, <port> and <name> can be anything you want. I’m not sure about protocol.

Here’s the corresponding one for the photo sharing:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<!-- See avahi.service(5) for more information about this configuration file -->
<service-group>
  <name>Beautiful Photos</name>
  <service>
    <type>_tivo-photos._tcp</type>
    <port>8001</port>
    <txt-record>protocol=http</txt-record>
    <txt-record>path=/tivo.cgi/photos</txt-record>
  </service>
</service-group>

http server

I set up port 8001 on apache and then wrote tivo.cgi. I wrote it in the least general way possible and so I don’t plan on publishing it–it just wouldn’t be of much use without specific perl data files that I already had lying around. They actually are for the Philips Streamium server that I heavily modified. And so I wanted to share them instead of putting a whole bunch of work into reinventing the interface…

I could publish my simple test server, if anyone is interested…

Photos

I haven’t looked too much at the photo sharing yet, but it looks exactly like the music sharing at first glance. In fact, you can see in the avahi configs I pointed photos and music to the same tivo.cgi. Only I don’t really check for /music or /photo yet and so a couple times I accidentally selected the photo option from the TiVo menu and it came up with a list of my music folders in the thumbnails (with the correct names and everything). So I think it really is the same.

Protocol

The protocol is pretty simple.

The tivo will send a request like this (each CGI parameter is on it’s own line for clarity and horizontal space conservation):

/tivo.cgi/music?Recurse=No
&Filter=x-container%2Ffolder,x-container%2Fplaylist,audio%2F*
&SortOrder=Type,Title
&ItemCount=0
&Details=Basic
&Format=text%2Fxml

The server should respond with something like this:

<?xml version="1.0" encoding="UTF-8" ?>
<TiVoContainer>
  <Details>
    <Title>Top</Title>
    <ContentType>x-container/tivo-music</ContentType>
    <SourceFormat>x-container/tivo-music</SourceFormat>
    <TotalItems>2</TotalItems>
  </Details>
  <ItemStart>0</ItemStart>
  <ItemCount>0</ItemCount>
</TiVoContainer>

The TiVo will then repeat the request with a higher ItemCount:

/tivo.cgi/music?Recurse=No
&Filter=x-container%2Ffolder,x-container%2Fplaylist,audio%2F*
&SortOrder=Type,Title
&ItemCount=2
&Details=Basic
&Format=text%2Fxml

This time the server should give some <Item>s:

<?xml version="1.0" encoding="UTF-8" ?>
<TiVoContainer>
  <Details>
    <Title>Top</Title>
    <ContentType>x-container/tivo-music</ContentType>
    <SourceFormat>x-container/tivo-music</SourceFormat>
    <TotalItems>2</TotalItems>
  </Details>
  <Item>
    <Details>
      <Title>Flac</Title>
      <ContentType>x-container/folder</ContentType>
      <SourceFormat>x-container/folder</SourceFormat>
    </Details>
    <Links>
      <Content>
        <Url>/tivo.cgi/25</Url>
      </Content>
    </Links>
  </Item>
  <Item>
    <Details>
      <Title>dsongs</Title>
      <ContentType>x-container/folder</ContentType>
      <SourceFormat>x-container/folder</SourceFormat>
    </Details>
    <Links>
      <Content>
        <Url>/tivo.cgi/26</Url>
      </Content>
    </Links>
  </Item>
  <ItemStart>0</ItemStart>
  <ItemCount>2</ItemCount>
</TiVoContainer>

The TiVo will now show a menu with 2 items (“Flac”, and “dsongs”). If you select “Flac” it will send a new query the same as the the first but to /tivo.cgi/25?Recurse=No&... etc.

So far I’ve only shown <Item>s that were folders. Here’s what a song looks like:

  <Item>
    <Details>
      <Title>Way Out -></Title>
      <SongTitle>Way Out -></SongTitle>
      <SourceBitRate>320</SourceBitRate>
      <AlbumTitle>The Middle Of Nowhere</AlbumTitle>
      <ArtistName>Orbital</ArtistName>
      <AlbumYear>1999</AlbumYear>
      <ContentType>audio/mpeg</ContentType>
      <SourceFormat>audio/mpeg</SourceFormat>
    </Details>
    <Links>
      <Content>
        <Url>http://10.0.0.1:8080/db/730f0108/01-iTunes-320.mp3</Url>
      </Content>
    </Links>
  </Item>

Note that the URL can point anywhere–to another port on my server in this case (running Apache::MP3 plus some transcoding extensions I wrote–that’s why the directory is labeled “Flac” but we’re pulling mp3s out of it).

I don’t show it in this case, but there are also <Duration> and <MusicGenre> tags. I don’t know if they are order dependent or not, but I copied the TiVoDesktop server which puts them after <SongTitle> and <AlbumYear> respectively.

The next thing that can happen is if you scroll through a list that has more than 8 items in it. When it needs to get the next set of items it issues something like this:

/tivo.cgi/22?Recurse=No
&Filter=x-container%2Ffolder,x-container%2Fplaylist,audio%2F*
&SortOrder=Type,Title
&AnchorItem=%2Ftivo.cgi%2F3
&AnchorOffset=-1
&ItemCount=1
&Details=Basic
&Format=text%2Fxml

So they added 2 new parameters: AnchorItem and AnchorOffset.

<rant>Whoever made up this part of the protocol was drunk. Notice that AnchorItem is given as a URL instead of an offset. This means if you have a list of <Item>s you have to search through them to find the URL of the AnchorItem. You could use a hash table to efficiently look for it but then you still need the original list so you can get the next 7 items. So for no good reason (that I can tell) this part of the protocol complicates your life. </rant>

Anyway, once you’ve found which <Item> the AnchorItem is referring to, you need to iterate AnchorItem by AnchorOffset + 1. Yes, +1. That’s why they almost always pass -1 as AnchorOffset. Lovely.

One other “interesting” thing: ItemCount can be negative. If you scroll up through a list it’ll start sending you negative counts. I deal with this like this:

if ($count < 0) {
    $start += $count;
    $count = -$count;
}

I also then make sure to clip $start and $count so they are within the list of items on that container.

When you select a track to play the TiVo will ask for details on just that item like this:

/tivo.cgi/4?Command=QueryItem
&Url=http%3A%2F%2F10.0.0.1%3A8080%2Fdb%2F730f0108%2F01-iTunes-320.mp3

The response looks like this:

<?xml version="1.0" encoding="UTF-8" ?>
<TiVoItem>
  <Item>
    <Details>
      <Title>Way Out -></Title>
      <SongTitle>Way Out -></SongTitle>
      <SourceBitRate>320</SourceBitRate>
      <AlbumTitle>The Middle Of Nowhere</AlbumTitle>
      <ArtistName>Orbital</ArtistName>
      <AlbumYear>1999</AlbumYear>
      <ContentType>audio/mpeg</ContentType>
      <SourceFormat>audio/mpeg</SourceFormat>
    </Details>
    <Links>
      <Content>
        <Url>http://10.0.0.1:8080/db/730f0108/01-iTunes-320.mp3</Url>
      </Content>
    </Links>
  </Item>
</TiVoItem>

That’s just a standard <Item> wrapped in a <TiVoItem> tag.

That is enough to set up a hierarchy, have the TiVo be able to maneuver around and play music. I wanted shuffle though so I enabled it in the TiVo expecting it to just jump around in the folders randomly. Nope. It sends this:

/tivo.cgi/1?Recurse=Yes
&Filter=audio%2F*
&SortOrder=Random
&RandomSeed=1172006919
&RandomStart=http%3A%2F%2F10.0.0.1%3A8080%2Fdb%2F0511ce12%2F02-iTunes-320.mp3
&AnchorItem=http%3A%2F%2F10.0.0.1%3A8080%2Fdb%2F5f103b07%2F04-iTunes-320.mp3&ItemCount=1
&Details=Optimal
&Format=text%2Fxml

RandomSeed and RandomStart are constant once you’ve started some music playing with shuffle enabled on the TiVo. Essentially It sends you the last song it played in AnchorItem and expects you to return a new song to play in <TiVoItem> form. For now I just key off of the SortOrder parameter and give a random item from the folder I’m in. This isn’t a good shuffle on small playlists since it will repeat songs while ignoring others, but it was simple.

In fact, the implementation I made only does the bare minimum to work. I ignore most of the parameters passed in (like Details and Filter) but it works just fine (for now).

Debugging

What helped me was pointing towards the TiVo Desktop server on my Mac and submitting test queries. I’d also always had the apache access.log open in my Emacs so I could see what the TiVo was trying to ask me. And if I didn’t understand I’d recreate the query on the Mac to see what it would do. I’ll try to post more later, especially if there is some interest.

Conclusion

I’m happy I did this. It isn’t complete but it good enough for me for the time being. It’s been working all day (I’m listening to music out of my substandard TV speakers as I write this).

Efficient JavaScript or Inefficient Browser?

I caught this article on Reddit this weekend and had definite mixed feelings about it. Many of their suggestions were obviously good ideas—calling SetTimeout() with a string is obviously stupid, as are 90% of the possible uses of eval() and Function(). The advice on what causes the DOM to repaint and reflow was interesting and informative.

It would have been a really good article except that some of their advice seemed weak. Some of what was masquerading as optimization “advice” seemed more like work-arounds for a substandard javascript implementation.

Take this for example:

var min = Math.min(a,b);
A.push(v);

These alternatives provide the same functionality, while performing better:

var min = a < b ? a : b;
A[A.length] = v;

Yes, I can see in someone’s world that might perform better. But come on, the compiler/scanner should be able to produce the equivalent code. Anything less is a failing of your browser and not an “unoptimized” program. Do you really expect people to crappify their programs because your browser can’t properly optimize javascript for you?

“Well,” I hear you say, “it’s interpreted, not compiled.” To which I say, yeah, well there’s your first problem. Fix that before telling me that I need to use stupid constructs to make my program go fast.

Here’s another example:

This example requires the script engine to create 21 new string objects, once for each time the length property is accessed, and once each time the charAt method is called:

var s = '0123456789';
for( var i = 0; i < s.length; i++ ) {
  s.charAt(i);
}

This equivalent example creates just a single object, and will perform better as a result:

var s = new String('0123456789');
for( var i = 0; i < s.length; i++ ) {
  s.charAt(i);
}

Again, this strikes me as being a failing of the compiler. I see no reason why the browser should instantiate a string object just to call its length(). A little bit of hard coding in your implementation can optimize that to a strlen() call or better—No object instantiation at all. What if someone overrode String.prototype.length()? That’s one if statement in C and some fallback code. Again, I don’t think my program should suffer because your browser doesn’t properly optimize things.

But then again, maybe I’m just crazy.

Calling Applescript from Perl

First off, I don’t like Applescript. In fact, I hate it with a passion. You would think the Apple would learn from Cobol that natural language programming languages just suck. But no, Apple created the abomination called Applescript anyway and foisted it upon us and now sometimes you just have to suck it up and use it. Preferably from a distance with gloves on (and maybe a 10 foot pole) so that it doesn’t dirty your soul too much. But I digress.

I needed to call some Applescript from perl and was quite proud of my end result:

sub osascript($) { system 'osascript', map { ('-e', $_) } split(/\n/, $_[0]); }

The usage looks like this:

osascript <<END;
 tell application "Finder"
 display dialog "Hello"
 end tell
END

So basically the Applescript sits right inside your perl program and gets launched without needing any temporary files on the disk.

How does it work?

If the perl line is incomprehensible to you, let me break it down. The inner most expression is:

split(/\n/, $_[0]);

That takes the first argument to the function, breaks it apart at the line endings and puts the parts into a list. If we were to dump the result of the split using the dialog example above it would look like this:

(' tell application "Finder"', ' display dialog "Hello"', ' end tell')

Next, it uses the map function to insert “-” before every item in the list. It does that by using the fact that perl flattens arrays. The dump of the output of the map function conceptually looks like:

(('-e', ' tell application "Finder"'), ('-e', ' display dialog "Hello"'), ('-e', ' end tell'))

but really perl immediately flattens it to:

('-e', ' tell application "Finder"', '-e', ' display dialog "Hello"', '-e', ' end tell')

At that point it calls the system function with “osascript” and that last array as arguments. When system is passed a list instead of a string it runs the command with the arguments directly instead of launching a shell. This way all the arguments are effectively quoted correctly but without ever having to worry about quotes. This is an extremely clean way of launching commands. You really should never pass system (or open) a string: (a) it needlessly launches a shell, and (b) you have to worry about quotes.

In the end, I like that it fits on one line and that it lets the Applescript sit in the program in its native form.

Edit: Also check out my Ruby version.

Darwin Ports Pextlib Problem

I decided to try Darwin Ports out today. I’ve used Fink in the past but I hate the fact that their unstable distro is source only. Try installing anything on fink and not having TeX build for an hour. So reacting on the bad taste that fink always leaves me with I set out installing Ports from source (since I want it in /usr/local/ports and not crappy “/opt”).

Everything appeared to install with no errors but when I’d run “port” I’d get this:

$ port sync
can't find package Pextlib 1.0
    while executing
"package_native require Pextlib 1.0"
    ("eval" body line 1)
    invoked from within
"eval package_native $args"
    (procedure "package" line 14)
    invoked from within
"package require Pextlib 1.0"
    (procedure "dportinit" line 310)
    invoked from within
"dportinit ui_options global_options global_variations"
Error: /usr/local/ports/bin/port: Failed to initialize ports system, can't find package Pextlib 1.0

Web searching turned up nothing helpful

After playing around and brushing up on my Tcl I discovered that /usr/local/ports/share/darwinports/Tcl/pextlib1.0/pkgIndex.tcl contained only a comment:

 # Tcl package index file, version 1.1
 # This file is generated by the "pkg_mkIndex" command
 # and sourced either when an application starts up or
 # by a "package unknown" script.  It invokes the
 # "package ifneeded" command to set up package-related
 # information so that packages will be loaded automatically
 # in response to "package require" commands.  When this
 # script is sourced, the variable $dir must contain the
 # full path name of this file's directory.

Seems something got screwed up in the installation process and didn’t generate the pkgIndex.tcl file correctly. On a hunch I checked and discovered I had an old PowerPC tcl sitting in /usr/local/bin (and not stowed — I hate stupid Mac installers that install into /usr/local. That is *my* directory). I deleted all the Tcl-ish things in

/usr/local/bin

(and didn’t find anything anywhere else, oddly enough), reinstalled Darwin Ports, and it worked! Yay.

Now I’ll see if it’s worth the 2 hours I spent debugging. 🙂

“Commit Patch”: Managing your mess

I consider myself to be chaotically organized. If you look at my workbench you’ll see stacks of papers and half finished projects. However, when I finally clean up I end up filing things away meticulously.

The same chaos applies to my development work. If I’m working on a bug or a new feature I end up making other random enhancements or fixes that aren’t at all related (except for the fact that they exist in the same file). Judging from other people I’ve worked with, I’m hardly alone in this behavior.

The problem comes when I’m ready to become organized and check in my work (the software equivalent of “file things away meticulously”). Version control systems don’t cater to the messy development model. Take CVS and Mercurial: They only let you commit a whole file at a time, not just pieces of it. Darcs is better, as it allows you to commit pieces of a file, but even it doesn’t let you commit just one part of a single line.

Why would anyone want to do this? Look at this made up example:

 void player_died()
 {
     game_over(score);
     char x[50];
     get_name(x);
     add_high_score(x, score);
 }

Say I need to change the way get_name() works—I need to make sure I don’t overrun my buffer. While I’m doing that I decide that “x” is a really stupid name for the variable, so I fix that too. The code now looks like:

 void player_died()
 {
     game_over(score);
     char name[50];
     get_name(name, sizeof(name));
     add_high_score(name, score);
 }

Fine. But now I have 2 unrelated changes that touch the same source line. In the past I’d just check them both in with a comment like this:

 - Pass in length with high score string storage.
 - x -> name for clarity.

Enter commit-patch With commit-patch I now can precisely specify the patch I want to commit and leave the rest behind as uncommitted changes in my working directory.

Ok, so how does it work?

First I use emacs to get a diff (I make sure I have an emacs library for whatever version control system I’m using). Here’s an example of what darcs might show me:

 diff -rN -u old-test/game.c new-test/game.c
 --- old-test/game.c     2006-08-22 23:09:44.000000000 -0700
 +++ new-test/game.c     2006-08-22 23:09:44.000000000 -0700
 @@ -1,8 +1,8 @@
  void player_died()
  {
      game_over(score);
 -    char x[50];
 -    get_name(x);
 -    add_high_score(x, score);
 +    char name[50];
 +    get_name(name, sizeof(name));
 +    add_high_score(name, score);
  }

Emacs has diff-mode which makes editing patches trivial. I now edit it down to this:

 --- old-test/game.c 2006-08-22 23:11:52.000000000 -0700
 +++ new-test/game.c 2006-08-22 23:11:52.000000000 -0700
 @@ -1,8 +1,8 @@
  void player_died()
  {
      game_over(score);
      char x[50];
 -    get_name(x);
 +    get_name(x, sizeof(x));
      add_high_score(x, score);
  }

Now I can check this in with commit-patch (along with the corresponding change in the get_name() function’s file). Since I’m using emacs I can just use the “C-c C-c” key sequence in any diff-mode buffer which runs commit-patch for me. Or I could save the patch out to a file and run commit-patch from the command line. Either way, it commits only what was in the the patch and leaves the working directory effectively untouched. To demonstrate, my diff after I’ve committed the above patch looks like this:

 diff -rN -u old-test/game.c new-test/game.c
 --- old-test/game.c     2006-08-23 14:58:54.000000000 -0700
 +++ new-test/game.c     2006-08-23 14:58:54.000000000 -0700
 @@ -1,8 +1,8 @@
  player_died()
  {
      game_over(score);
 -    char x[50];
 -    get_name(x, sizeof(x));
 -    add_high_score(x, score);
 +    char name[50];
 +    get_name(name, sizeof(name));
 +    add_high_score(name, score);
  }

Now I can check in the file again, this time for just the variable name change.

commit-patch allows you to keep your commits short and to the point instead of forcing you to check in a bunch of unrelated changes just because they happen to be in the same file (or the same line). It relies on you editing patches which I admit can be scary at first, but after a few times you really start to get the hang of it. I now always look over the diff of my uncommitted changes and weed out stupid changes in my code—debug prints, commented out dead code, and all the other junk I shouldn’t check in.

Ok, sounds great, what are the downsides?

Glad I asked. Checking in patches effectively means checking in untested code. Despite your best intentions, you can screw up the code when editing a patch. Generally the projects I work on where this really matters all have internal infrastructure to make sure builds work. One project I am involved with that uses CVS has an auto-builder to check for broken code. Another large darcs project I deal with requires ‘make’ to run successfully before it accepts a patch. Both of these provide good enough insulation for bad patches. The truth is, I’ve been using commit-patch for a few years now and I rarely have accidental errors checked in that were caused by bad patch editing. I would say I’m more likely to forget to check in a file completely than I am to check in a bad patch.

The only other downside is that it requires you to be able to edit patches. I use emacs which makes it pretty trivial. If you are not an emacs user then you’d have to look around and see what other patch editing options are out there. I honestly don’t know what other tools exists. So if you consider needing to use emacs a downside, then there you go. If you don’t, well then I salute you. 🙂

Conclusion

commit-patch fits my development style like a glove. I frankly cannot imagine working on a project without it any more. That is an interesting statement because before we wrote it, Jim and I felt that it would be a script we would use only occasionally. But once I started using it I found more and more situations where it would be useful.

I think commit-patch is one of those tools that you don’t realize you need until it suddenly becomes indispensable. So do yourself a favor and try it out for a week or two.

Get commit-patch here

Closures In Straight C

I’ve read a lot lately about closures. I’ve read articles where people wonder what they are, others where someone is lamenting the fact that they only recently discovered them. I too only recently discovered the joy of closures. I’ve been writing a lot of javascript recently and I love the way writing anonymous callback functions works. But it turns out I have already been using closures for a long, long time.

A friend and I were discussing C (our language of choice for embedded programming) and the things we like about it and somehow the subject of closures came up. I started thinking about what it would take to implement closures in C. Would you do it in the compiler as a language extension or just build them up by hand using in C itself in a cross-platform way?

And then it struck me: I’ve already been using, in straight, boring old C, the concept of closures. And I’ve been using them practically forever. I would say almost since I first learned C on my Macintosh back in high school.

The original Mac API (which still mostly lives on in Mac OS X’s “carbon” API) has an element in almost every major structure called “refCon”. The Apple documentation sorted of glazed over them (or maybe I glazed over when reading about them) and so I puzzled over these for a long time until I realized that they were pieces of memory that the Mac system would never look at or change. They are for the application writer to use as they see fit.

Apple also uses them when you register a function that the system will call at some later time. Here’s an example I grabbed from the current carbon window manager documentation:

OSStatus InstallWindowContentPaintProc(WindowRef window,
                                       WindowPaintUPP paintProc,
                                       WindowPaintProcOptions options,
                                       void *refCon);

What might you use one for? Well typically you’d create a structure with whatever local data needs to be associated with the call back function and put the pointer to that structure into the refCon. The callback function (since you’re writing it) knows to cast the refCon to the appropriate structure pointer so that it can get access to the fields of the structure.

This is by no means some great thing that Apple discovered. Almost every C API that has callbacks uses the same sort of concept. Here’s a function prototype from glib (one of the gnome base libraries):

guint g_io_add_watch(GIOChannel *channel,
                     GIOCondition condition,
                     GIOFunc func,
                     gpointer user_data);

Here “user_data” serves the same purpose as “refCon” did in the first example. The function you pass in “func” will get “user_data” passed to it (unmolested) by glib.

I’ve used this in almost every API I’ve ever written that has callbacks. Here’s another example from an embedded project I’m working on:

void add_timer(int (*timer)(), void *cookie, int timeout_ms);

I always name my refCons “cookie”, for some reason. Arguably “user_data” is a more descriptive name, but I sort of like the arbitrariness of “cookie”. Or maybe it just makes me think of cookies… MMMM… Cooookies…

So when I learned javascript and started coding Green Felt I wrote a little XMLHttp() wrapper to make the interface nicer (like every other ajax developer on the planet) and I put in a cookie. Later I realized that everything I was passing in the cookie was a local variable and therefore was already saved in the closure. So passing it explicitly through my ajax function was pointless.

It turns out that all those “cookie”, “user_data”, and “refCon” pointers that I had been using since I first learned C transform the user supplied function pointers into “closures”. They do it explicitly and without any syntactic sugar, but they are 100% the same conceptually. In effect you allocate some memory, put references to some of your local variables into the structure and then pass the structure along with the function pointer to be called back later. The callback can use the structure all it wants which ends up being equivalent to a real closure having direct access to those same local variables.

So… Closures aren’t anything new. C has had closures forever. Yes, you have to manually allocate and deallocate the memory for your variables without syntactic sugar… But, hey, that’s C!

3 big problems with Javascript

For years I had an irrational loathing of javascript. It was to the point where I’d always have it switched off in my browser. To me it was synonymous with bad web design and crappy menus that didn’t ever work in netscape. Then google maps came out and I was completely impressed. They did all that with javascript? I didn’t know javascript could *do* that. So about a year ago I decided to actually look at the language and see what you could do with it. I updated an old cgi script game of mine into a modern javascript game complete with XMLHttpRequest high score stuff. And I was generally very happy with everything. My friend Jim wrote a card game and we developed it into a very nice game site

I really love javascript now, but that doesn’t mean it doesn’t have its faults. I can make a number of little nitpicks but I want to focus on three major deficiencies of the language.

Problem 1: No include directive

This is particularly annoying when dealing with many libraries that also depend on other libraries. It forces you to know all the dependencies up front and hurts the modularity of your site.

I’ve read that an include directive was not ..er.. included in the language because they didn’t want to specify what it meant. The argument goes like this: Javascript is a nicely generic language and so if it was used as a scripting language of a video game (for instance) it would be annoying if the spec said “include” was a URL. I don’t see why that precludes adding the include directive to the language. I would argue that it’s better to have it in the spec with the argument being “implementation defined” and let the w3c guys say that it’s a URL and the game guys make it a virtual file inside some zip archive.

The other argument I’ve heard is that the language designer wanted something nicer than a simple “#include” style of include—something like the class importing of java or a “use” directive of perl. I can understand that. But I would argue that not having anything is much worse than having a simple file/URL based include system.

A third argument I’ve heard is that including files is gross since you have to protect against multiple inclusions with big ugly protection “if” statements like every C header file you’ve ever seen. I don’t buy this either—Objective C solved this 20 years ago. Objective C has “#include” and “#import”. “#include” includes the file verbatim and “#import” includes it only if has not been imported before. It’s very simple and it covers all the cases that might occur.

Problem 2: “this” is wrong in callbacks

Javascript programs, being functional and non-threaded, end up having a lot of callbacks. Especially if you use things like timers and XMLHttpRequest objects. If you pass a function to something and it runs it, your function gets run with all your local variables (since it’s a closure) but the “this” points to something stupid like the callee or the global context (I don’t remember but it’s not your local “this”).

I’m not sure why it does this but I imagine it makes some part of the language designer or language implementer’s life easier. The problem is that it makes every programmers life hell. Yes, it’s easy to get around: just make a local variable and set it to “this”. Look around greenfelt.net and you’ll see variable named “me” and “_this” littering the code. But I shouldn’t have to do that at all—the language should just do the right thing (maybe that’s the perl side of me talking).

My argument against this is that it is *never* what you want as a developer. *Never!* Why force every program to have stupid little work arounds just because it makes the spec of the language cleaner at some level? I think the spec writers and language implementors need to suck it up and take one for the team here.

How much time have I spent debugging a problem caused by the stupid “this” pointing to the wrong context? Days, I’m sure.

Problem 3: Inheritence sucks

The javascript object model is simple. I generally like it. But its simplicity causes problems when trying to do inheritence. I looked around for ways to do inheritence and everyone seems to have a different way of doing it. And all the methods seem to have some drawback. In the end I ended up settling on a method that seemed to minimize code, but it’s still ugly:

The base class:

function baseclass(a,b) {
    if (arguments.length) this.init(a,b);
}
baseclass.prototype.init = function(a,b) {
    ...
}

And the inheriting class:

function subclass(a,b) {
    baseclass.prototype.init.call(this,a,b);
    ...
}
subclass.prototype = new baseclass();

This allows for “new baseclass(a,b)” and “new subclass(a,b)”. But there’s a lot of scaffolding there. It really needs some good syntactic sugar.

From what I’ve read they’re trying to rectify this in the new version of javascript being developed.

I can only hope they also fix the other 2 problems.

Addendum

[Mon Mar 6 01:09:49 PST 2006]

Thanks to a comment from Pasha I’ve been experimenting with a different way of working around problem #2 using a the apply() function:

function wrap(context, f) {
    return function() {
        f.apply(context, arguments);
    };
}

Usage:

div.onclick = wrap(this, function() { ... });

I’m liking this better than my previous work around, the “var me=this” closure variable trick. It’s better because it actually makes “this” be correct and so the inner code doesn’t have to use a different variable name. The downside is you have to pass “this” into the wrap() function—it would be better if it could climb up the call chain and get “this” out of whatever context came before it.

Email: david+spam@porkrind.org
Last Modified on: Dec 31, 2014 18:59pm
Copyright © 1995-2023 David Caldwell