webloc shortcuts into DEVONthink Web Archives - python + automator + applescript

Recently I've been moving forward with my year-or-so long quest to organize my thoughts, readings and writings, and their associated notes, in a searchable format suitable for aiding me in deciding the order in which I should attack my ideas, and suitable for presenting me with relevant entries as I work through the execution of my ideas. To store and retrieve my documents I've decided to give DEVONthink a shot. A lot of searching, and a little bit of evaluating, went into this decision. I'm not 100% sold that it will meet my needs, but seems fit in with them pretty nicely. More on that another time, if I get around to it.

The Problem

In the past year I've accumulated 500 or so URL shortcuts on my Mac OS X desktop (more precisely, files with the extension .webloc that contain URLs in their resource forks). These were created by me dragging bookmark-like strings from Safari or Firefox to the desktop, ostensibly for future consideration and collation.

So, I have a collection of 500 files, each of which contains a URL, and I want each of those to become individual Web Archive entries in DEVONthink. This is versus its other record types that could be appropriate (PDF, URL, others).

A Solution

DEVONthink has no mass import feature that solves this problem exactly. Fortunately, and this is one of its outstanding features, DEVONthink is scriptable through AppleScript. In fact, there is a context-sensitive script "Add web document to DEVONthink" which appears in the menu bar's Scripts menu when Safari or Firefox are in the foreground. The contents of this script (~/Library/Scripts/Applications/Firefox/Add web document to DEVONthink.scpt) reveal how to send a URL, title and refering URL to DEVONthink for it to create a new record of the Web Archive type.

I then googled how to get the URL stored in a .webloc file as a string. I'm a Python programmer, and wanted to see how to do it in python and not AppleScript, so I ended up at http://toxicsoftware.com/webloc-to-pinboard/. The function infoForWebloc() does the trick.

I hadn't used Automator before so instead of tying together the AppleScript and Python directly I decided to make this my first foray into Automator. I created a Folder Action on a new folder (~/Desktop/url2devonthink/) with two actions: Run Shell Script and Run AppleScript.

The Shell Script portion executes a Python program to extract the URL and retrieve a suitable Title.

Shell: /bin/bash
Pass input: as arguments



Contents of ~/bin/webloc2url.py is as follows. I've removed some optimizations for clarity and brevity:


The AppleScript passes that on to DEVONthink for Web Archive creation. Note that I cut-and-pasted most of this from ~/Library/Scripts/Applications/Firefox/Add web document to DEVONthink.scpt.


Conclusion

Some caveats apply:

• The above was almost certainly not the most optimal way of accomplishing the goal
• The triple-pipe Cheap Hack (tm) was my lame workaround to not wanting to see how multiple arguments are passed between stages in Automator's pipes
  
• The actual process was iterative and error-prone. Some URLs had become bad, some I didn't want to include, and so on.
• Since this was a one-time task I have no need to create a more structured and reliable program than the above.
  

Now all I have to do is catalog all those new entries!

Why write essays? Why write essays concisely? Why write well-reasoned essays concisely?

I've been thinking a lot about writing recently. Specifically, I've been considering:

• What are my intrinsic motivators for wanting to write more? How about extrinsic?
• What's the extrinsic ROI? Intrinsic?
• Do I have the willpower to ignore something with a poor extrinsic ROI and a good intrinsic one? Should I ignore it, even if I do have the willpower?
• What about short and long term implications of the near-term decision I must make regarding spending more time on writing?

Of secondary concern to me are the topics of my writing, the format, the audiences and my writing's relevance. These are by no means unimportant. Indeed, they inform the answers to the questions I mention above, but they have not been the primary variables under consideration.

This morning I spent about 15 minutes and wrote a thousand or so words in a stream-of-consciousness format. I wrote about topics I wanted to write about related to my position at Cornell. I've been finding recently that I've been having a specific kind of communication problem with my staff. That is, with some members, it has become obvious to me that I'm assuming too much preexisting knowledge, or that I've been assuming that I've communicated that knowledge in the past. It is clear that I haven't, or didn't, communicate well.

Therefore, I'd like to start building up a library of essays on various topics related to what I do professionally. I'd like to use that exercise to both understand how to communicate my desires more effectively, and to provide a set of documents for future reference.

This got me to wondering: is a collection of short essays more valuable in this context than a collection of detailed compositions with reasoned arguments and citations? Academia has historically eschewed the blog entry as a medium for developing and delivering reasoned arguments, in favor of the well cited scholarly article. The writing I'm considering is not scholarly, but perhaps the motivations behind requiring arguments to be well considered and cited still apply? Is there room for scholarly-style writing in the realm of short essays on such mundane topics as "Why We Check In Our Code Periodically"?

I have several thoughts, but in the spirit of short essays without well reasoned, substantiated arguments and conclusions, I figure I should publish this entry before I add any of them!

Patching Mercurial and TortoiseHG for CUWebAuth

For a few months now I've been using a modified version of Mercurial (hg) to push/pull/clone repositories protected with Cornell's CUWebAuth. CUWebAuth is a single sign-on system implemented with Kerberos 5 which is primarily used to allow non-Kerberos-aware web browsers to use a trusted site (something like a web front end for kinit) to generate Kerberos credentials and store them in a cookie. The upshot of this arrangement is that an end user's credentials (username and password) aren't passed to the service needing the identifying information, but instead are only sent to the trusted server. The trusted server sends back Kerberos credentials to the browser, which sends them to the service needing the identification.

Ideally, I'd generate Kerberos credentials in python at the client (hg/tortoisehg), but I don't yet have the code in place to do that. What I did build was a urllib2 RedirectHandler which observes the CUWebAuth2 HTTP redirect, prompts the user for their username and password, sends these to the trusted server, acquires the Kerberos credentials and then replays the original HTTP request. A combination of Python features allowed this to be done relatively simply. Its simple, consistent module and namespacing system and the fact that it is interpreted allowed me to replace a function in one of hg's core libraries at runtime:

import mercurial.url
def opener(ui, authinfo=None):
   ....
   handlers.extend((urllib2.HTTPBasicAuthHandler(passmgr),
      mercurial.url.httpdigestauthhandler(passmgr),
      cornell.auth.CUWA.CUWARedirectHandler(passmgr, cookiejar)))
   opener = urllib2.build_opener(*handlers)
   ....
   return opener

It works even though it isn't the ideal solution (that being generating credentials locally). Cornell folks can get a copy of this library, and the prebuilt executable cuhg which you use in place of hg, from http://code.cals.cornell.edu.

One problem with this solution is that not all of my team members use the command line version of Mercurial. Some use TortoiseHG on Windows. Before I could deploy CUWebAuth protected Mercurial HTTP repositories for my group I needed my CUWebAuth patch to work with their client.

I spent about 10 hours over the past few months getting the TortoiseHG installer to build in one of my Windows VMs. It has a lot of dependencies (Free downloads), but is documented reasonably well with only a few rough edges. Most of that time was spent really getting to understand each step of the build process including PyGTK, Inno Setup, py2exe and several other tools. In the end I was able to build a version of the installer for TortoiseHG 0.8.1 with Mercurial 1.3.1 which functioned properly on a fresh Windows install.

But, it turns out I didn't need to go through all that hassle! The GUI components of TortoiseHG interact with Mercurial by behaving as if one were executing hg commands at the command line. As part of the build process py2exe compiles .pyc files of all the project's dependent libraries (including Mercurial itself) into a zip file called Library.zip. Adding Library.zip/mercurial/CUWA.pyc and updating Library.zip/mercurial/url.pyc with my modified opener() function was all it took to get my CUWebAuth redirect handler injected into TortoiseHG's HTTP client library!

I'm reminded of the old tale of the engineer and the factory. There's this factory which produces special boxes for packing delicate scientific equipment. Every hour the manufacturing line is not running the company loses thousands of dollars. They have staff on-site for dealing with day-to-day cleaning, lubricating and other maintenance tasks, but when something really goes wrong they have to call up the company that built the machine for support.

So one day, after the regular maintenance has been performed for that day, the machine wouldn't start up again. The foreman is stressing over how much money they're losing by not being able to operate so he calls the manufacturer to send over an engineer. The engineer inspects the line, then walks over to a maintenance panel. She opens it and turns an adjustment screw a quarter turn. The line starts back up and everything is back to normal.

A few weeks later the foreman gets a bill for $10,000. Furious, he calls up the engineer and asks why it cost $10,000 for a half hour of her time. He demands an itemized bill. A few weeks later the itemized bill arrives. It has two line items on it:

.5 hour labor: $150
knowing where to put the screwdriver and turn: $9,850

The point is that sometimes knowing exactly where to turn the screwdriver can be valuable both in terms of time saved and money spent.

What am I doing wrong?

So I got an auxiliary audio input installed in my car the other day. Excellent. I don't have to deal with the FM transceiver for the satellite radio or the iPhone anymore. They sound awful, and when I travel to populated areas I have to hunt around the crowded FM dial for weak stations I can override.

All I needed was a 1-2 meter (trying out the metric system here) cable with 1/8" (d'oh, failed already) TRS connector, a.k.a. "headphone plug"on both ends. We don't have too many options in Ithaca for procurement of such a cable. I made a valiant effort to "buy local": I asked one friend. She said Gallagher's. Unfortunately they went out of business years ago.

So, Best Buy, Radio Shack, Target or WalMart. I chose Target because I had to buy deoderant as well, and I don't have a high regard for Radio Shack brand deoderant.

I expected to pay around $5-$10 for this item. I walked into the electronics section of the store and immediately turned right into the first aisle. It had iPod accessories. I figured "hey, I'm connecting an iPhone, I bet this is where they'll keep the cable." Among the various incarnations of iPod-specific adaptors I found what I was looking for. Unfortunately, it was a Monster brand "iPod to Aux Input" 6 foot cable, with gold plating priced at $20.

No way! Directly above that they had a slot for another brand name "iPod to Aux Input" cable, priced at $14.99, but it was out of stock. Hmmm... still too much. I can do better. I turn around and see the MP3 player section.

Same $14.99 cable. This time labeled "MP3 player to Aux Input" cable. They had plenty of them. So now it's a game. What happens when I go find the portable CD player section? Does it get cheaper?

You bet! Same brand again, same cable, now $9.99, just labeled "Portable CD player to Aux Input". Now it is in my price range but I'm convinced I can still do better at this same store.

I go down the next aisle and find the generic Target brand cable section. They have all sorts of audio cables. RCA connectors. Cable TV connectors. HDMI. Just about everything. And there's the headphone jack cable. In a box that, essentially, says "connects Stuff to Other Stuff".

$3.69, including gold plating.

People are MAKING MONEY on people who DON'T KNOW WHAT THEY'RE DOING. Target is pulling off price discrimination on four things that do the same thing within 3 meters of one another! I'm in the wrong line of work!

And now you understand this post's title.

A Passive Infrared Motion Sensor Repeater/Extension?

The light switch for my office appears to be a SuperSwitch 2 passive infrared light switch which, according to the site, is

a motion sensing lighting control and conventional wall switch all-in-one

Great! I don't have to turn it on, and it turns off when I leave. What could be wrong with that? Nothing, I say!


But wait. Where did it get installed you ask? Is it behind the door?

D'oh! Yes, that's where it got installed.

FAIL.



If I leave the door completely open it can't detect my presence, and the light goes off after a bit. If I close the door people won't feel comfortable coming in. If I leave it halfway open (so the sensor can "see" me) visitors will invariably open it the rest of the way.

So, the question is, how to work around this? Relocation of the switch assembly is out of the question for a number of reason.

I can think of several options, some of which are "hard" and some of which are "dumb":

• Hack the sensor by removing the actual sensor from the switch assembly and running wires to where it could actually see me.
• Rig up some kind of secondary sensor attached to a heat source that can simulate body heat and movement that can be placed within the sight lines of the room's real sensor.
  
• Cut a sizable hole in the door.
• Deal with it.
  

I'd rather not do a, c or d. If you know of something that will do b, with the door completely open, and without annoying me (like a random incandescent strobe of sorts), I would be somewhat indebted to you!

now named blog - silent platform

On my way back from Boston and the #HEtweetup I was listening to some Rage Against the Machine. The lyrics to the song included the phrase "silent platform".

While the socio-economic-political issues in which I am interested aren't quite the same same as those covered in Township Rebellion, I'd still like to think that understanding their reality is critical to the long-term viability of academia and higher education, and that I can use this blog to publicly (Creative Commons-licensed) refine those thoughts, silently.

Thanksgiving won't be the same this year

For most of my life my mother's mother hosted Thanksgiving supper for their three children and their families. I was at them all until I moved to California in 1998, though I returned for a few after that. She stopped hosting them a few years ago due to declining health. I've joined her for several Thanksgivings since, either out or at another relative's home.

Well, I think I need to be more accurate. I was at almost all of them. I missed my first (in 1975) because I was in the hospital that day, having been born that morning.

Even when I wasn't there I could taste the special stuffing she would make, or the kielbasa she would serve on occasion. In fact, I can taste it right now even though I'm eating a chicken dish baked with potatoes, garlic and peas, which tastes nothing like home-made turkey stuffing complete with sausage bits.

This year, and every year from now on, I will still have those memories. But, she will only be joining me in spirit. She passed away early this morning in her sleep. She had been in declining health for several years with heart problems, but had been doing well recently. The news came as a surprise to me and others. Some good news is that she passed in her sleep at home, and not in a hospital bed.

Goodbye Grandmom, you will be missed.