Testing Adobe AIR applications with QUnit

I’d like to say I always write tests before I write any JavaScript code… Those times when I do write tests, I use the QUnit framework, which is used to test jQuery, so pretty well-tested itself.

I’ve recently been putting together an Adobe AIR application using HTML, CSS and JavaScript (aka Web Standards), which is very empowering and everything, but turned out to be a little tricky to write tests for. All the interesting stuff I wanted to do through AIR’s window.runtime object, such as opening native windows and messing with dock icons, is missing from the browser if you’re trying to test the JavaScript components in isolation. The ideal would be able to run your QUnit tests within the AIR runtime itself.

Fortunately, it’s straightforward to create a test app that wraps the QUnit test runner in the AIR runtime and exercises your code. Here’s how I got going:

Setting up the file structure

The folder structure I am using for the AIR app is like this:

/myApp
   myApp.html
   myApp-app.xml
   /js
      myApp.js
   /css
      styles.css

The myApp-app.xml file is the “application descriptor” that AIR uses to create the .air package when you build the app.

I added two folders at the same level as /myApp, to contain the test framework and test code; the complete structure looks like:

/myApp
   myApp.html
   myApp-app.xml
   /js
      myApp.js
      AIRAliases.js
      jquery-1.3.2.min.js
   /css
      styles.css

/myAppTests
   runner.html
   tests.js
   myAppTests-app.xml

/qunit
   testrunner.js
   testsuite.css

Configuring the test app

The application descriptor for myAppTests is not very different from the descriptor for myApp; here they are:

myApp-app.xml:
<?xml version="1.0" encoding="UTF-8"?>
<application xmlns="http://ns.adobe.com/air/application/1.5">
    <id>examples.html.myApp</id>
    <version>0.1</version>
    <filename>myApp</filename>
    <initialWindow>
        <content>myApp.html</content>
        <visible>true</visible>
        <width>400</width>
        <height>800</height>
    </initialWindow>
</application>

myAppTests-app.xml:
<?xml version="1.0" encoding="UTF-8"?>
<application xmlns="http://ns.adobe.com/air/application/1.5">
    <id>examples.html.myAppTests</id>
    <version>0.1</version>
    <filename>myAppTests</filename>
    <initialWindow>
        <content>myAppTests/runner.html</content>
        <visible>true</visible>
        <width>400</width>
        <height>800</height>
    </initialWindow>
</application>

The main structural difference is that myApp-app.xml has an initialWindow.content property set to myApp.html, whereas myAppTests-app.xml refers to the test runner from a directory above: myAppTests/runner.html. This is needed because the test app needs to have its working directory set to the top-level of the file structure, otherwise runner.html won’t be able to get at all the files it needs in other directories (more below).

Putting the correct files in runner.html

Here’s a runner.html that works with the folder structure described above:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
		<title>Test Suite</title>
		<link rel="stylesheet" type="text/css" href="../qunit/testsuite.css">
		<script src="../myApp/js/AIRAliases.js" type="text/javascript"></script>
		<script src="../myApp/js/jquery-1.3.2.min.js" type="text/javascript"></script>
		<script src="../qunit/testrunner.js" type="text/javascript"></script>
		<script src="../myApp/js/myApp.js"></script>
		<script src="tests.js" type="text/javascript"></script>
	</head>
	<body>
		<h2 id="banner"></h2>
		<h2 id="userAgent"></h2>
		<ol id="tests"></ol>
		<div id="main"></div>
	</body>
</html>

Running the test suite in debug mode

The easiest way to run the test suite is to use the built-in Adobe Debugger, which lets you test your app without having to build it into an AIR package:

adl myAppTests/myAppTests-app.xml .

Don’t forget the ‘.’ character on the end – this tells the debugger to run with the top-level directory as the working directory – without it, the default is the same directory as XML file.

A short jog with Eddie Izzard

About 1:45 yesterday afternoon, Nick and I stood in the rain near Stratford station, waiting to see if Eddie Izzard would run round the corner, so we could join him for the last 10 miles of a marathon finishing in Trafalgar Square. A marathon is never something to be sniffed at, but in this case, it was Eddie’s 43rd marathon in just 51 days, and the final leg of more than 1100 miles running around the British Isles for Sport Relief. (It’s not too late to donate! £200k so far…)

I had my Flip camera with me and took some videos as we ran round. Some of them are a little shakey (some of them are nausea-inducing). Nevertheless, I’m happy to have a record of Eddie’s gutsy sprint down the Mall for a 5-hour personal best, knowing literally how far he’d come but only guessing what must have been going through his mind.

Meeting Eddie

As we waited in the rain for Eddie to get to Stratford, I couldn’t help but notice that he has over 1 million Twitter followers.

Pit stop

You run 1000 miles and your crew can’t even keep up…

Eddie meets a fan

It was amazing how many people shouted encouragement or beeped their horns as we ran through London. One guy actually leaned out of his van to give Eddie a donation.

The rickshaw

The other side of the lens, this guy pedaled the camera crew smoothly through London traffic, displaying some fine maneuvers weaving between cones.

Catching up

Quite early on, I ran in the opposite direction to tell the producer we were moving on. I didn’t realise Eddie was going to set such a punishing pace all the way from the Olympic stadium to Aldgate East, where I finally caught them.

A chat with a doctor

Eddie’s put up pictures of his battered feet (I’m not providing a link!) to show the physical effects of all his running, but this doctor thought his shins must be taking a serious beating. Eddie did admit to all the muscles being “badly bruised”. Ouch.

Legging it

Eddie had an unnerving habit of picking up the pace just as you were getting comfortable. You wouldn’t know he’d been running for 50 days…

Over London Bridge

The traffic was blocked off from our side of the road and a helicopter flew overhead – it felt a lot like we were in a 5-man London Marathon.

Approaching St. Paul’s

Could seeing St. Paul’s and knowing there was only three miles left to go failed to have had an uplifting effect?

A new supporter

Danny joined at St. Paul’s. I was really expecting there to be a lot of people running in support, but there was just the three of us. I guess the torrential rain made a difference…

Embankment

Pit stop – the camera-crew needed directions once again.

South Bank

Eddie ran past the crowds by the London Eye and straight into a tourist, providing them with a pretty good story to tell back home.

Parliament Square

More speed from Eddie as St. James’ Park swung into view and we neared the last mile.

Minor drama

You can’t run 1100 miles and not have your sprint-finish captured on film, right? So where are the crew?

Buckingham Palace

Crew located, we sped on – just one straight, long road to go.

Down The Mall

This is where the temptation to attach “Chariots of Fire” gets very strong. I haven’t, so if you are that way inclined, please imagine it playing in your head.

Eddie’s Ices

Free ice-cream is never a bad thing.

Gracious End

Personally, I would have collapsed at this point, but Eddie had a nice long chat with the press and then hobbled down the steps of Trafalgar Square signing autographs and talking to people. He posed for some photographs with us. :)

Baselining my life (part 1, money)

Introduction

I began to seriously contemplate the idea of going freelance back in May, and realised I wasn’t very prepared. For one thing, I didn’t really know how much money I spent each month or what I spent it on. Nor did I have a good handle on how I spent my time. I wanted to see how I was behaving financially, so as to know how much I’d need to earn as a freelancer to maintain my lifestyle. I also wanted to know what made up this lifestyle to see how much time I’d have available for doing this freelance thing. I did go freelance at the end of June and carried on measuring things for a while so I’d have enough data for a meaningful result.

On the 6th June, I started keeping a record of my spending; on the 1st July, I began to keep a diary of what I was doing all day. At the end of August, I stopped measuring. This means I have four months of records of how I’ve spent money and three of how I’ve spent time.

Having this information is very useful, I think, so I’ve published all the data and analysis spreadsheets for you to play with. If you wanted to do something similar, they might be helpful. (The data are all published as Google Spreadsheets – see the links at the bottom of this post.)

Of course, it would be remiss of me, as a one-time scientist, if I were to look at data and deduce hypotheses after the fact. So, without prejudicing my judgement too much by figuring out a collection of perhaps-useful things in advance, I’ve written down some questions that I want answering and hypotheses to test for each.

Money

  1. How much money do I spend in a month?
    • Hypothesis: I think I spend £2400 a month
  2. What am I spending most of my money on?
    • I think the majority is being spent on rent, dining and boozing
  3. How much do I spend on booze?
    • I think I spend £400 a month on booze

Time

  1. How much time do I spend sleeping?
    • I think I spend 1/3 of my time sleeping
  2. How much time do I have free in a week after I’ve done all the habitual activities?
    • I estimated I have 31 hours a week to use (estimate recorded in Habits)
  3. What is the most expensive activity per time spent on it?
    • I think it is boozing (although dining will be a close second)

This post is going to cover the questions about spending money.

Constraints

  1. The Hawthorne effect – will I change my behaviour by measuring it?
  2. Not splitting money and time spending when on holiday – it just goes down as ‘holiday’

Measuring money

Answering the questions

How much money do I spend in a month?

Hypothesis: I think I spend £2400 a month
The data shows: Between May and August 2009, I spent an average of £2492.65 each month

What am I spending most of my money on?

Hypothesis: I think the majority is being spent on rent, dining and boozing
The data shows: rent 21.7%, holiday 20.6%, dining 10%, boozing 7.7%

How much do I spend on booze?

Hypothesis: I think I spend £100 a week or £400 a month on booze
The data shows: Between May and August 2009, I spent an average of £191.20 on booze each month

The data illuminates

The images below are generated from the summary spreadsheet, which contains live graphs you can play with – clicking on pie segments, for example, reveals the percentage and value attached to each.

Methods

Collecting data – iPod Touch & Notes

Online banking is a boon to the would-be data collector, as you can generally download a transaction history in csv format. Unfortunately, plenty of entries for “cash” creates a big unknown in the results. I decided to keep a record of when I spent cash during the day, what I spent it on and how much I spent. I didn’t start monitoring expenditure until the 6th May, so approximately £140 cash is unaccounted for over the four month period (I haven’t been worried sufficiently to alter any calculations to take this into account, as my monthly spending has a standard deviation of more than this).

I’d done something like this before, when I was having a crisis about where all my money was going shortly after I started working for BT in 2005. Back then, I used a pencil and notepad, and the prospect of copying all the records into a computer was so daunting that I never did. This time, I had the benefit of an iPod Touch (thanks Dad) and its “Notes” application. You can mail yourself a note, which I did every month or so.

Dealing with the data after it fell into my inbox was a matter of some search-and-replace in a text editor, to make the fields tab-separated so that they could be copy-and-pasted into Google Spreadsheets.

Analysing data – Google Spreadsheet

The first thing to do was enhance the data by tagging each expense with a chunky category like “rent” or “boozing”.

Using Google Spreadsheets to analyse an amount of information is a task with a fairly steep learning curve, since the examples for using the more complicated functions are not always that enlightening (I imagine if you are already an Excel functions master, it won’t be so hard). Nevertheless, the Google Docs forum is replete with the writings of two characters in particular – Otávio Alves Ribeiro, the Spreadsheet Ninja from Brazil, and “ahab“, the mysterious Google Docs Guru.

The performance of a Google Spreadsheet is generally quite good, although there is often slowdown in processing. I found, mainly through trial and error, that a good way to understand what is going on and get decent performance out of Google Spreadsheets is to separate worksheets between data collection and analysis; then perform any inter-spreadsheet aggregation by importing the plain data from each spreadsheet into a single worksheet and make all your calculations based on that.

Further work

I ended up with just over 20 different tags; if I were to do this again, I would split the ‘holiday’ tag into its constituent parts and I would split “dining” into “breakfast”, “lunch” and “dinner”.

It will be interesting to compare my monthly spend (which is the simplest data to measure) to the relatively steady pattern I have discovered over these four months to see how much effect measuring my spending had on dampening it; indeed, I could measure the historical monthly spends and perhaps see an effect that way.

Data

All the data for this experiment is published on various Google Spreadsheets. See the links below.

Coloured-in jaytoon

Nick’s been playing around with Illustrator and is pretty bowled-over by how easy it is to manipulate images. He did me this to show off his skills. I’m feeling most appreciative.

Here’s the original cartoon: Celebration

How-to log into HSBC online banking from the command-line

This is a description of the HTTP calls you need to make to achieve the log-in and account information retrieval I showed in my recent video.

Note that the content of cookies and session variables will be different each time, but examples are included here to illustrate what it going on.

Please let me know if you have similar information about a different online bank or find that this doesn’t work for you for some reason!

Getting to the login page

You can skip the HSBC homepage altogether and make a GET to:

http://www.hsbc.co.uk/1/2/HSBCINTEGRATION/

even without any cookies set. The response sets these cookies:

Set-Cookie: HSBC_COOKIEMI=e7f408d0-75f9-11de-8a43-000408000305;Domain=;Expires=Sun, 20-Jul-2014 13:25:06 GMT;Path=/
Set-Cookie: CAMToken=GCPyfe4/i1TFoVS3L/z4qRhzbNs=;Path=/1; Secure
Set-Cookie: piba=73472940.21248.0000; path=/

Submitting your Internet Banking number

Now POST to:

https://www.hsbc.co.uk/1/2/;jsessionid=0000EzVqNO88EsWt4lhQhIBMHVE:12c58sh8k?idv_cmd=idv.CustomerMigration

Note the jsessionid=... on the end of the URL, which is different for each session; the POSTed content:

userid=IBxxxxxxxxxx&StartMigration= 

The response is a 302 to (note the doubled jsessionid):

https://www.hsbc.co.uk/1/2/!ut/p/kcxml/04_Sj9SPykssy0xPLMnMz0vM0Y_QjzKLN4o38nYFSZnFm8cbm-pHmsWbxjt7QkQM4h3hAkH6BbmhEeWOiooAtvLMKw!!;jsessionid=0000EzVqNO88EsWt4lhQhIBMHVE:12c58sh8k;jsessionid=0000EzVqNO88EsWt4lhQhIBMHVE:12c58sh8k

that sets this cookie:

Set-Cookie: CAMToken=ItaH8ksgEuKY6vv6YC5nRJSTiEg=;Path=/1; Secure

The bits of your PIN that the system wants are contained in this structure in the content:

<div class="extPibRow hsbcRow">
<div class="logonPageAlignment">

         <strong>The</strong>
         <span class="hsbcTextHighlight">&#160;<strong>FIRST</strong></span>
         &nbsp;<strong>and</strong>
         <span class="hsbcTextHighlight">&#160;<strong>NEXT TO LAST</strong></span>
         &nbsp; <strong>and</strong>
         <span class="hsbcTextHighlight">&#160;<strong>LAST</strong></span>

Submitting login details

POST to:

https://www.hsbc.co.uk/1/2/;jsessionid=0000PPUs1fHtg7XOIHMqjvIsoHW:11j74lld0;jsessionid=0000PPUs1fHtg7XOIHMqjvIsoHW:11j74lld0?idv_cmd=idv.Authentication

with content:

userid=IBxxxxxxxxxx&memorableAnswer=xxxxxx&password=xxx

memorableAnswer is date-of-birth formatted ddmmyy
The response is a 302 with this location:

https://www.hsbc.co.uk/1/2/!ut/p/kcxml/04_Sj9SPykssy0xPLMnMz0vM0Y_QjzKLN4w38nYFSZnFm8cbm-pHoggZxDuiiZjEm7khhIL0C3JDQyPKHRUBimrSIQ!!;jsessionid=0000PPUs1fHtg7XOIHMqjvIsoHW:11j74lld0;jsessionid=0000PPUs1fHtg7XOIHMqjvIsoHW:11j74lld0

which sets these cookies:

Set-Cookie: CAMToken=1HOPXe7zOn2qQ/dp28qb1yeozS0=;Path=/1; Secure

Continuing with login

An additional step happens due to the JavaScript in the page. There is a GET to:

https://www.hsbc.co.uk/1/2/personal/internet-banking;jsessionid=0000PPUs1fHtg7XOIHMqjvIsoHW:11j74lld0;jsessionid=0000PPUs1fHtg7XOIHMqjvIsoHW:11j74lld0?BlitzToken=blitz

The response is a 200.

What you can get once you’re logged in – example – accounts’ balance

The account summary information on the left-hand column is in a structure like this:

<div id="jsAccountDetails" class="hsbcDivletBoxContent">
<div class="hsbcDivletBoxTitle" id="jsHideAccounts" style="display: none;">
<div class="hsbcDivletBoxRow">
			<a title="Hide my balances summary" onkeypress="this.onclick" onclick="hideAccounts(); return false;" href="#" class="hsbcDivletBoxRowText">
				<span class="hsbcDivletBoxRowImage">
					<img border="0" title="Hide my balances summary" src="/1/themes/html/hsbc_ukpersonal_ib/icon_balances.gif"/>
				</span>
				<span>Hide balances</span>
			</a></div>
<div class="hsbcSeparator"/></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">
				<strong>GRAD PLUS</strong>
			</span></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">LISTER JR</span></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">			40-27-16 91466410			</span></div>
<div class="hsbcDivletBoxRow hsbcDivletBoxRowNoLink">
			<span class="hsbcDivletBoxRowText">
				<strong> £ xx.xx  C</strong>
			</span></div>
<div class="hsbcSeparator"></div>
</div>
</div>

Online banking with JavaScript – first working thing

I recorded this at Islington Hackspace tonight, figured it would be cool to share visually rather than have a big blag-rant, (see my last post for one of those).

It ought to be self-explanatory. I’m thinking about wrapping this up in an iPhone app so I can pay people for free. Rather than use PayPal or something.

Sorry it’s a bit blurry.

Source (hopefully changing lots) on github.

How to build a DIY TwitPic (without any coding skillz)

Accounts you need

  1. A blog you can upload photos to from your phone (e.g. Blogger account on my Sony Ericsson K800i)
  2. Notify.me (or other email notification service)
  3. Gmail (or other email account with mail filters and auto-forwarding)
  4. Tarpipe
  5. Yahoo! Pipes
  6. Twitter

Coding skillz you need

  1. None

What is TwitPic?

In a nutshell, TwitPic lets you upload pictures and automatically tweet a link to them. You send photos by emailing them to TwitPic.

Why would I want to build my own TwitPic?

For me, it comes down to having an existing place that I post pictures to from my phone. In my case, I have a Sony Ericsson K800i, which offers me the option to ‘Blog this’ when I take a photo. If I choose to, the photo is uploaded to a Blogger blog. This is neat, but I’d really like to be able send out tweet with a link to new photos, as I generally take photos of things I’d like to show to people.

Another reason to want your own auto-tweeting photo-blog is that you are not able to use TwitPic because you can’t send email with photo attachments from your phone.

How to build your own TwitPic (without any coding skillz)

I’m going to run through how to make a complex system without writing any code. This involves mashing together existing services to get the desired cause and effect you’re looking for. The first thing to understand is the flow of data through the services we’re going to use. After that, I’ll step through how to set up each bit.

The flow

The flow looks like this: you take a photograph on your phone and choose to send it to your mobile blog (this is the point when you lose control); the blog system updates the RSS feed for your blog; Notify.me (or another notification service) notices the change and sends an email message to you with the body and title of the RSS item; Gmail (or another mail service) filters the incoming email and forwards it to a Tarpipe workflow; Tarpipe posts the email to a Yahoo! Pipe; Yahoo! Pipes extracts the URL of the new blog post and formats the text of the tweet, before returning the information to Tarpipe; Tarpipe tweets on your Twitter account.

Tabulated, the flow looks like this:

this thing does this to this
Person sends photo to Blog
Blog updates RSS feed
Notify.me reads change from RSS feed and emails Gmail
Gmail forwards email to Tarpipe
Tarpipe posts email to Yahoo! Pipes
Yahoo! Pipe extracts blog post URL and sends back to Tarpipe
Tarpipe posts a tweet to Twitter

Step 1 – sending a photo to your blog

In my case, the option to ‘blog this’ was already available on my phone’s camera menu, which posts a photo to a Blogger blog. It is not unusual for blog services to allow you to send new messages as email, so you could email photos to your blog as attachments.

Step 2 – updating the blog RSS feed

If your blogging service doesn’t already do this, it’s time to get a new blogging service…

Step 3 – getting an email about the change

Both Notify.me and Notifixio.us can email you if an RSS feed changes. They can take up to an hour to respond to changes, so this is the bottleneck in the flow. On the other hand, it is very simple to set one of these services up. There are other notification services, but they tend to require the writing of some code to work with them.

Assuming you’re using Notify.me, the subject of the email will be the title of the blog post prepended by “notify.me” and a space, and the body of the email will be the body of the blog post appended by a load of garbage advertising Notify.me.

Step 4 – responding to the incoming email

I expect that your email client allows you to set up filters to respond to incoming emails automatically. If it doesn’t, as above, it is time to get a new email client… Assuming you are using Notify.me as the notification service, you can filter out the incoming mail by creating a filter that selects email with a ‘FROM’ email address including ‘@notify.me’. Of these, you can get to the emails about your photo blog by selecting those whose subject includes your blog title. Set up the mail filter to forward to the email address you’ll get for the Tarpipe workflow we’re about to create.

Step 5 – the Tarpipe workflow

Tarpipe is a service for setting up workflows that respond to emails or API calls. Generally, the output will be an update to Twitter, Facebook or some other service that receives messages. Because a Tarpipe workflow can be kicked-off using email, and because you can post the email to any web service, it has opened the door to creating zero-code systems that are able to do interesting things. In short, hurray for Tarpipe.

We want to create a Tarpipe workflow that receives an email, posts it to Yahoo! Pipes, takes the response and tweets it. This is a simple chain of the “MailDecoder”, “REST” (with a “TextInput” connector to supply the URL to post to) and “TwitterUpdater” connectors. When you save the workflow for the first time, you’ll get the email address to use in the email filter from the previous step.

At the moment, Tarpipe has a shortcoming whereby the REST connector, which is what is used to send messages to any web service of your choosing, encodes the information in a structure that is very difficult to use directly with Yahoo! Pipes, despite Pipes’ obvious utility providing the string manipulation capabilities that Tarpipe lacks. You can use an intermediate service, such as this proxy, to munge the data into a more suitable format. The proxy takes a parameter in the URL to specify the ID of the Yahoo! Pipe to use. To get that ID, we’ll need to create a Yahoo! Pipe.

(That proxy URL in full: http://tarpipe-yahoopipes-proxy.smart.joyent.com)

Step 6 – getting at the blog post URL with a Yahoo! Pipe

Pipes has been around for a while now, and I believe it is mainly used to grab remote RSS feeds and other data, perform some operations such as selecting items from a feed, sorting them, turning them into Japanese and sending on the resulting feed. However, you can also use Pipes as a text manipulation machine, posting in some strings of text rather than grabbing them from somewhere else.

We’re going to create a Pipe to take the large and messy email body, delicately extract the URL of the blog post with our photo on, and append that to the subject of the email. We’ll also remove the string “Fwd: notify.me” (plus space) from the subject of the email.

You can get access to any of a list of parameters, posted as text to a Pipe, by adding a “Text Input” module from the “User inputs” list, one for each named parameter being sent, with the ‘name’ property of each module set to the name of the parameter. Using the proxy linked to in the previous step, the subject of the email will be posted to the Pipe as ‘title’ and the body will be posted as ‘description’. Set up your text input modules as shown below (only the ‘name’ property matters).

Now we have access to the strings, we have to extract the relevant strings from them. String extraction can be achieved directly by using the ’string regex’ module, which stands for ’string regular expression’. “Regex“, or “Regular expressions” is a language for describing patterns, such as “four letters, followed by a space, followed by a letter or a number”.

The string regex module doesn’t actually extract strings, it replaces any matches of a pattern with another string. However, we can make this a string extractor by replacing a pattern of “anything, followed by the pattern we want, followed by anything” with “the pattern we want”.

In regex, that looks like this:

replace: (?s).*(<the pattern we want>).*
with: $1

An explanation of the above runs like this: -
(?s) means “treat the string as a single-line” – without it, the pattern matching would stop at a line-break;
.* means “match zero-or-more of any character”;
(<the pattern we want>) puts <the pattern we want> into a “group”, to be used in…;
$1 means “the first group from the pattern”

The title is the simpler case, where the pattern we want to extract is [My Mobile Blog] - <the rest of the title>, from a string that looks like “Fwd: notify.me [My Mobile Blog] – super new photo”. To get what we want, connect a string regex module to the text input module named ‘title’ and put the following in the ‘replace’ and ‘with’ properties:

replace: (?s).*Fwd: notify\.me (\[My Mobile Blog\] - .*)
with: $1

The places where there is a \ preceding another character are examples of “escaping” characters that would otherwise have special meaning in regex. For example, [ would normally mean “start a set of characters to choose from”, so \[ just means “match [“.

The more complicated extraction is getting the URL from the email body. Here we want to get a URL of the form http://c.notify.me/F6d3HG out of a much longer string of text. This URL points to an address on Notify.me’s URL-shortening service, that will direct you to the blog post’s much longer URL. Connect a new string regex module to the text input module named ‘description’ and put the following in the ‘replace’ and ‘with’ properties:

replace: (?s).*(http:\/\/c\.notify\.me/\w{6}).*
with: $1

The \w{6} pattern means “6 word-characters”. A “word-character” is something you would expect to find in a word, rather than punctuation or a space. We don’t need to worry too much about what is and what isn’t a word-character, as we only expect to find letters, numbers and “_” in the 6 characters making up the end of the URL, which are definitely word-characters.

(If this has piqued your curiosity, there is a lot of information on regular expressions at http://regular-expressions.info.)

We’re nearly done. Now we have two strings that we want to join together as our tweet. To do this, use the ’string builder’ module and assemble them as “<title>” plus ” – ” plus “<URL>”. Before outputting this string, we’re going to create a new ‘item’, which is the unit of output from a Pipe – use the “item builder” module, setting the ‘title’ of the new item as the tweet string.

With that all done, we can save the Pipe. By clicking on the ‘run this pipe’ link, you can find the ID of the pipe in the URL of the page you’re taken to, named “_id”. To use this ID in the Tarpipe workflow, in the text input module which is connected to the REST connector, add the ID to the URL as a query-string parameter named “pipeId”: http://<proxy>/?pipeId=... . (Note the / between the domain of the proxy and the ?.)

Step 7 – tweeting from Tarpipe

If you have set up your pipe as above, the text to use for the tweet will be available from the ‘title’ terminal on the right-hand side of the REST connector. Wire this to the ‘title’ terminal of aTwitterUpdater connector.

You need to set up your Twitter account in the ‘accounts’ tab so Tarpipe can tweet with it. At the moment, you need to give out your Twitter username and password, which is the dreaded anti-pattern, and since Twitter now supports OAuth, it makes sense for Tarpipe to use that to get the appropriate level of access to your account. For the purposes of this example, you could set up a new twitter account to link to Tarpipe.

Save the workflow, and you’re done! If everything is configured correctly, sending a photo to your mobile blog will cause a tweet to appear on your timeline, linking back to your new blog post.

Testing the bugger

With a total of 5 remote systems (6 with the Tarpipe / Yahoo! Pipes proxy) between you and the intended tweet, there’s a lot of scope for things to go wrong. The weakest parts of the chain are going to be those you set up yourself, particularly the regex-heavy Yahoo! Pipe, however errors can happen at any point in the system, and problems caused by delays, caching or rate-limits can manifest themselves in various ways.

It’s a good idea to set up test accounts for any services you are hooking into that you use on a day-to-day basis, especially if they publish information to your friends or the public. If you don’t, as I didn’t for my Twitter account, you could end up with something like this mess:

The trick to understanding what is going on in these remote systems is to log activity whenever you get the chance. One useful service for this is PostBin, which lets you create a logger that records anything you post to it.

In the flow above, opportunities to post to this log are:

  1. Adding another REST connector to post what is received by the Tarpipe workflow
  2. Adding a ‘web service’ module to post what is received by the Yahoo! Pipe
  3. Adding a ‘web service’ module to post what comes out of the string extraction steps
  4. Adding another REST connector to post what is returned to Tarpipe from the Yahoo! Pipe

When using the Tarpipe / Yahoo! Pipes proxy, adding &postBinId=<id> to the URL in the text input which connects to the REST connector will cause the proxy to log at these four points:

  1. What comes into the proxy
  2. What is sends Yahoo! Pipes
  3. What comes back from Yahoo! Pipes
  4. What it sends back to Tarpipe

Here’s an example of the PostBin logs you get from the proxy:

As well as logging, some of the services mentioned provide help with the testing and debugging process:

  • Gmail puts mail it forwards due to filters into your ’sent items’; the exact text sent can be found by selecting ’show original’ from the ‘reply’ drop-down in an email
  • Tarpipe provides an ‘activity’ tab that shows what came in and went out (note that sometimes you have to click on ‘all’ to see workflow activity, particularly when workflows only contain REST connectors)
  • Yahoo! Pipes has an in-built debugger that allows you to examine the effect of your modules on the text you put into the ‘debug’ property of a user input
  • RegexPal lets you test out a regex on some text (such as the body text of your email)

Despite all these tools, debugging a chain of systems you have no control over can be a painful process. Be particularly careful when using Yahoo! Pipes, as it is not uncommon for results of a Pipe to be cached for 30 minutes. I have experienced this happening even if you change the input data. Lastly, note that should an error actually occur in your flow, it’s likely you’re not going to be told about it…

Good luck!

Notes on migrating an application from AppJet to Smart Platform

I’ve been wingeing quite a lot recently about AppJet’s demise and the shoddy state of potential replacements, and got quite a nice response from Jim Pick at Joyent. I decided today to have a play with their new Smart Platform and had a go at porting one of my AppJet applications over to Smart. Here’s my notes, fresh from the lab…

It’s not as simple as hitting an edit button yet, but it’s still worth having a play. The use of Sammy to handle and dispatch incoming requests, along with a single “bootstrap.js” file that controls all the SSJS, means that it ought to be fairly easy to port over single-page AppJet apps to Smart Platform. Here’s some notes I’ve gathered whilst porting over an application that acts as a gateway between tarpipe’s REST connector and a Yahoo! Pipe, allowing you to send data to a Yahoo! Pipe for processing.

Request object

In AppJet, you get the POSTed variables and the query-string parameters both in request.params. In Smart, the query-string parameters are in request.query with the raw string in request.queryString; POSTed variables are in request.body with the raw datastring in request.content.

Dispatching

In AppJet, you have the option to define different functions for handling different types of request e.g. GETs, POSTs and requests to different resources e.g. /this, /that. Smart uses Sammy, which gives you a similar mechanism to choose which functions respond to different incoming requests, although you also have the option to use a single function main(request) {...} to handle all responses. A concrete example is:

AppJet:
    function get_resource() { ... }
Smart:
    GET("/resource", function() { });

Writing a response

AppJet provides a default chrome that wraps an application, which you can remove by setting page.setMode('plain'); then you use functions like print(response) to send the respons. Smart does it slightly differently, where whatever you return from a function is sent as the response. It doesn’t seem possible to send HTTP header information.

Making HTTP requests

AppJet provides functions like wpost and wget to make HTTP calls. Smart provides the system.http.request function to do the same sort of thing. The difference in the returned response is that AppJet gave you a raw string of the response body, whereas Smart returns an object containing headers and content properties.

Deployment

The lack of an in-browser edit experience has been my complaint of late, but doing the Git bit to deploy my app to Smart was straightforward, and now it’s up at http://f7e3dbf.smart.joyent.com.

Try it out with:

curl -d 'pipeId=lAUEoB1R3hGTqt6hggSecQ&data={"items":[{"title":"A title","description":"Any description","link":"http:\/\/example.com"}]}' http://f7e3dbf.smart.joyent.com

Code

For the sake of completion, here’s the source code. I’ve gone from this:

/* appjet:version 0.1 */
import("lib-json"); 

var p = request.params;
var result = "";
var output = ""; 

function serialize(obj) {
    var str = "";
    for(var i in obj) {
        str += "&"+encodeURIComponent(i)+"="+encodeURIComponent(obj[i]);
    }
    return str.substring(1);
}
if(p && p.data && p.pipeId) {
    var data = JSON.parse(p.data);
    var pipeId = p.pipeId;
    var items = data.items;
    if(items) {
        var url = "http://pipes.yahoo.com/pipes/pipe.run?_id="+pipeId+"&_render=json";
        // tarpipe only POSTs 1 item
        var result = wpost(url,{title:items[0].title,description:items[0].description});
        if(result) {
            result = JSON.parse(result);
            if(result) {
                output = result.value;
                if(!output.items[0].title) {
                    output.items[0].title = "returned from Yahoo! Pipe at: "+pipeId;
                }
                output = JSON.stringify(output);
            } else {
                result = "bag result from Yahoo! Pipe at: "+pipe;
            }
        } else {
            result = "no result returned";
        }
    } else {
        result = "no items in data input";
    }
} else {
    result = "no data or no pipe input";
} 

page.setMode('plain');
output ? print(output) : print(result);

To this:

system.use("com.joyent.Resource");
system.use("org.json.json2");

function serialize(obj) {
    var str = "";
    for(var i in obj) {
        str += "&"+encodeURIComponent(i)+"="+encodeURIComponent(obj[i]);
    }
    return str.substring(1);
}

function main(request) {
	var p = request.content!="" ? request.body : request.query;
	var result = "";
	var output = "";
	/* for debug input
	for (var i in p) {
		output += i+", "+p[i]+"\n";
	}
	return output;*/
	if(p && p.data && p.pipeId) {
	    var data = JSON.parse(p.data);
	    var pipeId = p.pipeId;
	    var items = data.items;
	    if(items) {
	        var url = "http://pipes.yahoo.com/pipes/pipe.run?_id="+pipeId+"&_render=json";
	        // tarpipe only POSTs 1 item
	        var toPost = "title="+items[0].title+"&description="+items[0].description;
	        var result = system.http.request("POST",url,null,toPost);
	        if(result && result.content) {
	            result = JSON.parse(result.content);
	            if(result) {
	                output = result.value;
	                if(!output.items[0].title) {
	                    output.items[0].title = "returned from Yahoo! Pipe at: "+pipeId;
	                }
	                output = JSON.stringify(output);
	            } else {
	                result = "bag result from Yahoo! Pipe at: "+pipe;
	            }
	        } else {
	            result = "no result returned";
	        }
	    } else {
	        result = "no items in data input";
	    }
	} else {
	    result = "no data or no pipe input";
	}
	return output || result;
}

Adapting WireIt to work with independent data stores

I’ve been working during the last fortnight on the “Streams” project, which LShift, an IT consultancy in Shoreditch, are building in conjunction with the BBC. My part in this project has been to prototype the interface that a non-technical person would use to edit and create Streams, which are box-and-wire visual workflows similar to those found in Yahoo! Pipes or Tarpipe.

(The most interesting difference is that Streams combines Tarpipe’s event-driven model with Yahoo! Pipe’s modular data processing model. For more on this project and its ambition to open-sourcing, see LShift’s and the BBC’s blog posts.)

I have been using WireIt, an open-source JavaScript library, to aid with the interface prototyping. Fortunately for me, WireIt’s latest version (v0.4), announced shortly before I started this project, includes a fully-featured “Wiring Editor”, which gets you up and running with the familiar Yahoo! Pipes-style editor.

The brief for the prototype was to be able to complete a round-trip of loading up a Stream (stored as a document in CouchDB), edit it, save it (back to CouchDB) and be able to load it up again.

I thought it would be helpful to bring to your attention the things you need to be aware of if you want to use WireIt’s Wiring Editor and adapt it to work with an independent source of module and workflow data.

Stage 1 – different data formats

It is hardly worth saying that there is no point trying to persuade your data store owners to change their data format so your job is made easier. Apart from this obvious need to convert from one data format to another, you also have to deal with the fact that WireIt only keeps hold of the data it needs to display and configure its modules.

The best place I could find to hook into the loading process was by overriding the function called after the load button is pressed:

WireIt.WiringEditor.prototype._onLoad = WireIt.WiringEditor.prototype.onLoad;
WireIt.WiringEditor.prototype.onLoad = function() {
    this.service.listWirings({group:true}, {
        success: function(result) {
            var responseText = result.responseText;
            if(responseText) {
	        this.pipes = convertFromStreams(responseText);
                this.pipesByName = {};
                this.renderLoadPanel();
                this.updateLoadPanelList();
                this.loadPanel.show();
            }
        },
        failure:function() {
            console.log('failure when loading, args:',arguments);
        },
        scope: this
    });
};

The data format of a combination of blocks and wires in WireIt looks like this:

var modules = [
	{
		name: ,
		working: {}
	}
]
where <em>working</em> is defined as:
working = {
	"modules":[
		{
			"name":"FormContainer",
			"config":{
				"position":[115,58]
			},
			"value":{
				"email":"",
				"firstname":"",
				"happy":true,
				"lastname":"Dupont",
				"title":"Mr",
				"website":""
			}
		}],
	"properties":{
		"description":"",
		"name":"teste"
	},
	"wires":[
		{
			"src":{
				"moduleId":0,
				"terminal":"br"
			},
			"tgt":{
				"moduleId":1,
				"terminal":"tr"
			}
		}
	]
}

The convertStreams function changes each stream document available in CouchDB into the format above.

WireIt doesn’t keep hold of any properties you attach to blocks that aren’t used explicitly. This means you need to keep hold of any properties you need to save back to your data store. I did this by keeping a globally available variable, WireIt.customFields:

...
if(node.type) {
    WireIt.customFields.nodes[i].type = node.type;
} else {
    WireIt.customFields.nodes[i].terminal = node.terminal;
}
...

Stage 2 – loading the list of blocks from your data store

As well as loading up descriptions of a full combination of blocks and wires, I wanted to load all the available blocks in CouchDB into WireIt so they’d be available for making workflows with.

The WireIt Wiring Editor doesn’t load its blocks from a remote system, you have to provide all the definitions in a JSON “language” file when you create the Wiring Editor. I hooked into the setup of the Wiring Editor after it had finished loading its backend-describing SMD file (see below for explanation of this SMD malarkey):

WireIt.WiringEditor.prototype.onSMDsuccess = function() {
	WireIt.customFields = {};
	this.populateModules();
};

The populateModules function is responsible for turning the description of the block from CouchDB into the format WireIt uses for defining available blocks, which looks like this (note it is different to the format used to describe a block that is part of a workflow, described above):

var modules = [
	{
		"name": "detect_iplayer",
		container: {
			xtype: "WireIt.FormContainer",
			type: "regexp_replace",
			title: "Detect iplayer and remove",
			icon: "images/detect_iplayer.png",
			fields: [
				{"inputParams": {"name": "caseinsensitive", "label": "case insensitive?", "value":true}, "type": "boolean"},
				{"inputParams": {"name": "regexp", "label": "regular expression", "value": "(iplayer)"}},
				{"inputParams": {"name": "dotall", "label": "dot matches all?", "value": false}, "type": "boolean"},
				{"inputParams": {"name": "multiline", "label": "multiline?", "value": false}, "type": "boolean"},
				{"inputParams": {"name": "replacement", "label": "replacement", "value": "\\1"}}
			],
			width:250,
			terminals: [
				{"name": "input", direction: [-1,0], offsetPosition: {left: 110, top: -15}},
				{"name": "positive", direction: [1,0], offsetPosition: {left: 60, bottom: -15}},
				{"name": "negative", direction: [1,0], offsetPosition: {right: 60, bottom: -15}}
			]
		}
	},
	...
]

Stage 3 – adding default layout

The Streams data store doesn’t contain any information about how the blocks in a stream should be laid out, as this is not important to the execution of the streams. However, for loading a stream into WireIt, I wanted to give the blocks a default layout so they didn’t end up on top of each other.

I picked a layout that put the blocks equally spaced around a ring. This algorithm gives you the coordinates of a set of count blocks. It assumes the blocks are going to be square, and sets the radius of the ring so that the blocks would just touch if they were square.

function calculateLayout(count,width) {
	var coords = [];
	var n = count || 4;
	var h = width || 100;
	var radius = (n/Math.PI + 1.5)*h/Math.sqrt(2);
	var theta = 0;
	for (var i=0; i<n; i++) {
		theta = 2*Math.PI*i/n;
		coords[i] = [];
		coords[i][0] = radius*(1+Math.cos(theta)); // x
		coords[i][1] = radius*(1-Math.sin(theta)); // y
	}
	return coords;
}

Stage 4 – different load/save API (and the limitation of yui-rpc)

The Wiring Editor is hard-wired to use an implementation of JSON-Schema called “YUI-RPC”, along with a defining Service Mapping Description (SMD – think WSDL for JavaScript). The down-side to this is that the implementation does not currently support HTTP verbs other than GET and POST, nor is it obvious how to extend the implementation in line with the spec. If your API requires the use of PUT (as CouchDB does to update or create named documents), then you will need to do something about this.

YUI-RPC supported what I needed for loading, so supporting CouchDB was only as hard as changing one of the method specifications:

"listWirings": {
    "transport": "GET",
    "envelope": "URL",
    "target": "couchdb/_design/feeds/_view/join",
    "parameters": [
        {"name":"group", "type":"boolean"}
    ],
    "description": "Get a list of streams"
}

To support PUT for saving, I overrode WireIt.WiringEditor.prototype.saveModule and, deciding to go with what I know, stuck in a block of jQuery that I’d had hanging around from a test working with CouchDB’s API.

General comments on prototyping with single-page applications

I made the prototype as an application that could just be served as a single HTML page from a file: URI. I strongly support this over trying to make your prototype work with some web framework or other, as you will lose valuable time setting up the framework in the first place.

Working with HTML files loaded as files gives you a couple of special properties: you can read and write from the file system and make cross-domain XMLHttpRequests (on many browsers). I had to modify YUI’s Connect object a bit to support the cross-domain XHR requests. For more information on the special properties of HTML pages viewed over file: URI’s, see these sections of a W3C position paper about TiddlyWiki written by Osmosoftonian Paul Downey.

Joyent Smart Platform – a replacement for AppJet?

I’m totally two months behind with this news, but I noticed today that Joyent’s Smart Platform was Brian LeRoux’s top pick at JSConf back in May. This, coupled with poking from Joyent-er Jim Pick and his response to the Ajaxian coverage of AppJet’s closure, encouraged me to sign up to the Smart Platform beta today.

A month ago, I wrote about how sad I was that AppJet were closing their doors to developers, leaving those of us who want easy-to-edit server-side JavaScript without any decent options. A ray of light seemed to emerge in the form of JGate.de, but with uptime worse than Twitter’s, that appears to be unusable.

Jim poked me about Smart Platform because, I think, he knows I want hosted JavaScript that’s simple. And, if you’re even an entry-level web developer, Smart Platform looks pretty simple. Joining Heroku in the there-isn’t-any deployment model, pushing your code to your Joyent git repository is all you need to do to deploy your web app. Brian’s JSConf article claims the platform is super-scalable and all that, so from a distance, Smart Platform looks like a worthy addition to the hosted development (aka Platform as a Service) brigade.

However, I’m not coming at this from the point of view of a git-wielding web developer, I’m representing a swathe of people who can cope with a bit of JavaScript and logic, but have neither the time nor the inclination to get into the world of desktop development. When AppJet was still around, we had an edit button, a single page of a few dozen lines and a cross-domain AJAX API. This, surprisingly enough, was all you needed to apply some programmatic patching to a bunch of different use-cases: automated workflows, Facebook applications, mini web-services.

In fact that last one – mini web-services – is the crux of what I’m getting at here: programming doesn’t have to be about applications, it can be tiny web services that do something for you that you’ve cloned from someone else and tweaked until it works.

I’ve said it before and I’ll say it again – I will pay for what AppJet were providing. People will pay for what AppJet were providing. If you can’t get a squillion to provide this platform for free, that doesn’t mean you shouldn’t provide it for a charge. I’ll be your first customer.