I’ve been working during the last fortnight on the “Streams” project, which LShift, an IT consultancy in Shoreditch, are building in conjunction with the BBC. My part in this project has been to prototype the interface that a non-technical person would use to edit and create Streams, which are box-and-wire visual workflows similar to those found in Yahoo! Pipes or Tarpipe.

(The most interesting difference is that Streams combines Tarpipe’s event-driven model with Yahoo! Pipe’s modular data processing model. For more on this project and its ambition to open-sourcing, see LShift’s and the BBC’s blog posts.)

I have been using WireIt, an open-source JavaScript library, to aid with the interface prototyping. Fortunately for me, WireIt’s latest version (v0.4), announced shortly before I started this project, includes a fully-featured “Wiring Editor”, which gets you up and running with the familiar Yahoo! Pipes-style editor.

The brief for the prototype was to be able to complete a round-trip of loading up a Stream (stored as a document in CouchDB), edit it, save it (back to CouchDB) and be able to load it up again.

I thought it would be helpful to bring to your attention the things you need to be aware of if you want to use WireIt’s Wiring Editor and adapt it to work with an independent source of module and workflow data.

Stage 1 – different data formats

It is hardly worth saying that there is no point trying to persuade your data store owners to change their data format so your job is made easier. Apart from this obvious need to convert from one data format to another, you also have to deal with the fact that WireIt only keeps hold of the data it needs to display and configure its modules.

The best place I could find to hook into the loading process was by overriding the function called after the load button is pressed:

WireIt.WiringEditor.prototype._onLoad = WireIt.WiringEditor.prototype.onLoad;
WireIt.WiringEditor.prototype.onLoad = function() {
    this.service.listWirings({group:true}, {
        success: function(result) {
            var responseText = result.responseText;
            if(responseText) {
	        this.pipes = convertFromStreams(responseText);
                this.pipesByName = {};
                this.renderLoadPanel();
                this.updateLoadPanelList();
                this.loadPanel.show();
            }
        },
        failure:function() {
            console.log('failure when loading, args:',arguments);
        },
        scope: this
    });
};

The data format of a combination of blocks and wires in WireIt looks like this:

var modules = [
	{
		name: ,
		working: {}
	}
]
where <em>working</em> is defined as:
working = {
	"modules":[
		{
			"name":"FormContainer",
			"config":{
				"position":[115,58]
			},
			"value":{
				"email":"",
				"firstname":"",
				"happy":true,
				"lastname":"Dupont",
				"title":"Mr",
				"website":""
			}
		}],
	"properties":{
		"description":"",
		"name":"teste"
	},
	"wires":[
		{
			"src":{
				"moduleId":0,
				"terminal":"br"
			},
			"tgt":{
				"moduleId":1,
				"terminal":"tr"
			}
		}
	]
}

The convertStreams function changes each stream document available in CouchDB into the format above.

WireIt doesn’t keep hold of any properties you attach to blocks that aren’t used explicitly. This means you need to keep hold of any properties you need to save back to your data store. I did this by keeping a globally available variable, WireIt.customFields:

...
if(node.type) {
    WireIt.customFields.nodes[i].type = node.type;
} else {
    WireIt.customFields.nodes[i].terminal = node.terminal;
}
...

Stage 2 – loading the list of blocks from your data store

As well as loading up descriptions of a full combination of blocks and wires, I wanted to load all the available blocks in CouchDB into WireIt so they’d be available for making workflows with.

The WireIt Wiring Editor doesn’t load its blocks from a remote system, you have to provide all the definitions in a JSON “language” file when you create the Wiring Editor. I hooked into the setup of the Wiring Editor after it had finished loading its backend-describing SMD file (see below for explanation of this SMD malarkey):

WireIt.WiringEditor.prototype.onSMDsuccess = function() {
	WireIt.customFields = {};
	this.populateModules();
};

The populateModules function is responsible for turning the description of the block from CouchDB into the format WireIt uses for defining available blocks, which looks like this (note it is different to the format used to describe a block that is part of a workflow, described above):

var modules = [
	{
		"name": "detect_iplayer",
		container: {
			xtype: "WireIt.FormContainer",
			type: "regexp_replace",
			title: "Detect iplayer and remove",
			icon: "images/detect_iplayer.png",
			fields: [
				{"inputParams": {"name": "caseinsensitive", "label": "case insensitive?", "value":true}, "type": "boolean"},
				{"inputParams": {"name": "regexp", "label": "regular expression", "value": "(iplayer)"}},
				{"inputParams": {"name": "dotall", "label": "dot matches all?", "value": false}, "type": "boolean"},
				{"inputParams": {"name": "multiline", "label": "multiline?", "value": false}, "type": "boolean"},
				{"inputParams": {"name": "replacement", "label": "replacement", "value": "\\1"}}
			],
			width:250,
			terminals: [
				{"name": "input", direction: [-1,0], offsetPosition: {left: 110, top: -15}},
				{"name": "positive", direction: [1,0], offsetPosition: {left: 60, bottom: -15}},
				{"name": "negative", direction: [1,0], offsetPosition: {right: 60, bottom: -15}}
			]
		}
	},
	...
]

Stage 3 – adding default layout

The Streams data store doesn’t contain any information about how the blocks in a stream should be laid out, as this is not important to the execution of the streams. However, for loading a stream into WireIt, I wanted to give the blocks a default layout so they didn’t end up on top of each other.

I picked a layout that put the blocks equally spaced around a ring. This algorithm gives you the coordinates of a set of count blocks. It assumes the blocks are going to be square, and sets the radius of the ring so that the blocks would just touch if they were square.

function calculateLayout(count,width) {
	var coords = [];
	var n = count || 4;
	var h = width || 100;
	var radius = (n/Math.PI + 1.5)*h/Math.sqrt(2);
	var theta = 0;
	for (var i=0; i<n; i++) {
		theta = 2*Math.PI*i/n;
		coords[i] = [];
		coords[i][0] = radius*(1+Math.cos(theta)); // x
		coords[i][1] = radius*(1-Math.sin(theta)); // y
	}
	return coords;
}

Stage 4 – different load/save API (and the limitation of yui-rpc)

The Wiring Editor is hard-wired to use an implementation of JSON-Schema called “YUI-RPC”, along with a defining Service Mapping Description (SMD – think WSDL for JavaScript). The down-side to this is that the implementation does not currently support HTTP verbs other than GET and POST, nor is it obvious how to extend the implementation in line with the spec. If your API requires the use of PUT (as CouchDB does to update or create named documents), then you will need to do something about this.

YUI-RPC supported what I needed for loading, so supporting CouchDB was only as hard as changing one of the method specifications:

"listWirings": {
    "transport": "GET",
    "envelope": "URL",
    "target": "couchdb/_design/feeds/_view/join",
    "parameters": [
        {"name":"group", "type":"boolean"}
    ],
    "description": "Get a list of streams"
}

To support PUT for saving, I overrode WireIt.WiringEditor.prototype.saveModule and, deciding to go with what I know, stuck in a block of jQuery that I’d had hanging around from a test working with CouchDB’s API.

General comments on prototyping with single-page applications

I made the prototype as an application that could just be served as a single HTML page from a file: URI. I strongly support this over trying to make your prototype work with some web framework or other, as you will lose valuable time setting up the framework in the first place.

Working with HTML files loaded as files gives you a couple of special properties: you can read and write from the file system and make cross-domain XMLHttpRequests (on many browsers). I had to modify YUI’s Connect object a bit to support the cross-domain XHR requests. For more information on the special properties of HTML pages viewed over file: URI’s, see these sections of a W3C position paper about TiddlyWiki written by Osmosoftonian Paul Downey.

I’m totally two months behind with this news, but I noticed today that Joyent’s Smart Platform was Brian LeRoux’s top pick at JSConf back in May. This, coupled with poking from Joyent-er Jim Pick and his response to the Ajaxian coverage of AppJet’s closure, encouraged me to sign up to the Smart Platform beta today.

A month ago, I wrote about how sad I was that AppJet were closing their doors to developers, leaving those of us who want easy-to-edit server-side JavaScript without any decent options. A ray of light seemed to emerge in the form of JGate.de, but with uptime worse than Twitter’s, that appears to be unusable.

Jim poked me about Smart Platform because, I think, he knows I want hosted JavaScript that’s simple. And, if you’re even an entry-level web developer, Smart Platform looks pretty simple. Joining Heroku in the there-isn’t-any deployment model, pushing your code to your Joyent git repository is all you need to do to deploy your web app. Brian’s JSConf article claims the platform is super-scalable and all that, so from a distance, Smart Platform looks like a worthy addition to the hosted development (aka Platform as a Service) brigade.

However, I’m not coming at this from the point of view of a git-wielding web developer, I’m representing a swathe of people who can cope with a bit of JavaScript and logic, but have neither the time nor the inclination to get into the world of desktop development. When AppJet was still around, we had an edit button, a single page of a few dozen lines and a cross-domain AJAX API. This, surprisingly enough, was all you needed to apply some programmatic patching to a bunch of different use-cases: automated workflows, Facebook applications, mini web-services.

In fact that last one – mini web-services – is the crux of what I’m getting at here: programming doesn’t have to be about applications, it can be tiny web services that do something for you that you’ve cloned from someone else and tweaked until it works.

I’ve said it before and I’ll say it again – I will pay for what AppJet were providing. People will pay for what AppJet were providing. If you can’t get a squillion to provide this platform for free, that doesn’t mean you shouldn’t provide it for a charge. I’ll be your first customer.

I announced my intention to leave Osmosoft in a blog post just over a month ago, and it became real this week. When I posted my resignation, it was all about me – why I was leaving, what my plans were – but said nothing about my relationship with BT and Osmosoft, so I thought I’d rectify that.

I have something to say on how I feel personally about Osmosoft, my opinion about Osmosoft’s “success” and why BT will get its biggest return on investment from those of us that leave.

First of all, I am and will always be grateful to Jeremy and JP for creating and maintaining Osmosoft’s environment, as it allowed me to transform from some sort-of technical consultant, with little track record of actually building things, into the fully-functional member of society you know today.

This is the environment I’m talking about: one that allowed me to learn systems development on the job, encouraged everyone to follow their creative hunches and organise their own time, broadcast our individual personalities on to the web and into the conference arena and gave us the time to shape our attitudes about open source and the web into a pretty homogenous set of principles (in effect, if not in concrete).

Over two years, Osmosoft has transformed from a vehicle for one man’s project and the incipient commercialisation thereof, through being a public flag-bearer for a private corporation, into a software house specialising in collaboration systems and web prototyping. At Osmosoft’s offsites, held roughly once every six months, we regularly brought up two questions (@nickwebb deserves a lot of credit for not letting these go unanswered): “what is Osmosoft for?” and “how do we know if Osmosoft is successful?”. We never came up with good answers. Retrospectively, I think now I’d answer them like this (and as I write this, I am aware that Jeremy will likely maintain that this represents the truth as he has always seen it):

Osmosoft is for developing and demonstrating what BT thinks about the web and about open source software. Although Jeremy bears the title “Head of Open Source Innovation” at BT, Osmosoft is as much about attitudes towards the web as it is attitudes towards open source. And the funny thing is, BT has, like, a million people writing systems that you use through a web browser. AND we’ve been using linux and php and mysql and tomcat and asterisk and ALL THAT for years, but it’s taken a dude deposited into the executive with a job-title like Jeremy’s, narrowly focussing on a toolset like TiddlyWiki, before anybody has accepted that BT has a line on either of these things.

Does this mean that BT is justified in spending circa £2,000,000 over 2 years to keep us in pains aux chocolate and conference passes? Depends how you see it. What else does £2m buy you in BT? 2% of the Prison Phones contract; 0.2% of the NHS IT project; 0.02% of the Next Generation Network. Maybe a worthwhile investment if the practices and principles Osmosoft represents does, as Jeremy’s boss JP hopes, save Global Services (which, for the 99% of you who have never heard of them, currently earns 33% of BT’s income and about 100% of it’s net loss).

I’ve left all this behind and as I’ve already claimed via the title to this post, I think that’s great for BT. As I mentioned above, I was not a fabricator when I joined Osmosoft, tiny as it was, two years ago; now I think nothing of erecting savage Frakensteins to get an idea across, or delicately architecting part of the web’s nervous system (or concocting flamboyant metaphors). The magnitude of this change and the Osmosoft-centredness of it is not lost on me. Now I’m free of the obligations that go with someone paying your wage, it is natural to feel that I am also breaking free from Osmosoft and BT. However, this isn’t right. I am an Osmosoftonian, and as an alumni I will always be an Osmosoftonian. And it is here that BT should look for their return on investment.

In case you have just elevated your eye-brows, this is what I mean: To be optimistic would be to hope that in a year’s time I will have a non-zero portfolio of happy clients, for whom I have done interesting and impactful work to improve their individual measure of success. I will have done this by being an Osmosoftonian: using the tools, principles and brilliant brains I have got used to having around me.

At the start of Osmosoft’s journey with BT in May 2007, we thought we’d have overcome some pretty impressive hurdles if we could be a catalyst for changing views, both inside and outside BT. I think there will be no greater testament to the success of this ambition if BT’s leaders can point to its own successors and effectively say, with pride, “we made that”.

At the quiet intersection of the busy Harrow Road and lengthy Elgin Avenue, a gourmet food and gifts market has been set up by Westminster Council, opening this weekend and breathing new life into a space that has functioned as a haunt for local drunks and drug dealers.

The cheery green and white awnings of the dozen-or-so market stalls almost entirely obscure the green and white “Costcutter Express” sign that would otherwise set the tone. The sky-blue Maida Hill Gallery is clearly in sight and, as I write this at a plastic garden table, its artificial grass lawn out front is hosting a number of happy shoppers with hog roast sandwiches.

The reason I am here is to give some support to Fabio and Robert, the baristas of Alto Café, which operates five days a week from a small van parked on Strutton Ground, almost within the shadow of Osmosoft’s 4th floor office in Westminster Telephone Exchange.

Apart from anything, we owe a great debt of gratitude to Fabio, Brazil’s champion barista, for saving the coffee of SW1. Until we became aware of Fabio’s and, as he was trained up, Robert’s delicious and caffeinated creations, the only way to get a decent cup was to trot 20 minutes down Buckingham Palace Road to Tomtom.

It looks like Alto (the UK’s only branch of the award-winning French franchise) are going from strength to strength – they have recently introduced Square Mile coffee as an alternative to the Alto blend, stock an expanding range of Villandry pastries and now have secured a Saturday slot at the Maida Hill market. On top of this, Alto are going up to Edinburgh for two weeks in August to run three coffee stalls during the festival.

Fabio and Robert represent exactly the kind of business I am leaving Osmosoft to help, so I have started to poke them with some ideas like using Twitter for taking local orders. It would be awesome to figure something out for the Edinburgh festival, perhaps a micro-site for festival-going coffee lovers promoting the locations of their three stalls; or something to take the pain out of managing the stock of the three locations.

If you’re in North West London on a Saturday, go spend some time at Maida Hill market. The square has a very intimate feel to it and is a bit like someone’s transplanted a chunk of Borough market, without all the tourists. And say hi to Fabio and Robert from me. Oh, and try a vanilla shakerato, it’s a few shades of awesome.

[MyVillage.com has a video here]

I saw Mike posting recently about how he is using TiddlyWeb with theTiddlyDocs and TiddlyGuv projects and I thought I would put something out about how we’ve been using TiddlyWeb with another Osmosoft project – the redesign of ILGA.org for the International Lesbian and Gay Association.

The ILGA project has taken our use of TiddlyWeb in unexplored directions. The essential problem that we’re using this system to solve is that we have a set of people around the world who wish to contribute content to a website, which is then stored and displayed to a visitor when they access the site. It is, simply, a CMS problem as solved a hundred times over by projects, public and private, around the world.

Chris Dent did not want to create a framework when creating TiddlyWeb , but with this is what he has done. With ILGA, we have been determined to push the limits of the system and expose new patterns of use and holes for further development.

The following describes how we’ve used TiddlyWeb to solve the problems of content-editing and publishing. The natural progression has been to work with a TiddlyWeb plugin, cheekily called “jinx”, when the existing toolset didn’t seem to be adequate. As you’ll see, revisions of our methods have used more and more of TiddlyWeb’s innards as we’ve gone along, showing what you can do with the system in its native form.

Accepting content from contributors

Where and how to publish

TiddlyWeb uses the concepts of “bags“, which are spheres of permissions, and “recipes“, which collect bags and pour out the tiddlers through filters. Tiddlers themselves do not have permissions assigned to them, which in a sense, helps you with your decision about where to let people put content. Each contributor gets their own bag, and anything that they contribute ends up in this bag.

In a finally enlightening cycle of development, we discovered that the connection between a recipe and the bags it references is implemented in a useful and elegant way: when you create a new tiddler and save it back to TiddlyWeb, it will end up in whichever is the first bag you have permission to PUT to, counting up from the bottom of the recipe. This means that if your own bag is the last line in the recipe, any new tiddlers will be assigned to that bag. A neat syntactical trick in creating the recipe can make it generic for any contributor:

recipe
   /bags/system/tiddlers
   /bags/common/tiddlers
   /bags/{{ user }}/tiddlers

This simple construction saved us large chunks of code that had been written as TiddlyWeb plugins before this mechanism was properly understood.

The TiddlyWiki as content manager

Although the pages that we wanted to show to site visitors had to behave like ordinary web pages, there is no such limitation on pages that the relatively small number of contributors use to edit and publish content, as they are effectively private. This allowed us to take advantage of the full power of TiddlyWiki as an editor and manager of the content in someone’s bag.

Draft vs. publish

One feature we added to the TiddlyWiki editor early on was a button to publish a piece of content. Initially, this involved a lot of custom code to handle PUTing an article, until the understanding of the bag and recipe concept outlined above encouraged us to use the default behaviour. Setting a flag on a tiddler that indicates which bag it comes from overrides the recipe-based bag selection and keeps publishing to different bags simple.

Displaying content to visitors

Now that we have content safely tucked up in bags, the question is how to get it out again for visitors to the site, outside of the shackles of a standard TiddlyWiki. This is because TiddlyWikis are not suitable as our ordinary web pages for reasons of accessibility, page size and search engine optimization. This has so far been the area most in need of attention and experimentation, as TiddlyWeb does not make content presentation as obvious as content saving.

Getting stuff out – recipes, filters and tiddler selection

[Update: this techniques described in this section continue to evolve, but I think it is worth posting this as it shows where we came from and does include some useful generic techniques.]

The complement to the content-storing bags mentioned above are the recipes, which deliver a filtered set of tiddlers that come from any number of bags. We’ve set each person up with their own bag to store their tiddlers. We found two difficulties with using recipes to deliver content:

1. You can’t dynamically add the title of a tiddler you’re looking for to a recipe, unless you have a single recipe for e.g. each news article. This means that our recipes deliver a large number of tiddlers, which then have to be filtered further in some way, either in the template or in the jinx plugin;
2. A further consequence is that the recipes get longer as you add more contributors, and have to be maintained. It would be easier to be able to point to all contributors’ bags in a recipe.

As mentioned above, further selection is required after a recipe has delivered a set of tiddlers. This is where the jinx plugin has been developed (and repeatedly re-worked!) the most. Because we can examine the URL requested in the plugin, there is an interdependency between the URL’s structure and the filtering applied to a set of tiddlers returned by a recipe. The URL structure itself is examined further down.

The scenarios we have for wanting to get content out of TiddlyWeb and the corresponding recipes and tiddler selection we’ve set up in the jinx plugin are:

• Pages with one news article (a tiddler tagged ‘article’)

recipe
   /bags/JON/tiddlers?filter=[tag[article]]
   /bags/BOB/tiddlers?filter=[tag[article]]
   /bags/PETER/tiddlers?filter=[tag[article]]

selection
   get_tiddler_from_recipe(tiddler_name,recipe_name)

• Pages with many news articles

recipe
   /bags/JON/tiddlers?filter=[tag[article]]
   /bags/BOB/tiddlers?filter=[tag[article]]
   /bags/PETER/tiddlers?filter=[tag[article]]

selection (no further filtering)
   get_tiddlers_from_recipe(recipe_name)

• Pages with country profile information (tiddlers tagged ‘countryprofile’ from a single contributor), news articles for that country, a tiddler defining themes for a map and a tiddler for the country itself

recipe
   /bags/countries/tiddlers?filter=JAPAN
   /bags/BOB/tiddlers?filter=[tag[countryprofile]]
   /bags/JON/tiddlers?filter=[tag[article]location[JAPAN]]
   /bags/PETER/tiddlers?filter=[tag[article]location[JAPAN]]
   /bags/BOB/tiddlers?filter=[tag[article]location[JAPAN]]
   /bags/ILGA/tiddlers?filter=[tag[maptheme]]

selection (no further filtering)
   get_tiddlers_from_recipe(recipe_name,named_tiddlers_array)

The final example, although still developing, illustrates how difficult it can be to get the tiddlers you need in a way that you can tell them apart. We added the second argument to the get_tiddlers_from_recipe function so that we could pass in tiddlers as named variables to the next step, HTML templating, and refer to them directly rather than iterating through a huge loop of tiddlers each time we wanted to find one.

Formatting stuff – templates

Right from the word go, we’ve wanted to “template” tiddler content through HTML templates, inserting bits and pieces of tiddlers into the relevant places in a mainly static page. The approach we’ve developed so far involves using the jinx plugin to map a particular URL path, say /articles/MyArticle, to a function that collects a set of tiddlers as described above (in this case probably “articles”) and pushes them through a templating process to generate the HTML pages.

A technique we’ve found ourselves using quite frequently is iterating through the delivered set of tiddlers to find the one we want. For example, on country page, we want to be able to isolate the tiddlers which are news articles. At the moment, we can do that like this:

{% for tiddler in tiddlers %}
   {% for tag in tiddler.tags %}
      {% if tag == 'article' %}
         <h2>{{tiddler.title}}</h2>
      {% endif %}
   {% endfor %}
{% endfor %}

This structure is less than ideal and it would be much easier to be able to access the groups of tiddlers directly, which is why the named_tiddlers_arrayparameter was created for get_tiddlers_from_recipe. However, this doesn’t feel very solid yet.

Accessing stuff – URL’s

The URL used to access a tiddler can be accessed and acted on by the jinx plugin, so we have used it to determine what is presented and how, through two methods: first, by mapping various URL structures to functions inside the jinx plugin; second, by using the sections of a URL’s path as variables in these functions, determining things like which template is used, or which tiddler is retrieved.

The mappings we are using are listed below. TiddlyWeb differentiates between/resource and /resource/, so there are double entries for most mappings:

config['selector'].add('/index.html', GET=get_index)
config['selector'].add('/myactivism', GET=get_myactivism)
config['selector'].add('/myactivism/', GET=get_myactivism)
config['selector'].add('/allies', GET=get_allies)
config['selector'].add('/allies/', GET=get_allies)
config['selector'].add('/article/{name:segment}', GET=get_article)
config['selector'].add('/countries/{country:segment}', GET=get_country_section)
config['selector'].add('/countries/{country:segment}/', GET=get_country_section)
config['selector'].add('/countries/{country:segment}/{section:segment}/', GET=get_country_section)
config['selector'].add('/countries/{country:segment}/{section:segment}', GET=get_country_section)

It seems that, in the pursuit of genericism, GET /path/to/resource should be understood as GET /path/to/recipe/tiddler - accessing a recipe, pushing the results through a template (perhaps with the same name as the recipe), and selecting the named tiddler.

Late last year, I helped Dan Morris with a project called “Geekmap“. The idea was to create “a social infrastructure for geeks”, as he put it, which resembled a social network like LinkedIn in that it allowed people to make professional contact, but the whole interface was focussed on navigating via geography and people’s skills. The original idea sounded pretty cool when Dan described it in March, but we didn’t start doing anything serious with it until November. Three months later, in February 2009, we launched the first version and followed it up in March with another release.

geekmap v2

The problem is, we haven’t done anything significant since that March release. In fact, we had a crisis of confidence when we realised that despite our best efforts, the people who we had in mind to first use the site didn’t have a reason to use it. Our reaction was to plan to add some things for some different people and see how that went; except, with the wind already falling from our sails, this lack of encouragement made it very difficult to maintain any momentum and we put the project on hold shortly thereafter.

Today, just over three months since that second release, I think I know why that momentum died – it comes down to our use of Agile methodology to guide our development. Agile starts from the people using the site and their goals, and encapsulates these in “User Stories“, which are your units of development. Having used it all the way through Geekmap and in other projects, I think it’s great for prioritizing your development energy and keeping close to all the non-developers.

I think that the problem was that we used Agile right from the start.

With the best intentions, we used User Stories to figure out the most basic goals of the people using Geekmap and built from there. The eventual outcome was that the site received a fair amount of development attention, but crucially, without any compelling features being built. This was becuase the Agile approach encouraged us to choose the lowest-complexity solution for any given problem – the innovations conceived in the seed idea were always tabled for some time in the near future (always the near future…).

Building to throw away

I think the answer lies in prototyping. We should have started by building a prototype from the original, visionary ideas.

With hindsight, it seems like this would have been very useful for three reasons:

1. unknown or complicated ideas could have been explored in order to find their real utility, which would have made them candidates for fulfilling User Stories
2. people other than Dan could have used the prototype to get to the point where they believed in what they were building and shared the vision
3. it’s a lot easier to find people prepared to become the first customers and become a part of the development if they can see you are making exciting things

I’ve heard plenty of times that the best thing to do with a prototype is to throw it away. I can understand that now in terms of the benefits of starting fresh with all the experience of making that prototype. (Maybe don’t throw it away, open source it…)

AppJet is dead. Long live AppJet.

Aaron Iba, AppJet’s CEO, has made the decision to close AppJet because they realised that EtherPad, their real-time document collaboration product, is commercially viable. Never mind that there is a growing community using the free product: hosting for Server-Side JavaScript with an in-browser editor.

I think this is the worst possible reaction AppJet could have had to the realisation that EtherPad could make them some money. I think this because AppJet is currently the only tool out there where you get the in-browser editor combined with hosting of applications.

The closest competitor to AppJet is Jaxer’s Aptana Cloud service which, although it has a free account, charges $20/month for its cheapest paid-for package. It’s not apples for apples in this case, as Aptana Cloud provides a great deal more in the hosting side of things, such as infrastructure scaling and integration with databases.

However, Aptana Cloud relies on Aptana Studio, which is a chunky piece of desktop software used for developing web software. AppJet gave you a tiny application editor that you accessed through your browser. This encouraged you to think of your applications as pieces of the web, capable of working with other pieces of the web, such as the libraries and applications that other AppJet users had built. The requirement for free hosting was that your application source was published.

These patterns encouraged a vision of the community providing little applications that performed singular tasks.  This is a much different construction to the Aptana Cloud product, which is essentially individualistic hosting of entire web applications and web sites.

It’s true that focussing on a core product is often a good thing to do, and EtherPad is a pretty cool real-time document collaboration tool. The problem that I have with this is that people are already doing that.

I guess this move will work out well for AppJet in the end. I’m just hoping someone somewhere will pick up from where they left off.

Tomorrow is Osmosoft’s 2nd birthday.

Today I’m handing in my resignation: this is it. In just over four weeks, I’ll be working for myself. In a recession? Yeah.

I think it’s a fair assumption that I’ll take on some freelance work to pay the bills, but hey, everyone like’s to dream, so here’s the plan:

I want to help independent, creative people use technology, particularly the web, to expand their market or make their company more efficient. They could be a t-shirt seller in a market, an independent pub-owner, or even run a magazine. It’s a pretty broad definition, but the people I’m thinking of are doing what they’re doing because of both the aesthetic of the thing and because they love it.

I think people like this share a couple of characteristics:

• they are under a lot of pressure to grow their company and make it a success, so they are unlikely to have the time to figure out how to use technology to their advantage;
• they don’t have a lot of capital hanging around to pay for consultants to help them out

I can empathize with this second point, as it’s exactly the position I find myself in now, so I’ve decided on doing things like this: I don’t want to charge people big fees, I want to be a part of their success. This means I’d do something like take a part of whatever increase in profits I can help them make. Which means I need to measure them, but I think that’s a pretty cool thing anyway.

But, like I said, I’ve got to sort out how to pay those regular bills first. So… if you fancy paying me for anything you see me writing about on this blog, please get in touch!

Yesterday, we had a “code jam” at Osmosoft Towers, with the aim of releasing, at 6pm GMT, a polished and usable version of “MediaWiki Unplugged”.

MediaWiki Unplugged is a web application you run locally (in your browser), that downloads content from a MediaWiki, allows you to edit it and sync’s back. The idea, a collaboration between Globant and Osmosoft, is that you’d use this when your Internet connection is flakey or absent, or just to give you a bit of breathing time when composing edits. We did it and the results are here: http://mediawikiunplugged.com – the hashtag is #mediawikiunplugged.

from genglebienne's flickr stream

Here’s the ‘ting – it’s a bit shit. I mean, it works and everything, but it’s not something that we’re really PROUD of. This is a problem for some. For me, this is one of the progressive things we’ve done for a long time and, in my opinion, should set the precedent for how Osmosoft functions as a team. MediaWiki Unplugged might be rough; however, it is new, potentially very helpful and, fundamentally, THERE.

There’s a principle in Agile Development, which Google Code appears to have taken as its motto – “Release early, release often”. This is a scary thing for many people creating new things – there’s always something that can be improved or a problem to fix. The “sensible” way to release new things is to keep them hidden from view until they are polished to a perfect shine, then release to a triumphant fanfare in the form of a press release and flurry of feathery tweets. Sadly, this expectation of great consequence will turn out unmet, the carefully composed press release will go uncited and the tweetstorm will fail to materialise.

To mitigate against this disappointment and optimize your chances of making a big splash with your new stuff, I suggest you instantly adopt the practice of releasing early and often. Here’s why:

1. Other people might not care about the things you think they care about; and there’s a good chance they won’t care about the things you care about. When you’re developing early-stage stuff, this is really important – you give out your product and everyone starts to use it in ways you hadn’t imagined, they put up with the things you thought would kill the experience and they complain about things you’d overlooked. Get your stuff out there earlier, with your hypotheses in your head, and watch and learn as people surprise you.
2. If you let out something you’re not proud of (a software fart, perhaps), it gives you one helluva motivation to sort it out. There are few things more stimulating than criticism or the fear of criticism.
3. You’ll do a better job next time. The eternal feedback loop has commenced and it feels hard. That’s because you’re learning about what shape your things need to be before people will love them. You’d pay people thousands of pounds for this kind of information in the controlled laboratory of market research.

Back to MediaWiki Unplugged then: my ears are open. As are the ears of everyone in the Osmosoft/Globant team. Roll on the next version.

from genglebienne's flickr stream

[Globant CTO Guibert Englebienne took lots of photos and posted them to Flickr, along with a video of Jeremy talking about MediaWiki Unplugged as a TiddlyWiki application, another of himself talking about the day; he also set up a tumblr for the day]

I went to the first day of Yahoo! Open Hack London this weekend – three rooms at Covent Garden’s Congress Centre, a couple of hundred hackers and liberal supplies of catering coffee.

For those unacquainted with the Yahoo!-brand recipe for hack days, their flavour goes something like this: a morning’s talks about Yahoo! hack-worthy technologies and general interest, followed by a 24-hour hackathon started by a supposedly inspiring video and keynote (aka “gas”) from a Yahoo! founder. The day later, the hackers present their produce and prizes are awarded. Everyone goes home to sleep off their pizza hangover.

I wasn’t around on the second day to see the presentations, which is a shame, as I missed out on some good shit: the Dr. Who theme played on an orchestra of 6 iphones and 2 wiimotes, a site that lets you know whether it’s a good or bad day for the UK, a search engine for FreeCycle, a side-scrolling single-touch accessible Google search, an EU-wide TheyWorkForYou (with translations!), a searchable database of all the nation’s blue (and black) plaques, a Twitter-powered taxi-sharing service, a one-shot guest pass for sharing your location…

(The full list of hacks shown is here, but not all have URL’s; see this Tweetmeme page for popular links.)

In the wake of this massive amount of creativity and technical competence, I was thinking about the hacker’s response to problems, as reflected in the frequency of this type of event. That is, put a problem in front of a hacker, and they will try to concoct some sort of solution to it. They will build their way over the problem. “Hacker” here is a description of a mindset, not a specific label for the techno-geek. Hackers can be public-service policy wonks, or garden-centre owners as well as PHP nerds.

I think there’s a lot of this “hack-ish” response to things. To shoot straight to the point I want to elaborate, I think that often, it can be disadvantageous, wasteful of resources and time and disappointing. There is a need for events like this weekend’s hack day, and the smart thing to do is recognise what the effect of 52 interesting hacks is – it is not 52 viable new products and services on the marketplace, it is 52 influences and idea-forming pressures in the heads of the people who, at the end of the day, are the builders and maintenance men of the web. To hack within a hack day is right, it helps. To expect that you can go much further with the hacks is wrong. Hacking on a problem must be understood for its exploratory and inspiring nature, it does not solve the problem.

There’s a phrase used in big biz that describes the hacker’s response – “not made here”. This is applied to product development groups who ignore all of the useful resources outside of the company in favour of building from scratch using the talent in their team (often ignoring the talent in other parts of the company). This is supposed to be a BAD THING, something we are supposed to strive to change. I have spent the last two years working in a team dedicated to the adoption of open source software, possibly the world’s biggest melting pot of already-fixed problems. I know how true it is that there is no such thing as a new idea. And yet, I’m often guilty of assuming that everyone else has done it the wrong way or is irrevocably committed to the wrong path, that the way to implement such-and-such a new idea is to do it again from scratch, myself.

And how many times I’ve been proven wrong! How many half-baked tools and un-loved applications sit on the shelves of my source code repositories. Bah.

Sure, hack. But don’t think that you can transfer a hack to a new, real thing and not have it consume your life. Products take love, care and commitments (like kittehz). They take some big-ass mutherfucka sitting on someone’s head until they fix the bug that no-one wants to tackle but kills it for 66% of people. They take late night seances with the Google PageRank devils. They take more time, effort, pizza and people than you ever thought they would. And they take all these things over and over again.

Someone else is doing all that right now, with your idea. So give over with your glory-seeking and start co-operating.