I’ve been working during the last fortnight on the “Streams” project, which LShift, an IT consultancy in Shoreditch, are building in conjunction with the BBC. My part in this project has been to prototype the interface that a non-technical person would use to edit and create Streams, which are box-and-wire visual workflows similar to those found in Yahoo! Pipes or Tarpipe.
(The most interesting difference is that Streams combines Tarpipe’s event-driven model with Yahoo! Pipe’s modular data processing model. For more on this project and its ambition to open-sourcing, see LShift’s and the BBC’s blog posts.)
I have been using WireIt, an open-source JavaScript library, to aid with the interface prototyping. Fortunately for me, WireIt’s latest version (v0.4), announced shortly before I started this project, includes a fully-featured “Wiring Editor”, which gets you up and running with the familiar Yahoo! Pipes-style editor.
The brief for the prototype was to be able to complete a round-trip of loading up a Stream (stored as a document in CouchDB), edit it, save it (back to CouchDB) and be able to load it up again.
I thought it would be helpful to bring to your attention the things you need to be aware of if you want to use WireIt’s Wiring Editor and adapt it to work with an independent source of module and workflow data.
Stage 1 – different data formats
It is hardly worth saying that there is no point trying to persuade your data store owners to change their data format so your job is made easier. Apart from this obvious need to convert from one data format to another, you also have to deal with the fact that WireIt only keeps hold of the data it needs to display and configure its modules.
The best place I could find to hook into the loading process was by overriding the function called after the load button is pressed:
WireIt.WiringEditor.prototype._onLoad = WireIt.WiringEditor.prototype.onLoad;
WireIt.WiringEditor.prototype.onLoad = function() {
this.service.listWirings({group:true}, {
success: function(result) {
var responseText = result.responseText;
if(responseText) {
this.pipes = convertFromStreams(responseText);
this.pipesByName = {};
this.renderLoadPanel();
this.updateLoadPanelList();
this.loadPanel.show();
}
},
failure:function() {
console.log('failure when loading, args:',arguments);
},
scope: this
});
};
The data format of a combination of blocks and wires in WireIt looks like this:
var modules = [
{
name: ,
working: {}
}
]
where <em>working</em> is defined as:
working = {
"modules":[
{
"name":"FormContainer",
"config":{
"position":[115,58]
},
"value":{
"email":"",
"firstname":"",
"happy":true,
"lastname":"Dupont",
"title":"Mr",
"website":""
}
}],
"properties":{
"description":"",
"name":"teste"
},
"wires":[
{
"src":{
"moduleId":0,
"terminal":"br"
},
"tgt":{
"moduleId":1,
"terminal":"tr"
}
}
]
}
The convertStreams function changes each stream document available in CouchDB into the format above.
WireIt doesn’t keep hold of any properties you attach to blocks that aren’t used explicitly. This means you need to keep hold of any properties you need to save back to your data store. I did this by keeping a globally available variable, WireIt.customFields:
...
if(node.type) {
WireIt.customFields.nodes[i].type = node.type;
} else {
WireIt.customFields.nodes[i].terminal = node.terminal;
}
...
Stage 2 – loading the list of blocks from your data store
As well as loading up descriptions of a full combination of blocks and wires, I wanted to load all the available blocks in CouchDB into WireIt so they’d be available for making workflows with.
The WireIt Wiring Editor doesn’t load its blocks from a remote system, you have to provide all the definitions in a JSON “language” file when you create the Wiring Editor. I hooked into the setup of the Wiring Editor after it had finished loading its backend-describing SMD file (see below for explanation of this SMD malarkey):
WireIt.WiringEditor.prototype.onSMDsuccess = function() {
WireIt.customFields = {};
this.populateModules();
};
The populateModules function is responsible for turning the description of the block from CouchDB into the format WireIt uses for defining available blocks, which looks like this (note it is different to the format used to describe a block that is part of a workflow, described above):
var modules = [
{
"name": "detect_iplayer",
container: {
xtype: "WireIt.FormContainer",
type: "regexp_replace",
title: "Detect iplayer and remove",
icon: "images/detect_iplayer.png",
fields: [
{"inputParams": {"name": "caseinsensitive", "label": "case insensitive?", "value":true}, "type": "boolean"},
{"inputParams": {"name": "regexp", "label": "regular expression", "value": "(iplayer)"}},
{"inputParams": {"name": "dotall", "label": "dot matches all?", "value": false}, "type": "boolean"},
{"inputParams": {"name": "multiline", "label": "multiline?", "value": false}, "type": "boolean"},
{"inputParams": {"name": "replacement", "label": "replacement", "value": "\\1"}}
],
width:250,
terminals: [
{"name": "input", direction: [-1,0], offsetPosition: {left: 110, top: -15}},
{"name": "positive", direction: [1,0], offsetPosition: {left: 60, bottom: -15}},
{"name": "negative", direction: [1,0], offsetPosition: {right: 60, bottom: -15}}
]
}
},
...
]
Stage 3 – adding default layout
The Streams data store doesn’t contain any information about how the blocks in a stream should be laid out, as this is not important to the execution of the streams. However, for loading a stream into WireIt, I wanted to give the blocks a default layout so they didn’t end up on top of each other.
I picked a layout that put the blocks equally spaced around a ring. This algorithm gives you the coordinates of a set of count blocks. It assumes the blocks are going to be square, and sets the radius of the ring so that the blocks would just touch if they were square.
function calculateLayout(count,width) {
var coords = [];
var n = count || 4;
var h = width || 100;
var radius = (n/Math.PI + 1.5)*h/Math.sqrt(2);
var theta = 0;
for (var i=0; i<n; i++) {
theta = 2*Math.PI*i/n;
coords[i] = [];
coords[i][0] = radius*(1+Math.cos(theta)); // x
coords[i][1] = radius*(1-Math.sin(theta)); // y
}
return coords;
}
Stage 4 – different load/save API (and the limitation of yui-rpc)
The Wiring Editor is hard-wired to use an implementation of JSON-Schema called “YUI-RPC”, along with a defining Service Mapping Description (SMD – think WSDL for JavaScript). The down-side to this is that the implementation does not currently support HTTP verbs other than GET and POST, nor is it obvious how to extend the implementation in line with the spec. If your API requires the use of PUT (as CouchDB does to update or create named documents), then you will need to do something about this.
YUI-RPC supported what I needed for loading, so supporting CouchDB was only as hard as changing one of the method specifications:
"listWirings": {
"transport": "GET",
"envelope": "URL",
"target": "couchdb/_design/feeds/_view/join",
"parameters": [
{"name":"group", "type":"boolean"}
],
"description": "Get a list of streams"
}
To support PUT for saving, I overrode WireIt.WiringEditor.prototype.saveModule and, deciding to go with what I know, stuck in a block of jQuery that I’d had hanging around from a test working with CouchDB’s API.
General comments on prototyping with single-page applications
I made the prototype as an application that could just be served as a single HTML page from a file: URI. I strongly support this over trying to make your prototype work with some web framework or other, as you will lose valuable time setting up the framework in the first place.
Working with HTML files loaded as files gives you a couple of special properties: you can read and write from the file system and make cross-domain XMLHttpRequests (on many browsers). I had to modify YUI’s Connect object a bit to support the cross-domain XHR requests. For more information on the special properties of HTML pages viewed over file: URI’s, see these sections of a W3C position paper about TiddlyWiki written by Osmosoftonian Paul Downey.
