The dataset we will use is Leeds Buildings 2011 – it’s shows all the buildings in the District of Leeds, UK, that were in the OpenStreetMap database as of July 2011. I made it from the England shapefile export from GeoFabrik.de, and can highly recommend checking out the download pages at GeoFabrik if you want to play with some OpenStreetMap data yourself.

Leaflet is a modern, lightweight BSD-licensed JavaScript library for making tile-based interactive maps for both desktop and mobile web browsers, developed by CloudMade. I think it’s pretty nice, and they have some good examples. I’d be interested to hear if you can view this map with mobile devices.

Get Leaflet.

First go ahead an get Leaflet from http://leaflet.cloudmade.com/. Make sure you add on your page the link to the correct js file in the dist directory.

Make a Map

We set up everything on the init function called by the body onload event. var myMap = new L.Map('map'); sets everything up with a div in the body with the id as “map” <div id="map" style="height: 90%;"></div>

Add Acetate tile layer

Acetate is the result of GeoIQ’s set of Mapnik stylesheets and algorithms to create good quality basemaps especially for overlaying data on top.
It’s a tile layer, and tile layers are usually defined according to the ZXY scheme. An example tile URL would be http://a2.acetate.geoiq.com/tiles/acetate/13/4059/2638.png

var acetateUrl =  'http://{s}.acetate.geoiq.com/tiles/acetate/{z}/{x}/{y}.png';
var acetateAttrib = '2011 GeoIQ & Stamen, Data from OSM and Natural Earth';
var acetate = new L.TileLayer(acetateUrl, {maxZoom: 18, attribution: acetateAttrib, subdomains: ['a1', 'a2', 'a3']});

The first line, we define the pattern for Leaflet to understand. The second line defines the attribution to show with the layer, and the last line, creates the TileLayer and passed in some options, the main thing here is the use of the {s} variable in the url and the subdomains option. {s} will be replaced with a1, a2 or a3, to create a proper image, like above.

var leeds = new L.LatLng(53.796, -1.551);
myMap.setView(leeds, 14).addLayer(acetate);

Now we create a LatLng object, centred over Leeds, UK, and then set the view for the map, and method chain adding the acetate layer to it. Once you’ve done this, you will actually see a map!

GeoiQ Features API & Get JSON Features

The features REST API allows a developer to get all the features within a radius, specifying the lat lon of the centre of the radius and the distance.

var geojsonLayer = new L.GeoJSON();

var url = "http://geocommons.com/datasets/168923/features.json?lat=53.796&lon=-1.551&radius=3&callback=?"; 

jQuery.getJSON(url, function(data){
	jQuery.each(data, function(i,f) {
		f.properties = {name: f.name, osm_id: f.osm_id, type: f.type};
		f.type = "Feature";
		// Now Push into geojson layer each feature
		geojsonLayer.addGeoJSON(f);
	});
});

myMap.addLayer(geojsonLayer);

lat=53.796&lon=-1.551&radius=3 Is our spatial filtering. It says, given a point at Lat 53.796 and Lon -1.551, give me all the features within a radius of 3KM. Interestingly, we can also use the units param to specify the units of distance, if you like your imperial units!

I’m using JQuery here, as it’s got some nice helpers. This grabs the JSON, and loops over the features, adding features to the geojson layer. Using the addGeoJSON method call.

Did you know that the features API also has the ability to filter by many different ways, for example by polygon, and by buffered polygon? polygon=-77,34,-78.5,34.5,-82,32,-79,30,-77,34&radius=100. I particularly like the filter functionality. In our example, we could have just filtered the map to just show hospital buildings.
http://geocommons.com/datasets/168923/features.json?filter[type][][equals]=hospital&callback=?

Add popup

We have to use the featureparse event on the geojsonLayer, and specify this before any things are added to the map.

geojsonLayer.on("featureparse", function (e) {
if (e.properties && e.properties.name){
   e.layer.bindPopup(e.properties.name +"<br />"+e.properties.type);
}

The featureparse event is called just before features are added to the layer. Here it’s looking at the feature, and if it has a name in the properties, then to create a popup, showing the name and type

Add click handler

We add a click handler within the same feature parse event handler.

if (e.properties && e.properties.osm_id) {
 e.layer.on('click', function(ee){
   Query(".info").html("<a href='http://www.openstreetmap.org/browse/way/" +
   e.properties.osm_id+"'>OSM ID: "+e.properties.osm_id + "</a><em> Name: </em>"+
   e.properties.name + "  (<em>type: </em>" +e.properties.type+")");
 });
 }

Here, we use a bit of Jquery to update the value of a div with the details of the feature. It shows the osm id, and a link to the openstreetmap website for that feature, and some values.

Add categorical styling

This bit is a bit more complex, but essentially, the SVG features are styled using CSS, and can be styled accordingly. Within the same featureparse event handler, again, set up before features are added to the layer, we look to see if the feature has a type (like, hospital, retail etc), and if it has, then it’s given a style. We only really change the style colour.

if (e.properties && e.properties.type && e.layer.setStyle){
      var color = "#334DCC";
      var t = e.properties.type;
      var colormap = {
        "hospital":"#CC3399",
        "residential":"#CC6633",
        "house":"#CCB333",
        "school":"#99CC33",
        "retail":"#D864B1",
        "commercial":"#006600"};
      
      color = colormap[t] || color;

 e.layer.setStyle({"color":color,"weight": 2,"opacity": 1 });
}

Give it a go!

Try out the playable demo, and if you want you can have a look at the code too.
All the code can be found here:
And a playable dem

Loading KML Files

Frankly, its not exactly obvious from our API documentation what the best way to do this is. And if you aren’t familiar with Curl, the examples are probably not going to help you much, so I’ll be doing this code in Java. Of course, we here at GeoIQ are Ruby programmers, and thus have a natural disdain for anything to do with Java, so I’m probably losing serious Ruby street cred just posting this, but anything for the good of the cause. We will be using the occasionally obtuse Geocommons REST API, but I’ll try to steer you around some of the not so obvious pitfalls.

The basic idea of the program is that you would run it from the command line, passing in your login, password, and a directory. It then scans the directory for KML files and uploads them to geocommons. Fortunately, Java elegantly handled getting the files using a FileFilter:

public class Loader {

    public static void main(String[] args) {
        for (String filePath: kmlFilesIn(dirPath)) {
            // load file here
        }
    }

    /**
     * Determines if file will be uploaded
     *
     * @param dirPath path to file
     * @return array of filenames for kml files in specified directory
     */
    private static String[] kmlFilesIn(String dirPath) {
        String[] filePaths = new File(dirPath).list(new FilenameFilter() {
            public boolean accept(File dir, String name) {
                return name.endsWith(".kml");
            }
        });
        if (filePaths == null) {
            throw new RuntimeException( "Invalid directory: " + dirPath);
        }
        return filePaths;
    }
}

POSTing the file

cat 98633.csv | curl -i -X POST -u "username:password" --data-binary @- -H "Content-Type: text/csv" http://geocommons.com/datasets.json

You might think we could do something similar and send the raw kml data in the post with the content type set to “application/vnd.google-earth.kml+xml”, but I’m not sure this will work. Even if it did, as a general approach, it has some major shortcomings — for example, you can’t send any other data in the request. Instead, we will be sending the data as if it came from a form, with a Content Type of “multipart/form-data”. While it may be more complex at first, it can be used in virtually all cases where we are POSTing or PUTing data to GeoCommons.

For some reason, multipart form data  doesn’t seem to well supported in Java’s networking libraries (or Ruby’s, for that matter). The code to do format multi-part form data isn’t particularly hard to write, but can be a little tedious to get correct, particularly if you can’t see what’s happening on the server side. Instead we’ll be using the HttpComponents project from Apache. You can download HttpCore and HttpClient (I got versions 4.1.3 and 4.1.2 respectively) from here.

So, lets add a method to our class to post a file:

/**
 * Posts a file to a given URL
 *
 * @param url URL string to post to
 * @param file file to post
 * @return location returned
 * @throws LoaderException if we don't get a good response from the server
 * @throws Exception if file fails to load
 */
private String post(String url, File file) throws IOException, LoaderException {
    HttpClient httpclient = new DefaultHttpClient();
    HttpPost request = new HttpPost(url);
    MultipartEntity entity = new MultipartEntity();

    entity.addPart("dataset[kml]", new FileBody(file));
    request.setEntity(entity);
    HttpResponse response = httpclient.execute(request);
    if (response.getStatusLine().getStatusCode() == 201) {
        return response.getFirstHeader("Location").getValue();
    } else {
        throw new LoaderException("Failed to process file: "
            + EntityUtils.toString(response.getEntity()));
    }
}

Basically, we create HttpClient and HttpPost objects, add our file, submit the request, and handle the response. In handling the response, we have a couple of options. When you POST a new object to GeoCommons, it should respond with a representation of the data posted (based on the extension you gave in the url, “json” in this case). I recommend in most cases getting your response as json, since it is handled more consistently across the GeoCommons API. In addition to the JSON, the response will have a URL in the “Location” header you can use as an HTTP end-point for further REST requests. Since 90% of the time we are going to need that URL for reasons that will be come obvious later, this method return the URL rather than parsing the JSON content.

You might be wondering what is in that JSON response:

{ "success": true, "state": "processing", "id": 1234, "title": "myfile", "desc": "" }

The two important fields here are the id, which you are going to need to do things like add a dataset to a map, and the state, which will be described later. You can actually get the id from the URL in the location field (or vice-versa), since it would just be “http://geocommons.com/1234.json”.

Authenticating

HttpClient seems to have support for doing authentication, but after half an hour of searching the JavaDocs, reading the examples, and trying out code, I was unable to figure out how its supposed to work. Fortunately, all we really need to do for Basic Authentication is set  a header in the request with the username and password encoded using Base64 encoding, so we can skip all those confusing authentication classes and just do:

/**
 * Add basic authentication header to request
 *
 * @param request
 */
private void addAuthentication(HttpRequestBase request) {
    String usernamePassword = login + ":" + password;
    String encodedUsernamePassword
 = DatatypeConverter.printBase64Binary(usernamePassword.getBytes());
    request.addHeader("Authorization", "Basic " + encodedUsernamePassword);
}

Waiting for Completion

• complete – we’re done with this file, and can move on
• processing – GeoCommons is still processing the file, we need to wait
• errored – Some unrecoverable error occurred; we’re done with the file, but should inform the user
• anything else – Geocommons stopped processing the file, and its now in some intermediate state waiting for further input.

Unfortunately, Java (at least as of version 1.6) doesn’t have a built in library for parsing JSON, but as usually, there are a ton of third party libraries that will do JSON just fine. Being a simple person, I ended up using json-simple. You can download version 1.1 from here.

So, our code to check the state of the dataset looks like:

/**
 * Checks the specified dataset url to see if the dataset has completed.
 *
 * @throws IOException in the unlike event that the response can't be processed
 * @throws LoaderException if we don't get a good response from the server or geocommons
 * has a problem with the file
 */
private boolean isComplete(String url) throws IOException, LoaderException {
    String datasetJson = get(url);
    JSONObject dataset = (JSONObject) JSONValue.parse(datasetJson);
    String state = (String) dataset.get("state");
    if (state.equals("errored")) {
        throw new LoaderException("Error processing file");
    }
    return !state.equals("processing");
}

/**
 * Sends a GET request to a URL, and returns the response body as a string
 *
 * @param url url to get
 * @return string representing the response content
 * @throws IOException in the unlikely event that the response body can't be processed
 * @throws LoaderException if we don't get a good response from the server
 */
private String get(String url) throws IOException, LoaderException {
    HttpClient httpclient = new DefaultHttpClient();
    HttpGet request = new HttpGet(url);
    addAuthentication(request);
     HttpResponse response = httpclient.execute(request);
     if (response.getStatusLine().getStatusCode() == 200) {
         return EntityUtils.toString(response.getEntity());
     } else {
         throw new LoaderException(
             "Failed to get dataset: " + UntityUtils.toString(response.getEntity()));
     }
}

We are sending the GET request, then parsing the response content as JSON and pulling out the “state” property.

Putting it all together, the loader will now load each file, wait until it finishes loading, then move on to the next

/**
 * Loads multiple kml files to GeoCommons.
 *
 * This program will upload all kml files in the specified directory to geocommons.
 *
 * To run:
 *
 * java com.geoiq.Loader mylogin mypassword /dir/to/files
 */
package com.geoiq;

import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FilenameFilter;
import java.io.IOException;

import javax.xml.bind.DatatypeConverter;

import org.apache.http.HttpResponse;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.entity.mime.MultipartEntity;
import org.apache.http.entity.mime.content.FileBody;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
import org.json.simple.JSONObject;
import org.json.simple.JSONValue;

public class Loader {

    public static final String SITE_URL = "http://geocommons.com";

    private String login;
    private String password;

    public Loader(String login, String password) {
        this.login = login;
        this.password = password;
    }

    /**
     * @param args command line arguments
     *
     * args[0] - geocommons login
     * args[1] - geocommons password
     * args[2] - directory
     */
    public static void main(String[] args) {
        String login = args[0];
        String password = args[1];
        String dirPath = args[2];

        Loader loader = new Loader(login, password);

        for (String filePath: kmlFilesIn(dirPath)) {
            loader.load(new File(dirPath, filePath));
        }
    }

    /**
     * Determines if file will be uploaded
     *
     * @param dirPath path to file
     * @return array of filenames for kml files in specified directory
     */
    private static String[] kmlFilesIn(String dirPath) {
        String[] filePaths = new File(dirPath).list(new FilenameFilter() {
            public boolean accept(File dir, String name) {
                return name.endsWith(".kml");
            }
        });
        if (filePaths == null) {
            throw new RuntimeException( "Invalid directory: " + dirPath);
        }
        return filePaths;
    }

    /**
     * Loads the specified kml file to geocommons, tracking state until load completes
     *
     * @param file file to load
     */
    public void load(File file) {
        System.out.println("loading" + file.getPath());
        try {
            String url = post(SITE_URL + "/datasets.json", file);
            while (!isComplete(url)) {
                Thread.sleep(3000);
            }
        } catch (LoaderException e) {
            System.err.println("Failed to load " + file.getPath());
                e.printStackTrace(System.err);
            } catch (IOException e) {
            System.err.println("Failed to load " + file.getPath());
            e.printStackTrace(System.err);
        } catch (InterruptedException e) {
            System.err.println("Interrupted");
        }
    }

    /**
     * Checks the specified dataset url to see if the dataset has completed.
     *
     * @throws IOException in the unlike event that the response can't be processed
     * @throws LoaderException if we don't get a good response from the server or
     * geocommons has a problem with the file
     */
    private boolean isComplete(String url) throws IOException, LoaderException {
        String datasetJson = get(url);
        JSONObject dataset = (JSONObject) JSONValue.parse(datasetJson);
        String state = (String) dataset.get("state");
        if (state.equals("errored")) {
            throw new LoaderException("Error processing file");
        }
        return !state.equals("processing");
    }

    /**
     * Sends a GET request to a URL, and returns the response body as a string
     *
     * @param url url to get
     * @return string representing the response content
     * @throws IOException in the unlikely event that the response can't be processed
     * @throws LoaderException if we don't get a good response from the server
     */
    private String get(String url) throws IOException, LoaderException {
        HttpClient httpclient = new DefaultHttpClient();
        HttpGet request = new HttpGet(url);
        addAuthentication(request);
        HttpResponse response = httpclient.execute(request);
        if (response.getStatusLine().getStatusCode() == 200) {
            return EntityUtils.toString(response.getEntity());
        } else {
            throw new LoaderException(
                "Failed to get dataset: " + EntityUtils.toString(response.getEntity()));
        }
    }

    /**
     * Posts a file to a given URL using basic authentication
     *
     * @param url URL string to post to
     * @param file file to post
     * @return location returned
     * @throws LoaderException if we don't get a good response from the server
     * @throws Exception if file fails to load
     */
    private String post(String url, File file) throws IOException, LoaderException {
        HttpClient httpclient = new DefaultHttpClient();
        HttpPost request = new HttpPost(url);
        addAuthentication(request);	 

        MultipartEntity entity = new MultipartEntity();
        entity.addPart("dataset[kml]", new FileBody(file));
        request.setEntity(entity);
        HttpResponse response = httpclient.execute(request);
        if (response.getStatusLine().getStatusCode() == 201) {
            return response.getFirstHeader("Location").getValue();
        } else {
            throw new LoaderException(
                "Failed to process file: " + EntityUtils.toString(response.getEntity()));
        }
    }

    /**
     * Add basic authentication header to request
     *
     * @param request
     */
    private void addAuthentication(HttpRequestBase request) {
        String usernamePassword = login + ":" + password;
        String encodedUsernamePassword
            = DatatypeConverter.printBase64Binary(usernamePassword.getBytes());
        request.addHeader("Authorization", "Basic " + encodedUsernamePassword);
    }

    /**
     * Exception thrown when geocommons doesn't behave as expected.
     */
    private static class LoaderException extends Exception {
        private static final long serialVersionUID = 1L;

        public LoaderException(String messge) {
            super(messge);
        }
    }
}

It’s a C port of the Java Topology Suite but they both do the same things very well – basically they do spatial functions for processing geometries.

What does it do?

The kinds of things it can do include:

• Supports all OGC Geometry types, like Points, Polygons etc
• Area calculations
• Distance between geometries
• Point in polygon calculations
• Intersects, overlaps, touches, contains operations
• Buffer, difference, union
• Geometry simplification

And we use it for pretty much all of the above!

How to get it

Since we use Ruby, we use the GEOS Ruby Bindings. I get mine from the UbuntuGIS repository and install the libgeos-ruby1.8 package (and the associated dependencies). I’m assuming you would know how to go ahead and get it.

You can also try out the newer FFI bindings for GEOS and let me know how you get on, and you can also use JTS with JRuby if you feel like it too!

Donut Photo by http://www.flickr.com/photos/amylovesyah/

How to use it

Donut Buffers

Here’s a snippet of code that is used when you want to buffer a dataset.

def buffer(feature, radius, min_radius=0.0)
   wkb_writer = Geos::WkbWriter.new
   wkb_reader = Geos::WkbReader.new

   geom =  wkb_reader.read_hex feature.geometry
   buffer_geom = geom.buffer(radius)

   if min_radius != 0.0
      hole_geom = geom.buffer(min_radius)
      buffer_geom = buffer_geom.difference(hole_geom)
   end

   feature.geometry = wkb_writer.write_hex buffer_geom
   feature
end

Going through the code, the method takes in a feature object, which has a geometry attribute which in this case is in  the HEX WKB format  (01010000002FDD2406816154C0C66D3480B7A03B40 for example)

Geometries can be stored in many formats, and GEOS can read them. But the key bit is that we have to use a GEOS::WkbReader to read in a string object and convert it into a geometry.

POINT(-81.5235 27.6278) Would be the same point geometry but in Well Known Text, or WKT.

So, we read the geometry in, and convert it into a special GEOS object, ready to be worked upon.
The next line down is where the buffer operation takes place.

buffer_geom = geom.buffer(radius)

Pretty simple, eh?

This makes a polygon around a buffer, with the radius variable as distance.

The next few lines are donut making – we see if the method was called with a min_radius value, buffer the same point but with the smaller radius and then do a difference operation on that!

buffer_geom = buffer_geom.difference(hole_geom)

And the final of the method uses the geos wkb_writer to write back the new donuted geometry for that feature.

There are small touch-ups throughout the platform, Semprebon‘s preview images of datasets and maps, to larger capabilities such as data editing and versioning (did someone say “git for data?”). And clearly the most pronounced change was an entire rewrite of “the App formerly known as Maker” (TAFKAM) – simply put, our map visualization. We’re still using the very powerful and capable Flash engine, and Andrei has performed some magic to render a lot of features (think 100′s of thousands) but we’ve moved the rest of our entire interface to JavaScript and HTML thanks to Derek. The benefit of that is that with our beta Polymaps support from Chris you can now not only view maps on your iPad, but you can build visualizations and analyses on your iPad!

And lastly, but most importantly yet not nearly as noticeable, we’ve swapped out our infrastructure. Matt and Justin have raised GeoCommons completely to the cloud where we’re now able to easily scale out any piece of our infrastructure on demand. Combine that with Tim‘s work with our asynchronous processing we have truly brought geospatial data management, visualization and spatial analysis to the cloud – free for everyone to use.

Stay tuned for more details on these features I’ve mentioned, including some technical details on how we’ve scaled out our system to provide unique and capable data analysis to the web. And don’t think we’re stopping there – this new infrastructure has already enabled us to quickly add new features and tweaks. We’d love to hear your feedback, praise, or any issues you run into on our Community forum. And if you’re a passionate GeoCommons user, a developer and want to work for a fun, dynamic, and exciting team – drop us a line.

There were several talks that were really great, and we were able to meet some truly amazing developers while we were there too. Here’s a short list of some of the talks and techs that I found to be interesting:

The first talk of the conference was really good. It was titled Bytes and Blobs by the one and only David Flanagan, and it focused new HTML5/JS APIs for handle all sorts of binary data structures in the browser. Though its still early to start playing with, the raw file data API (which centers of passing around BLOBs) seems like it holds large amount of potential. Reading client-side files is just one of many features of this API, and one that could have a direct implication on the geo-spatial industry over the next years. Think about the ability to map data without having to actually upload it, or even doing format conversion in javascript without ever touching a server? interesting idea for sure. The entire group of these client-side binary data APIs was very impressive.

An early talk in the conference “B Track” was Tim Caswell on Node Powered Desktop Apps. A lot of people might ask “why JS on the desktop?” and this talk was a good introduction to a sub-theme of the conference: JavaScript everywhere. Why? cause JavaScript is clean, simple and becoming more and more engrained within every part of the web. It makes sense that more developers would be comfortable writing more backend applications in JS. So with the explosion of interest in Node.js people are really starting catch onto the idea that JS is not limited to just the browser and that it possesses some potent capabilities for app development in other areas. One of the libraries Tim mentioned and showed off was node-GTK which is a node binding to libgtk – awesome.

Later on in the first day we saw Avni Kharti speak about visualization tools on the web. Her talk was okay, but was as close as anyone seemed to get to mentioning geo-spatial tools on the web. She focused on several tools and APIs that most geo geeks would be really familiar with (basically the Google toolkits like Google Maps and Fusion Tables). I was shocked she didn’t once mention GeoCommons in her talk, it truly and utterly shocking. Every downside of every tool in her talk is handled elegantly by GeoCommons. I made a point to tell her all about it afterward, and she admitted the next day that she wished she’d seen it before her talk.

The second to last talk of the conference was Brendan Eich & Jeremy Ashkenas. Brendan is one the fathers of Javascript and gave a large audience a glimpse into some of the future features that we might expect in JS. There was strong theme of writing less code, and being more efficient in JS within his slides (and I was excited to the idea of list generators creep in there). Next Jeremy Ashkenas (creator of Backbone.js and CoffeeScript) gave a thorough talk on the concepts, benefits and features of CoffeeScript. It was great to see CoffeeScript in action and get a first hand description of how it works from the creator himself.

Of course this is a very short list of only a few of the many awesome talks this year. I heard over and over again that people’s brains were exhausted from all the great new libs and ideas that were being talked about. Even though this was my first JSConf, I have a feeling it won’t be my last, and I am really looking forward to it next year.

We’ve recently added some new features to our JavaScript API as well as redone the documentation to make it easier to read. Chris has been hacking on Node.js and Polymaps. Derek’s been working on our charting ability with Raphael. Come see what we’ve done and bring your own project ideas.

So sign-up and see you there!

Step 1 – Roles

Define who is going to use it. Understand at a high level what they are going to want to do. Instead of this “bob wants to click on ‘click here to map’ and then wants to click ‘analyze my data’” do this “bob wants to make a map and analyze data”. Don’t presuppose the UI flow in your definitions. Use as much or as little detail as you need to capture the mindset of a few different kinds of people. If you can talk to actual customers in these roles, do it. If you can observe them using an existing UI then even better.

If you can, try to define roles that defy the biases of you and your team. If you’re slow and methodical, imagine someone who isn’t. If you are fast and impatient, try using your product with your opposite hand or a different input device. The consumer products company OXO, renowned for their understanding of ergonomics, keeps a large collection of gloves of all shapes and sizes so their designers can better understand the roles of those they serve.

Step 2 – Requirements – What does it need to do?

I use the term requirements very loosely. This is about collecting all the ideas, thoughts and hunches of everyone involved in the project. It’s about finding the specific mandates and vague notions about the design. Make a list anywhere, on paper, online, on a napkin, in a spreadsheet, wherever, of all the things people want it to be. Don’t confuse this with the massive requirement documents made by sweaty guys in dimly lit conference rooms used to enslave Java developers in the 90′s.

Step 3 – Absolute Requirements – What does it absolutely need to do now?

Next, cut the list down to the absolute must-haves. It’s great to have a huge list of requirements, but be super critical about which ones are important for the current phase.

Prioritize the list by looking at two items and thinking “If I could keep just one, which would it be”. Decide what on the list has to be handled in the first phase and what can iteratively follow in subsequent phases.

The full list will inform your choices for later, you may decide to allow something to have a flexible height so it can accommodate future requirements when they become critical. But for now, just cut it down to the must-haves.

Step 4 – UI Inventory – A list of what needs to be designed

Now, take the requirements list and turn it into an inventory of UI elements. If you’re designing a home page it might look like this:

Header
   Logo
   Tagline
   User Menu
       Login Name
       Sign Out
       Account Settings
       Thumbnail
Marketing Message
   Headline
   Slideshow
   Call To Action button
   Secondary Action
Content
   Case Studies
   Product Gallery
   ...

The idea here is to capture everything that will need to fit on the screen. List in great detail all the elements and sub-elements that you can think of. If something is vague, it’s ok, but try to make it more specific if you can. For our search results redesign we listed “quality”, maybe that can be better stated as “quality icon” or “quality gauge”.

I’ve found that the more time I put into the UI inventory the easier the design goes once I jump into drawing. I’ve also found it valuable to share the inventory with the team as a sanity check. I haven’t gotten much feedback on my inventories since they can be a bit daunting, but it at least gives everyone a glimpse into the design process and they can see how their personal favorite ideas are being worked into the design.

The inventory serves as a great place to record thoughts that would be hard to illustrate. I often put notes about interactivity into the inventory:

For Each Layer:

• Title (reveals layer menu on click)
• [menu] hint (appears on hover)
• Attribute Icon (reflects current styles)
• Attribute Name (truncated if necessary)

These things are hard to illustrate prior to designing and going through the process of illustrating them may not add much value, so I make a note of it and move on.

I consider the inventory to be one of the most helpful ways of organizing UI projects. In one particularly complex redesign, I spent almost full week in just a text editor planning out the UI inventory, no sketches or photoshop comps at all. It was worth it as it provided a lot of direction for the steps that followed.

Step 5 – Prioritize the inventory

Now start removing stuff. Be ruthless as you cut and demote items. Remember that everything on the list will cost the company time and effort to build and maintain. Each item on the list will result in a series of bugs that keep you working late on a Friday.

Eliminate till it breaks. Keep removing and demoting items on the list so that you have as few elements on screen as possible. The highest priority items should be critical to your entire audience, the kinds of things that the product simply won’t function without. Stop when you eliminate a feature that 20% of your users will miss and will whine endlessly about. If you’re not sure, cut it and see if anyone notices. I’ve cut out a lot of features this way that and haven’t heard a peep.

Be ruthless even with your own favorite features. For example it would be cool to have a glossy sliding panel that bounces into place but that feature is going to cost time and effort. At this point it’s all about what people need to see rather than what they want to see. It will be easier to add the wants later, though more often than not I’ve found that addressing the needs counts for much more.

By the end of this step, you’ll have a good idea of what needs to be addressed in the wireframes. You’ll be able to focus all of your mental capacity on layout, geometry, balancing prominence, etc. You won’t have to worry about forgetting a critical feature. I use this process because it greatly reduces the number of times that I have to work late redesigning an entire flow simply because I forgot to include a small but important element.

Step 6 – Sketch artist

Start sketching, with a real pen on real paper. Steal some of those 11×17 sheets from the bottom drawer of the copier. They’re free (to you), disposable and you won’t be afraid to fill them up with scribbles. Think of your inventory as a puzzle. Brainstorm out all of the possible layout solutions. Use a broad pen, don’t sweat the details, avoid writing words. The goal is to do this until you feel like you have a good idea that everything on the inventory has a sensible place.

The big thing here is to figure out how much space is needed for different things. I call it the Prominence Hierarchy and for me it’s one of the most important parts. The key is to figure out, separate from the design, what needs to be most important and what can be the least. What happens if you dedicate two thirds of the space to an image versus one third? What happens if you put a lot of emphasis on the title? Or, on the navigation options? Get a good feel for the different ways pieces and parts can fit together.

Consider different ways of hiding elements. What happens if you use an Accordion control vs. tabs?

I’ve found that using pen and paper allows a creative flow that is much harder to initiate on a computer screen. Try getting away from your computer (and smart phone) and go somewhere where you can focus completely uninterrupted.

Step 7 – Wireframes

At some point, something will click in your head and you’ll want to rush to your computer and get started. Or you might just need to place some things on screen to see how much room they really take up. Go for it.

Use a program like omnigraffle, visio, axure or anything that lets you draw up some simple wireframes. Just boxes and lines. Treat it like a whiteboard to verify the direction you’re going and communicate your design to a couple people to validate the direction.

I find it helpful to show my work to one or two people as I go. The earlier the design the fewer people I show it to. This eliminates the group think and design-by-committee mentality that can kill good design. It usually only takes one or two people to point out the biggest flaws in the early designs. Any more people than that and you’re just wasting time. Then as more people are brought into the loop the design has a bit more momentum and maturity resulting in less wasted time for everyone.

Step 8 – Graphical Design

One advantage of this process is that it can easily be handed off to a designer at any point. Even if all you’ve done is work through the roles, that can be enormously helpful to a designer. Having a firm grasp of the requirements is even more helpful. Having a list of what needs to be designed per screen, even better. Some designers will disagree and may prefer to work out some of those steps for themselves. If so, great, that’s a mark of a good designer (my opinion). At the very least you’ve built a common vocabulary and you can weigh the designer’s solution against yours and have a deeper combined understanding.

If you’re not doing the graphical design, bundle up everything that you’ve done and send it to a designer. Also send them your favorite inspiration from dribbble.com, 365psd.com and screen grabs of elements from sites you like. Avoid sending things that are not clearly related to what you’re doing. Write a few lines about what you like from each example if it isn’t already obvious. i.e. you “love the typography but not the colors”, you “love the layout but feel free to experiment with other options”.

If you are responsible for the graphical design, jump into photoshop or the program of your choice and start making it real. Avoid going directly to code and css, you’ll lose a lot of flexibility that is critical at this stage.

Either way, it pays to have a basic understanding of design principles. Here’s the set of principles I measure my designs against.

Consistency

Use the same font and size whenever possible. I didn’t do this in my early design with Fortius and it shows. Lately I’ve standardized on 14px Helveitca Neue for everything. It’s done a lot to smooth out the UI and brought some consistency to it.

Repetition and Rhythm

Learn about grids. In photoshop set your grid to 16px with 4 divisions. When in doubt, align it to the grid. Look at the Layers palette PSD and you’ll see how the height of the header, the spacing between lines, the padding within layers, all aligns to the grid in some way. This took a few extra hours to perfect but the results are worth it. There’s a rhythm and simplicity to it that is hard to get when you’re simply eyeballing the dimensions.

Alignment

One of the big differences between novice design and expert design is the number of alignments the design contains. If you have a text on several different lines and they all start at different horizontal positions, they’re not aligned. Align them and the design will feel more cohesive. Align boxes and UI elements with each other both horizontal and vertical if you can. Align the top of the zoom bar with the top of the layers palette and the style palette.

Position

Keep like things together. Don’t put logout far away from the user name. Keep attribute count near feature count. Don’t force people to scan from left to right just to match something up.

Keep in mind the human fovea is small, if you put related things too far away from each other they’ll fall into the peripheral vision and people simply won’t see them. You’ll see it because you know it’s there, but others wont. Look at one point in the design, how much can you see at one time, if you’re very focused it’s probably just a few square inches. This is why designs with too much whitespace can actually hinder usability.

Consistency, Repetition, Alignment, Position: CRAP for short.

Resources

At any time in this process, get inspiration.

• smashingmagazine.com – general layout, color and design inspiration.
• patterntap.com – A brilliant site where you can browse neatly categorized design examples in categories like Forms, Navigation, Signup, Layout, Introductions, etc.
• dribbble.com – Like twitter for graphic designers. See what top designers are making right now.
• 365psd.com – Tons of downloadable photoshop examples. Lots of silky smooth buttons and UI elements. I wish I knew about this a year ago when they started.

Also spend some time at sites that do things well like mint.com, delicious.com, squarespace.com, any of the 37signals products.

Train your eye for what designers are doing and often more importantly, what they’re not doing. For example, back when browsers introduced repeating backgrounds, I felt compelled to add funky textures to every page I designed. Then I looked around the web at the sites made by the top designers in the field and I saw white. Lots and lots of white. No one was using funky textures and for good reason.