Teaching self-driving cars how to see

context

Spare5 is a pair of web and mobile apps with a global community of users who get paid to tag, categorize, and draw shapes on images using our consumer-level annotation tools. These annotated images subsequently get used by our customers to train computer vision algorithms. The following are a few examples of interaction design artifacts I created in the process of developing various aspects of the tools.

Outlining shapes

In order to satisfy the customers using our service we needed a way to cleanly outline shapes from busy backgrounds, so we designed and built a vector-based tool for drawing polygons in our mobile app. I created mockups and several iterations of a clickthrough prototype which enabled us to envision the experience and come to consensus on what we would build.

Segmentation Tool

Our customer needs evolved into a desire for full image segmentation (where every area of the image gets an outline and corresponding label), and we took learnings from the polygon tool to begin design and development for both web and mobile versions of a segmentation tool.

When we got to a place where the mobile design seemed to be working well, and our developers had released the feature on iOS, we moved on to designing the web version of the same tool.

Optimizing for humans

Segmenting an image could be a slow and laborious process for a single user, so we designed a few different ways that the work could be shared.

Below are a few examples of images that were segmented using our tool.

Sequencing & Complexity

As the complexity of our annotations grew we needed a way to simplify the annotation process and enable more user control while enabling them to track objects within a sequence. So we redesigned our web interface to include an annotation list, timeline, and details panel. These mockups show a user doing a bounding box task.

Review

Creating annotations (labels and shapes) on sequenced images was one challenge, but then designing a way for users to review the annotations was another. I created some user flows and wireframes to try and make sure our users would have an intuitive system to use for review.

I put the wireframes below into a step-by-step clickthrough to thoroughly explain the user flows.

Conclusion

Our annotation tools are used by a global community — hundreds of thousands of people who task on our platform. These tools, especially within the Spare5 web app, have dramatically increased our company’s reach and enabled us to annotate millions of images at high velocity. New users come onto our platform every day and are able to begin annotating and earning money in a very short amount of time.