Teaching self-driving cars how to see

Example_SLU-Mercer.png

The task

In the summer of 2016 Spare 5 (now Mighty Ai) began getting requests from customers to have our community outline objects in images in order to train their computer vision models. After months of iterative development we designed and implemented a tool in our Spare 5 web and iOS apps that would enable our community to efficiently segment and label images completely.

9 months of development

The following images and videos describe how we developed a tool that has put Mighty Ai in a position of strength for automakers and software companies poised to bring self-driving cars to life.

The Polygon Tool

PolygonAllUP.png

starting with polygons

Before we even knew we needed a segmentation tool we designed a developed a product that enabled our users (iOS primarily) to outline objects in images. We had customers who wanted common objects and company logos isolated from their backgrounds.

Opinionated approach

Based on my experience with graphical software tools I knew we shouldn't waste any time developing wand tools or fancy lassos. We needed a vector tool, pure and simple, to cut through the cruft of whatever our customers threw at us. I started the design process for this tool by throwing some mockups into a keynote...

Initial success

We went back and forth a bit with our development team on whether or not and how we should group shapes, and how we should deal with negative space, and within a month had a live polygon feature out in the wild on the Spare 5 iOS app.

On to the big challenge

Designing the polygon tool had its challenges, but our focus was shifting to an even bigger challenge. "How can we design a tool that will enable our community to efficiently outline and label 100% of the objects in an image down to a very minute level of detail?" Again we started by designing for mobile first...

The Segmentation Tool

When we got to a place where the mobile design seemed steady, and our developers had released the feature on iOS, we moved on to designing the web version of the same tool.

 
Sample of redlines I provided. We were working in Sketch + Keynote so .pdfs like this helped our front-end magicians craft an intentional layout.

Sample of redlines I provided. We were working in Sketch + Keynote so .pdfs like this helped our front-end magicians craft an intentional layout.

Optimizing for humans

It was the fall of 2016 and the tool we built was working IRL. But our users were making a lot of mistakes, and a single segmentation could take an individual a really long time. We knew from the beginning these tasks would be daunting for our community, but we also had some ideas about how to make their lives easier by breaking up the process.

 

Results

On the business side

The development of this tool created successes for the company on several fronts. 

  1. In the first half of 2017 we more than doubled the number of tasks we had normally done in 6 months.
  2. Customers with rigorous quality requirements were pleased with the results.
  3. Proven success has generated substantial new business.

On the human side

We implemented a variety of these optimizations and were pleased to find that our community was able to successfully annotate these images at scale. A few sample images are below.