February 2011 Archives

Bjoern Hartmann's Crowdsourcing Seminar at Berkeley read my VizWiz paper today, and his students provided written comments.  He asked me to share a response, which I did and which I have included below.

Looking back on VizWiz, I think what was interesting was that it showed (i) you can do crowdsourcing in something close to real-time (regardless of how you do it) and (ii) you can use resulting "deployable Wizard-of-Oz" prototypes to learn more about your population.  Through the VizWiz prototype, we found and verified (Soylent ref?) a number of visual questions that blind people might want answered, and isolated problems that effective tools will need to address to make them feasible.

I think nearly all of Bjoern's students found such problems in reading through the VizWiz paper, which is great!  I hope they'll be inspired to go solve them to make a VizWiz-like tool even better and more useful.

==Enabling Blind People to Take Pictures==

Blind people don't have nearly the trouble taking pictures that one might imagine. Think about all of the great non-visual clues that are available!

Nevertheless, here are a number of interesting approaches one might take to either help blind people take better pictures, or lessen the impact of poor-quality pictures. We explored some simple approaches in the paper (darkness/blur detection), and have expanded the capabilities a lot since then while working with some local blind photographers (who already take some pretty great pictures). Our current version gives users the option to record a video instead of still photos, although at the cost of latency to send a larger file. The best improvement so far came by simply upgrading to the iPhone 4 and its better camera and flash! To me, what I think is interesting about the VizWiz study is that it showed how far you can get with low-fidelity input and crowdsourcing, especially when the capture of that input is mediated by human intelligence (the blind user). Generally, what happens when a question comes back with an answer like "the picture is bad" is that the user will take another picture and ask again.

==Answer Quality==

Our mechanism for dealing with answer quality was to present multiple answers to users. Most strategies you might consider to ensure answer quality end up delaying the answer -- for instance, waiting for other users to verify. We decided to rely on the user to make sense of the answers, especially given that answers were correct the majority of the time. We actually saw zero malicious answers.

Depending on how you look at it, a correct (and quick) "there's nothing in this image" is a great answer because it signals that the person should try again. But, users wanted more interactivity. VizWiz highlights challenges that research could address in facilitating such interactivity -- you can get very interactive responses by pairing a user with a particular worker, but how do you keep that worker around for the whole interaction?  What if that worker ends up not being a "good" worker? How do you pair a user with a group of workers and have the interactions still make sense?  All great questions.

==Other Latency-Reducing Strategies==

What I like about this paper is that (I think) it introduces the idea that crowdsourcing could happen in something like real-time. I agree that strategies like signaling to workers when work is available may be more cost effective than keeping them busy-waiting, but busy-waiting ends up being cost effective if you have enough users. The additional complexity of the signaling system may not be worth it in the end. Yet another great problem to explore more.

You can see the original article and Bing's response.  Bing essentially admits that for users who opt-in, it records usage data (like clicks on Google) and uses that as one of many factors determining search engine results. Search engines for a long time have used people's observed behavior on search results pages to influence ranking -- if people keep clicking (and staying) on the result the search engine is returning 3rd for a query, then it might get promoted to 1st. This would be the first time that a search engine (has been caught) doing this with another search engine's results.

So, what's the difference?  Google would have you believe that Bing is stealing their search results, and that's partially true. Search result quality can be measured along two axes, precision and recall.  Roughly, higher precision mean better search results up top, and higher recall means the page you're looking for is more likely to appear in the search results at all.

In my opinion, what Bing is doing to up their precision is mostly legitimate.  They're presenting users with a bunch of choices of links to click (which happen to come from Google), and using their behavior to influence Bing's sense of what makes a good search result for a query. But, a less morally clear side effect is to up their recall -- if a user clicks on a link that Google includes in their search results that Bing does not, then the sting would have us believe that Bing will add that URL to its results, thereby increasing its recall on Google's back. But, both Bing and Google include most popular web pages in their search results already, so what we're really arguing about is recall at the long tail of the web.

The long tail of the web are the massive number of web pages that are highly relevant to a small number of infrequently search phrases. Since these aren't popular web pages, many try to argue that they don't matter, but, in fact, these are precisely the pages over which the next search engine battle is likely to be fought. Bing and Google both have money, they can pay an army to crowdsource good results for the top 10k, 100k, 1m results. But, they can't pay for good results for the next 100m or 1b queries, so it might make sense to seed Bing's view of the long tail on Google's back - even if they get caught in a sting.

It will be interesting to see what happens here. Is your usage data on Google the de facto property of Google, or can you choose to donate (or sell) it to the likes of Microsoft? Does this come down to Microsoft attempting to use its dominance in the browser market to hedge in on Google's dominance in web search?  Will Danny Sullivan discover that Steve Balmer is his long lost step brother in law?  Only time will tell.