I love social media. Not only is it a great place to rant about politics and share cute baby photos, but it's also great for answering questions.

Yesterday, I asked the following to Twitter and Facebook:
"looking for reference for 'ppl will do more for free as volunteers than they will if you pay them a low wage' -- know where this came from?"

I got a lot responses, indicating that this is a pretty well-studied question with a lot of different results pointing to the general idea.  Thought it might be useful to list them here for other interested folks.

For my purposes, the following seem to be the best:

Uri Gneezy and Aldo Rustichini
(found by Kyle Murray and Eytan Adar)

Winter Mason and Duncan Watts
(found by Clemens Drews, Walter Lasecki)

Other interesting pointers:

Alfie Kohn
(found by Daniel Lowd)

Examples from Freakonomics, specifically the chapter about the Israeli daycare's experience with charging parents for late pickup of children causing more to be late. 
(found by Dan Weld @dsweld, Matt Post)

Dan Ariely's work, for instance http://web.mit.edu/ariely/www/MIT/Papers/zero.pdf"How Small is Zero Price? The True Value of Free Products"
(found by Bertil Hatt  @bertil_hatt)

Enhanced by Zemanta

Sam White, an undergrad in my group, created the turk-powered bartender that you see below.  The gist is that it's a machine that makes rum and Cokes, and Mechanical Turk workers get to decide how strong you'd like your drink. It's really well done (especially for a weekend project) and I'm sure it's a great conversation starter at parties. It also brings some of the unaddressed issues with human computation into the fore.

For instance: What happens if turkers serve drinks that are a bit too strong - are they culpable for bad things that happen?  Do turk bartenders need to be licensed?  If so, where -- does or should a turker in India serving real-world drinks need to comply with liquor laws in New York State?  Do workers understand that the abstract notion of choosing an alcohol level has real-world impact -- aka, they are *really* mixing a drink that someone will consume? If they knew it was real, would they behave the same way?

I don't have answers to these questions, but asking them is becoming increasingly important as the crowd makes it into our everyday lives. In the meantime, please don't let turkers mix your drinks :)
Dear Colleagues,

After months of high-level discussion with university administrators,
I am happy to report that effective Fall 2011, all curriculum in
computer science (courses with CSC designation) will be required to
incorporate components on Human-Computer Interaction, hereafter
referred to as HCI. This decision was reached after extensive research
and discussion, which concluded that people (aka, humans) are in fact
one of the primary end users of all computing systems [1].

Although this decision will affect our broad offering of courses in
different ways, some examples include the following:

- Artificial Intelligence (CSC 242) will be required to include explicit modules on Intelligent User Interfaces and mixed-initiative interaction.
- Operating Systems (CSC 256) will devote at least 51% of classroom time to discussing Windows XP (the most popular human OS [2]).
- Advanced Algorithms (CSC 284) will include RTF (run-time feeling) analysis as a first-class dimension on which algorithms are analyzed.
- Human-Computer Interaction (CSC 212) will remain unchanged.

If you have any questions, please direct them to either Dean Clark (CC'd here) or me.

I look forward to working closely with all of you to implement these changes over the coming months and years.


[1]  UR Technical Report #04012011:  http://bit.ly/ur-tr-04012011
[2]  Web Statistics and Trends: http://www.w3schools.com/browsers/browsers_os.asp
iPhone 4

Image by Witer via Flickr

Bjoern Hartmann's Crowdsourcing Seminar at Berkeley read my VizWiz paper today, and his students provided written comments.  He asked me to share a response, which I did and which I have included below.

Looking back on VizWiz, I think what was interesting was that it showed (i) you can do crowdsourcing in something close to real-time (regardless of how you do it) and (ii) you can use resulting "deployable Wizard-of-Oz" prototypes to learn more about your population.  Through the VizWiz prototype, we found and verified (Soylent ref?) a number of visual questions that blind people might want answered, and isolated problems that effective tools will need to address to make them feasible.

I think nearly all of Bjoern's students found such problems in reading through the VizWiz paper, which is great!  I hope they'll be inspired to go solve them to make a VizWiz-like tool even better and more useful.

==Enabling Blind People to Take Pictures==
Blind people don't have nearly the trouble taking pictures that one might imagine. Think about all of the great non-visual clues that are available!

Nevertheless, here are a number of interesting approaches one might take to either help blind people take better pictures, or lessen the impact of poor-quality pictures. We explored some simple approaches in the paper (darkness/blur detection), and have expanded the capabilities a lot since then while working with some local blind photographers (who already take some pretty great pictures). Our current version gives users the option to record a video instead of still photos, although at the cost of latency to send a larger file. The best improvement so far came by simply upgrading to the iPhone 4 and its better camera and flash! To me, what I think is interesting about the VizWiz study is that it showed how far you can get with low-fidelity input and crowdsourcing, especially when the capture of that input is mediated by human intelligence (the blind user). Generally, what happens when a question comes back with an answer like "the picture is bad" is that the user will take another picture and ask again.

==Answer Quality==
Our mechanism for dealing with answer quality was to present multiple answers to users. Most strategies you might consider to ensure answer quality end up delaying the answer -- for instance, waiting for other users to verify. We decided to rely on the user to make sense of the answers, especially given that answers were correct the majority of the time. We actually saw zero malicious answers.

Depending on how you look at it, a correct (and quick) "there's nothing in this image" is a great answer because it signals that the person should try again. But, users wanted more interactivity. VizWiz highlights challenges that research could address in facilitating such interactivity -- you can get very interactive responses by pairing a user with a particular worker, but how do you keep that worker around for the whole interaction?  What if that worker ends up not being a "good" worker? How do you pair a user with a group of workers and have the interactions still make sense?  All great questions.

==Other Latency-Reducing Strategies==
What I like about this paper is that (I think) it introduces the idea that crowdsourcing could happen in something like real-time. I agree that strategies like signaling to workers when work is available may be more cost effective than keeping them busy-waiting, but busy-waiting ends up being cost effective if you have enough users. The additional complexity of the signaling system may not be worth it in the end. Yet another great problem to explore more.

Enhanced by Zemanta

Is Bing Copying Google? - Jeff's Research Blog

| No Comments | No TrackBacks
Reports have surfaced alleging that Bing has been copying Google's search results.  Google claims to have verified this with a "sting" operation in which they rigged their search engine to return a bogus page for terms for which no one would normally search, had some employees search for those terms using Internet Explorer with the Bing toolbar from home, and lo and behold Bing started returning the same bogus pages for the same meaningless keywords.

You can see the original article and Bing's response.  Bing essentially admits that for users who opt-in, it records usage data (like clicks on Google) and uses that as one of many factors determining search engine results. Search engines for a long time have used people's observed behavior on search results pages to influence ranking -- if people keep clicking (and staying) on the result the search engine is returning 3rd for a query, then it might get promoted to 1st. This would be the first time that a search engine (has been caught) doing this with another search engine's results.

So, what's the difference?  Google would have you believe that Bing is stealing their search results, and that's partially true. Search result quality can be measured along two axes, precision and recall.  Roughly, higher precision mean better search results up top, and higher recall means the page you're looking for is more likely to appear in the search results at all.

In my opinion, what Bing is doing to up their precision is mostly legitimate.  They're presenting users with a bunch of choices of links to click (which happen to come from Google), and using their behavior to influence Bing's sense of what makes a good search result for a query. But, a less morally clear side effect is to up their recall -- if a user clicks on a link that Google includes in their search results that Bing does not, then the sting would have us believe that Bing will add that URL to its results, thereby increasing its recall on Google's back. But, both Bing and Google include most popular web pages in their search results already, so what we're really arguing about is recall at the long tail of the web.

The long tail of the web are the massive number of web pages that are highly relevant to a small number of infrequently search phrases. Since these aren't popular web pages, many try to argue that they don't matter, but, in fact, these are precisely the pages over which the next search engine battle is likely to be fought. Bing and Google both have money, they can pay an army to crowdsource good results for the top 10k, 100k, 1m results. But, they can't pay for good results for the next 100m or 1b queries, so it might make sense to seed Bing's view of the long tail on Google's back - even if they get caught in a sting.

It will be interesting to see what happens here. Is your usage data on Google the de facto property of Google, or can you choose to donate (or sell) it to the likes of Microsoft? Does this come down to Microsoft attempting to use its dominance in the browser market to hedge in on Google's dominance in web search?  Will Danny Sullivan discover that Steve Balmer is his long lost step brother in law?  Only time will tell.  
Enhanced by Zemanta
I just submitted short descriptions of three research projects that undergraduates can get involved in -- I thought I'd post them here so you get an idea of some of the cool work we're doing at ROC HCI.

Real-time Human Computation
Over the past few years, human computation -- integrating the intelligence and decision-making skills of people in computational processes -- has been shown a practical means to add true intelligence to computer programs today.  As an example, computer vision is difficult, and so it can make sense to have a computer program query humans out on the web when it needs information about an image, instead of trying to do this automatically. Research goals include (i) developing methods for quickly integrating the input of dynamic pools of workers into actionable decisions, (ii) designing and implementing toolkits that enable developers to easily include real-time human computation as part of their own programs, and (iii) devising methods for estimating the expected latency for answers from different sources of human computation from past experience. Students working on this project will participate in the design of methods for achieving effective real-time computation and contribute to an open source toolkit allowing others to use real-time human computation in their own projects.

Human-Backed Access Technology
The past few decades have seen the development of wonderful new computing technology that serves as sensors onto an inaccessible world for disabled people - as examples, optical character recognition (OCR) makes printed text available to blind people, speech recognition makes spoken language available to deaf people, and way-finding systems help keep people with cognitive impairments on track. Despite advances, this technology remains both too prone to errors and too limited in the scope of problems it can reliably solve to address the problems faced by disabled people in their everyday lives. A promising approach for enabling people with disabilities to take advantage of this technology now is to let the error-prone technology fall back to human-powered services when it fails. For instance, if an OCR program is unable to recognize text, it may query always-available workers on services like Amazon's Mechanical Turk. In this project, students will extend an iPhone application that we have created called VizWiz that lets blind users take a picture, speak a question, and receive answers back in less than 30 seconds from workers on the web. Students will add in new automatic services, such as OCR and simple computer vision components (color detection, darkness detection, etc), and enable questions to be sent to social networks like Facebook and Twitter. Students will need to address the research and design challenge of helping users decide where to send their questions based on dimensions such as latency, accuracy, privacy, and anonymity.

Cloud-Based Assistive Technology in the Classroom
Millions of students with disabilities in the United States use assistive technology programs to help them use computers and learn classroom material. These programs range from screen reader programs that convert the visual information on a computer screen to audible speech for blind people, to speech recognition programs that enable people with physical disabilities to control their computers, to reading programs that speak and highlight words as students read. A primary problem with this technology is that it is not available on every computer that students access, and, even when the technology is there, the specific settings and preferences of the students must be repeated.  A promising solution to these problems is to host assistive technology in the cloud so it can be accessed from anywhere and from any device with a web browser.  In this project, students will design and build web applications that can replicate the complex, multimodal transformations of traditional assistive technologies within the restrictive web sandbox, and investigate the potential of these web applications by disabled students in local schools.
I've been railing for weeks against the short-sighted and downright ignorant attack on science funding from House Majority Whip Eric Cantor -- check out the website where he's inviting the public to help slash projects they don't understand.
Representative Eric Cantor of Virginia

The man trying to undercut the USA's dominance in science and engineering.

Image via Wikipedia

Not understanding science leads to silly recommendations, often directly opposing your own goals. For instance, the following examples of wasteful spending provided by Republican Eric Cantor apply directly to defense: (i) computer models to analyze the on-field contributions of soccer players, and (ii) modeling the sound of objects breaking. Automatically determining contributions of soccer players is directly applicable to analyzing and understanding troop movements and battlefield dynamics. A member of my department at the University of Rochester has DARPA funding to understand player coordination in games of Capture the Flag, so the Department of Defense obviously gets the connection.  Similar technology is all the rage in prison technology to help understand inmate movements, prevent violence before it happens, and protect guards. I'll leave it to the reader to figure out why scientists are modeling games first before moving on to real battlefield situations.

As for modeling the sound of objects breaking -- that's directly applicable to automatically understanding what's happening during battles, especially the small urban battles characteristic of today's military engagements. The wall above me was hit by a bullet, wouldn't it be great if I knew from the sound of the brick breaking what kind of bullet it was and where it came from?

What makes all of this even more ridiculous is the scale of the funding.  Sure, a grant worth several hundeds of thousands of dollars seems like a lot.  But, it's likely spread over several years, and provides funding and education to at least 3-4 people over that period.  Most of the funding goes to students, who in exchange for their education do advanced work in the national interest for far below the going rate.  The annual funding of NSF is less than $8 billion, which amounts to less than a month in Iraq, even by conservative estimates.  But then, we got such a good return on investment for the $1 trillion we've spent there and are continuing to spend.

It seems that this backlash on science is part of a broader trend. Somehow the conventional wisdom has become that scientists are out of touch, flitting away tax dollars on meaningless projects.  We as scientists may be partly to blame for this and should do more to connect with the broader community, but it doesn't lessen the importance of scientific research. The reality is that science is what made and maintains the dominance of the United States across everything from economics to medicine, and materials to the military.  Without research funding for science, the Internet and Google may not exist.  But, more likely, they would have been invented slightly later in Europe or China. For more of the historical importance of revving the science engine of the states, see George Will's Op-Ed in the Washington Post.

In closing, my statement to Congress:
"Keep science strong. Keep the U.S. strong. Don't succumb to short-sighted, ignorant pressure to gut innovation by eliminating science funding."
Enhanced by Zemanta
The National Federation of the Blind (NFB) filed a complaint with the Department of Ed against Penn State University last Friday due to the inaccessibility of its technology properties (primarily its web site(s), speaker podiums, and debit cards).  This is worth paying attention to for a lot of reasons.  Foremost in most peoples' minds is that the NFB recently won
The Nittany Lion and a Penn State cheerleader ...

Image via Wikipedia

a large settlement (multiple millions of $$) against Target Corporation for the inaccessibility of their web site, but to me it's more illustrative of a broader trend that reflects growing frustration among people with disabilities on the lack of improvement in accessibility. For years, it seemed that education was the answer for making web sites accessible -- most of the accessibility problems are really easy to fix if they're done at the time the site is created, but developers just don't know that they're important. But, after years of this not working, and, after high-profile successes like the Target case, I think we'll increasingly see more cases like this.

The NFB states in its complaint that this problem isn't unique to Penn State, and having been at several universities myself over the past few years I can certainly confirm that.  If the NFB is successful in its complaint (even in part), it will be interesting to see what changes occur at universities. There are a number of legitimately difficult problems to solve to make university web pages accessible -- first, they have a ton of legacy content that would be incredibly costly to fix up if it wasn't created correctly in the first place.  Secondly, universities often use complicated (and universally reviled) third-party software packages to which they are all but locked-in -- the NFB lists ANGEL at Penn State; at the University of Rochester we use Blackboard (which was recently certified by the NFB - we joke it is now equally horrible for everyone); at UW they used an in-house project (that was actually accessible, and decently usable). Finally, there is not currently one (or even several) central authorities for university web pages -- at a minimum, each department and organization in the university has its own web presence. These are managed by companies, contractors, in-house people, and students, and so it's not always clear who to blame. Course homepages are even worse as many of these are created by the faculty themselves!

Increasingly, we're seeing people with disabilities resort to legal challenges to accessibility problems - I think this is a natural response to the dismal job we've all done in giving our best effort to create accessible web content. Universities are an interesting case because of the distributed nature of web production, and so it will be especially interesting to see the outcome of this case and what universities do in response.
System Diagram that connects Google App Engine, to Twitter, to EchoFon on iPhones

I really like Google Voice, but Apple refuses to accept it in the iPhone's App Store. Google provides a web-based Google Voice - the version just released today (1/26) looks pretty much like the Google Voice iPhone application that was rejected from the store.  The problem is that the web page is pull-only - and without push you don't get notified of new text messages or voicemails - you have to actually visit the site.  I set out to try to fix this, and came up with a solution that works really well and is completely legitimate -- no Apple approval required!

The summary is this - my Google App Engine application polls Google Voice every 15-20 seconds looking for new text messages. If a new text message is found, then the application sends a direct message containing the sender of the text message and the message itself to my Twitter account from a Twitter account that I set up for this purpose. I have EchoFon running on my phone, which does Push notifications (I needed the $4.99 Pro version of EchoFon to do this), so that when a new DM arrives my phone vibrates and rings.

The end effect is that when I get texts on my Google Voice account, my phone acts almost exactly like it would if I had received a text message from AT&T.  I've timed the round trip and it's less than 90 seconds on average. I can respond to the message using the web-based version of Google Voice by clicking a link included with the message.

Because this uses Twitter, there's nothing Apple or AT&T can deny from the App Store unless they agree to deny all Push applications (instead of Twitter, these could be Facebook messages, emails, fake point-of-interest near your location) - so, you don't need approval to run it!

I'll update this post with details of this solution in a few days so you can get free Google Voice push on your phone too!
Reblog this post [with Zemanta]

Science Editor:
 Hey boss, we just got an incredible submission!  It's titled "Merry-Go-Round Warp Drive" and it explains how to exceed the speed of light!  See, all you gotta do is get the merry-go-round going up to like 99.9% of the speed of light and then make it really big.  The outside part will then be going way faster than the speed of light.

Science Senior Editor: Who is this guy?  What lab's he at?

Science Editor:  That's the thing - this is a 12-year-old genius savant.  He turned it in for his sixth-grade science homework.

Science Senior Editor: Buy the rights. In fact, buy the rights to the whole damn class' homework assignments - we'll bundle 'em all up into a special issue, maybe sell it to Nature!

Better yet, let's bundle 'em, then auction them off paragraph by paragraph!

Grad Student: But the articles won't make any sense then.

Dr. Bigwig, Ph.D.:  Exactly!  They'll be difficult to understand, causing everyone to believe they must be incredibly ground-breaking, driving up prices (and our reputations).

...a few months later in a lab near you

Grad Student:  I've got this great idea for how to cure cancer.  If we can get people moving up to the speed of light then they'll become super-conductors and we can zap them and all the free radicals in their bodies without hurting them. It could stave off or even cure existing cancers!

Dr. Bigwig, Ph.D.:  I'm a bit skeptical - how would you get them up to the speed of light, isn't that impossible?

Interference of split light beams

Image via Wikipedia

Image via Wikipedia

Grad Student:  No!  There's this new result out of McKinley Middle School that shows how to exceed the speed of light - we'd just have to spin them really, really fast and then zap them.

Dr. Bigwig, Ph.D.: You know, I've heard that there's a new result showing how if you spin something fast enough it can be made to move faster than the speed of light.  We could do that...and then zap them.

Grad Student: Right that's what I ...

Dr. Bigwig, Ph.D.: Ok, then we're all set!  What a great idea I had!  I'm going to be famous!

Grad Student: But can we get IRB approve it?

Dr. Bigwig, Ph.D.: Ha!  IRB's are a worry of the past - the past few administrations have realized that the IRB was stifling creativity - they still exist, but their responsibilities have been reduced to getting coffee for one another and hassling HCI students wanting to do surveys. They've realized that we wouldn't dare do something dangerous - it would hurt our reputations, so subject protection is self-moderating.

Grad Student:  Great!  So I'll schedule the first couple of participants and see how they react.

Dr. Bigwig, Ph.D.: That's so slow, I need pay-off now.  Bring in 100 people, do it all at once, you'll save time and money.

Grad Student:  You're so smart Dr. Bigwig, Ph.D.  I'll schedule them all as soon as I can attach this platform to this blender motor and rig up this wire for them to hold to an electrical outlet.

...a few days later at a hospital near you

Resident: Doctor!  We have just received 100 patients all with electric shocks, burns, and dizziness. I've quickly surveyed them and it appears the patient John Q. is in the worst shape - we should get him to surgery stat!

Doctor: Damn residents! Get this jerk out of here, bring in C. E. Oppenheimer, and tell that jackass to keep up with the latest medical research before gambling with people's lives.

Now C., I'm glad I was able to intervene and see you first - your chart indicates your high levels of cash - is there anything I can get you?  Coffee, juice?

...at a conference on a small island paradise off Morocco

Dr. Bigwig, Ph.D.:  "...and so after 2 years, of the 100 subjects in the experimental condition, all of whom were previously diagnosed with stage 4 cancer, NONE had died from cancer."

... later that evening

Dr. Bigwig, Ph.D. (on phone with dean): "The university has onlly awarded me 20 x current tuition for my annual bonus - this is simply unacceptable. My contract clearly states that I am to be awarded 30 x annual undergrad tuition. You know the Ivies have been calling me - if you want to remain keep the top talent, you need to reward me for my stellar performance."

Dean: "But Dr. Bigwig, you killed 100 people...I know you're worth 30 x annual undergraduate tuition, but you know the parents and alumni - they just don't understand. How about I give you 10 full undergraduate tuition tickets that you could sell to future undergraduates in a year after these crazy politics have died down?"

Stay tuned for next week's article ---  what if banking were like science?
Reblog this post [with Zemanta]