Jeff's Research Blog

Sam White, an undergrad in my group, created the turk-powered bartender that you see below.  The gist is that it's a machine that makes rum and Cokes, and Mechanical Turk workers get to decide how strong you'd like your drink. It's really well done (especially for a weekend project) and I'm sure it's a great conversation starter at parties. It also brings some of the unaddressed issues with human computation into the fore.

Dear Colleagues,


After months of high-level discussion with university administrators,
I am happy to report that effective Fall 2011, all curriculum in
computer science (courses with CSC designation) will be required to
incorporate components on Human-Computer Interaction, hereafter
referred to as HCI. This decision was reached after extensive research
and discussion, which concluded that people (aka, humans) are in fact
one of the primary end users of all computing systems [1].


Although this decision will affect our broad offering of courses in
different ways, some examples include the following:

- Artificial Intelligence (CSC 242) will be required to include explicit modules on Intelligent User Interfaces and mixed-initiative interaction.
- Operating Systems (CSC 256) will devote at least 51% of classroom time to discussing Windows XP (the most popular human OS [2]).

Reports have surfaced alleging that Bing has been copying Google's search results.  Google claims to have verified this with a "sting" operation in which they rigged their search engine to return a bogus page for terms for which no one would normally search, had some employees search for those terms using Internet Explorer with the Bing toolbar from home, and lo and behold Bing started returning the same bogus pages for the same meaningless keywords.

I just submitted short descriptions of three research projects that undergraduates can get involved in -- I thought I'd post them here so you get an idea of some of the cool work we're doing at ROC HCI.

Real-time Human Computation
Over the past few years, human computation -- integrating the intelligence and decision-making skills of people in computational processes -- has been shown a practical means to add true intelligence to computer programs today.  As an example, computer vision is difficult, and so it can make sense to have a computer program query humans out on the web when it needs information about an image, instead of trying to do this automatically. Research goals include (i) developing methods for quickly integrating the input of dynamic pools of workers into actionable decisions, (ii) designing and implementing toolkits that enable developers to easily include real-time human computation as part of their own programs, and (iii) devising methods for estimating the expected latency for answers from different sources of human computation from past experience. Students working on this project will participate in the design of methods for achieving effective real-time computation and contribute to an open source toolkit allowing others to use real-time human computation in their own projects.

Human-Backed Access Technology
The past few decades have seen the development of wonderful new computing technology that serves as sensors onto an inaccessible world for disabled people - as examples, optical character recognition (OCR) makes printed text available to blind people, speech recognition makes spoken language available to deaf people, and way-finding systems help keep people with cognitive impairments on track. Despite advances, this technology remains both too prone to errors and too limited in the scope of problems it can reliably solve to address the problems faced by disabled people in their everyday lives. A promising approach for enabling people with disabilities to take advantage of this technology now is to let the error-prone technology fall back to human-powered services when it fails. For instance, if an OCR program is unable to recognize text, it may query always-available workers on services like Amazon's Mechanical Turk. In this project, students will extend an iPhone application that we have created called VizWiz that lets blind users take a picture, speak a question, and receive answers back in less than 30 seconds from workers on the web. Students will add in new automatic services, such as OCR and simple computer vision components (color detection, darkness detection, etc), and enable questions to be sent to social networks like Facebook and Twitter. Students will need to address the research and design challenge of helping users decide where to send their questions based on dimensions such as latency, accuracy, privacy, and anonymity.

Cloud-Based Assistive Technology in the Classroom
Millions of students with disabilities in the United States use assistive technology programs to help them use computers and learn classroom material. These programs range from screen reader programs that convert the visual information on a computer screen to audible speech for blind people, to speech recognition programs that enable people with physical disabilities to control their computers, to reading programs that speak and highlight words as students read. A primary problem with this technology is that it is not available on every computer that students access, and, even when the technology is there, the specific settings and preferences of the students must be repeated.  A promising solution to these problems is to host assistive technology in the cloud so it can be accessed from anywhere and from any device with a web browser.  In this project, students will design and build web applications that can replicate the complex, multimodal transformations of traditional assistive technologies within the restrictive web sandbox, and investigate the potential of these web applications by disabled students in local schools.

I've been railing for weeks against the short-sighted and downright ignorant attack on science funding from House Majority Whip Eric Cantor -- check out the website where he's inviting the public to help slash projects they don't understand.

Not understanding science leads to silly recommendations, often directly opposing your own goals. For instance, the following examples of wasteful spending provided by Republican Eric Cantor apply directly to defense: (i) computer models to analyze the on-field contributions of soccer players, and (ii) modeling the sound of objects breaking. Automatically determining contributions of soccer players is directly applicable to analyzing and understanding troop movements and battlefield dynamics. A member of my department at the University of Rochester has DARPA funding to understand player coordination in games of Capture the Flag, so the Department of Defense obviously gets the connection.  Similar technology is all the rage in prison technology to help understand inmate movements, prevent violence before it happens, and protect guards. I'll leave it to the reader to figure out why scientists are modeling games first before moving on to real battlefield situations.

As for modeling the sound of objects breaking -- that's directly applicable to automatically understanding what's happening during battles, especially the small urban battles characteristic of today's military engagements. The wall above me was hit by a bullet, wouldn't it be great if I knew from the sound of the brick breaking what kind of bullet it was and where it came from?

What makes all of this even more ridiculous is the scale of the funding.  Sure, a grant worth several hundeds of thousands of dollars seems like a lot.  But, it's likely spread over several years, and provides funding and education to at least 3-4 people over that period.  Most of the funding goes to students, who in exchange for their education do advanced work in the national interest for far below the going rate.  The annual funding of NSF is less than $8 billion, which amounts to less than a month in Iraq, even by conservative estimates.  But then, we got such a good return on investment for the $1 trillion we've spent there and are continuing to spend.

It seems that this backlash on science is part of a broader trend. Somehow the conventional wisdom has become that scientists are out of touch, flitting away tax dollars on meaningless projects.  We as scientists may be partly to blame for this and should do more to connect with the broader community, but it doesn't lessen the importance of scientific research. The reality is that science is what made and maintains the dominance of the United States across everything from economics to medicine, and materials to the military.  Without research funding for science, the Internet and Google may not exist.  But, more likely, they would have been invented slightly later in Europe or China. For more of the historical importance of revving the science engine of the states, see George Will's Op-Ed in the Washington Post.

In closing, my statement to Congress:
"Keep science strong. Keep the U.S. strong. Don't succumb to short-sighted, ignorant pressure to gut innovation by eliminating science funding."

The National Federation of the Blind (NFB) filed a complaint with the Department of Ed against Penn State University last Friday due to the inaccessibility of its technology properties (primarily its web site(s), speaker podiums, and debit cards).  This is worth paying attention to for a lot of reasons.  Foremost in most peoples' minds is that the NFB recently won a large settlement (multiple millions of $$) against Target Corporation for the inaccessibility of their web site, but to me it's more illustrative of a broader trend that reflects growing frustration among people with disabilities on the lack of improvement in accessibility. For years, it seemed that education was the answer for making web sites accessible -- most of the accessibility problems are really easy to fix if they're done at the time the site is created, but developers just don't know that they're important. But, after years of this not working, and, after high-profile successes like the Target case, I think we'll increasingly see more cases like this.

The NFB states in its complaint that this problem isn't unique to Penn State, and having been at several universities myself over the past few years I can certainly confirm that.  If the NFB is successful in its complaint (even in part), it will be interesting to see what changes occur at universities. There are a number of legitimately difficult problems to solve to make university web pages accessible -- first, they have a ton of legacy content that would be incredibly costly to fix up if it wasn't created correctly in the first place.  Secondly, universities often use complicated (and universally reviled) third-party software packages to which they are all but locked-in -- the NFB lists ANGEL at Penn State; at the University of Rochester we use Blackboard (which was recently certified by the NFB - we joke it is now equally horrible for everyone); at UW they used an in-house project (that was actually accessible, and decently usable). Finally, there is not currently one (or even several) central authorities for university web pages -- at a minimum, each department and organization in the university has its own web presence. These are managed by companies, contractors, in-house people, and students, and so it's not always clear who to blame. Course homepages are even worse as many of these are created by the faculty themselves!

Increasingly, we're seeing people with disabilities resort to legal challenges to accessibility problems - I think this is a natural response to the dismal job we've all done in giving our best effort to create accessible web content. Universities are an interesting case because of the distributed nature of web production, and so it will be especially interesting to see the outcome of this case and what universities do in response.

Science Editor:  Hey boss, we just got an incredible submission!  It's titled "Merry-Go-Round Warp Drive" and it explains how to exceed the speed of light!  See, all you gotta do is get the merry-go-round going up to like 99.9% of the speed of light and then make it really big.  The outside part will then be going way faster than the speed of light.