Taking Touch beyond the Touch Screen A prototype tablet can sense gestures, and objects placed next to it.

Taking Touch beyond the Touch Screen A prototype tablet can sense gestures, and objects placed next to it.

  10:05:00 AM    Science Lover

---

A tablet computer developed collaboratively by researchers at Intel, Microsoft, and the University of Washington can be controlled not only by swiping and pinching at the screen, but by touching any surface on which it is placed.

In touch: The spacecraft on this tablet's
screen can be controlled by maneuvering
the toy on the table next to it. Credit: Intel

Finding new ways to interact with computers has become an important area of research among computer scientists, especially now that touch-screen smart phones and tablets have grown so popular. The project that produced the new device, called Portico, could eventually result in smart phones or tablets that take touch beyond the physical confines of the device.

"The idea is to allow the interactive space to go beyond the display space or screen space," says Jacob Wobbrock, an assistant professor at the University of Washington's Information School, in Seattle, who helped develop the system. This is achieved with two foldout cameras that sit above the display on either side, detecting and tracking motion around the screen. The system detects the height of objects and determines whether they are touching the surrounding surface by comparing the two views captured by the cameras. The approach make it possible to detect hand gestures as well as physical objects so that they can interact with the display, says Wobbrock.

In one demonstration, software tracks a small ball as it moves across the surface the tablet sits on. As the ball strikes the side of the tablet, a virtual ball appears on-screen following the same trajectory, as if the physical ball had entered the device. In this way the ball can be used to score on-screen goals. In another demonstration, the angle of a toy spaceship placed on the table next to the tablet controls the angle of a virtual spaceship onscreen, allowing the user to shoot down "asteroids."



Wobbrock says the same approach would work on smart phones and other pocket-sized devices. "As devices continue to shrink, they compromise the screen space. But with Portico you can reclaim the surrounding area for interactivity," he says.

With the tablet, Portico increases the usable area sixfold, says Daniel Avrahami a senior researcher at Intel Labs Seattle, who came up with the idea for Portico, and led its development, with help from Shahram Izadi at Microsoft Research in Cambridge, UK. For a 12-inch tablet, "that's the equivalent of a 26-inch screen," says Avrahami, who will present the work in October at the ACM User Interface, Software and Technology Symposium in Santa Barbara, California.

Eventually, says Wobbrock, it may be more practical, especially from a commercial standpoint, to use clip-on cameras instead of foldout ones, which tend to break more easily. But he also notes that the entire display might be replaced with a fold-up frame containing both cameras and a pico projector to produce the image on the surface below.

Eva Hornecker, a lecturer specializing in human-computer interaction at the University of Strathclyde, in Glasgow, Scotland, says there is growing interest in using cameras to detect hand gestures and objects among researchers.

"The problem with touch screens is you can't detect anything that's happening over the surface," Hornecker says. However, she notes that allowing interaction beyond the screen could introduce new challenges such as how to provide feedback so the user knows where the interactive area starts and ends.

Read More

A Search Engine for the Human Body

A Search Engine for the Human Body

  2:12:00 AM    Science Lover

---

Microsoft software recognizes organs and other structures in medical images.

A new search tool developed by researchers at Microsoft indexes medical images of the human body, rather than the Web. On CT scans, it automatically finds organs and other structures, to help doctors navigate in and work with 3-D medical imagery.

Inside out: A close up of a CT processed by new software from Microsoft.
Credit: Microsoft Research

CT scans use X-rays to capture many slices through the body that can be combined to create a 3-D representation. This is a powerful tool for diagnosis, but it's far from easy to navigate, says Antonio Criminisi, who leads a group at Microsoft Research Cambridge, U.K., that is attempting to change that. "It is very difficult even for someone very trained to get to the place they need to be to examine the source of a problem," he says.

When a scan is loaded into Criminisi's software, the program indexes the data and lists the organs it finds at the side of the screen, creating a table of hyperlinks for the body. A user can click on, say, the word "heart" and be presented with a clear view of the organ without having to navigate through the imagery manually.

Once an organ of interest has been found, a 2-D and an enhanced 3-D view of structures in the area are shown to the user, who can navigate by touching the screen on which the images are shown. A new scan can also be automatically and precisely matched up alongside a past one from the same patient, making it easy to see how a condition has progressed or regressed.

Criminisi's software uses the pattern of light and dark in the scan to identify particular structures; it was developed by training machine-learning algorithms to recognize features in hundreds of scans in which experts had marked the major organs. Indexing a new scan takes only a couple of seconds, says Criminisi. The system was developed in collaboration with doctors at Addenbrookes Hospital in Cambridge, U.K.

The Microsoft research group is exploring the use of gestures and voice to control the system. They can plug in the Kinect controller, ordinarily used by gamers to control an Xbox with body movements, so that surgeons can refer to imagery in mid-surgery without compromising their sterile gloves by touching a keyboard, mouse, or screen.

Body search: This CT image shows organs and other features identified by the Microsoft software. A list of these features appears at left.
Credit: Microsoft Research

Kenji Suzuki an assistant professor at the University of Chicago, whose research group works on similar tools, says the Microsoft software has the potential to improve patient care, providing it really does make scans easier to navigate. "As medical imaging has advanced, so many images are produced that there is a kind of information overload," he explains. "The workload has grown a lot."

Suzuki says Microsoft's approach is a good one, but that medical professionals might be more receptive to the design if it indexed signs of disease, not just organs. His own research group has developed software capable of recognizing potentially cancerous lung nodules; in trials, it made half as many mistakes as a human expert.

Criminisi sticks by the notion of using organs as a kind of navigation system but says that disease-spotting capability is also under development. He says, "We are working to train it to detect differences between different grades of glioma tumor"—a type of brain tumor.

The Microsoft group also intends the tool to be used at large scales. It could automatically index a collection of 3-D scans or other images, making possible new ways of tracking medical records, says Criminisi. Today, records are kept as text that describes scans and other information. A search tool that finds the word "heart", for example, would not know if that meant it appeared in a scan or was mentioned in another context. If a hospital's computer system indexed new scans, the Microsoft software could automatically record what was imaged in a person's records and when.

Read More

Software that Learns by Watching

Software that Learns by Watching

  8:38:00 PM    Science Lover

---

KarDo learns how to perform common IT support tests by observing what the experts do.

Overworked and much in demand, IT support staff can't be in two places at once. But software designed to watch and learn as they carry out common tasks could soon help--by automatically performing the same jobs across different computers.

Task manager: A screenshot 
shows KarDo performing administrative jobs 
via a graphical interface.
Credit: KarDo

The new software system, called KarDo, was developed by researchers at MIT. It can automatically configure an e-mail account, install a virus scanner, or set up access to a virtual private network, says MIT's Dina Katabi, an associate professor at MIT.

Crucially, the software just needs to watch an administrator perform this task once before being able to carry out the same job on computers running different software. Businesses spend billions of dollars each year on simple and repetitive IT tasks, according to reports from the analyst groups Forrester and Gartner. KarDo could reduce these costs by as much as 20 percent, Katabi says.

In some respects, KarDo resembles software that can be used to record macros--a set sequence of user actions on a computer. But KarDo attempts to learn the goal of each action in the sequence so it can be applied more generally later, says MIT post-graduate Hariharan Rahul, who codeveloped the system.

When IT staff want KarDo to learn a new task, they press a "start" button beforehand and a "stop" button afterwards. During a "learning phase," KarDo will attempt to map each of the actions performed in the graphical user interface, such as clicking on particular icons or buttons, with system-level actions, such as starting or closing a program, or opening a Web page. This allows a task to be applied across machines running different software, says Katabi. "I can go to my desktop, click on the Internet Explorer icon, go to a website, and then click on a particular link to download a file," she says. The same actions could then be applied by KarDo on a machine running a different Web browser like FireFox or Chrome. KarDo compares actions performed during the learning phase with a database of other tasks.

KarDo is able to reliably infer how to reproduce each of the subtasks after watching it being performed just once, says Rahul. For example, after watching an e-mail account being set up using Microsoft Outlook, it can do the same on other computers running different e-mail software. KarDo has been tested on hundreds of combinations of real tasks by IT staff at MIT and was found to get tasks right 82 percent of the time. When KarDo doesn't perform a task correctly, the results aren't serious, Katabi says.

The ultimate goal is for KarDo to intervene completely automatically, although this has not yet been tested. The idea is that when a user sends a request to the IT department , KarDo would perform the task automatically.

This sort of "programming by demonstration" is not a new idea, says Stephen Muggleton, an expert in machine learning at Imperial College London. But the approach has remained very much a research curiosity, he says. "An obvious concern from a user point of view will be the accuracy of the learned model," says Muggleton. Normally it takes relatively large amounts of data to generate error-free machine learning models, he notes.

"There's a great deal of promise in learning procedures and plans by watching," says Eric Horvitz of Microsoft Research in Redmond, WA. However, in general, this is very challenging to pull off. It is usually hard to do anything useful without constraining the nature of the task, says Horvitz.

KarDo was announced last week as the winner of the Web/IT track of MIT's $100K Entrepreneur Competition.

Read More