A Search Engine for the Human Body

A Search Engine for the Human Body

  2:12:00 AM    Science Lover

---

Microsoft software recognizes organs and other structures in medical images.

A new search tool developed by researchers at Microsoft indexes medical images of the human body, rather than the Web. On CT scans, it automatically finds organs and other structures, to help doctors navigate in and work with 3-D medical imagery.

Inside out: A close up of a CT processed by new software from Microsoft.
Credit: Microsoft Research

CT scans use X-rays to capture many slices through the body that can be combined to create a 3-D representation. This is a powerful tool for diagnosis, but it's far from easy to navigate, says Antonio Criminisi, who leads a group at Microsoft Research Cambridge, U.K., that is attempting to change that. "It is very difficult even for someone very trained to get to the place they need to be to examine the source of a problem," he says.

When a scan is loaded into Criminisi's software, the program indexes the data and lists the organs it finds at the side of the screen, creating a table of hyperlinks for the body. A user can click on, say, the word "heart" and be presented with a clear view of the organ without having to navigate through the imagery manually.

Once an organ of interest has been found, a 2-D and an enhanced 3-D view of structures in the area are shown to the user, who can navigate by touching the screen on which the images are shown. A new scan can also be automatically and precisely matched up alongside a past one from the same patient, making it easy to see how a condition has progressed or regressed.

Criminisi's software uses the pattern of light and dark in the scan to identify particular structures; it was developed by training machine-learning algorithms to recognize features in hundreds of scans in which experts had marked the major organs. Indexing a new scan takes only a couple of seconds, says Criminisi. The system was developed in collaboration with doctors at Addenbrookes Hospital in Cambridge, U.K.

The Microsoft research group is exploring the use of gestures and voice to control the system. They can plug in the Kinect controller, ordinarily used by gamers to control an Xbox with body movements, so that surgeons can refer to imagery in mid-surgery without compromising their sterile gloves by touching a keyboard, mouse, or screen.

Body search: This CT image shows organs and other features identified by the Microsoft software. A list of these features appears at left.
Credit: Microsoft Research

Kenji Suzuki an assistant professor at the University of Chicago, whose research group works on similar tools, says the Microsoft software has the potential to improve patient care, providing it really does make scans easier to navigate. "As medical imaging has advanced, so many images are produced that there is a kind of information overload," he explains. "The workload has grown a lot."

Suzuki says Microsoft's approach is a good one, but that medical professionals might be more receptive to the design if it indexed signs of disease, not just organs. His own research group has developed software capable of recognizing potentially cancerous lung nodules; in trials, it made half as many mistakes as a human expert.

Criminisi sticks by the notion of using organs as a kind of navigation system but says that disease-spotting capability is also under development. He says, "We are working to train it to detect differences between different grades of glioma tumor"—a type of brain tumor.

The Microsoft group also intends the tool to be used at large scales. It could automatically index a collection of 3-D scans or other images, making possible new ways of tracking medical records, says Criminisi. Today, records are kept as text that describes scans and other information. A search tool that finds the word "heart", for example, would not know if that meant it appeared in a scan or was mentioned in another context. If a hospital's computer system indexed new scans, the Microsoft software could automatically record what was imaged in a person's records and when.