Positive Technology Journal

07 April 2005
Exclusive from New Scientist Print Edition

Jenny Hogan Barry Fox

Imagine movies and computer games in which you get to smell, taste and perhaps even feel things. That's the tantalising prospect raised by a patent on a device for transmitting sensory data directly into the human brain - granted to none other than the entertainment giant Sony.

The technique suggested in the patent is entirely non-invasive. It describes a device that fires pulses of ultrasound at the head to modify firing patterns in targeted parts of the brain, creating "sensory experiences" ranging from moving images to tastes and sounds. This could give blind or deaf people the chance to see or hear, the patent claims.

While brain implants are becoming increasingly sophisticated, the only non-invasive ways of manipulating the brain remain crude. A technique known as transcranial magnetic stimulation can activate nerves by using rapidly changing magnetic fields to induce currents in brain tissue. However, magnetic fields cannot be finely focused on small groups of brain cells, whereas ultrasound could be.

If the method described by Sony really does work, it could have all sorts of uses in research and medicine, even if it is not capable of evoking sensory experiences detailed enough for the entertainment purposes envisaged in the patent.

Details are sparse, and Sony declined New Scientist's request for an interview with the inventor, who is based in its offices in San Diego, California. However, independent experts are not dismissing the idea out of hand. "I looked at it and found it plausible," says Niels Birbaumer, a pioneering neuroscientist at the University of Tübingen in Germany who has created devices that let people control devices via brain waves.

The application contains references to two scientific papers presenting research that could underpin the device. One, in an echo of Galvani's classic 18th-century experiments on frogs' legs that proved electricity can trigger nerve impulses, showed that certain kinds of ultrasound pulses can affect the excitability of nerves from a frog's leg. The author, Richard Mihran of the University of Colorado, Boulder, had no knowledge of the patent until New Scientist contacted him, but says he would be concerned about the proposed method's long-term safety.

Sony first submitted a patent application for the ultrasound method in 2000, which was granted in March 2003. Since then Sony has filed a series of continuations, most recently in December 2004 (US 2004/267118).

Elizabeth Boukis, spokeswoman for Sony Electronics, says the work is speculative. "There were not any experiments done," she says. "This particular patent was a prophetic invention. It was based on an inspiration that this may someday be the direction that technology will take us."

More to explore

Virtual Reality
Transcranial Magnetic Stimulation
Matrix: the movie

Press releases

Australian IT - Sony patents 'real' Matrix
News24 - Sony eyes 'real-life Matrix'
Forbes.com - Sony takes first step to patent 'real-life Matrix
Dawn International - Real life Matrix in the making
Timesonline - Sony takes 3-D cinema directly to the brain
Australian Financial Review - First step to patent 'real-life Matrix'

By Geoff McMaster, ExpressNews Staff - The closest many of us have come to imagining virtual reality is the holodeck, a fantasy playground featured on the television series Star Trek.

Such flights of fancy are no longer the stuff of science fiction, however. Computer scientists at the U of A have already created technology allowing people to sit across from three-dimensional recreations of each other, even though in reality they may be thousands of miles apart.

It's only the beginning of a revolution in virtual reality technology expected to take us by storm in the next decade or so, says Dr. Pierre Boulanger, a U of A computer scientist who just received an iCORE/TRLabs Industrial Research Chair worth a total of $1.7 million to develop his groundbreaking work in collaborative virtual environments.

Imagine a world, for example, where professors of surgery transmit hand and scalpel movements, as well as what they see while operating, thousands of miles across a computer network, where it is recreated in an operating room.

"The student will actually look at that and actually feel what the doctor is doing," said Boulanger. "On the other hand, the doctor can feel what the students are doing and give them a nudge in the right direction--It's like being in virtual residence with doctors."

Families separated by travel will spend meals together through what is called "telepresence," said Boulanger. "You would wear special goggles--and we're working on that--which would allow you to see your wife sitting in front of you, having a day-to-day conversation. In the future you will have virtual encounters like this, people you want to be part of a meeting sitting beside you virtually and having a conversation."

At a press conference on campus Tuesday to celebrate his chair, and that of Dr. Christoph Sensen at the University of Calgary, Boulanger explained how scientists are now able to create and manipulate a model of the earth's core by feeding computers highly sophisticated mathematical equations. Once recreated in 3D, the average person is fully capable of understanding such complex physical phenomena, he said. "People can actually interact with it, and say, "What happens if we have that instead of this?'"

"A three-dimensional visual model will allow you to explain complex systems, and understand how the world runs. People understand complex systems because daily life is actually very complex--the new technology is truly human-centred. Computers are smart enough today to adapt to people, and that's really a recent shift in computing."

The two chairs held by Boulanger and Sensen will focus on a variety of pursuits that will benefit from virtual reality technology, including engineering prototypes, testing medical procedures and conducting scientific research. Sensen, who has received $1 million for his research, is developing new tools to virtually work in the human body.

Boulanger was recruited to the University of Alberta's computing science department in 2001 from the National Research Council of Canada, where he spent 18 years as a senior research officer. He is also an adjunct scientist and principal investigator for new media at TRLabs, Canada's largest not-for-profit information and communications technology research consortium , and at the Banff Centre.

His new chair includes an iCORE Industrial Chair Establishment grant of $50,000 per year for five years, in addition to further grants from TRLabs, the University of Alberta, the Canada Foundation for Innovation, the Natural Sciences and Engineering Research Council, the Canadian Network for the Advancement of Research in Industry and Education and other industry partners.

"From my perspective, an absolutely essential part of Pierre's work on the collaborative virtual environment is described by the word collaborative," said dean of science Gregory Taylor. "It's at the very forefront of what I like to call the new science, interdisciplinary science where the collaborative team becomes the vehicle for discovery."

iCORE was established six years ago by the provincial government to support university research that supports information and communications technology. There are now 20 chairs focused on emerging areas such as wireless communications, artificial intelligence and quantum nanocomputing

From Gizmodo

I4U has a few details on an impressively lightweight LCD screen from Scalar Corp. that can be mounted on ordinary glasses that weighes a scant seven grams. The screen offers a simulated 14-inch screen with 180,000 pixels (roughly television resolution). It's not terribly expensive, either, at just $460 or so.

Contemporary art is exploring the link between mind, brain and technology. Artists are scientists who experiment reality in a different way. They don't use the language of maths and physics, but the universal language of sensory experiences. There are concepts that cannot be expressed using numbers. For example, how would one represent the infinite complexity of the relationship between the mind and the brain? And how to describe the invisible co-evolution between natural and artificial, between biological and technological networks?

Bianco and Valente, two artists working in Neaple (Italy) propose an original answer to these questions. The focus of their work is "on perception phenomenons and brain dynamism that enable us to retain the memories of our experiences and perceive mind images, through which we can conceive an evolving reproduction of the external reality A very fascinating subject for us is the body-mind duality: a flesh organic structure, finite and dependent on space and time that carries about the mind, a spontaneous phenomenon without visible boundaries, totally free and self-referential. We search for boundaries of this immaterial space living into the cerebral convolutions, trying to understand if and where it is possible to locate a point of contact linking indivisibly the two domains, the material and the immaterial"

More to explore

Bianco-Valente Web site (only Italian, but with a few English articles about their work)

From the "Presence" listserv

By ERIN BELL
September 23, 2004
Special to The Globe and Mail

Anyone who's used Sony's EyeToy for the PlayStation 2 console knows how much fun it can be to put yourself into a video game with the help of a camera. More advanced technology developed in Canada is being used to help people with special needs learn skills and experience things that used to be beyond their reach.

The technology comes from a partnership between integrator Xperiential Learning Solutions Inc. and Toronto developer Jestertek Inc. (formerly the Vivid Group), whose virtual reality systems have been installed in places ranging from museums and science centres to the hockey and basketball halls of fame.
The pair's Experiential Learning Product Suite is aimed at people with physical, mental or behavioural disabilities.

Xperiential's founder, Theo D'Hollander, has a son with autism and relatives who suffer from cerebral palsy and Down syndrome. "It made me realize that this whole area [of technology for people with disabilities] is almost like a generation behind, it's in the industrial age when the rest of the world is in the Internet age," he said. "Today's technology is great for things like information gathering, but it has actually created more distance between people with disabilities and the life or job opportunities they need. The answer is to use technology to help create new experiences for them."

Like Sony's EyeToy, the Experiential Learning Product Suite uses cameras to capture a person's image and project it onto a monitor or large screen, combining it in real-time with the computer-generated action. The player can participate in virtual reality scenarios such as snowboarding, soccer, boxing, racing, and even mountain climbing, controlling the action by moving parts of their body. It's there the similarities cease, however.

Using cameras that capture at least 30 frames a second and hardware much more powerful than a game console, the suite can adapt to a player's physical characteristics and abilities.
Sensitivity, speed and range of motion are adjustable, allowing people to control programs with tiny gestures -- from a shrug to a toe-twitch -- letting a bedridden person see what it's like to ride a horse, or someone without the use of their hands play a virtual musical instrument.

The profile for each user can be fine-tuned as their mobility or skills improve. The partners are also working on a way to allow people to compete or collaborate on-line.

"It's very much a motivational experience for the kids or adults with disabilities who use the system," Jestertek president Vincent John Vincent said.

"Children with cognitive disabilities have short attention spans,"Mr. D'Hollander added. "With these programs, they're engaged by the games and music. There's something enticing about seeing themselves on television, and the idea that they're inside a computer game."

A study by the University of Ottawa is looking at ways to use the suite to make children's home-exercise rehabilitation programs more engaging.

"This gives people, especially those with some cognitive impairments or disabilities, the opportunity to have an experience that could not be possible otherwise," explained Dr. Heidi Seistrup, associate professor in the University's School of Rehabilitation Sciences.

"For example, they could play volleyball even though they're in a wheelchair, just by moving their fingers. The environment can be tweaked to allow someone with a very limited range of motion to play against someone who has a full range of motion. You put them on a level playing field, which you can't do in real life very easily."

In other cases, the suite is used to teach life skills. There are modules that can train people to sort a load of laundry, teach basic traffic safety, or show how to serve customers in a doughnut shop.

The system can be a social behaviour coach, too, Mr.
D'Hollander said. "Having a child with autism go to a family gathering at Christmas or go into a crowded mall is a big issue, because the initial encounter is so intense they can't handle it.
We can create a tape of family settings or a mall, and allow children to get used to it by interacting with [the virtual crowd] before they encounter the real thing, helping them over that social hurdle."

A site licence for the Experiential Learning Product Suite is $5,600 (including hardware for a single user and several dozen applications), and systems for additional users can be purchased for around $750 each. There's also a basic unit that sells for around $400, including about a dozen games, to deliver extra entertainment and exercise-related programs in a home setting.

Over the past several months, Xperiential has sold about 30 site licences to community living homes, rehabilitation centres and school boards across Ontario, as well as to customers in the United States and Europe, Mr. D'Hollander said.

A unit was recently installed at Community Living Oakville in Oakville, Ont. "We use it three times a week, and it's awesome,"
day service worker Kelleigh Melito said. "They love the dancing, racing and snowboarding. Because of our location, they can't get out much to go for walks, because there are no sidewalks; this is how they get their exercise."

Don Seymour, executive director of developmental services for Lambton County, Ont., said he hopes to use the virtual reality units in all the special needs homes under his jurisdiction.

"I watched a fellow we support who is in a wheelchair get in front of the camera and look at the screen, and all of a sudden realize he was in a racecar," Mr. Seymour said. "All he had to do was move his shoulders to race this car around a track. For a person who has limited mobility with their arms or legs, to be able to steer a racecar on a big screen was incredible."

"You hear about this stuff a dozen times in a year, and then to actually find something that has immediate flexibility to our folks is quite something," he added.

Maria V. Sanchez-Vives and Mel Slater have published a review article entitled "From Presence to Consciousness through Virtual Reality" on the prestigious journal Nature Neuroscience (ASI impact factor 2003: 27.007).

In this article, the authors argue that presence - the sense of "being there" perceived during the exploration of a virtual environment - is worthy of study by neuroscientists, and that it might aid the study of perception and consciousness.

Maria V. Sanchez-Vives, Mel Slater, From Presence to Consciousness through Virtual Reality, Nature Reviews Neuroscience 6, 332-339 (2005); doi:10.1038/nrn1651