The Japan News by The Yomiuri Shimbun

Home > Research > Waseda research zone - latest news on project research


Waseda Research Zone - Latest News on Project Research -

Information Technology Research Organization Digital Entertainment Laboratory

Realizing Movies in Which Viewers are the Performers
- "Dive into the Movie"

Professor Shigeo Morishima, Director of the Digital Entertainment Laboratory

©dentsu/dentsu tec: A digital entertainment system in which viewers themselves can become the main character in a movie (model is Professor Morishima).

The originality found in the work of Japanese artists who are active in a variety of areas is praised as "cool Japan". Japanese movies, animation, comics and Otaku culture has become known throughout the world. However, when examining the overseas sales ratio of Japanese work in the contents industry, the numbers show an extremely small percentage. Japan accounts for 3% (2002 report by the Ministry of Economy, Trade and Industry) of the overseas contents industry as opposed to the 17% held by America. Furthermore, the scale of the Japanese contents industry is below the world average (as a percentage of GDP: 2.2% for Japan, 3.2% for world average, 5.1% for America) and the sluggish growth rate of the industry is also a major issue.

In the midst of these developments, the Japanese government has raised the goal of making Japan into a world-class digital contents superpower. Debate, formulation of concepts, and even specific measures to achieve this goal has been undertaken by scholars in Japan. Currently, a goal has been established to develop the scale of the contents industry to 20 trillion yen by 2015 (approximately 13.8 trillion yen as of 2007, with 2.7 trillion yen in digital contents). Digital contents are expected to compose a large portion of this growth.

The key to this growth is held in soft power such as the development of high-level creators who will undertake the development of new digital media technology, as well as the application and industrialization of that technology. However, in Japan, this soft power is overwhelmingly weak compared to America, particularly in the area of tying technology into application and industrialization of entertainment.

In these circumstances, the "Dive into the Movie" project of the Waseda University Digital Entertainment Laboratory is showing great spirit and creating new programs. The laboratory is developing a completely new type of digital entertainment technology in which viewers themselves appear as performers within authentic video works. We spoke with Dr. Shigeo Morishima, Professor in the Faculty of Science and Engineering and Director of the Digital Entertainment Laboratory.

The laboratory is performing development unique to the entertainment field. Such development includes providing a system in which users can select from a number of hairstyles and other traits, while also pursuing a "likeness of the person" by estimating features such as face, expression, shape of the rear of the head, thickness of neck, and body type.

New technology which emerged at the Aichi Expo

Photographs of viewer's faces were taken during waiting time and then synthesized into the faces of the movie characters.

©dentsu/dentsu tec: A world was realized where viewers enjoy a movie in which they themselves are the characters.

The Aichi Expo was held in Aichi Prefecture in 2005. Future Cast速, a completely new digital entertainment system featuring participation from viewers, made its debut at the Mitsui-Toshiba Pavilion of the exposition. The pavilion contained three theater halls with a capacity of 80 people each, allowing a total of 240 people to participate in each rotation of the 40 minute movie time. The theater halls showed the film "Grand Odyssey", a movie which was set in the earth and universe of the future. This movie realized a world-first system in which all of the viewers were incorporated into the acting of the film. More than 1.63 million viewers experienced this new entertainment system.

Grand Odyssey was created by employing first-class staff in every field. The elemental technology of this digital entertainment system is the Future Cast® system, the core of which is supported by estimating and synthesizing the face and expression of human beings. Professor Morishima has worked in this technology for many years.

First, while participants are waiting in line, photography is performed for the faces of all viewers. The images, taken by 7 digital cameras, are read into the system and synthesized as the faces of characters appearing in the movie. The role played by each person is automatically assigned according to factors such as the age and gender of the participants. "We used ordinary and inexpensive digital cameras. However, by photographing from 7 different angles, we worked to produce a face and expression close to that of each individual, with the highest possible degree of accuracy. The viewers were not informed before the movie regarding the role which they were assigned. This created the delight of searching for one's own face while watching the movie. The movie also featured complex story development and was designed to be enjoyed like a video game." (Professor Morishima)

The movie system was prepared over a period of approximately 2 and a half years. When Professor Morishima was first visited by a representative from the major advertising agency which was producing the movie, he decided to take a relaxed role in the project as a simple advisor. "However, when the project actually started, I suddenly found myself in the middle of things. Together with movie creators and many other staff members, I ended up working my hardest to develop a type of entertainment that the world has never seen before. It was also a good experience for students who actually worked to implement the system."

Experiencing content from the perspective of performers

For "Dive into the Movie" the individual's physique and back of head are estimated in addition to the face. For this purpose, 3 cameras are used and photography is performed to the area around the individual's ears. Compared to previous versions, the system has been greatly simplified and realizes much greater accuracy and faster processing, while also achieving a low cost.

In order to pre-develop a default model for the movement of muscles for facial expression, an experimented was repeated in which individuals change their face from expressionless to smiling in front of a camera. During operation, participants simply perform the same actions in front of a camera, and a smile model which fits that person is instantly selected and synthesized from within the default variations.

By considering the thickness of facial fat layers, the variety of expressions is increased for the area around the mouth and the cheeks of the movement model of muscles for facial expression. This increases the reality of the image.

After the Aichi Expo, Grand Odyssey became a permanent attraction at Huis Ten Bosch, a theme park in Nagasaki. The contents are the same as during the Aichi Expo, but the system has been made much more practical. For example, the time required for photography and modeling of faces has been shortened to 10 seconds from the previous 2 to 3 minutes.
"At the exposition, we were exacting in our pursuit of accuracy. Through that experience, we were able to find a point of compromise at which a likeness of the individual is captured and over-specification in eliminated. As a result, we placed emphasis on analysis technology for the shadows of the grid placed on faces and succeeded in reducing the number of digital cameras required to just one. This created dramatic improvements in processing speed." (Professor Morishima)

Furthermore, in order to create further advancements in the system, a research project entitled "Research of New Image Technology Used in 'Dive into the Movie'"(*) was held for 3 years beginning in 2006. This project was held in cooperation with ATR (Advanced Telecommunications Research Institute International) and Osaka University. This project raised two main research goals. The first goal was to pursue the reality of an individual-likeness by moving beyond face and expression to replicate other individual characteristics such as physique, voice, and way of walking. The second goal was to replicate a world of subjective images in which the individual sees from the perspective of the character being performed in the image. In Future Cast®, it was possible to objectively view oneself within the image. However, the project seeks technology which enables viewers to enter the very scene of the movie and experience the world as viewed by the performer.

The Osaka University team was responsible for technology used in walking and full-body modeling, as well as in generation of a panoramic image from the perspective of the movie character. ATR was responsible for voice research used to dub movie lines into the individual's voice and for technology used to replicate a 3D acoustic environment. Waseda University conducted research related to expression modeling, full-body action modeling, and real-time character animation. Waseda also served as a project leader by summarizing the technology as a verification system and evaluating the overall system. The project contained an overwhelming amount of technical issues. However, it was difficult to realize practicality in the system simply by pursuing only the improvement of elemental technology. This is particularly true considering the goal of creating a new digital entertainment system. Important development targets centered on questions such as how to minimize the load placed on participants and what kind of performances are needed to entertain participants.

"For example, during experiments, we were able to perform measurements by having participants wear a swimming cap to remove their hair from the area being measured. However, there we cannot actually ask real viewers to wear a swimming cap. It is important to develop estimation technology and variations of default models for cases in which the shape of a person's face makes it difficult to perform modeling for the back of their head. Another important example is development of realistic performance methods for use in entertainment, such as making a person's neck thinner in order to make the person look younger." (Professor Morishima)

(*) "Dive into the Movie" was conducted through "Research of New Image Technology Used in 'Dive into the Movie'", a project selected in 2006 as a theme for the "Research and Development Program for Resolving Critical Issues" sponsored by the Special Coordination Funds for Promoting Science and Technology and by the Ministry of Education, Culture, Sports, Science and Technology.

"Dive into the Movie" seeks to create an even more advanced digital entertainment system in which an individual can subjectively view the world and experience acoustics from the perspective of the character which they are performing. 360 degree image from subjective perspective (provided by the Osaka University Institute of Scientific and Industrial Research).

Seeking an environment in which high-level specialists can perform

"Dive into the Movie: Opening the Curtain on New Entertainment", an event held at the National Museum of Emerging Science and Innovation.

Professor Morishima has always been involved in research regarding muscles for facial expression in humans. He has engaged exclusively in fundamental research such as performing experiments by constructing mechanical models of muscles and by observing anatomical dissections of the human body in order to study facial muscles. The impetus for his interest in entertainment was a small service which he encountered in Hollywood during a business trip to America. The service consisted of a booth similar to those used for making "print club" (Japanese name for stickers made from photographs). The booth allowed users to purchase a souvenir of a composite photograph of their own face featured on the cover of a famous magazine. Fortunately or unfortunately, Professor Morishima though it would be interesting to make a movie version of this booth.

In a short while, Professor Morishima had conceived a practical model in which the user enters into famous films or Hollywood movies. He continued to conduct trial-and-error experiments with this model. However, even if verification experiments and demonstrations were performed, nothing could be done unless the copyright issue was cleared. When working in practical development in the world of digital entertainment, there is a great need for a business model and for support with solid backing in the area of capital.
"I had just encountered these obstacles and was starting to lose my motivation when I was contacted regarding the Aichi Expo. I lost myself in my work during development, but I was able to obtain the cooperation of first-class creators to create completely new and unique contents based on my own technology. It was a wonderful opportunity." (Professor Morishima)

These kinds of favorable circumstances are extremely rare in Japan. In the area of digital contents development, there are very few opportunities for performance by technicians who possess a high level of specialized ability. "For example, consider students in America who have obtained a Doctoral Degree at MIT (Massachusetts Institute of Technology) and who have presented their thesis at SIGGRAPH, an international conference in the field of computer graphics. It is common for such students to gain employment under very good conditions in the R&D departments of entertainment corporations. Revolutionary entertainment is continually created precisely because of the depth of the R&D environment. Unfortunately, in Japan, even if students that possess outstanding abilities are developed, the students have no other option than to obtain employment in completely unrelated fields such as the manufacturing or financial industries." (Professor Morishima)

It is necessary to break free of this situation in order to transmit revolutionary digital contents from Japan to the world. Professor Morishima seeks to do more than develop immediate technology. He is involved in a long-term project to expand a network with various individuals in industry, academia and government, and to conduct changes from business systems and project development systems.

©dentsu/dentsu tec

Related Links

Waseda University, Digital Entertainment Laboratory

Waseda University, Information Technology Research Organization

Waseda University, Faculty of Science and Engineering

Waseda University, Morishima Laboratory

"Dive into the Movie"?MEXT: Project for Special Coordination Funds for Promoting Science and Technology?

Digital Animation Laboratories?Japan Science and Technology Agency: Crest Project?