CartooNect: Sensory motor playing system using cartoon actions

 FUJIMURA Wataru1, IWDATE Shoto1, SHIRAI Akihiko1

1Kanagawa Institute of Techonology, Atsugi Japan

cartoonect at shirai.la

Keywords-KINECT; Natural Input, OpenNI, Manga, Interaction, drawing, sensory-motor play

  

I.           Introduction

This project is focusing to find a new VR game system and its contents using current emerging video game technologies. Microsoft KINECT [1] (formerly Natal) is a most advanced device in this moment to detect player dynamic motions by a depth camera and its software.

As we did before at Nintendo Wii [2] and WiiRemote, a lot of hackers had tried to reveal and release the technology. However, it was rapidly shared via OpenNI[3] framework by the key technology suppliers, PrimeSence, Willow Garage and SideKick by free.

Then this project had thought again about what is a state of innovation then it took an orientation to realize “sensory motor play” in children imagination.

II.         Sensory motor play in children immagination

“Sensory motor play” is the first stage of Piaget's cognitive development [4]. It can be observed at 0 to 24 months period basically, but it constructs most important function of intellectual development through their playing naturally. It can be explained as obtaining the external world to make interact with something from the brain. Then this stage is not limited in a baby, childhoods to teenagers are also studying slowly when they face to a new behavior in the external world.

Piaget also defined “Symbolic Play” and “Rule Play” stages after the sensory motor stage in development. They are very important to develop human linguistic skill and logical thinking and relations but they are also important to work out a game play in current video games design.

In a classic video games design, it always based on rule plays to define winners or losers. Logical human interfaces like buttons and sticks were functioned as logical input to measure players' rapidness, preciseness and toughness in a virtual with some scenario, logics and renditions. It may affect to children’s sensory motors if their parents do not limit the playing time in their development stage. Because of obtaining the sensory motor, it can give an amusing sense breezy when children just run in a field. It means the gaming system should focus to move players than showing complex things.  This hypothesis can be applied not only for children but also adults in Virtual Reality entertainment system because they can be just a beginner like a baby in a virtual world in development.

III.        Playing using cartoon actions

Cartoons like Manga, Bande-Dessinee (B.D.) and picture books give a great part of imaginations to children. Now we have a various contents from these cartoons to media like cinema, video games and TV animations. However, these media are mainly based on consuming models and most media are non-interactive exclude video games. It sounds a normal sense but it might limit human imagination and innovation into a consuming model. Because children can draw their original worlds and personas by themselves thanks emulating some creations of cartoons.

This project focuses to apply cartoon actions in virtual world to enhance such a creativity and motivation. The aim is not to develop a character merchandising game. It should be shared and be motivated the human creations which include drawing, motion and reactions in a virtual world.

IV.        Technology

“CartooNect” is a code name of supposed system. It is a coined word of Cartoon and KINECT. “KINECT” is also a coined word of kinematics and connect. Then it might be a best word to explain the technical characteristic “Connecting into cartoon world using Kinect” for the public.

A.    Installation

 

Figure 1.   Concept sketch of project

CartooNect is very smart system. It is configured by KINECT, PC and a projector in 3x3m space. Through our experiments, it should keep 2.5m distance from KINECT to a player and each player should keep about 1m diameter or more. In our testing space, we had used an ultra-short range projector (NEC NP-U310WJD) with an aspherical mirror. It can be replaced by rear projection in an exhibition.

B.    Software and Algorythms

OpenNI is an open and emerging framework to create “Natural Input”. It contains communication drivers to KINECT OS “XIRON” via USB connection for several platforms. It also has useful APIs to handle human kinematics like skeleton structure, transformation functions from depth map to real coordinates and/or projection coordinates with actual sample code. However it glows day by day then it should be refer the latest information at the website (www.openni.org) to obtain the technical details.

In the alpha development, we had tried to develop our own algorithms to recognize human motions to detect cartoon hero style actions. It had designed as Figure 3.

 

Figure 2.   Example of  a rule based motion detections

Figure 3 describes a series of motion detection to realize homage of a manga “DragonBall” fighting game. Through the development and testing, we had found it limits the player’s actions which just replay “pre-recorded actions” in a rule play using matching of motion. It is normal sense if we will apply well prepared model in a character merchandizing product but it is completely opposite with our concept. Additionally, the matching was also not worked well because it needs very precise motion in 3D and some player who has different shape could not control it well.

 

Figure 3.   Example of a figure caption. (figure caption)

Figure 4 describes our final implementation, “part kinematics based detection” in a same context. In the detection of “Guard” and “Crouch”, it compares a part of series of bones with a target motion in 2D coordinates using simple dot products. An evaluation function for a target motion multiples Cosines of angles between target bone matrices and current player’s bone matrices.  If each bone has very close value of angles, the output becomes to 1.0. This model can detect combination motion like “Crouching + Guard” by multistep values.

 

Figure 4.   Natrual condition in prototype application

 

Figure 5.   “Crouching” motion detected (face and background)

Figure 6.   “Guard” motion detected. (a picture overrayed)

 

 

Figure 7.   A combination detection and effect of Guard and Crouching.

Figure 4 – 7 are screenshots from our prototype application. To describe, they shows observing kinematics as lines on depth shadow of player. When it detects “Crouching” motion by legs, it changes the background and picture on the face. And when it detects “Guard” by arms, it shows a large picture on the player. Figure 7 is a result to play to make a combination effect of Guard and Crouching.

V.         Contents and Experience

We are improving the system to create a valuable experience through “CartooNect”, a code name of proposing content. It is a coined word of Cartoon and KINECT. (“KINECT” is also a coined word of kinematics and connect.) Then it might be a best word to describe the technical characteristic “Connecting into cartoon world using Kinect” for the public.

Figure 8.   Realized experience of “CartooNect”

Figure 9.   Top view of floor plan in Laval Virtual ReVolution 2011

Figure 10.    Scanning process to generate the scene.

Figure 11.    A scene of CartooNect (in development)

In the experience, the visitors can choose a world or draw it by themselves. Background, props and cartoon calligraphies can be put by hand drawing and scanning. This design has a role to make immerse players by commitment to generated world.

All target motions can be saved as effects with drawn textures, calligraphies with particle effects. This is very simple implementation but it allows creating new motion interaction between players.

Figure 12.    Players had invented their own harmonized action

Detected kinematics could be applied to a realistic 3D model but it looked not funny like a combat figure then we will develop a special character dynamic animation engine to improve CartooNect.

VI.        Conclusion

CartooNect had been developed in a very short time (approximately 2 weeks in training period). It will be tested in more actual playful conditions to add new functions and experiences. Through this project, we are touching a new generation which is sensory-motor oriented computer entertainment. This point of view is more important than just developing an application of KINECT content. We would like to continue to share our knowledge and methodologies into actual virtual reality industry and next generation of video games.

Acknowledgment

The project thanks to Taisuke YAMASHITA, Tomohiro MIYAGAWA, Yukua KOIDE, Haruka OGUMA and Motofumi HATTORI for their creation, management and encouragement.

References

[1]     Microsoft KINECT_TM, http://www.xbox.com/kinect

[2]     Nintendo Wii®, http://www.nintendo.co.jp/wii/

[3]     OpenNI, http://www.openni.org/

[4]     Tracy S. Kendler, Levels of cognitive development. Routledge, 1995 

[5]     TORIYAMA Akira, Bird Studio, “Dragon Ball”, Shueisha.

[6]     ARAKI Hirohiko, LUCKY LAND COMMUNICATIONS,  “JoJo’s Bizzare adventure”, SHUEISHA, 1987-2005. 

溜めを検知を出来たことで、プレイヤーのリアルタイムモーションとCGキャラクターのアニメーションを同調させることができ、プレイヤーがより一体感を得ることができる。

If you need academic papers, here is link of PDFs. 関連論文は以下のダウンロードページにあります．

https://www.shirai.la/downloads