GREE VR Studio Lab, Intern Report — Rex Hsieh
March 1st to November 11th, 2019
I started my internship at GREE’s VR Studio Lab headed by Akihiko Shirai sensei on March 1st 2019 and will be ending my internship on November 11th due to having finished my projects in VR Studio Lab and needing to finish my PhD dissertation. During this almost 9 month long internship, my main tasks revolve around the evaluation of vTuber character and voice transformed audio in university classes as well as conducting Chinese vTuber platform and business research. Other activities I was involved in were onsite support and research presentation during SIGGRAPH 2019 in Los Angeles and VRSionUP! meetings that used to take place once per month on Friday. In this report, I will first describe the nature of VR Studio Lab, then move on to describe my activities, and finally to recommendations for people interested in joining the VR Studio Lab as an intern.
VR Studio Lab, to my understanding, is a research and development (R&D) division in Wright Flyer Studios, a game developer and subsidiary company of GREE, Inc. While Wright Flyer Studios (WFS) focuses on mobile game development and occasionally collaborated with other companies such as Japan Aerospace Exploration Agency (JAXA) developing contents (web, JAXA, presentation in CEDEC2019), the VR Studio Lab focuses primarily on vTuber application research and development with Wright Flyer Live Entertainment (WFLE).
Almost all of my internship work in VR Studio Lab revolves around evaluating and analyzing effectiveness of vTubers in university environment, attending events and trying to advertise for the company’s products, as well as provide assistance during business meetings, especially with Chinese companies which seemed to be the newest focus of VR Studio Lab. In short, VR Studio Lab does not do game development but does a lot of technical research and data analysis on vTuber.
My main focus up until the end of August in VR Studio Lab is the evaluation of effectiveness of vTubers in a processing class for freshman students in the Kanagawa Institute of Technology. In this research project, I used an application developed by WFLE called “REALITY” to create vTuber avatars to be used in eLearning videos. Another technology I used for this project is the “Roland VT-4 Voice Transformer” which can adjust the pitch, formant, balance, and reverb of a speaker’s voice in real time.
Using “REALITY” in conjunction with “Roland VT-4”, I created 3 visuals named: Real Lecturer (R), Male Styled Avatar (A), and Female Styled Avatar (B), as well as 2 audios: Original Audio (O) and Transformed Audio (T). Using these 5 elements, 6 eLearning videos (RO, AO, BO, RT, AT, BT) bearing the initials of these elements were created each week and shown to students taking the processing class. The students’ subjective impression of the videos and their grade performance as a result of watching these videos were gathered and evaluated. In the end, it was determined that Female Styled Avatar (B) is rated as the best avatar visually but does not contribute to a student’s grade performance. The usage of anime styled avatar A and B does contribute to an increase in video watch duration; however, with B contribute to about a 5% increase in video view duration and A contribute to a 3% increase in video view duration in comparison to R. Transformed audio (T) results in a decrease in video view duration of up to 10% compared to their Original Audio (O) counterparts and when O and T audio were given to students, the student’s grade performance dropped. This research has since been submitted and accepted to various international conferences such as Intelligent Virtual Agents (IVA) in France, SIGGRAPH 2019 in Los Angeles and in annual conference of VRSJ (Virtual Reality Society of Japan).
- Rex Hsieh, Akihiko Shirai, and Hisashi Sato. 2019. Evaluation of Avatar and Voice Transform in Programming E-Learning Lectures. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents (IVA ‘19). ACM, New York, NY, USA, 197–199. DOI: https://doi.org/10.1145/3308532.3329430
- Rex Hsieh, Akihiko Shirai, and Hisashi Sato. 2019. Effectiveness of facial animated avatar and voice transformer in elearning programming course. In ACM SIGGRAPH 2019 Posters (SIGGRAPH ‘19). ACM, New York, NY, USA, Article 82, 2 pages. DOI: https://doi.org/10.1145/3306214.3338540
- 謝 レックス(神奈川工科大学/グリー株式会社), 白井 暁彦(GREE VR Studio Lab, グリー株式会社), 佐藤 尚(神奈川工科大学), “プログラミング教育向けeLearningにおける顔アニメーションアバターとボイスチェンジャーの影響”, 第24回日本バーチャルリアリティ学会大会, Tokyo.
https://twitter.com/VRStudioLab/status/1171607490296856576
Experiences in SIGGRAPH
Shirai sensei encouraged to contribute to the community. So I have been long time contributed to ACM SIGGRAPH as a student volunteer not only as a researcher and/or presenter. (Profile in SIGGRAPH official web).
Onsite assistance during conferences is another main function I have as an intern. As the best English speaker with extensive experience in the United States, I acted as the main communicator during the lab’s journey to SIGGRAPH 2019 in Los Angeles and helped out during the 2 birds of a feather sessions: International Collegiate Virtual Reality Virtual Reality Contest and Virtual Beings World as well as the Appy Hour “REALITY” showcase.
During my presentation in IVA France and VRSJ in Japan, I also introduced the “REALITY” application in AppyHout to visitors and many seemed to be excited about the prospect of seeing the application’s release abroad.
After SIGGRAPH2019, we had opportunity to see Facebook and Google VR.
Other Social implementation outreach
Other events that I assisted with include VRSionUP! event that used to take place once per month on Friday. VRSionUP! is an event hosted by my mentor Akihiko Shirai with the focus of bringing virtual reality enthusiasts from all across the society. Attendees range from high school students to company employees, reporters, professors, manga artists, and renowned researchers. The event consisted of 3 main parts, with the 1st part being long presentation, 2nd part being 3 minute pitches, and the last part socializing. Some of the topics discussed during VRSionUP! include: Laval Virtual 2019 in Laval, France, one of the largest European virtual reality conference, SIGGRAPH 2019 in USA, and industry related topic like VR in manga and the art of acting as a live streaming vTuber. During these days, I would work as volunteer coordinating visitors and assist with presentation setup. The work hour during VRSionUP! is from 12:00 pm to about 23:00 pm compare to normal workday’s 12:00 pm to 18:00 pm. Here are some of the videos taken during VRSionUP! such as the VRSionUp! 3 event that took place on April 19, 2019 and VRSionUP! 4 event on May 17th, 2019.
I have also experienced GREE Family day in July. The event is welcoming company family and well designed to encourage explore inside of company for young family. GREE VR Studio Lab is organized a zone of “Research Lab” and we have demonstrated our latest technology and field testing for children in a casual environment.
VTuber and live streaming market research for China and Overseas
In the past 2 months since the end of SIGGRAPH 2019 in Los Angeles, my focus in VR Studio Lab has been changed from a vTuber avatar data analyst to English storyboard artist for SIGGRAPH Asia 2019 and Chinese vTuber market researcher. Since the lab’s submission has been accepted to SIGGRAPH Asia 2019 Real Time Live, I have been given the task of creating a storyboard for the 10 minute presentation. Recently GREE has also seen an increase in the interest of the Chinese vTuber market and had several meetings between Chinese businesses to discuss the possibility of future collaboration has taken place. Furthermore, Shirai-sensei has also visited the 2019 World Conference on VR Industry hosted in Nanchang, China from Oct 19 — Oct 21. Due to this new interest, I have been researching Chinese video streaming websites and social media sites in the last several days of my time here in VR Studio Lab. Since China uses completely different social media platforms and websites and do not have access to popular sites such as YouTube, it is important to research into the best website for releasing vTuber videos. Currently some of the good sites we have in mind include: Bili Bili, which is one of the premiere video uploading websites in China.
Roppongi Life
Aside from everything that is happening at VR Studio Lab, life and facility inside GREE and its’ surrounding that is Roppongi Hills are also very impressive. Roppongi Hills itself is a major landmark in Tokyo and companies situated inside Roppongi Hills are considered top notch. One can arrive at Roppongi Hills from the Roppongi station which has a passageway that leads directly to front garden of Roppongi Hills. The daily life inside GREE revolves around arriving at around 10 am and working until 18:30 pm, which is a very good and reasonable amount that allows one to get a lot done without overworking oneself. On the 12th floor there is a large cafeteria full of vending machine that even sales bento boxes so one never have to look far when desiring a snack. Outside of company there are also a lot of exhibitions around Roppongi Hills. Past exhibitions include The Science Behind Pixar while future exhibition include the 32nd Tokyo International Film Festival 2019. On the 5th and 6th floor of Roppongi Hills are also some restaurants (albeit a bit leaning towards the high-end side).
In conclusion
My 9 month journey in VR Studio Lab from March 1st to November 11th consisted of vTuber analysis research in the first 6 month with bits of onsite exhibition assistance intertwined. In the last 3 months, my focus has turned into that of English storyboard artist and Chinese market researcher. Other members of the lab are developing vTuber haptic devices or doing data analysis and visualization. Yusuke Yamazaki, another PhD 3rd year candidate currently doing his internship at GREE is working on the usage of haptic device in vTuber broadcasting system. Yamazaki is also the CEO of his startup company Hapbeat. While the lab does not do game development, for people interested in virtual youtuber research and attending exhibitions abroad, doing an internship at VR Studio Lab headed by Akihiko Shirai-sensei is well worth the time not only due to its achievements but also for its activeness in both the business and academic circle.
See you again in SIGGRAPH ASIA 2019!
