Antonio Gomes

BitDrones is a toolbox for building interactive real-reality 3D displays that use nano quadcopters as self-levitating tangible building blocks. Our prototype is a first step towards interactive self-levitating programmable matter, in which the user interface is represented using Catomic structures. We discuss three types of BitDrones: PixelDrones, equipped with an RGB LED and a small OLED display; ShapeDrones, augmented with an acrylic mesh spun over a 3D printed frame in a larger geometric shape; and DisplayDrones, fitted with a thin-film 720p touchscreen. We present a number of unimanual and bimanual input techniques, including touch, drag, throw and resize of individual drones and compound models, as well as user interface elements such as self-levitating cone trees, 3D canvases and alert boxes. We describe application scenarios and depict future directions towards creating high-resolution self-levitating programmable matter.

References
Antonio Gomes, Calvin Rubens, Sean Braley, and Roel Vertegaal. BitDrones: Towards Using 3D Nanocopter Displays as Interactive Self-Levitating Programmable Matter. CHI'16 [PDF]

Antonio Gomes, Calvin Rubens, Sean Braley, and Roel Vertegaal. BitDrones. Interactions 23, 3. April 2016. [PDF]

Calvin Rubens, Sean Braley, Antonio Gomes, Daniel Goc, Xujing Zhang, Juan Pablo Carrascal, and Roel Vertegaal. 2015. BitDrones: Towards Levitating Programmable Matter Using Interactive 3D Quadcopter Displays. UIST'15 Adjunct [PDF]

WhammyPhone is a completely new way of interacting with sound using a smartphone. It allows for the kind of expressive input normally only seen in traditional musical instruments. WhammyPhone features a 1080p flexible OLED touchscreen display. The display shows keys that can be used to play sounds on sound synthesis software running on a computer. WhammyPhone is also equipped with bend sensors, which allows for the user to bend the phone as a means of manipulating the sound.Bend input can be used to simulate bending a string on a virtual guitar, providing Hendrix’ style feedback sounds. In another example, the phone is used to simulate the bowing of a simulated violin. Here, the bending of the phone provides the same kind of experience as exerting pressure on a real bow. A final example shows how WhammyPhone can be used to control loops in Electronic Dance Music, making it more intuitive for DJs to interact with their instruments.

References
Antonio Gomes, Lahiru Priyardarshana, J.P. Carrascal, and Roel Vertegaal. WhammyPhone: Exploring Tangible Audio Manipulation Using Bend Input on a Flexible Smartphone. UIST'16 Adjunct [PDF]

DisplayCover is a peripheral cover designed for compact touch-enabled laptops. A tactile keyboard affords users with the comfort and ease of use provided by physical keys. A thin-film e-ink display with a resolution of 1280x305px extends the available screen real estate of the slate device by up to 8%. An e-ink display as chosen due to the bistable nature of electrophoretic ink, reducing the secondary screen's impact on battery life. DisplayCover explores the ability to dynamically alter the peripheral display content based on usage context, while extending the user experience and interaction model to the horizontal plane, where hands naturally rest. For example, stylus annotation can be performed directly on the peripheral cover, reducing the need need for users to routinely home their hands between the slate display and the physical keyboard. DisplayCover was developed during my internship with the Applied Sciences Group at Microsoft.

References
Antonio Gomes, Tristan Trutna, and Roel Vertegaal. DisplayCover: A Tablet Keyboard with an Embedded Thin-Film Touchscreen Display. MobileHCI '15. [PDF]

PaperFold is a novel shape changing mobile device with multiple reconfigurable touch sensitive thin-film electrophoretic magnetic display tiles. PaperFold explores the perceived benefits of having multiple computing devices combined into a single mobile device featuring multiple detachable displays. In PaperFold, each display tile can act independently or as part of a single system. Advantages include better support for performing tasks that traditionally require multiple devices, as well as physical manipulation and sharing of views. Touch and Inertial Measurement Unit (IMU) sensors embedded in each display tile allow users to dynamically manipulate content.

References
Antonio Gomes and Roel Vertegaal. PaperFold: Evaluating Shape Changes for Viewport Transformations in Foldable Thin-Film Display Devices. TEI'15 [PDF]

Antonio Gomes and Roel Vertegaal. Paperfold: a shape changing mobile device with multiple reconfigurable electrophoretic magnetic display tiles. CHI'14 Extended Abstracts. [PDF]

MorePhone is an actuated flexible smartphone with a thin-film E Ink display. MorePhone uses shape memory alloys to actuate the entire surface of the display as well as individual corners. We conducted a participatory study to determine how users associate urgency and notification type with full screen, 1 corner, 2 corner and 3 corner actuations of the smartphone. Results suggest that with the current prototype, actuated shape notifications are useful for visual feedback. Urgent notifications such as alarms and voice calls were best matched with actuation of the entire display surface, while less urgent notifications, such as software notifications were best matched to individual corner bends. While different corner actuations resulted in significantly different matches between notification types, medium urgency notification types were treated as similar, and best matched to a single corner bend. A follow-up study suggested that users prefer to dedicate each corner to a specific type of notification. Users would like to personalize the assignment of corners to notification type. Animation of shape actuation significantly increased the perceived urgency of any of the presented shapes.

References
Antonio Gomes, Andrea Nesbitt, and Roel Vertegaal. MorePhone: a study of actuated shape deformations for flexible thin-film smartphone notifications. CHI'13. [PDF]

This project features an interactive installation created for the Madeira Whale Museum located in Caniçal, Portugal. The exhibition conveys the message of clean ocean in the form of a cooperative video game. The game, built in the Unity Game Engine, allows three guests to act as an underwater litter cleaning team. Using a wheel, guests control a submarine to clean and explore the ocean floor. Litter is picked up using the two joysticks, which controls the left and right arms of the submarine. Sea creatures occasionally are caught in the litter, requiring guests to work together to remove the trash and free the animals. [PDF]
 

Interactive art has emerged as a distinctive genre in media art that relies on digital contents to express the artist’s message. Situated within this field, this work presents an approach to multimedia storytelling that allows audience members to control separate but overlapping parts of the story chapters. We believe that the system engages its audience with a high level of immersion due to its combination of digital computation and tangibility; the tangible system supports a stronger connection to the storytelling than traditional screen-based systems, helping to bridge the gap between the physical world and cyberspace within the field of multimedia storytelling. Consequently, it offers significant potential to share storytelling among a group based its immersive environment and support for embodied interaction paradigms.

This work was done in collaboration with Hyunjoo Oh who kindly shares the inspiration for this installation as follows:

"Through the dead (or dying) branch, audience can see it's memory of lifetime. My grandmom who has Alzheimer's disease often said with awareness of herself that her life is over. Her memories are not connected. Sometimes, she became a middle school student and sometimes she said that she needs to go to the kindergarten to pick me up. She kept swimming in her memories that were not organized without noticing who I am or who she is. I grab her hands and listen her stories. For me, she exists here. People often say that old creatures also had their beautiful time before; they are still beautiful like that dead branch is also beautiful. So it seems too hard to define which part of life is more meaningful. Just, through this work, I want to say that all creatures blossom in the beauty of each moments involving the moment which recalls the past."

References
Hyunjoo Oh and António Gomes. The memory of a tree: an interactive storytelling installation. TEI '12. [PDF]

This was one of the capstone projects towards my Masters in Entertainment Technology. Our client was the Madeira Theme Park, which is located in Madeira, Portugal. Inspired by traditional music and instruments, Ilha Musical is an interactive experience in which guests help growing flowers and creating buildings to create a village of joy and beauty to the Island of Madeira by playing instruments and singing through kiosk and microphone.

Reference
Antonio Gomes, Hyunjoo Oh, Yoram Chisik, and Monchu Chen. Ilha Musical: A CAVE for Nurturing Cultural Appreciation. IDC '12.

This installation features a unique space for a yoga practitioner to achieve a deeper level of realization. This studio can be used by a regular daily practitioner, and can also be utilized for learning the poses in a more accurate way and encourages longer holdings of the each poses. The interactive visuals not only keeps the interest constant throughout the practice, but also guides the practitioners to stay in the poses for a healthy duration.

Reference
Monique Park, Mário Pinto, Monchu Chen and Antonio Gomes. Enhancing the Experience of Yoga Practice through an Immersive Interactive Studio. APCHI'12. [PDF]