The XBPMMM meeting on 01.12.21 started with a guest presentation by AURORA developer and media computer scientist Leonid Barsht about locomotion, interaction and user interface in virtual reality.
During the following discussion it became clear that we will probably test some VR applications as references for locomotion and interaction in the near future – we will keep you posted! For XBPMMM a combination of different locomotion and interaction possibilities (possibly depending on the level of the multiplayer) is planned.
And traraaaa: At this meeting the Unity project was restructured based on the preliminary work of Janne and Anton. That means that now – parallel to the development of the storyboard – we can start with the development!
The focus of storyboard and development is first on level 0, where you arrive, and the third and thus last scene – both in combination with the softrobots and the MQTT protocol. The goal is to think and develop the (browser-based) WebGL and VR multiplayer in parallel.
At this meeting, our new expert board member Sebastian Keppler presented the BMBF-funded research project VitraS, in which he is involved as a project collaborator (lead at HTW: Prof. Habakuk Israel, also expert on the board of XR_Unites).
The focus of VitraS is on virtual reality therapy by stimulating modulated body perception, for example in obesity patients – it is thus a true embodiment project in a medical context. Based on Sebastian’s talk, a mutually fruitful exchange developed at the intersection of art, humanities, informatics and health.
Common topics were, for example, normative body images and barely existing »marginal body forms« in the digital world, facial expressions of avatars, counter-gendered hand models, body awareness and the potential of the mirror – symbolizing self-knowledge in visual art – to mediate perception of one’s own (avatar) body. The fact that body images are always mediated by media was also discussed – a fact that has enormous relevance for body perception in times of Instagram face filters and remote conferencing and reinforces the often unhealthy orientation towards idealized norms in our contemporary Western society.
VitraS is a cooperation of HTW Berlin with the University of Würzburg (lead), University of Bielefeld, SRH University of Health Gera, The Caputry GmbH, TU Munich, brainboost GmbH, CBMI.
We thank Sebastian Keppler for the great lecture and exchange!
Anton Krause presents the WebGL-Multiplayer. Photo: XR_Unites, HTW Berlin
XBPMMM (AT) addresses a huge range of exciting computer science areas – most of which have to do with embodiment experience in VR. On 10/27/21, the second meeting on this took place at the Research Center for Culture and Informatics (FKI) at HTW Berlin. The artists shared their experiences with the MQTT protocol, we played the already developed WebGL multiplayer, Janne Kummer brought softrobot forms cast out of silicone and skills, responsibilities and necessary technologies were important topics.
Potential work areas for further development:
Cross Platform Development (XR)
Multiplayer in WebGL and VR (bring together)
Integration of MQTT on different hardware and software and the interpolation of data between devices
Physicality and locomotion in VR
Deformability of meshes (avatar body).
Materials, textures (size, scaling, export from Blender)
Mixed reality installation with HoloLens 2 Collaboration by reVerb, Chitrasena Dance Company, Sri Lanka, and XR_Unites, HTW Berlin Media Theatre, Humboldt-Universität, Berlin, 2021
Dance: Thaji Dias Choreography: Heshma Wignaraja Dramaturgy: Susanne Vincenz Sound: Mareike Trillhaas 3D scans & management: Umadanthi Dias Developers: Christoph Holtmann, Laura Magdaleno Amaro und Ekaterina Losik Video & Scenography: Isabel Robson
Within XR_Unites funded by the European Regional Development Fund (ERDF) in the INP-II program. It is also supported by the national performance network (npn) – stepping out, funded by the Federal Government Commissioner for Culture and the Media as part of the initiative Neustart Kultur. Aid Program Dance.
Exciting! A first meeting took place on 7 October, which mainly served to get to know each other and make initial plans for cooperation. The next step will be a joint hackathon at HTW Berlin on 27 October!
From 3 September to 5 October 2021, the mixed reality installation TRANSIENT EXPOSURE can be experienced in Berlin-Mitte! It is the result of the collaboration between the INKA project XR_Unites at the HTW Berlin, the artists’ collective reVerb and the Chitrasena Dance Company – the result of XR_Unites’ first OPEN CALL.
The performance venue is the Media Theatre of the Faculty of Humanities and Social Sciences at the Humboldt-Universität Berlin.
Two to three visitors at a time can enter the installation with the mixed reality glasses HoloLens and immerse themselves in a 15-minute multimedia experience: In the physical world, the installation was conciously kept very minimal with rattan blinds, a fan and a large metal box. Archive material from the Chitrasena Dance Company in combination with a 3D sound collage from Colombo and partly interactive 2D and 3D elements formed the digital level of the installation, which thrilled many visitors.
»How seemingly simple and natural analogue and digital space interpenetrate here, how skilfully you guide the visitors, how lovingly and yet astutely you curate and stage the archive material – it all makes you want more!« wrote one visitor shortly after the experience.
The development with the VFX graph is part of our digital media production. In TRANSIENT EXPOSURE, the graph is combined with Kinect recordings of the dancer Thaji Chitrasena. It is a powerful free tool from Unity that is quite easy to learn even if you are not a computer scientist – especially if you have previous knowledge of software like Blender or Bolt, because the VFX graph is also based on nodes. That means you don’t have to work with code – instead, the user interface shows blocks (nodes) that can be connected to each other via edges. And what’s the point of all this?
As a depth camera, the Kinect generates data that are not visible at first. The VFX graph, on the other hand, consists of a visual particle system that can be edited via the nodes, e.g. in their shape, color or number of particles. This creates an effect through which the Kinect data can be made visible – e.g. via a pixel cloud.
An avatar serves as the medium between the VFX graph and the Kinect data: It is synchronized with both the Kinect data and the particle system of the VFX graph. To let the effect resemble the dancer in the end, the avatar still has to be edited so that its anatomy roughly corresponds to that of Thaji.
In TRANSIENT EXPOSURE, we work with the Azure Kinect, a sensor bar that contains advanced AI sensors, a depth sensor and a spatial microphone array. The device can track body movements (keyword body tracking) and save them as a data set. The AI combines all collected data with a rough skeleton model.
To enable our team members in Colombo, Sri Lanka, to easily make recordings with the Azure Kinect for Unity, a recorder is being developed in XR_Unites which currently includes the following features:
Capture motion via the Kinect’s body tracking feature
Transferring these movements to a human 3D model
Save the motion as an animation of a 3D model in a separate file
Simultaneously record audio via a microphone
Saving the audio in a separate (.wav) file
Specifying a concrete recording time
Simultaneous playback of audio and animation
Loading and deleting the saved recordings via a menu
TRANSIENT EXPOSURE is an artistic experiment with the Hololens 2, the latest Mixed Reality goggles from Microsoft, which some may already know from artistic works such as BLOOM: Open Space by Brian Eno and Peter Chilvers or Concrete Storm by Studio Shift (both 2018). Usually, it is used in industry – for example, in manual production by digitally overlaying additional information and instructions. So far, it has not been used much in art, which may also be due to its high price of over € 3,000 – however, it is sometimes also lent by Microsoft for artistic projects, for example in the two cases mentioned above. The Hololens 2 features 6 degrees of freedom, area- and space-, as well as gesture-, hand- and speech recognition. It is also suitable for people wearing glasses. In XR_Unites, the Mixed Reality application for TRANSIENT EXPOSURE is being developed with the Unity game engine.
The concept of the mixed reality installation Transient Exposure won our first Open Call in early 2021. We interviewed Isabel Robson, Mareike Trillhaas and Susanne Vincenz – as the artist collective reVerb they are behind the project idea in cooperation with the Chitrasena Dance Company (Colombo, Sri Lanka).
XRU: How would you describe your collective?
Isabel: We see ourselves as a video art collective. We explore spatial imagery in relation to performance. Over time, technologies have changed and so now we are on our first attempt with Mixed Reality (MR).
XRU: How did the collective come about and what was your vision at the time?
Susanne: reVerb was always linked to the question of how to integrate documentary material as a moving image in a live performance. This has evolved piece by piece over time. We worked with different artists internationally, on urban development in China, on the protest movement in Iran, and the videos took on a three-dimensionality in connection with the performance. This led to the next step, which was to think more in terms of installations. Starting with György Kurtág’s Kafka fragments, we had an installation at the Jewish Museum in 2013. That was our first joint work as reVerb.
XRU: And then joinedMareike. Right?
Mareike: Exactly. I already knew Isabel from other projects. We met for the first time at the Maxim Gorki Theater and then sporadically did some smaller things together. I joined reVerb last year.
XRU: Transient Exposure is your new project in cooperation with the Chitrasena Dance Company in Sri Lanka. How did this collaboration come about?
Isabel: I was lucky to spend three and a half years of my childhood in Sri Lanka. Every Saturday morning, my mother and I attended the Chitrasena dance school. There was a fascination with the movement and the music from that time. reVerb, meaning reverberation, is the program in this respect. After several visits in recent years, we are now in the middle of a joint project. The collaboration with XR_Unites is one chapter of this rapprochement and collaboration.
XRU: What is the appeal and potential of Transient Exposure for you?
Isabel: I have a great desire to experiment with HoloLens 2, the mixed reality glasses. Compared to the closed virtual reality experience, MR offers more possibilities for integrating a live performance. In this first stage, we won’t be able to work with live performance. It’s more about user interactivity. But what’s exciting is that content is spatially anchored and that you as a user decide what you see and what you don’t. And above all, the answer to the question: What can this technology of live performance bring on a scenographic level – and what not?
XRU: What can the audience expect?
Mareike: I imagine it so far like a mixture of claimed communication and real communication, because the dancer is supposed to connect with the people who are also in that space. Since it’s not live, the communication can’t really be direct. But by being able to use gesture control or maybe acoustic control, there is still that kind of interaction. The communication can still be between the two or three people who are watching and listening to it. And then there are also these levels between times, because the past also plays a role, and between places like Sri Lanka and Berlin.
XRU: You have chosen a form of presentation in which dancers are represented as point clouds. These can give the impression that what is represented with them is not tangible – it can also easily disintegrate or dissolve. Why this decision?
Isabel: Our title suggests that. There were always encounters of the Chitrasenas through tours and travels between their traditional dance and other stage artists in the Soviet Union, also here in the former GDR, for example with Gret Palucca. For a moment there is an exposure – one is exposed to another language or art form. What remains of the actual dance? That’s a question we ask with the Chitrasena Company archive. The Point Clouds move away from photorealism, and if you really want to focus on movement, then an abstract form is more appropriate.
XRU: The application will not only bring together dancers from different generations, but will also let the rhythm of traditional Sri Lankan dances like the Kandy Dance play a central role. How do you coordinate the music of old choreographies with new ones?
Susanne: In fact, they are the same choreographies that have been passed down through three generations. These are dances that came from the ritual contexts of Sri Lankan tradition and then were adapted for the stage in the 1940s. They were further developed by the women of the company and passed down through three generations. We work with overlays that are meant to make these layers of time visible. Thaji Dias still dances the choreographies that her grandmother danced in a different way.
XRU: Will this overlay also be in the music?
Mareike: It will be a mix between new and real sounds from Colombo. The heavy traffic, for example, plays an important role. But there can also be more abstract sounds that don’t reflect one-to-one how it is now or how it was, but that awaken associations. It can be sounds of costumes – there is a costume with a lot of metal that is very noisy. I’d like to integrate something like that, too.
XRU: To what extent will the political-historical context factor in, for example the Civil War?
Isabel: Political background, also where the country is right now, is important. We learn about the hurdles and challenges that each generation has experienced. When we look at the work of Upeca Chitrasena in the 1980s, it’s in the context of the beginning of the civil war, which lasted so long. Susanne: I think that in Transient Exposure you can’t directly experience such a complex background. But what you might get is that it’s a transitional situation. Basically the questions: What is the place of this dance today in Sri Lanka? To what extent does it have something to do with contemporaneity? What does this dance tell today? What does it trigger? Where is its place? That is something that we are also trying to convey with the MR installation.
XRU: What will happen with Transient Exposure after the XR_Unites grant?
Susanne: We will definitely take it to Sri Lanka. There is a possibility that it will be shown in different places on the island. We are in contact with various institutions, including the Goethe-Institut. One of the challenges in this collaboration is actually to develop something that can work for an audience in Sri Lanka as well as in Germany. And perhaps this new technology is particularly well-suited for this purpose, because neither we, nor an audience in Sri Lanka, have had so much contact with it so far.
The interview was conducted by Laura Magdaleno Amaro from XR_Unites on 03.03.2021.
