Tech Project

Open Call
In Residency
Residency Outcome
A photorealistic sign-interpreter avatar inserted into a personalised TV stream for the Deaf
From Sept. 1, 2017 to Aug. 31, 2020
  • -

Description of the challenges faced by the Tech Project

Content4All exploits the relationship between the Human Being and her description with visual and linguistic content. The big challenge of the project is to develop the avatar signing the TV content. The creation of a photorealistic 3D avatar based on cameras with complementary spatial information implies the analysis of a huge amount of data. E.g. here is also the question how to represent the avatar visually (how should it look like). Also important is how to communicate our intermediate data clearly and user friendly, e.g. workflows and early prototypes to the user as we conduct a user-centered design process.

Brief description of technology

2 calibrated professional cameras and 1 Microsoft Kinect will record and stream a human sign-interpreter for (Swiss-)German and (Flemish-)Dutch sign languages in the domain of news – in particular sport and weather forecast. The data captured (video, positions) and metadata (subtitles, speech-to-text, scene script) will be analyzed and synthetized into language models and will serve to reproduce the human sign-interpreter into a photorealistic avatar. The vision of the CONTENT4ALL project is to translate spoken language in TV automatically into sign-interpreted language and present it via a 3D representation of a human sign-interpreter (photorealistic avatar). All the data captured and metadata streamed will be made available to the artist and we can investigate the possibility of sharing part of the avatar model, too.

What the project is looking to gain from the collaboration and what kind of artist would be suitable

We are an interdisciplinary team of experts of different domains (e.g., computer sciences, data modeling and broadcasting). We have also experts developing the user interaction, but rather from a functional point of view. What we lack is an expert, who communicates our technical possibilities to the addressed community with a high graphical feeling. We are open to any kind of artist’s ideas, but could imagine artist’s support in the way of e.g.: • Visualizing projects (intermediate) data, e.g. in form of graphic user stories or visual demos (videos) or the like. • An original and novel interpretation of the visual and spatial information of the sign interpreter captured. • Discussion with the artist to exploit the relationship between the human being and its synthetic representation In addition to this, the evaluation phase with the end users might be influenced and better described thanks to the artist point-of-view.

Resources available to the artist

We imagine the artistic residency to take place in Berlin. Two partners are based in Berlin, whose work is closely related to user needs and user representation. These partners would both, provide meaningful input regarding the user group und profit from artistic creativity the most. HFC Human-Factors-Consult is responsible for assessing user needs and requirements, designing the system user interface as well as conducting user tests, and Fraunhofer HHI is developing the 3D representation of the signer. Content4all will support the artist with in kind-support and mentorship on the project technologies and data. These will also be made available to the artist for producing his/her artworks.