Data-driven development of Virtual Sign Language Communication Agents

Abstract

Engaging deaf and hearing people in common discussions requires interfaces to help them understand each other, such as robot agents that translate spoken language into Sign Language (SL) expressions and vice-versa. However, the recognition and generation of signed sentences is a complex task of high dimensionality that cannot be solved in sufficient quality yet. Thus, it is necessary to develop new technologies of improved performances. The sequence to sequence neural network model, traditionally used for machine translation, is adapted to the above two tasks by treating a SL sequence as a multi-dimensional sentence. We defined an encoding of the SL annotations and conducted experiments on the network structure to define a most accurate translation model. This study proves the network trainable and possibly applicable in real-life with an extended dataset, which shall be tested for deployment in virtual translation assistants in the following.