Rybena: a java R&D project boosting the DIGITAL INCLUSION of INDIVIDUALS with Special Needs, Through Mobile Communication.

THE RYBENÁ PROJECT

The team’s ultimate goal is to enable communication between a blind and a deaf in the future through the use of a cell phone. Since this project requires knowledge with which the team is not familiar and for the purpose of obtaining intermediary results, a simplified model was developed to allow hearing individuals to talk to deaf individuals through a Telco operator. The speech is processed through a Via Voice acknowledger where it is converted into text and then sent to the LIBRAS word interpretation service. To make it possible, the system establishes a connection with a Prevalence Data Model (PDM) (Villela 2002, Wuestefeld 2001, Wuestefeld 2002) where a LIBRAS set of animations is stored. For each recognized word, a corresponding LIBRAS animation is transmitted to the client’s cell phone. In case one word does not have a correspondent symbol at the PDM, images of all its characters are transmitted letter by letter (spelling module) to the mobile device (Fig.1).

Figure 1. Rybena Project Architecture

Figure 2. View of Project Support Classs


Figure 3. View of presentation tier

Figure 4. View of business logic tier

The architecture may be currently broken down into five work forces, the first three of which are in charge of developing the infrastructure required for communication services, the fourth will validate the project’s concept, and the fifth will provide a universal Portuguese/LIBRAS translator.

Figure 5. View of data source tier

      Setting up the LIBRAS library
As the first effort to import some words (LIBRAS animations) to a cell phone, five animated gifs were created, corresponding to the phrase “welcome to Banco do Brasil.”1 The images were generated according to restrictions on readability and memory costs. Each image occupies 11 KB on average.
Among the social inclusion programs developed by the DFJUG in partnership with the CTS Institute, the deaf community was contemplated with a training program in the Java language. This activity resulted in the need to create a LIBRAS terminology to represent programming concepts and, in particular, object-oriented programming. Table 4a shows the set of terms developed by students for such concepts.

a) words produced in a tailor manner (concept and development)

(b) words produced based on INES dictionary (development only)

(c) words being produced based on the INES dictionary

Table 4. The first words produced

New terms were aggregated to the animations to facilitate teacher-student communication in the classroom. An additional set of words was then created, since such words are used in general communication. Therefore, another ninety-six words were incorporated to the database (Table 4b).
A considerable effort was required in developing each animation because conceiving and creating animations is a time-consuming job. On the other hand, INES, of the Brazilian Ministry of Education, provides a Brazilian LIBRAS dictionary in the form of AVI-formatted images, currently consisting of a little more than two thousand words. A set of words was then selected to supplement the words already existing (Table 4c) and made into a three-hundred-word database2. The process gained agility and confidence since animations are currently developed based on the pattern proposed by INES.
Like the hearing community, the deaf community also develops regionalisms (slang). The authors noted that the deaf involved in image development often questioned the signs presented by the INES. The authors’ decision to maintain the pattern for convenience purposes has caused some stress among deaf team members. Such problems will be dealt with in the future to avoid further conflicts.

    JMS service implementation
Access to heterogeneous clients (PC/cell, cell/cell, cell/PDA, and others) was defined as the desired characteristic for the Rybena project. Therefore, the team decided to use a JMS server (JMS, 2004; Monson-Haefel & Chappell 2001) to provide the messaging infrastructure enabling client integration within this context.
Such decision was based on the fact that the JMS technology was the technology that best met the project needs. In addition to the messaging service, the JMS technology will also allow Rybena to integrate with other services such as databases, third-party services and different computer platforms. The JMS technology also rules out the need for services of a telephone operator since it requires only a TCP/IP connection between the (heterogeneous) clients and the server.
The Peer-to-Peer connection will be used as required, always passing through the JMS server, thus allowing a PC to chat with a PDA, for instance.
An immediate advantage is the total control over Rybena’s interface with the JMS server, enabling, for instance, that critical message services are signaled until the (deaf) client removes it from the informational context with which it is associated.
The JMS server may provide the Rybena project with an architecture that supports a set of plug-ins directly connected with the services available.
At present, the team is working with the JORAM3 server, developed by the Institut National de Recherche en Informatique et en Automatique, (INRIA - France) in a joint venture with Bull and France Télécom. The fact that this is a free software project was a decisive factor.

    Adapting prevalence data model to cell phones
Prevayler4 is a Java application, written in only 350 code lines, enabling object persistence without using a database owing to a model that takes snapshots of memory at predefined times (Birrell et al., 2001; Wuestefeld, 2001, 2002; Villela, 2002). This system, implemented on higher than 1 Terabyte bases, provides response times one thousand times faster than through JDBC connections even with cache.
The project intends to implement this technology in mobile devices, since at present there are no databases available to cell phones that meet project specifications, that is, a low-cost cell phone that supports J2ME applications.

    Universal translator

Figure 6. Rybena Mobile to Person

While discussing the possibilities raised by the Rybena project to enable communication between a deaf and a speaking individual holding a Rybena-enabled cell phone, the team decided to use the cell phone as a translator. This would work as follows:

1.The speaking person turns the cell phone screen to the deaf and speaks on the back of the equipment (in the test with Sony Ericsson P800 the cell phone was able to pick up the voice from the back);
2.The voice is sent to the Telco operator, which, upon recognizing the equipment as one used to communicate with deaf individuals, transfers the voice to the Voice Transformation Module (VTM). After recognizing the voice, the VTM selects the indexes of the proper LIBRAS gestures and sends them to the cell phone;
3.The cell phone then uses the indexes to retrieve the appropriate sequence of LIBRAS gestures from its storage and displays the resulting gesture composition on the screen;
4.The deaf would be able to understand the message if he/she is conversant in LIBRAS.

Obviously this is only a one-way communication but it can be upgraded to a two-way communication should the deaf be able to tap the LIBRAS code into the cell phone and the Telco operator send the corresponding sound to the cell phone. This would require that, upon picking up the cell phone, the speaking person would tap a command to release the voice to the cell phone loudspeaker.

    LIBRAS training course on cell phone
Within this context, the proposal of a virtual teaching-learning environment emerged on the LIBRAS (Brazilian Sign Language) web site. This particular environment should offer support to people integration, computer systems and contents, allowing cooperative and collaborative interactions among students and between students and teacher, including communication using cell phones.
http://www.cts.org.br/aula/work/org/apache/jsp/Aula1/Aula1Exemp.jsp
Six web pages were created on the application developed for the project. See below the web page list:

1 - Content introduction
2 - List of examples
3 - Exercise treatment
4 - Evaluation treatment

The purpose of the first web page is to introduce the LIBRAS alphabet content. The page consists of 26 images representing alphabet letters in addition to other 10 images representing numerical digits. The images are accompanied by their respective symbols for easy identification. The page includes HTML content and is intended to be an introduction only, being therefore a static page.

The presentation of the page listing examples enables the user to watch an animation and view the word relating to it. The user may repeat the animation as many times as he/she wishes and request that other words be included in the example. Examples may be generated on a randomized basis and changed according to the content of the words existing on the database. The page results from HTML, JavaScript, and JSP integration. The JavaScript part handles the animation and the JSP part the class performing database access functions and treatment of searched data. This page not only generates the loop where the user searches as many examples as he/she wishes, but also allows the user to proceed to the exercise page at any time or return to the main menu.

The exercises (Figure 7) are divided into two categories to allow the development of different skills which are randomly presented to the user. The first category asks the user to train gesturing of a word and then check if it is correct. The second category shows an animation and asks user to indicate which word the animation refers to. This manner of organizing the exercise activity allowed the exercise page to have a structure with one page shows the exercise and related answer (for one exercise category) and another shows the answer to an exercise generated on a previous page (for the other exercise category). Animations are handled in JavaScript and the treatment in which exercise category is shown is handled by JSP on this page. The JSP part also handles exercise randomness. The page allows user to proceed to the evaluation section or return to the main menu.

The evaluation section consists of 10 exercises intended to evaluate the student’s understanding and assimilation of the alphabet letters presented on the content page. The evaluation is set up in a dynamic and randomized manner, always seeking to generate different evaluations and to diversify the content addressed in the previous stages. Page animations are handled in JavaScript and database access and page evaluation setup in JSP. For each animation there is a set of items where one is the answer and the others are randomized items to fill the number of exercise answer options. Exercise answer preparation does not follow a predefined criterion. The page allows the user to return to the main menu and continue navigating on the web site or return to the content to continue to work on page contents.

Figure 7. Example of a LIBRAS class

introduction | content | the project | current situation | final remarks