Kanav Kahol Website

Research Summary

Visual sensation and perception allows sighted individuals to perceive distal environment beyond the reach of their hand: an area known as the kinesphere. The ability to sense and perceive objects beyond our reach is central to many day-to-day activities such as driving, navigating and so on. Vision provides sighted individuals the ability to plan and execute spatial tasks. Additionally, the correlation between visual, auditory and tactile sensation builds redundancy in spatial representation that allows sighted individuals to exploit intermodal transfer and coordination mechanisms for spatial tasks. Unlike sighted individuals who perceive distal environmental stimuli primarily through visual sensory channels, individuals who are blind perceive environmental stimuli through haptic and auditory channels. Studies have reported that individuals who are blind are proficient and skilled in many spatial tasks such as texture discrimination, texture description, shape description, shape discrimination, object recognition and object description. However a major limitation of the haptic modality is that it limits the perceptive field of individuals to the kinesphere. Secondly while individuals who are blind build intercoordination models between auditory and haptic channels, psychology research has shown that these modalities are not adapted to perceive distal stimuli as much as vision. In this project we are conducting experiments and developing algorithm that can aid in distal environment perception by converting visual data into haptic data.

Research Approach

Figure 1 shows our overall research approach in this project.

Figure 1. Overall Research Approach

We use conventional digital video camera to capture objects. This capture process is aided by existence of a haptic object database: a database of objects whose profiles we have stored and we actively search for those objects. This capture process is also aided by storing what is called as the haptic exploration of an object. Haptic exploration of an object refers to the hand movement used by humans to perceive the object. The gestures used by humans to perceive features of an object have a structure and give clues into what the feature is. For example lateral motion of index finger of an object represents coarse texture while smooth movement of index finger means a smooth texture. This is an example of how visual capture can be aided by storing the haptic or touch profile of an object in the form of hand movements: A truly multimodal system. Once an object has been captured visually in the training phase, we let the user perceptually categorize the shape, size, weight, texture and material of the haptic object. This provides the training data for us to generate computational models of the haptic features of an object. These computational models are developed based on neuropsychological and cognitive psychological models of visio-haptic feature analysis. These computational models lead to the development of visio-haptic prototypes or visio-haptic templates of objects. These are datastructures that store both visual as well as haptic features of the object and then develop automated mapping between the various visual and haptic features. These prototypes are stored in the haptic object database along with the haptic exploration strategies.

In order to develop such systems, we are currently collecting a database of 50 objects. Figure 2 shows an image of the objects in the database.

Figure 2. Haptic Object Database.

The objects represent a controlled variation of shape, size, material and texture. We visually capture the objects images from -90 degrees to +90 degrees using our face recognition setup. Then we have human subjects who come in to our lab and when blind folded categorize the haptic features of each object. This process gives us the ground-truth on haptic features of the object. This work has been submitted to the IEEE International Conference on Computer Vision 2005 to be held in Beijing China.

Currently we are in the process of collection of data. The objective of this research is to essentially explore relations between visual and haptic features. We are also conducting psychological experiments to analyze visual and tactile space using multidimensional scaling with the help of Dr Donald Homa

As an immediate application of this research, we have developed a haptic glove based block diagram rendering software. This system allows blind individuals to perceive information presented in block diagrams. We use image processing techniques to extract information from block diagrams and develop an annotation of the diagram. The user can then haptically explore the block diagram with the system presenting haptic and audio feedback about the block diagram to the user. We are publishing results and methodology of this in the First IEEE Workshop on Computer Vision Application for Visually Impaired 2005 to be held in conjunction with CVPR 2005.

Publications

K Kahol, P Tripathi, T McDaniel, S Panchanathan, "Hand anatomy based modeling of manual haptic gestures" submitted for review at First International Conference on Pattern Recognition and Machine Intelligence (PReMI'05), to be held in Kolkata, INDIA.

K Kahol, T McDaniel, P Tripathi, S Panchanathan, "Rendering Block Diagrams Accessible through an audio-haptic interface", submitted to IEEE Computer Vision for Visually Impaired Conference to be held in conjunction with IEEE Computer Vision and Pattern Recognition Conference 2005, to be held in San Diego, CA 2005.

T McDaniel, K Kahol, P Tripathi, S Panchanathan, "Visio-Haptic database of objects for automatic content creation in multimodal environments", submitted to IEEE International Conference on Computer Vision 2005 to be held in Beijing China.

K Kahol, P Tripathi, S Panchanathan and M Goldberg, "Formalizing Cognitive and Motor Strategy of Haptic Exploratory Movements of Individuals who are blind", presented at IEEE Haptic Audio Visual Environment Workshop HAVE 2004 held in Ottawa, Canada 2004.