The IMAGE Project

If you are blind or low vision, have you ever wanted to hear a photograph or pie chart, and not just a description of what is in it? The IMAGE project (Internet Multimodal Access to Graphical Exploration) adds controls to graphics in your browser so you can activate it on photographs and some charts and maps, on any webpage you visit. You will then receive experiences that combine spoken words with other sounds you hear around your head, that indicate details such as where things are, how large they are, or other information depending on the graphic. We are also working on touch experiences using the Humanware Monarch braille pin array tablet, but those IMAGE experiences are not yet released.

To try IMAGE right away, jump to the tutorial, or install the browser extension from the IMAGE Project Chrome webstore page.

Our Approach

We use rich audio (sonification) together with the sense of touch (haptics) to provide a faster and more nuanced experience of graphics on the web. For example, by using spatial audio, where the user experiences the sound moving around them through their headphones, information about the spatial relationships between various objects in the scene can be quickly conveyed without reading long descriptions. In addition, rather than only passive experiences of listening to audio, an optional haptic device can help the user literally feel aspects like regions of a landscape, objects found in a photo, or the relationships between components of a textbook diagram. This permits interpretation of maps, charts, and photographs, in which the visual experience is replaced with audio and touch information.
Sample renderings returned by the IMAGE client Google Chrome extension.
Sample renderings returned by the IMAGE client Google Chrome extension

Engaging the Community

Partners Gateway Navigation CCC Ltd and the Canadian Council of the Blind (CCB), a consumer organization of Canadians who are blind, have helped us to ensure that our system is in line with the needs of the community. In addition, we have conducted user studies with teachers and students using the Monarch braille tablet. We are always looking for feedback from users, so do not hesitate to contact us with questions, comments, or suggestions!
Monarch Braille tablet
Monarch Braille tablet

Developers

We know that our small team will never be able to create experiences for all kinds of graphics, so we have built IMAGE to be modular (Docker microservices) and fully extensible. You can create your own microservices, building on top of the ones already created, to focus on just the content and experiences that matter most to you. Then you can run your own IMAGE server that services client requests, and use it by entering the URL of your own server in the IMAGE browser extension options. IMAGE is an open source project, with github repos for Server components and our Chrome browser extension.

For detailed information on the IMAGE architecture, read our paper IMAGE: An Open-Source, Extensible Framework for Deploying Accessible Audio and Haptic Renderings of Web Graphics to get an overview of the system as a whole, then consult the wiki for the IMAGE server repository for more specific implementation details. We are happy to receive pull requests. Please see our Contributor License Agreement.

If you are thinking about using IMAGE as a foundation for your project, we would love to hear from you, since we would be delighted to see IMAGE become a playground for researchers, developers, designers, and others to create and deploy new ways to help make web graphics more accessible.

Researchers

IMAGE is an ongoing research project at McGill University’s Shared Reality Lab. Research topics are constantly changing as students come in and out of our lab, but in general, we seek to:
  • Run user studies on how best to convey information, which varies depending on the user’s background and needs. For example, can we make a photograph more comprehensible to someone who is blind from birth by transforming the photo into a soundscape based on the relative “depth” of the objects it contains, rather than just their locations on a 2D plane?
  • Optimize learnability of rendering techniques, such as spatialized audio, for people who are not familiar with it.
  • Experiment with haptic hardware devices, such as the Haply 2diy force feedback device and the Monarch pin array tablet. Although not yet widely available to end users, using touch in conjunction with audio can better convey size and relationships.
  • Seek to create tools to support designers in creating haptic effects without the need for programming. For example, can we use machine learning techniques to guide a haptic designer in creating effects (Investigating Haptic Co-creation with Reinforcement Learning)? Can we create a visual representation of haptic effects to help designers more quickly and easily understand the results of their design changes?
  • Work with teachers and students who want support for using graphics in a classroom environment. How would we link multiple haptic devices in a classroom to the teacher’s IMAGE extension, and create tools for teaching multiple students with these devices? Can we create tools that let teachers manually modify the experiences that IMAGE creates, to tune them for their specific classroom needs, and fill in details that automated methods still miss?
  • Continue building out the IMAGE technical platform to make it easier to use, and able to create richer experiences, as documented in our publication “IMAGE: An Open-Source, Extensible Framework for Deploying Accessible Audio and Haptic Renderings of Web Graphics”
If you are planning to evaluate IMAGE, please make sure to review the README files for the preprocessors in the IMAGE server repository for up to date information on the status of each preprocessor.

Funding and Partnerships

This project is carried out by McGill University’s Shared Reality Lab (SRL), in strategic partnership with Gateway Navigation CCC Ltd and the Canadian Council of the Blind (CCB). We gratefully acknowledge numerous sources of funding:

Press

Talks and Presentations