How do I use IMAGE?

Overview, controls, installation and hardware

Installing the IMAGE browser extension #

The IMAGE browser extension works on non-mobile versions of Chrome, also including Edge, Brave, Safari, and Opera. It does not yet work on iPhone or Android devices. To use IMAGE, install the extension in your web browser from the Chrome webstore. Once installed, a button “Interpret this graphic with IMAGE” appears below graphics, charts, and maps. By activating these buttons, you’ll send the graphic to the IMAGE server. The IMAGE server then sends back one or more interpretations, which are presented to you in a list. Note that the full graphic is sent to a server, so we recommend not using IMAGE on graphics that contain sensitive or personal information.

Understanding the interpretations #

Photos #

Regions #

Regions are broad areas of photos made up of similar objects, like trees in a forest, or large things, such as a wall or floor. This is to give you an idea of where the larger components of the photo are relative to other elements of the graphic, rough shape, and spatial extent.

This is what three regions sound like together:

Regions example (boat, 2 segments).

To understand what you are hearing or feeling, imagine that, using the buzzing sound, you are stretching your arm out in front of you to feel the edge of the region, and tracing the contours of the edge clockwise, with the pitch also changing as it traces up and down.

Things and people #

Things and people are found by our object detection programs. Things includes a wide range of objects such as cars, animals, signs, cups, neckties, and many others. People are humans, which may be an entire person, or just a face.

Here is a recorded example of things and people:

Things and people example.

In that example, there were two different thing groups, as well as people, and you should have heard their relative locations in the scene as popping noises.

Try it out: A bear in the woods #

A black bear in a forest. Below is a photograph of a black bear in a forest. If you have the browser extension installed, you can:

Tab to navigate to the graphic
Tab once more to select an invisible button “Interpret this graphic with IMAGE”.
Activate the button. You will be notified when when the graphic is sent, when the data is being processed, and when the renderings window opens.
When it appears, the renderings window has a list of “Interpretations”.
Activate one of the Interpretation headings to expand it, and access the text or controls for that interpretation.
For example, in the “Rich audio description”, you can play it by activating the “Full rendering” button, plus there are additional buttons for selecting a specific portion of the rendering you want to hear.

Try it out: A street intersection #

A street level view of downtown Toronto, Queen Street West in front of City Hall. Try the same steps on a more complex photo, to get a feeling for where all the things are in the photograph.

Maps #

Points-of-Interest #

Imagine that you’re standing at the map’s location, facing north. You’ll hear a little jingle in a direction relative to due North followed by the name of a place. The volume of the jingle indicates how close it is: louder if it’s closer, quieter if it’s farther.

Here are 5 points of interest centered around a popular tourist spot in Toronto, Ontario. Two of them are to the right, one is behind, and the other two are to the left.

Recorded example of 5 POIs.

Try it out: The Royal Tyrrell Museum in Drumheller, AB. #

You can explore points of interest around a location using an embedded map. Follow these steps:

Tab on the embedded, interactive Google Map focused on the Royal Tyrrell Museum in Drumheller, AB.
Keep pressing Tab to move past the map controls to the dropdown “Select IMAGE Options” just below the map.
Expand the dropdown menu using the Space or Enter.
Within the options, select “Interpret this map with IMAGE”. You will be notified when when the request is sent, when the data is being processed, and when the renderings window opens.
Tab to navigate the “Interpretations”. Similar to the dropdown, expand the “Interpretations” using Space or Enter.
Play the audio rendering generated under “Points of interest around the location in the map.” to discover the location’s surroundings.

Charts #

IMAGE can interpret graphs and charts made using Highcharts.

Line graphs #

Currently, IMAGE can turn pie charts and line graphs with a single variable into spatialized audio. You will hear the title of the chart, followed by the website the chart is on, and the variable being measured. You will then hear a noise going left to right as it goes from the start of the x-axis to the end of the x-axis. The noise goes up in pitch as the value it represents goes up and down in pitch as the value goes down.

Here is a recorded example of what IMAGE produces for a financial chart.

Line graph (Etherscan) example.

To use IMAGE to create that interpretation dynamically, navigate to a Highcharts chart on the web (try this!) by activating the IMAGE button below the chart on that page.

Try it out: Live Highcharts #

A live Highchart you can experience with IMAGE. Follow these steps:

Tab to the chart on the right.
Tab once more to select the “Interpret this chart with IMAGE” button.
Activate the button. You’ll be notified as the request is sent and processed, and when the renderings window opens.
In the renderings window, expand the Interpretation and play the audio.

Pie charts #

IMAGE can also turn pie charts into spatialized audio. Each slice’s sonification and loudness is proportional to slice share.

Try it out: Quebec COVID statistics pie chart #

Below is a screenshot of the INSPQ chart. To hear a spatialized rendering, you’ll need to go to the actual page:

Open the INSPQ page.
After the page loads, tab to just past one of the pie charts and activate the "Interpret this graphic with IMAGE" button.

Tips and Tricks #

IMAGE Launch Pad: Pressing Alt+I on your keyboard pops up the IMAGE Extension Launch Pad. Here, you can activate IMAGE on a file stored on your computer, manage the extension options, or access this page if you need help.
Instead of tabbing to photographs, you can bring up a context menu (e.g., by right-clicking a photo), and activate IMAGE from there.
We are expanding support to the Humanware Monarch pin array tablet. When enabled, it will add additional options that let you send experiences directly to the Monarch, or edit them before sending. This is in beta, and does not yet work on production Monarch tablets.