The problem of accessing non-textual information on the web

The problem of accessing non-textual information on the web

John A. Gardner, Hadi Bargirangin, Vladimir Bulatov, Hans Kowallick, and Randy Lundquist
Department of Physics and Science Access Project Oregon State University


People who cannot see the screen can nonetheless access most of the important information contained on most well-constructed web pages. Most information is present as text, and people can use screen reader software to read text audibly or through an on-line braille display if they wish. However there are many common non-textual methods of conveying information visually that are much more effective (for people who can see and understand what they see) than text. Maps, Charts, graphs, diagrams, math equations and many other common display methods often are indeed worth a thousand words. It is possible to use non-visual (audio, tactile, or audio/tactile) methods to access non-textual information. In principle, such access is possible with the technology of today, although improved tactile display technologies could make them easier. In practice, access to non-textual information requires that the information, not just a picture of that information, be available to the browser. We review possible authoring tools and file structures that could make such non-textual information more accessible to people who cannot see or understand visual data, more useful for all users, and more compact than bit-mapped images.


Non-textual information is currently represented on the web as bit-mapped pictures. Unless these pictures are accompanied by a complete ALT TEXT description, information in the bit-mapped image is not accessible. It is not accessible to people whose vision is too bad to see the image, to people with visual dyslexia, or to search engines. Moreover authors often find it difficult to present complex information in pictures that are easily understood even by people without print disabilities - in effect making that information inaccessible in practice to nearly everybody.

All these accessibility problems arise fundamentally from the "printed page" paradigm of bit-mapped graphics and could be greatly reduced by intelligent use of the capabilities of electronic media. We present a brief overview of the concepts of "smart graphics" - "information behind a picture" and intelligent graphics browsers for accessing such information. We also discuss briefly the presentation methods by which such information could be used by readers, including those who are blind or have severe print disabilities.

Nearly every part of smart graphics technology exists today, but to our knowledge there is no complete package that incorporates everything necessary to author a smart picture, incorporate it into an electronic document, and display it intelligently. This paper is a discussion of what could be, not what is.


We limit discussion to pictures that can be described as a collection of objects with quantitatively-definable properties and position. Our focus and examples are all two-dimensional and static, but the concept need not be so limited. Virtual Reality Modelling Language (VRML) and other three dimensional modeling languages have exactly this kind of object structure.

Charts, tables, graphs, diagrams, math equations, maps, etc, are common examples of pictures that can be represented by such object-oriented descriptions. There are undoubtedly many examples of representations (eg Escher's famous drawings of three dimensional impossibilities) that would be difficult or impossible to cast in a quantifiable-object-oriented form, but the vast majority of images intended to convey information could be so described.

The "information behind" such representations consists of a list of each object's properties and its position and orientation. Depending on which properties are included, the list may or may not be really "informative".

Consider as an example the simple equation, x equals the fraction whose numerator is y to the power n and whose denominator is 2. It is pictured as: image of x equals y to the power n divided by 2

It is within the capabilities of current-technology optical character recognition to recognize that this picture shows a character x on the left, a double horizontal bar next followed by a horizontal line above which is a character y with a raised character n to its right. Below the horizontal line is the character 2. The double horizontal bar may or may not be recognized as an equals sign, and the raised n may or may not be recognized as the superscript of the y character.The font type would probably be classified as italic. To our knowledge there is no commercial OCR software that would recognize the fact that the structure on the right is a fraction.

An information map identifying the x, y, and n as math font characters, the double horizontal bar as an equals sign, the n as a superscript of y, and the right-hand structure as a fraction would be very helpful. This information is equivalent to the page markup for the equation. The maximum amount of information would be a full computable object list that identifies the superscript as a power (as opposed to a superscripted index for example).

The kind of information contained in such lists is often internally stored in the authoring program that created the picture. If the picture is created by a computational program, the program knows that the picture is of a math equation and exactly what its various parts are. If the authoring tool is a common math editor, the information is less complete, because the meaning of the superscript n is not normally defined. However if the author has used the math editor correctly, the semantic content stored internally is equivalent to that of the printed picture.

Unfortunately, this internal information is lost if the bit map image is the only "information" retained in the published electronic document. If this "information behind the picture" were made part of the electronic file, then any application smart enough to retrieve and interpret the information has full access.

We emphasize that the process described above requires absolutely no additional effort on the part of the author. Of course it is not always possible to use authoring tools that are smart enough to know the full information content without some additional input from the author, but there are many programs other than math editors that are capable of creating information lists with little or no extra effort from authors. Examples include some chart-drawing programs, flow-diagram creators, electronic circuit diagram CAD programs, and scientific graphing programs.


A bit-map image is seldom the most efficient (smallest bandwidth) way to store non-textual information. If the information is stored in an an object-oriented form and if object properties include the information necessary to create the visual image, a graphics browser can simply construct the image when needed. A number of current programs (eg Community Place, CosmoPlayer, Liquid Reality, Live3D ) do exactly that.

If the graphics browser were also sufficiently intelligent, the reader could alter the picture in order to obtain more information or better information than could be presented in a single picture. A simple option that allows the reader to show some objects and not others would, for example, permit a reader to simplify many complex figures by viewing one part at a time.


Introspective scientists have long dreamed of a method by which the data illustrated in scientific articles could be made easily available. Scientific advances often suggest reinterpretation of previously-published results, but published figures are seldom accurate enough to allow data to be reconstructed from the printed image. If all the data, formulae used to fit or illustrate results, etc. were contained in an information file accompanying the figure, these dreams would become reality.

Many such reasons for wanting data rather than pictures of data exist in nearly all professional fields. For example, an electrical engineer who wanted to improve upon a published electronic circuit would find it extraordinarily convenient to have the ability to just read the electronic circuit information into a design program, make modifications, and proceed. This ability would avoid not only a great deal of human labor but also greatly reduce the possibility of errors resulting from human frilties in translating pictures into the information they represent.

Information lists also provide an extraordinarily flexible method of enriching information. Authors would like for non-textual information to be displayed in an elegant picture whose qualitative impact is often more important than the quantitative details. Nonetheless they must somehow indicate those details, and the result is often a cluttered picture that is so confusing that its message becomes lost in the clutter. Authors often find it necessary to write many pages of description about various details of pictures. Such information could often be better included as a field keyed to the object under discussion. Details not critical to understanding the article would not need to be included in the text but could be incorporated as information accessible in a field associated with the object(s) of concern.

Authors could also make good use of display options in an intelligent graphics browser. For example an author could provide a menu permitting several specialized views of different layers of a picture, or separate views of each data set in a group of overlapping data, etc.

Smart graphics could also provide a powerful new tool for educators. Authors of pedagocic treatises could write a simple introductory level text and supplement it with very detailed examples illustrated and explained in figures. A great deal of teaching is done by graphical illustrations, and it is known that a substantial fraction of students simply don't understand many such concepts (eg vectors). Students may be able to grasp such concepts more readily if they had the ability to ask figures to reveal information.


People whose vision is good enough to see objects on the video display can choose to access information by indicating, eg by clicking with a mouse, on the object and opening up a dialogue box that displays information about that object. As with any other textual information, the dialogue box could be browsed with speech or braille as well as read visually. The combination of visual access to the picture and visual or audio access to the information is likely to provide greatly improved accessibility for people who have visual dyslexia or to people with only moderate visual disabilities.

An intelligent browser would include some keyboard and- mouse-accessible method to access the object list. Some well-defined objects such as math equations, simple tree diagrams, and many kinds of charts and graphs may be browsed and understood without ever seeing the picture. However there are many kinds of non-textual information that can be understood only through spatial relationships. The information conveyed by such pictures is accessible only if the image can be seen.

For a blind person, a picture can best be "seen" tactually. Consequently, full access requires some device that can display a tactile image of the objects on the video screen. At present there is no commercially-available affordable device that provide such a tactile image. A considerable amount of research worldwide is being devoted to R&D on possible refreshable tactile display technologies, and we can only hope that some adequate device becomes available in the not-too-distant future. One possible device of the future is describe by Fricke and various discussions of the subject are included in the information linked to Equal Access to Software and Information (EASI).

Until that time, the quickest access method for blind people is to print a hard copy tactile image of the desired picture and use that tactile image to locate objects and "click" on them for information. The technology of using a tactile image, an external digitizing pad, and an object description of the image accessed in this way was pioneered by Prof. Donald Parkes with his Nomad tablet. The technology is also discussed by Loetzsch.

We suspect that use of moderatly intelligent browsers and the full range of these alternate access technologies will be sufficient to make most non-textual information accessible to people with print disabilities. Accessing the more complex figures will require skill on the part of the reader in using the ability to pick and choose what objects to represent. It is certainly true that not everything presented by every author will be directly accessible however. Nonetheless, the need for human intervention would certainly be a great deal smaller, and consequently information much more accessible, if graphics were smart enough.