ChromeVox: built-in spoken feedback for Chrome OS

By T.V. Raman, Research Scientist

Cross-posted with the Chromium Blog

We recently unveiled ChromeVox — a built-in screen reader for Chrome OS — during Google I/O 2011. This is an early developer beta that is designed to help authors of web applications come up to speed with platform accessibility on Chrome OS.

ChromeVox is built as a Chrome extension. This means that unlike most accessibility software, it is built using only web technologies like HTML5, CSS and Javascript. As the built-in accessibility solution for Chrome OS, it can help users with special needs access modern web apps, including those that utilize W3C ARIA (Access to Rich Internet Applications) to provide a rich, desktop-like experience.

ChromeVox leverages two of Chrome's experimental extension APIs, the experimental.tts API for cross-platform text-to-speech, and the experimental.accessibility API that lets an extension listen for accessibility events in Chrome's menus and toolbars. In turn, ChromeVox exposes a simple screen reader API to web developers who want to further customize the ChromeVox user experience. Thus, within your application, you can:

• Automatically generate spoken messages and earcons.
• Set ChromeVox to synchronize with your application's current focus.

ChromeVox also comes with an interactive online tutorial that demonstrates how users of spoken feedback interact with webpages. Examples range from static content to interactive applications. You can test these same navigation techniques within your own applications to quickly verify users can reach all portions of your application using the keyboard and obtain meaningful feedback. You can then annotate your application with the necessary ARIA properties and other accessibility enhancements to ensure that blind and visually impaired users gain complete access to your application. Please see our Google I/O 2011 talk for more.

Details on enabling accessibility in Chrome OS can be found on the Accessibility help page, and the Chrome extension is available for download from our Wiki page. For now, ChromeVox is targeted at end-users on Chrome OS, but it may also prove a useful tool to web developers using Chrome on all major platforms. We welcome your feedback via our Open Source project website at http://google-axs-chrome.googlecode.com.


T. V. Raman is a research scientist at Google. He leads a team of engineers building innovative user interfaces on Android and Chrome OS, and researches creating highly efficient eyes-free interfaces.

Posted by Scott Knaster, Editor

Benetech: enabling reading for all

By by Gerardo Capiel, VP of Engineering, Benetech

This post is part of Who's at Google I/O, a series of guest blog posts written by developers who are appearing in the Developer Sandbox at Google I/O.


Benetech is a different kind of technology company, where we measure our success not on ROI, but Return to Humanity. Benetech is a non-profit organization that builds software solutions to address large scale and global social needs in literacy, human rights and the environment. Many of our software solutions are delivered via an open source model.

One of our latest literacy projects has been to develop an Android-based book e-reader for people with print disabilities. Print disabilities, such as blindness, paralysis, or dyslexia, effectively prevent a person from reading traditional print books. Many of those people qualify to have free or inexpensive access to books thanks to an exemption in U.S. copyright law called the Chafee Exemption. Bookshare, one of Benetech’s largest solutions, provides over 100,000 e-books in the accessible DAISY format (similar to ePub) to over 130,000 Chafee-qualified people in the U.S. Once downloaded from Bookshare, DAISY books can be consumed using Assistive Technology (AT), which employs Text to Speech technology (TTS), electronic refreshable braille, or large fonts for low vision users. Bookshare was originally built 10 years ago on a PHP architecture, was migrated to a Java/Hibernate/MyBatis framework and we recently migrated the content repository to S3.

Bookshare has a public REST-based API, which enables AT developers to directly integrate their applications with our API. Through the API, an AT application can enable a user to directly search for books, browse books based on category or recently added books and download a book packaged up as either a DAISY file or a BRF file commonly used by an electronic refreshable Braille display, such as HumanWare’s BrailleNote. Being able to directly download from the AT application simplifies the potentially frustrating experience of having to manually transfer the books from a PC to the AT software or device. The API supports anonymous use, which provides access to freely available books and open educational resources that have no copyright restrictions. Only qualified Bookshare members can access the copyrighted books and periodicals. To learn more about the Bookshare API and obtain a developer key, visit http://developer.bookshare.org/.

Recently Benetech challenged a group of volunteers to build a free, open source, mobile and accessible e-reader which leverages the Bookshare API. The volunteers chose to extend FBReaderJ, a popular open source e-reader for Android which leverages Android’s TTS API (android.speech.tts.TextToSpeech). The project is a work in progress, but so far the volunteers have added DAISY format support and Bookshare API integration. They are now working on improving the accessibility of the application and are evaluating different user interaction experiences to making it easy for print disabled users to access books. To learn more about Android accessibility, check out http://eyes-free.googlecode.com/ and to check or contribute to the project visit http://github.com/amahule/fbreaderj.

Ultimately, we believe this Android e-reader could also benefit people who don’t qualify under Chafee, but who have other disabilities, such as Attention Deficit and Hyperactivity Disorder (ADHD), which makes it hard for them to enjoy printed or even traditional e-books. Furthermore, TTS technology and the Google Translate API may help us use inexpensive Android devices to distribute valuable knowledge locked up in print to illiterate populations in developing countries. Accessing knowledge for illiterate populations will be critical to the success of emerging democracies.

We welcome ideas you may have about our app or Benetech in general. We particularly welcome anyone interested in contributing product development skills to our Android e-reader project or any other open source projects Benetech is working on regarding literacy, human rights or the environment. To learn more about how you can volunteer your skills and time, please go to http://benetech.org/join_us/volunteer_opportunities.shtml.


Come see Benetech in the Developer Sandbox at Google I/O on May 10-11.

Gerardo Capiel is a two-time Internet entrepreneur turned social entrepreneur. When he's not geeking out for humanity, he's looking for inside tips on the best food in San Francisco.

Posted by Scott Knaster, Editor

Apps4Android: Developing accessibility apps for Android

By Steve Jacobs, President, IDEAL Group, Inc., and CEO, Apps4Android, Inc

This post is part of Who's at Google I/O, a series of guest blog posts written by developers who are appearing in the Developer Sandbox at Google I/O.


IDEAL Group's Android Development Team has developed and released several apps in the Android Market. In this post, we'll highlight three of our apps which capture some of the best aspects of developing on Android.

IDEAL Magnifier

Android smartphones can have amazing hardware, and the platform gives developers the ability to tap into that power. Traditionally, handheld video magnifiers have been standalone, dedicated, hardware devices that can cost hundreds of dollars. Thanks to Android's Camera APIs, we're able to offer similar functionality in the form of a free, open source app.

In addition to using Android's zoom and flash features to make things easier for our users to see, we also enable our users to apply color effects such as converting everything to monochrome and even inverting the colors to improve contrast. Despite the wide variety of Android devices available, we found it relatively easy to support multiple devices since Android enables developers to check what the maximum zoom level is and what color effects are supported. Here's a YouTube video demonstrating IDEAL Magnifier in action.

IDEAL Item Identifier including Talking Barcode Maker

Thanks to Android's Intents system and its MediaRecorder and Text-To-Speech (TTS) APIs, we were able to produce an open source app which turns a user's phone into a talking barcode reader. Talking barcode readers enable blind and visually impaired users to scan the barcode of a product and hear what that item is. In addition, many of the higher end models offer the ability to let users create their own barcodes which they can stick onto items. Unfortunately, like video magnifiers, these devices have traditionally been quite expensive.

We solved the problem of detecting and reading barcodes without spending any development time by simply delegating this task to the ZXing Barcode Scanner. Once we get the UPC code of a product, we do a lookup of that UPC and speak the name of that product.

For custom labels, we record what the user is saying and save it to a file locally. We then use the Send Intent to enable users to email themselves a QR code which contains the automatically generated filename of that recording so that we play back that file when users scan this code. Users can print out the QR code on any sticky label, and voila, their very own custom label. Here's a video demonstrating IDEAL Item ID in action.

Vista Center

The Vista Center is a Palo Alto, California-based organization that helps the blind and visually impaired. We volunteered to create an Android app for them to help users access their educational materials which include topics such as how to use ticket machines and how to set up Android phones for accessibility.

This turned out to be a much easier project than expected, thanks to Android's accessibility features and the strong open source culture that is part of the Android platform's DNA. Specifically, we were able to take advantage of the Google Accessibility Team's I/O challenge which encouraged contestants to open source their submissions. We modified the ccTube app so that it always does a search on startup for videos from the Vista Center, and since Android has accessibility built right into the platform, we didn't need to do anything special to make it work with the TalkBack screen reader.

(Hat tip to Google's Charles L. Chen for helping us connect with the Vista Center and pointing us to Google I/O's Accessibility Challenge, and to Casey Burkhardt, who wrote ccTube and open sourced his code.)

Android is a tremendous platform for building tools that empower people. We're very excited by the fast pace of Android evolution and can't wait to see what the next iteration of this wonderful platform will have to offer.


Come see Apps4Android in the Developer Sandbox at Google I/O on May 10-11.

Steve Jacobs’ greatest passion is to enhance the independence, quality of life, education and mobile communications experiences for tens of millions of consumers with disabilities, senior citizens (like Steve), people who never learned to read, and everyone else.

Posted by Scott Knaster, Editor

Moving accessibility forward on Android

By Eduard Sánchez of Code Factory

This post is part of Who's at Google I/O, a series of guest blog posts written by developers who are appearing in the Developer Sandbox at Google I/O.


For the last 8 years we at Code Factory have been making software that helps the blind and the visually impaired access their mobile phones. We’ve created this software for several different platforms. Last year we decided it was time to start doing something for the Android platform, due to its growing popularity and variety of devices.

From our past experience, developing a screen reader for a new platform required a lot of work, hacks, and investigation. Almost none of the previous platforms we supported implemented any sort of Accessibility API that we could use. Android, we thought, would be no exception to this rule. We were very wrong.

Starting at version 1.6, the Android operating system comes with a built-in Accessibility API that makes our application a lot easier to develop. All you do is create a service, which implements the AccessibilityService interface, declare it in your manifest and voilà! The system will start sending events, such as button presses, list navigation, focus changes, etc. to your service. You then convert this information to voice using a Text-to-Speech engine, and you have a screen reader.

The Accessibility API is not yet as complete as what you can find on a desktop PC, but it's good enough to provide the users with basic user interface navigation, and we have no doubt that, as the Android platform evolves, so will the built-in Accessibility API.

We also wanted our application to go beyond a screen reader and provide an intuitive, easy-to-use UI that allowed the blind and visually impaired access to most of the phone's functionality, such as messaging, web browsing, contact management, and so on.

We were pleased to see that we could do this Android. The existing set of UI controls, such as buttons and lists, can be overridden in order to provide custom functionality, such as speaking the text of the control. This made it possible for us to keep the user interface of our application consistent with Android, while at the same time providing the speech feedback that our users require.

By intercepting touch events within our application and using the gesture detectors that Android provides to developers, we were also able to make the touch screen accessible to our users, so they can use gestures like swipes to move through items of lists, or double-taps to activate items.

We really like how much we can accomplish with Android with so little code. Want to let a blind person create an SMS or email using voice? Simply use the SpeechRecognizer class. Want blind users who are walking on the street to know their exact location? Just use the LocationManager and Geocoder classes to give their exact street name and number.

Android lets us do a lot in a very efficient way. It wraps a whole bunch of cool technology into well-defined classes and interfaces. And if at any given time you need to know how something works behind the scenes, you just take a look at the source code, which is freely available to everyone.

We just can't wait to do more on this platform.


Come see Code Factory in the Developer Sandbox at Google I/O on May 10-11.

A pioneer in assistive technology for mobile phones, Eduard Sánchez is the brain behind all Code Factory software applications. His greatest satisfaction is to use his passion for programming to make a positive difference in the lives of people with disabilities.

Posted by Scott Knaster, Editor

Interactive Transcripts and Automatic Captions for Developer Videos

Did you notice the new Interactive Transcript feature that lets you scan quickly through the full text of any owner-captioned video that you’re watching on YouTube? For videos from I/O, that means you can quickly scan through a 60 minute talk to find just the part of the talk that you need to see. Or use your browser search with the Interactive Transcript to find a mention of an API call, and then click on a word in the transcript to jump straight to that part of the video.

Because developers don’t all speak English (and because some developers speak really fast when presenting) we caption every video that we post to http://www.youtube.com/googledevelopers. Most of the year, that’s a pretty easy thing to keep up with. But last year, when we posted all the videos for Google I/O 2009, it took us months to get everything done.

This year, we captioned everything within 24 hours or less of the videos going live. I’m excited about that, because it wouldn’t have been possible without the new auto-caption and auto-timing features in YouTube. We also did something a little nerdy -- we used four different methods of captioning.

If you use YouTube to share talks from your own developer events, you might find this summary useful.

The two fastest options for producing and cleaning up our captions used auto-timing. We uploaded a transcript and had YouTube’s speech recognition calculate the timecodes for us.

The two auto-timing methods were:

• CART live real-time transcript + auto-timing
  Because we had professional real-time transcriptionists at I/O, we could instantly caption anything that had a live session transcript. That’s how we got the keynotes captioned on the day of the event. We also used this method for the android talks.

• Professional transcription + auto-timing
  This was less expensive than CART, and faster than full captions with timecodes, but slower than real-time transcription because we had to get video files to the transcribers.

Although these methods were fastest, auto-timing turned out not to be perfect for all videos. When mic quality varied, or we had too many speaker changes in a short period of time (e.g panel discussions or fireside chats), the timing sometimes slipped out of sync. You can still use the Interactive Transcript to see what was said, but it’s not ideal.

The two slower methods that we used were:

• Pure 'traditional' captioning
  This is what we did last year for Google I/O 2009 videos. It’s slower, and more expensive, because you have to transcribe and set all the timecodes correctly. But the end result is 100% accurately timed. We did this to fix a video that the auto-timing had a lot of difficulty with.

• Speech recognition (auto-captions) with human cleanup and editing
  This gave us perfect timecodes, just like traditional captions, and took less time than traditional captioning. It took slightly longer than auto-timing alone because we had to download the machine-generated auto-captions from YouTube to do the edits.
  
  Automatic captions are fantastic if you don't have time or budget to put any work into your captioning. But for I/O, we wanted our captions to be perfect on technical terms, so fully automatic captions weren't the best fit.

Not all of these methods are equal in terms of quality, but it’s interesting to compare. To see which method was used on a video, look for the track name in the caption menu. To compare owner-uploaded captions with pure machine-generated auto-captions, you can always choose ‘Transcribe Audio’ from the caption menu for our videos.

If you’d like to help improve caption quality, please watch a video and fill out our caption survey to tell us what you think of these captions! We know some of them are going to be a little off -- if you report issues, we’ll fix them.

By Naomi Bilodeau, Google Developer Programs

New! Caption files for Google Developer Videos

by Naomi Bilodeau, Google Code Team

Last year, YouTube launched a Captions and Subtitles feature. In addition to launching a new playlist for captioned Developer Videos, we're also kicking off an Open Source project to host caption files that anyone can reuse under the terms of the Creative Commons 3.0 BY license.

We're hoping that developers will come up with interesting uses for caption data, once it's in the public domain. You can use transcripts as a corpus for training speech-to-text algorithms or testing applications that read and write caption files. Or, combine timepoint data with YouTube's URL support to jump to a specific point in a video.

Caption tracks make YouTube videos accessible to a wider audience. For example, try a search on [RESTful protocol YouTube] and you'll find search results from the captions on Joe Gregorio's recent talk.

While we're delighted that Kevin Marks' captioned English accent can be more easily understood by Americans, we've also translated the caption files and provided tracks in multiple languages for a few of our captioned videos. For all other videos, YouTube can perform Auto-Translate on caption text using Google Translate technology.



To learn more about YouTube caption file formats, take a look at the YouTube Help Center. If you're interested in contributing caption files for videos on Google channels, or making translations available, please consider joining the project.

We hope you'll find these additions useful. Happy reading!

Introducing AxsJAX -- Access-Enabling AJAX

Posted by Charles L. Chen and T.V Raman

As the developer behind Fire Vox I've always wanted to make AJAX web applications truly usable for the blind and visually impaired. The challenge is that these users have to deal with a much higher learning curve than sighted users. Instead of simply learning the controls for a web application, they have to also learn how to get their assistive technology of choice to go to the interesting parts of that application to find out what is currently there.

When I started as a Noogler, I was extraordinarily impressed with the tools that T.V. Raman had built into Emacspeak for efficiently performing specific tasks. An insight that I gained from watching him use Emacspeak is that the application should just say the right thing in response to user actions; users should not have to do an action in the application and then use their assistive technology to go hunting around the screen to figure out what happened.

In my first week at Google, I discovered Google Reader a highly optimized feed reader with very good keyboard support. For my starter project at Google, I decided to access-enable this application using W3C ARIA. Using Greasemonkey, I could inject JavaScript code to add the needed ARIA bits to make Google Reader say the right things at the right time.

Connecting The Dots

Based on the experience of access-enabling Reader, we have now refactored the code to come up with a common JavaScript framework for enhancing the accessibility of AJAX applications. This framework is called AxsJAX, and it was refined in the process of access-enabling Web Search.

We're now excited to open-source this framework since we believe that there is nothing Google-specific in the techniques we have implemented. We invite the Web developer community to help us collectively define a robust framework for rapid prototyping of accessibility enhancements to Web 2.0 applications.

The ability to rapidly prototype end-user interaction has led to an explosion in the number of AJAX applications; until now, visually impaired users have been left behind in this process. We hope that the AxsJAX framework encourages the Web community to bring the power of Web 2.0 development to solving the problem of accessing rich Web interaction in an eyes-free environment.

Google Developer Podcast Episode Nine: The status of accessibility on the Web

By Dion Almaer, Google Developer Programs

Using iTunes?

T.V. Raman is a Research Scientist at Google who knows a thing or two about accessibility. We took the opportunity to interview him, and Hubbell, his seeing-eye dog (who was nice and quiet).

We started out by asking the honest question that developers ask about accessibility: "What is in it for me?". T.V. discusses the practical issues, and what you should be doing with respect to accessibility, and how it is one piece of the usability picture.

We then delve into the problems of developing accessible websites, and solutions to some of the problems.

If you listen to the interview you will learn:

• How not to develop in a user-agent specific manner
• Fun issues with screen readers
• How audio CAPTCHA brings equality to the pain of CAPTCHA, and how people who can see use the audio ones
• How painful is the Web to view for a blind person
• Using the Google Web Transcoder (the other GWT!) to clean up pages
• How CSS hasn't been as leveraged as much as we would like
• How the increase in mobile and widget platforms has a side effect of accessible views
• How RIA applications deal with accessibility
• How T.V. has written custom clients for Google APIs
• What standards groups are doing in the accessibility space
• Dealing with Python, a language that cares about whitespace, as a blind man.
  

You can download the episode directly, or subscribe to the show (click here for iTunes one-click subscribe).

Also, check out an accessible web search for the visually impaired.