One thing that the Dragon NaturallySpeaking speech engine could do better is hyphenation. I don’t mind so much when I say something that should be hyphenated and it’s not. I can always say the NaturallySpeaking command “hyphenate that” or the UC command “1-10 Hyphenate” after the fact if the NaturallySpeaking engine leaves out the hyphenation. I can also specify hyphenation when I want it, e.g. “on hyphen the hyphen fly” will type “on-the-fly”.

If I have something that’s not hyphenated and should be, it’s either a mistake or something I accidentally left out.

But if NaturallySpeaking puts in hyphenation where I don’t want it, there are two problems. First, there’s not an easy way to remove hyphenation after the fact — I have to select the phrase, then say it again in two phrases so it won’t be hyphenated, which is 3 steps. Second, there’s no way to specify no hyphenation.

If NaturallySpeaking over-hyphenates and I don’t notice, it looks like I’m consciously adding hyphens where they shouldn’t be. There’s nothing more annoying than having another entity introduce mistakes into your work.

Because the minuses of over-hyphenation are larger than the minuses of not hyphenating enough, when I see a phrase hyphenated when it’s not supposed to be I remove the hyphenated version from Natspeak Vocabulary so it won’t happen again.

For instance, I removed “follow-up”, which I often put as a stand-alone tag in my todo list. It’s a clunky workaround, but it’ll have to do until speech engines get better at analyzing hyphenation.

To remove a vocabulary word say “NatSpeak Vocabulary”, say the or phrase you want to delete, “Under d c” to delete and close the window, and “Enter” to confirm the change.

I think Nuance could mitigate this problem with a pair of in-line commands: “no-hyphen that” would remove hyphenation in the last phrase and “no-hyphen” would specify that something not be hyphenated, parallel to the “no-caps” command. I’m adding this to the Nuance wish list.

My stock answer is I prefer the 2003 drop-down menus to the 2007 ribbon. It’s funny, at the same time as Office made the switch from drop-down menus to the more Web-like ribbon, the Web application Google Documents made the opposite move — changing from a tab-based interface to drop-down menus. Out of the box, 2007 is less efficient — it takes up more screen space and requires more steps than 2003.

Having said that, the 2007 interface is also very configurable. You can put any drop-down menu or menu item on the Quick Access Toolbar that runs across the very top of the screen. And you can hide the ribbon. If you take the time to put the items you use most on the Quick Access Toolbar, you can make Office 2007 much more accessible.

For details on setting things up and using Microsoft Office 2007 with Utter Command, see UCExchange: UCandOffice2007 .

What’s your opinion on 2007 versus 2003? Reply here or let me know at info@ this website address.

It looks like what you see changes what you hear. Researchers from Haskins Laboratories and MIT have found that different facial expressions alter the sounds we hear.

This shows that the somatosensory system — the mix of senses and brain filtering that determines how you perceive your body — is involved when you process speech.

This doesn’t have a whole lot to do with speech commands except to show that it’s easy to underestimate the complexity, and subtlety, of our perception of spoken language.

Resources:

Somatosensory function in speech perception
www.pnas.org/cgi/doi/10.1073/pnas.0810063106

3

The way we control computer programs is fairly inefficient.

Keyboard shortcuts are underused in favor of using a mouse to click through menus. This is short-term practical — it takes less thought to browse through menus than to remember a keyboard shortcut. But it’s not very productive.

Look at the whole picture and you find good reasons to make the less productive choice. There are barriers to using keyboard shortcuts. Help and learning tools for keyboard shortcuts are scant at best. And inconsistencies across programs make the learning task larger.

So how do we improve things?

We can (continue to) encourage software makers to improve keyboard shortcut documentation and consistency. This is important, but it’s not going to change the world.

I think things would improve greatly given universal abilities to

1. name our own keyboard shortcuts — this currently exists in some but not all programs
2. share sets of shortcuts
3. see all shortcuts for a given application, and even compare shortcuts across applications

This would provide a good mental map of functionality — both of individual programs and across the landscape of suites.

It would make efficient functionality accessible across the board. It would enable individuals, organizations, departments or corporations to make applications more efficient and even standardize shortcuts across applications. The ability to share shortcuts would put a lot of brains on the problem and make the process efficient and evolutionary.

Given a map of all shortcuts, you could make things even better by allowing the user to mark the map — maybe using color labels.

Tools like this are the equivalent of a downhill groove for water– it would make it easy to be more efficient.

Keyboard shortcuts are least standard and most lacking in Internet applications. I’m thinking an ability like this could be built into or be an add-on to a browser.

And in addition to increasing productivity across the board, keyboard shortcuts are central to accessibility. The blind community relies on keyboard shortcuts. And speech commands often tap keyboard shortcuts — they’re often the hooks people use to write custom macros, and Utter Command allows you to speak and combine keys including keyboard shortcuts.

So who’s going to step up to the plate?

3

Computer commands of all kinds — speech, keyboard and mouse — are much easier to use when they’re consistent across programs.

At the base level, it’s important that common elements like drop-down menus act the same. You control drop-down menus without thinking — click on an element or use the Left, Right, Up, Down and Enter keys.

Consistent commands are the real-world equivalent of having the same gravity in every room, or keys turning the same way to unlock.

Web applications are looking more and more like standard computer programs, but sometimes the elements that look familiar don’t act the way we’re used to. Drop-down menus usually respond in a familiar way to the mouse, but often don’t respond to the Up, Down and Enter keys.

But perhaps things are getting better.

The first drop-down menus to show up on Google Docs didn’t respond to Left, Right, Up, Down and Enter. Then most of the folder-view drop-down menus were arrow key/Enter enabled, but not document menus. A few months ago document menus changed from looking tab-like to looking more menu like, but still didn’t respond to arrow keys and Enter. Then, sometime in the last few weeks, the Doc menus were arrow key/Enter enabled (the change didn’t show up on the update notice).

The keyboard shortcuts enable better speech navigation as well. I can say, for instance, “3 Down Enter” to choose an item in an open menu, “3 Down 2 Right Enter” to choose a color on the open color menu, or “7 Right Wait 3″ to take a three-second peak at each of the seven successive menus starting with the file menu open.

This is a great trend.

Now all we need is keyboard shortcuts to open the menus in the first place. We also need the same kind of control in all Web applications, including Google spreadsheets.

3

NatSpeak boilerplate Text and Graphics commands allow you to insert any text or graphics into a document using a single speech command. These commands can be very powerful — they’re good for adding text and graphics that you use often, such as your address or a set of directions.

The NatSpeak Vocabulary editor allows you to add words or phrases to your vocabulary that have different spoken and written forms. This allows you to make words like your email address easily pronounceable.

The key to using boilerplate and vocabulary commands is being able to remember them.

There are two ways to make these types of commands easy to remember:

1. Word them consistently

2. Make them easy to look up

I find the easiest way to remember boilerplate Text and Graphics commands is to simply say the first part of the text you’re inserting followed by “Full”. So “Redstart Full” prints the full name and address of Redstart Systems. If you have two different versions of the address, add a number. “Redstart Full 1″ prints the same address in a different format.

You can use the Utter Command Clipboard facility to make anything easy to look up. Once you name your Text and Graphics command say “Line Copy To” followed by the name of the UC Clipboard file and you’ve got it recorded. For example, to keep your boilerplate commands in “UC List 1″ say “Line Copy To List 1″.

Now any time you want to consult your list of commands say “List 1 File”. You can also print it out.

I also use the start-to-say method for vocabulary words that have different written and spoken forms. I’ve put my Redstart email address in as a vocabulary word with the spoken form “Kim at Red” and my Gmail address in as a vocabulary word with the spoken form “Kim at G Mail” (in address commands I use “Kim” whether or not the actual address is just Kim or something longer).

One caution in using vocabulary in this way — make sure commands are at least two words and make sure the two words are not a common phrase that you’d want to say as is. If you need to, use the “Full” method above to avoid this problem. Also make sure to save your user after adding vocabulary words.

If you wish, keep vocabulary words that have different written and spoken forms on the same list as your boilerplate commands.

The difference between boilerplate commands and written/spoken vocabulary words is a block of boilerplate is returned exactly as written, while vocabulary commands are treated like words, with appropriate spacing before and after them.

UC Commands Tip: say “NatSpeak” followed by the first one or two words in a NatSpeak dialog box title to call up that dialog box.

Commands for the dialog boxes mentioned above:

“NatSpeak My Commands” calls up the NatSpeak My Commands dialog box where you can write a boilerplate Text and Graphics macro

“NatSpeak Vocabulary” calls up the NatSpeak Vocabulary Editor dialog box

Whenever I talk to people who use speech commands to control a computer I encourage them to complain. Something that frequently comes up is it’s a drag having to say “page down” so much.

We’ve come up with several ways to diminish the drag:

1. Several screens at once

First, “Page” is a back-of the-mouth word, which is more difficult to say than words that only use sounds that originate in the front of the mouth. This isn’t a problem for commands you don’t use frequently, but looms large when you have to repeat something over and over again.

And when you say “Page Down”, you’re really moving by screen, not by page. This is fortunate, because “Screen” is easier to say than “page”.

Using Utter Command you can say “Page Down” and “Page Up” to hit the page up and page down keys, but you can also say “Screen Down” and “Screen Up”. And you can move multiple screens: “2 Screen Down”, “5 Screen Up”

2. Right to the point

You can also go to a given screen. “Screen 3″, for instance, jumps you right to the third screen of information in a document.

And in programs whose Find facilities recognize page numbers, including pdf’s, you can go right to a given page by saying, for instance, “Find Page 22″. You can try this out on a UC lesson document: “UC Lesson 1″.

3. Wait

It’s still tedious to say “Screen Down” every couple of seconds when you want to glance quickly at subsequent pages. Try this: “3 Screen Down Wait 5″. This moves down a screen, waits 5 seconds, moves down another screen, waits 5 seconds, then moves down another screen.

4. The right tool for the job

It’s also important to look at exactly why you’re going through a document screen by screen. Often you’re looking through pages for a certain section. In this case the screen-by-screen facility isn’t the right tool for the job, but you may be using it because usually it’s the best tool available.

If you’re looking through a document that has numbers, letters or symbols to differentiate sections you can use the UC Keywords facility go directly to any of these. To see what I mean say “Find 1 Period”, “Find 3 Period” in this document. Now picture a longer document with more and longer sections, and a section outline along these lines:

2. Speech Command

You could say, for example “Find 1 Period”, “Find 1 Point 1″, “Find 1 Point 2″ and “Find 2 period” to jump among these sections.

Using the UC Keyword list you can use any section organization scheme you want — numbers, letters, numbers and letters (1a., 1b….) or heading words themselves (“Find Introduction”, “Find Summary”). Sometimes I put tildas (~) at key points in a document so I can jump to those points (“Find Tilde”). I also use the word “PLACEHOLDER” this way (“Find Placeholder”).

You can also use “Wait” with keywords. I use this one to scan a document for placeholders: “Find Placeholder Wait 2 Repeat 5″.

Keyboard shortcuts are powerful tools for the speech interface because they work across all programs and they can be combined — you can say several keyboard shortcuts in one phrase to speed things up.

This is why we encourage all software makers to make all features available via keyboard shortcuts.

Google is experimenting with adding keyboard shortcuts to search results. Here are the experimental keyboard shortcuts:

And here’s how to speed things up further with Utter Command combinations:

To try these out

1. Go to the Google experimental page www.google.com/experimental/1
2. Under the Keyboard Shortcuts heading click “Join Experiment”
3. Go to regular Google search www.google.com2 or Advanced Google search www.google.com/advanced_search?hl=en3, type a query, then try the shortcuts on the results.

As long as you’re logged in you’ll be able to use these shortcuts in the regular and advanced Google search pages.

Note: the Join Experiment button uses cookies. If your browser is set to remove all cookies at the end of a session and you want to retain this setting add www.google.com to your exceptions list (Firefox: Tools/Options/Privacy/Exceptions; Internet Explorer: Tools/Options/Privacy/Sites).

The SpeechTEK speech conference has a lot to say about the state of the desktop speech interface. The exhibits in and 2006 and 2007 were largely about where all the speech interface action is these days — not on the desktop, but over the telephone with interactive voice response (IVR) systems.

I went to several sessions aimed at the voice user interface designers (Vuids) who construct telephone speech command interfaces (even though I’m something of an imposter as a desktop voice user interface designer — I guess Dvuid would be the appropriate term).

We’re dealing with a lot of the same issues, though often with different twists:

• Making sure people know what to say and stay oriented in the system
• Accommodating beginners and experienced users
• Making the process as fast and efficient as possible so people won’t hit the operator button or hang up (or not use the software — many people who buy desktop speech recognition software end up not using it)
• In both cases the communications relationship is between a person and machine

And we’re looking at similar answers:

• Making commands consistent
• Avoiding ambiguity
• Doing user testing
• Thinking about configuring information in a certain order to make it more memorable (good mental maps and appropriate training wheels)
• And above all avoiding the trap of thinking that people can just say anything because even if you truly could just say anything you still don’t know what to say

I’ve also been thinking about the differences between IVR and the desktop speech interface — these differences make the challenges more difficult or easier for each of the systems.

• Desktop users tend to follow a more predictable curve — they get more experienced or drop it, while for some IVR systems you have occasional users.
• People are more often forced to use IVR, while most people can easily avoid the desktop speech interface if they wish.
• The desktop is capable of both visual and audio feedback, while IVR systems tend to only have audio feedback. (Interestingly, even though most speech engines come with the ability to speak, desktop computer interfaces generally don’t use this feedback channel. We’ve had positive results in user testing of judicious use of audio feedback.)
• Both systems suffer from the widespread use of pseudo natural language. Natural language doesn’t really exist on either type of system and trying to fake natural language creates its own problems.