How to speak with your fingers

Now for the big question: Why do this? Why spend time and money developing a free program to let people type at 250 words per minute? What is steno good for?

I can think of a few groups that might benefit from free steno technology:

* People who don't use their voices to communicate, who would benefit from using a synthesized voice that can speak at conversational speeds.

* Writers, programmers, and other people who spend a lot of time working with text, who would like to set down their thoughts in a more fluent, natural way, rather than having to type them out laboriously, letter by letter.

* People who want to avoid or ameliorate the risk of repetitive motion disorders such as tendonitis or carpal tunnel syndrome.

* People interested in mobile/wearable computing and augmented reality.

* People who want to break onto the high score tables of online typing games, or who spent months teaching themselves the Dvorak keyboard layout for -- at most -- a 20% increase in speed, because they want to do everything as quickly and efficiently as possible.

* People who might be interested in court reporting, captioning, or CART (communication access realtime transcription) as a career, but who don't want to invest thousands of dollars in equipment, software, and training fees before knowing for sure whether it's the job for them.

I'll take the first group first and follow up on the other ones in subsequent blog posts.

Many people communicate without using their voices. Some are Deaf, some have cerebral palsy or other muscular disorders, some are on the autistic spectrum, and some have issues with their brain, tongue, jaw, larynx, or lungs which prevent them from producing comfortable or intelligible speech. Some of these people communicate using sign language. Others use assistive technology, including speech synthesizing software.

For example:

* Stephen Hawking

* Roger Ebert

* Schuyler Rummel-Hudson

* Alan Doherty

Not everyone who uses a speech synthesizer could benefit from steno. People with paralysis, like Stephen Hawking, don't have full control over their fingers, so they input text using rocker switches, sip-and-puff devices, or eye-gaze cameras. Schuyler Rummel-Hudson, who's 10, currently uses a small computer with pictographic symbols that stand in for common words and phrases. As she gets older, she might choose to switch to an alphabet-based system, but pictographs are sometimes more useful for people who have difficulty with fine motor control.

Steno would almost certainly be a great help to people like Roger Ebert or Alan Doherty, though. They don't have lower jaws, but their fingers work just fine. Currently they're forced to type everything they want to say on an ordinary keyboard, either letter by letter or using predictive text systems, which require around three to four keystrokes per word. The very best qwerty typist can get up to around 130 words per minute, but normal conversational speed is usually around 180, and often goes as high as 260 words per minute. People who rely on qwerty keyboards to communicate face the choice either of playing back pre-written sentences or requiring long pauses between each sentence. Either way disrupts the natural rhythm of conversation.

Steno, by design, can be written as quickly as English can be spoken. It also allows for greater fluency of thought (which I'll touch on more in the segment about writers and programmers), because it works syllabically rather than letter by letter. For instance, I'll take a random sentence:

"Whether or not the application is completed depends on your full cooperation."

That's 78 keystrokes on a qwerty keyboard. On a steno keyboard? Twelve strokes in all, making it over six times more efficient.

Has anyone used steno for accessibility before? Well, obviously it's been used for nearly two decades by CART providers like me on behalf of Deaf and hard of hearing people, but none of my research has turned up any accounts of disabled people using steno on their own behalf to communicate. (If you do know of anyone, though, please contact me! I'd love to hear about it.) The reason is not too hard to guess: Steno is ridiculously expensive, and it's got a relatively steep learning curve. Steno software (which costs around $4,000) is designed primarily for court reporters, and is not completely compatible with text-to-speech applications. Steno machines ($2,000 to $5,000) are -- with a few exceptions -- heavy, bulky, and anachronistic-looking.

Virtually everyone who learns steno these days does it because they intend to make a career out of it. The cost of the technology is prohibitive to dabblers, hobbyists, and people who don't have the time or inclination to undergo intensive court reporter training. Plover is an attempt to eliminate the $4,000 software cost. I'm currently looking into low-cost solutions for steno hardware. Once those two barriers are removed, I think the training will largely take care of itself. It took me a year and a half of intensive practice to get from 0 to 225 words per minute, but I was writing at 100 words per minute after only a few months, and that was true of nearly all my classmates as well. Steno is so vastly more efficient than qwerty that even a beginning stenographer can outstrip the best qwerty typist relatively quickly.

People who use assistive technology are uniquely suited for the task. They're often at their computers for many hours a day and have a high incentive to learn everything they can about their equipment. It's almost a truism that people with disabilities are usually the earliest adopters and most dedicated power users of almost any new technology. Voice-to-text software, which has proven invaluable to people who are unable to type, requires as much if not more training time than steno, as I mention in my post on Voice Writing.

Steno can provide the same benefits in the opposite direction. In addition to people like Mr. Ebert and Mr. Doherty, who can't speak but are able to hear, steno technology could do a lot for my own clients, who are primarily Deaf and hard of hearing. Without having to hire a CART provider, people who are hard of hearing and don't know sign language can speak to each other using steno as quickly as thought, with no potential for misunderstanding. People who are Deaf and do use sign language rather than spoken English can communicate with Hearing English speakers by writing what they want to say in steno and using the voice synthesizer to speak for them, rather than having to rely on hand-written notes when interpreters aren't available.

Even people who don't have any problems with hearing or speaking might start to use steno, as a way to communicate in noisy clubs or libraries, or as a high-speed substitute for texting. I'll write more about that in my wearable computing post, but the bottom line is this: The more people communicate using steno, the more universally accessible our society will be.

Qwerty is a venerable and popular input system, but at its best it's only a third as efficient as human speech. Steno is every bit as efficient as human speech, and it's been forced into undeserved obscurity by its high entry cost and inaccessible design. Plover is an attempt to fix that. I'll keep you posted.

The Sidewinder X4 is here

It arrived today. This evening I cut up a couple of my old leather steno keypads, and laid out a steno machine on its keys.

This is a terrible webcam shot (I'll try to get a proper one with my partner's digital camera later), but you can sort of see the shape of the thing if you look for the non-illuminated black squares:



And here it is with the equivalent steno letters superimposed on the image by means of Microsoft Paint. Yeah, we're pretty high tech up in here.



I've tried it out, and the touch is actually not that bad. Some of the twistier steno outlines are a little awkward because the keys are quite a bit closer together than on a regular machine, but the anti-ghosting is working like a dream. I wrote the phrase "So is this working?" in steno six times and got this:

avarwdvlru
vaarwdlvru
avawrldvru
avarwldvru
avarwdvlru
avarwvdlru

AV or VA is "so" (SO)
A is "is" (S-)
WR or RW is "this" (TH-)
DVL or DLV or LDV or VDL is "working" (WOG)
RU or UR is "?" (HF)

The order doesn't matter in steno, so this amount of key-recognition consistency should really be all we need to make a relatively functional machine. I'm very excited. My Python tutor can't meet next week, but the week after that we might try to get the Sendkeys issue resolved, or we might backtrack a bit and see if we can get this $60 steno machine (plus $10 worth of keypads) working with our dumb terminal version of Plover. How amazing would that be, huh?

Figuring it out

Well, my tutor and I didn't get any code written today; the lesson was taken up with trying to make SendKeys work, with next to no success. The problem is that SendKeys was written for Python 2.1 and was last updated in 2003. We've been writing Plover in Python 3.1, so it doesn't work. Our options are:

* Figuring out a way of updating SendKeys so it's compatible with Python 3.
* Revising Plover so that it's compatible with SendKeys as it is (might be tricky because pyserial requires Python 2.3 or later; plus my heart bucks against the idea of regressing backwards into the unsupported Pythonic past.)
* Finding an alternative to SendKeys -- possibly using the script found in the last comment of this?

On the hardware side of things, I just ordered a Sidewinder X4, which I mentioned in an earlier post. If it's really true that multiple keys of the keyboard will be recognized as chords, maybe I can train my fingers to ignore the slight misalignment between rows and learn how to play the qwerty keyboard like a steno machine. That is:

QWER   TY   UIOP[
ASDF   GH   JKL;'
    CV   NM

would map to

STPH   **   FPLTD
SKWR   **   RBGSZ
    AO   EU

I tried a little bit experimentally on my laptop's keyboard, just mashing keys into a Vim window ("just mashing keys into a Vim window" came out as "mapfdsxkejrlscn;wxoianfmpdimn", but I think some of the letters weren't recognized, since my Lenovo's keyboard doesn't have any anti-ghosting abilities), and it feels weird, but I think it might be possible with practice. The main barrier I'm worried about is that of determining spaces between strokes. Steno machines register keypresses once all of the keys have been lifted; qwerty keyboards register keypresses as soon as each one is pushed down. Will it be possible to put in some sort of software hack that registers each stroke as finished every time the keyboard realizes that none of its keys are being pressed?

Wacom + Keypads = Steno?

Could this be an answer to our steno hardware problem? It talks about unlimited multitouch, up to 20 fingers or more. If you program in a steno keyboard layout and stick some keypads on there for ergonomic feedback, could that serve as a poor man's Lightspeed? The question is whether it'll actually be cheaper. Low-end drawing tablets have gotten quite reasonable in price; I saw one at Staples the other day that boasted multitouch (meaning up to three inputs at a time) and went for only $50. But this looks like a flagship model, and I know some of the top-of-the-line drawing tablets go for several thousand. Still, there's a lot bigger market out there for those than there currently is for steno machines, so if the price comes down in a year or two we might have our solution.

Numbers are working. Now what?

I just had a great Python session with my tutor, and we've gone about as far as we can go with our current dumb terminal version of Plover. Everything is translating properly, including numbers. The new version of the dictionary contains all the entries I use with my proprietary software. It's all very exciting.

But I'm still the only person using Plover. It's very useful when I'm outputting my CART display to one computer using my proprietary software and echoing it on another computer using Plover. That's a pretty specific activity, and if there's anyone besides me who thinks it would come in handy for them, they haven't said anything to me about it. No, in order for this thing to do what it needs to do, it has to output to the operating system, just like any ordinary qwerty keyboard does when you plug it in and start typing.

My first volunteer beta tester uses Fedora, so whatever I use to rig the output has to work on Linux. I'm on XP, so we need a Windows version too. I wish I could use the same code for both -- and if anyone reading this has an idea of how to do that, please let me know -- but from the brief research my Python tutor and I did before the end of today's session, it looks like we're going to have to use something like SendKeys for the Windows version and... I don't know. Whatever GOK uses in Linux? This is all very much uncharted territory for me, so I'd love advice from anyone who knows how to get input from a Python program and use it to emulate keyboard output. If you've got any ideas, comment here or email me: plover@stenoknight.com.

Plover is getting uglier...

But it's all to the good! In the first version of the dictionary file I uploaded, I just took out all the entries that weren't plain text, so Python wouldn't blarf on them. This time I've actually gone through and changed the metacharacters to typographic symbols like ^ (caret); this makes the output of Plover look much uglier than before, but it means I now have my fingerspelling alphabets back, and as soon as we pipe this stuff out to external programs, we'll be able to use those characters to tell Plover when to capitalize a word, when not to insert spaces between translations, and all that good stuff. So hold your nose and have a screenshot!

Why Plover?

I first came up with the idea of learning how to program by writing open source steno software more than three years ago. I've been geeky all my life. I've been an internet addict since I first discovered Gopher and Usenet in '93. As a teenager, I spent my time reading The Jargon File, watching Sneakers, and calling up every BBS in my area code repeatedly and obsessively. I steeped my brain in old-school cyberpunk and thought somehow that coding knowledge would come to me automatically, by osmosis. Never happened. Both my brothers and many of my friends are programmers, but somehow I never wound up actually sitting my butt down and teaching myself how to code. Now I'm pushing 30, making a good living at something I love, but I feel my brain getting slower and less plastic every year, and so at long last I'm taking steps to fix the mistake I made in my youth. I'm learning Python, one hour a week, and in the process I'm making something that might do some good for a fair number of people.

I'm planning a post that'll talk about the various things I think Plover might be good for, but for now I just want to talk briefly about how the project got its name. I remember walking around Fort Tryon Park, trying to think up something snappy, easy on the ears, not too flashy, that actually had something to do with stenography. At first I was fond of Stentor, but it turned out to be taken already. I tried anagrams of "stenotype" and "stenographer" but got only "Testy Peon" and "Nasty Gopher", neither of which inspired me. I thought I'd go to the great natural philosopher Nicolas Steno, but when I proposed naming my program after his Latin name, Stenonis, my better half quite rightly laughed herself senseless.

Finally I decided that, since none of the obvious "This is a program about steno!" names seemed to work, I'd come up with a name wasn't necessarily related to steno, but which could be used to show how insanely cool steno actually was, once you figured out how it worked. I wanted to show the powerful, flexible ambiguity of the keyboard, the way a single chord could translate to "slung" or "shrunk", "apple" or "amp", "squarer" or "sierra", "castle" or "cavil". I wanted to show how a single steno stroke could represent two syllables and half a dozen letters. I also wanted it to sound distinctive and useful, a word that was familiar enough not to sound alien, but not so common that it was mundane.

Finally I hit on it: PHROFR. In most steno dictionaries, the steno outline "PHROFR" is defined as "moreover". PH- is pronounced as "M", and the "R-OFR" bit sounds pretty much the way it looks; put them together and you get mrovr, which is easy enough to parse. But because HR- can also be pronounced as "L", PHROFR is also the way you stroke the word "plover", a charming little wading bird of the subfamily Charadriinae. Court reporters use the word "moreover" a lot more than normal people do, because it's used so often in legal contexts. Since I intended to write the first steno program that almost entirely neglected the very specific needs of court reporters (sorry, guys), I thought that using the zoological rather than the legal reading of the stroke would not be out of place.

Plover is also one of those words that's not pronounced the way it's spelled. It's PLUH-VER rather than PLOE-VER, and the idea of using a non-phonetic word for a phonetically based system amused me. Finally, there's the whole obnoxious Web 2.0 trend of naming programs after random verbs and then adding an -R to the end without an intervening E, and it drives me mad. I figured Plover would be my way of fighting back; it sounded like a helpful utility to use whenever you needed to get some serious plovving done (whatever that might mean) but it was a proper English word, with a proper -ER ending instead of the ubiquitous tacked-on -R.

So... Plover. One stroke, two syllables. Counter-intuitive pronunciation to give the in-crowd a feeling of leetness. Cuddly little bird logo just waiting to be drawn. It might not be hard-bitten and foreboding like what I would have chosen back in the '90s, when I dreamed of being a teenaged gargoyle; but now that I'm grown up and boring, I think it'll do quite nicely.

Steno machines on the cheap

I bet you that there's plenty of people who are interested in tripling their typing speed but don't intend to make a career out of it. Steno could be of great benefit to them, but steno machines cost big bucks. Right now you can get ugly, bulky, ancient student machines for about $700 at the cheapest, and they have none of the ergonomic benefits of a machine like a Gemini ($1,000) or a Revolution Grand like I've got ($2,400), or even a Lightspeed ($2,000). Incidentally, did you know that the technology used in the Apple iTouch was purchased from a company that had been using it to make low-cost steno machines? Apple put a stop to that, of course, and we've been waiting for an affordable touch-based machine ever since. Story here and here.

I have high hopes for the MSI computer I posted about below, but the video showed only two channels of multitouch. That might be the case only for the prototype; if it's that way in the final version as well, we'll just have to keep on waiting. Then, of course, there's the question of how the haptics will work even if the multitouch is fully functional. Will they be laggy? Insufficiently well delineated? So well delineated it's impossible to touch two keys with one finger? Who knows? Maybe a keypad overlay system, similar to the LightSpeed's, can work where haptics don't; it's something to consider.

But until our dual-screen fully multitouch magic bullet comes along, what solution is there for people who don't want to drop a minimum of a grand on some bit of tech that they're not sure they'll ever use? There's a cheap ($60) qwerty keyboard from Microsoft that can supposedly recognize up to 26 keypresses at once. How possible would it be to strip the keys off and use the key-recognition circuitry with a new steno-based layout? I know very little about hardware hacking, but my Python tutor has done a fair bit of it, so I intend to bring this up to him once we're a bit further along in Plover's development. My instinct is that it will probably be possible to rig up something that works but that it'll be trickier to come up with something that's both comfortable and reliable. I'm holding out hope for the dual-screen solution because that makes it as easy as pressing a key to switch between qwerty and steno layouts, which seems pedagogically ideal. But if we could really rig up working steno machines out of $60 keyboards plus some minor parts and labor customization -- maybe even make it into a kit for hobbyists to assemble themselves -- it might give people a low-risk incentive to pick one up and play around with it.

Getting there

It's been a very productive few days. We've made it so that any number of asterisk keys are translated as only one asterisk; same thing for initial S- keys. We've made it so that an asterisk pressed by itself will delete the last stroke from the buffer. Finally, I changed stenoworking.py (the version I've been using as a dumb terminal while providing CART for a client using two computers with my Revolution Grand's dual Bluetooth output streams) so that it now displays the entirety of the buffer with each stroke, rather than just the last stroke in the buffer. It might look busier, but it's actually quite useful, because now when I delete misstrokes on the client's display (which is running Eclipse, the $4,000 proprietary steno software) I can confirm that I haven't deleted too few or too many. Here, have a screenshot!