ShapeWriter (1999 - 2010)
The ShapeWriter research project was founded in 1999 at the IBM Almaden Research Center against the backdrop of 1. User computing was moving beyond the desktop but user interaction, particularly text entry, remains a key challenge to the future of mobile computing. 2. Writing systems are rarely scientifically researched or systematically designed but new forms of computing bring forth opportunities to invent and research novel writing methods. 3. The tension between ease of use for novice users and high efficiency for expert users, critically raised and studied in the research of marking menus, should be a central and general research question in Human-Computer Interaction. Text entry, being an intense interaction process, is a critical test bed for such research.
Through over 10 years of research, publication, software release and finally a start-up company, the ShapeWriter project pioneered the gesture keyboard paradigm of information input on touch screens.
Gesture Keyboard Concept and Theory
Tapping on individual letters on touch screens is a tedious and redundant process from an information theory point of view (over specification). Instead, the stroke that swipes through all letters of a word on the keyboard can be viewed as a geometric code of the word. Each word in a lexicon (a list of words) has its own code on a given keyboard layout. There are two extremes in drawing such a geometric code: visual tracing, and gesturing. Visual tracing from letter to letter on the keyboard is a visually guided, closed-loop, recognition-based, and relatively slow but easy action. Humans are good at remembering stroke patterns. In use, these stroke patterns can be remembered by the user (mostly subconsciously) so the shape writing processing gradually shifts from visual tracing toward the gesturing extreme: a memory recall based, open-loop and relatively fast and efficient action. Users will treat the entire stroke as a single chunk of information to produce. In this sense, shape writing is a new shorthand writing method, one word per stroke. Due to the Zipf's law effect, a small number of words are used disproportionally frequently and their stroke patterns get memorized rather quickly. Longer and less common words are typically made of common fragments whose shapes can also be quickly remembered.
KEY PROPERTIES & TECHNOLOGY
Pattern recognition: The trace of each word in a large lexicon (e.g. 50,000 words) forms a geometric pattern on a keyboard. Using computational intelligence ShapeWriter finds the closest word whose pattern matches the input stroke gesture on keyboard in real time (with imperceptible time delay). Due to the large amount of white space in a lexicon, an intended word can still be recognized although irrelevant letters between intended letters are crossed or even if some of the letters in a word are missed in the stroke gesture. ShapeWriter is therefore fundamentally error tolerant. The best way to do shape writing recognition is to take multiple sources of information (shape, location, path, context etc) and integrate them in a probabilistic framework.
Ease of error correction: alternative words that closely match the user gesture stroke are presented to the user so even if the user stroke is in fact closer to another word, the intended can still be easily selected.
Unified command and text input: the same mechanism for shape writing text is also used for issuing commands by Command Strokes. For example if the user swipes a stroke from a command key to c-o-p-y, ShapeWriter copies the selected text.
Dynamic and adaptive lexicon: ShapeWriter gives preference to the most common active vocabulary of 10,000 or so words in the 50,000 (or more) words lexicon. In fact only active words can come out by default. A passive word in the lexicon can only be presented as in a suggestion list. However, if the suggestion is selected, it will instantly become an active word. This quick expansion scheme means that ShapeWriter can rapidly adapt to an individual user's writing vocabulary. For a name or a word not in the lexicon, the user can tap it once and add it to the lexicon for shape writing next time.
Keyboard layout optimization: The layout of ShapeWriter's soft keyboard defines the ideal trace of each word. The familiar Qwerty layout is currently the default ShapeWriter keyboard but it is also possibly the worse possible layout. Shape writing tends to zig-zag left and right in a similar fashion for many words. In the long run, an optimized layout has great potential in further improving shape writing efficiency and experience. Learning shape writing on a new layout is much easier than learning typing on a new layout because of human sensitivity to geometric patterns.
This list also includes papers on soft keyboard and gesture interface related to ShapeWriter. Recommended readings on ShapeWriter are marked with ****.
*** Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Peng, S., Liu, L. Dunnigan, A., Shapewriter on the iPhone: from the laboratory to the real world. ACM CHI 2009 Conference on Human Factors in Computing Systems Extended Abstracts. pp. 2667-2670.
Lee, S., Zhai, S., The performance of touch screen soft buttons. Proc. of ACM CHI 2009 Conference on Human Factors in Computing Systems. pp. 309-318.
Appert, C., Zhai, S., Using strokes as command shortcuts: cognitive benefits and toolkit support. Proc. ACM CHI 2009 Conference on Human Factors in Computing Systems. pp. 2289-2298.
Zhai, S., On the ease and efficiency of human-computer interfaces. Proc ACM ETRA 2008: ACM Eye Tracking Research & Applications Symposium, pp 9-10 (keynote address abstract, slides)
Zhai, S. Kristensson, P.O, Interlaced QWERTY: accommodating ease of visual search and input flexibility in shape writing. Proc of CHI 2008: ACM Conference on Human Factors in Computing Systems, pp 593-596.
**** Per Ola Kristensson, Shumin Zhai. Improving word-recognizers using an interactive lexicon with active and passive words. Proc ACM Intelligent User Interfaces 2008, pp 353-356
**** Kristensson, P.O., Zhai, S. Command Strokes with and without Preview: Using Pen Gestures on Keyboard for Command Selection, Proc. CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.
Cao. X, Zhai, S. Modeling Human Performance of Pen Stroke Gestures, Proc. CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.
Cockburn, A. Kristensson, P.O., Alexander, J. Zhai, S. Hard Lessons: Effort-Inducing Interfaces Benefit Spatial Learning, Proc. CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.
Kristensson, P.O., Zhai, S. Learning Shape Writing by Game Playing (Interactivity Paper), CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.
Moran, T. P., Zhai, S. Beyond the Desktop Metaphor in Seven Dimensions, closing chapter in V. Kaptelinin and M. Czerwinski (Eds), Designing Integrated Digital Work Environments: Beyond the Desktop Metaphor.
** Kristensson, Per Ola, Discrete and Continuous Shape Writing for Text Entry and Control, Linköping University, Department of Computer and Information Science, PhD thesis, 200 pages, Supervisor: Zhai, Shumin, Opponent: Buxton, William, 2007. ISBN: 978-91-85831-77-7
**** Zhai, S., Kristensson, P.O., Introduction to Shape Writing, IBM Research Report RJ10393 (A0611-006), November1, 2006 (also as a Chapter 7 of by I. S. MacKenzie and K. Tanaka-Ishii (eds), Text Entry Systems: Mobility, Accessibility, Universality, Morgan Kaufmann Publishers, pp 139, 158)
*** Kristensson, P-O., Zhai, S. Relaxing Stylus Typing Precision by Geometric Pattern Matching. Proc. IUI 2005, ACM Conference on Intelligent User Interfaces, 2005, pp. 151 - 158, ACM Press.
Zhai, S., Kristensson, P-O, Smith, B.A., In search of effective text interfaces for off the desktop computing, Interacting with Computers, 17(3):229-250, 2005.
**** Kristensson, P.O., Zhai, S., SHARK2: A Large Vocabulary Shorthand Writing System for Pen-based Computers, Proc. ACM Symposium on User Interface Software and Technology (UIST 2004), Oct 24-27, Santa Fe, New Mexico, CHI Letters 6(2), pp. 43 - 52, ACM Press.
**** Zhai, S., Kristensson, P.O., Shorthand Writing on Stylus Keyboard, in Proceedings of CHI 2003, ACM Conference on Human Factors in Computing Systems, Fort Lauderdale, Florida, April 5-10, 2003. pp 97-104.
Zhai, S., Sue, A., Accot, J., Movement Model, Hits Distribution and Learning in Virtual Keyboarding, in Proc. of CHI'2002: ACM Conference on Human Factors in Computing Systems, Minneapolis, Minnesota, April 2002. pp 17-24.
Zhai, S., Smith. B. A. Alphabetically Biased Virtual Keyboards Are Easier to Use - Layout Does Matter, in Extended Abstracts of CHI 2001, ACM Conference on Human Factors in Computing Systems, Seattle, Washington, 31 March-5 April 2001 (Short Talk), p321-322.
Smith, B.A., Zhai, S. Optimized Virtual Keyboards with and without Alphabetical Ordering - A Novice User Study, in Proc. of INTERACT 2001: Eight IFIP Conference On Human-Computer Interaction, Tokyo, Japan, July 9-13, 2001. p92-99.
Zhai, S., Hunter, M., Smith, B.A., The Metropolis Keyboard -- An Exploration of Quantitative Techniques for Virtual Keyboard Design. In the Proceeding of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST 2000), November 5-8, 2000, San Diego, California. pp 119-128.
AWARDS & RECOGNITIONS
Hailed as "A revolutionary breakthrough" by James Kendrick's Top Ten Tech blog, upon the first public release of SHARK in 2004 from IBM Alphaworks
From 2003 to 2005, the project was reported by major news organizations including The New York Times, Financial Times, San Jose Mercury News, and the BBC.
Google Android Developer's Challenge prize in 2008
Top 11 iPhone Applications by Time.com
14 must-have iPhone apps by Examiner.com
Top 10 mobile applications by 2009 Razorfish Digital Outlook
Top 10 iPhone Apps for Girls by TotallyHer.com
Best iPhone Apps: Office and Personal Productivity by PC World
"IBM Research Accomplishment" recognition in 2010
Based on a technology licensing and service agreement with IBM, ShapeWriter Inc was founded in 2007 to further develop and commercialize ShapeWriter technology. The company's mission "is to make mobile phones, tablet computers, game pads, and all other touch-screen devices more useful - and user friendly - with software based upon shape writing technology". ShapeWriter Inc was acquired by the market leader in information input, Nuance Communications, in May 2010.
"ShapeWriter" and the ShapeWriter Logo are trademarks of ShapeWriter Inc.