ShapeWriter Research Project Home Page

[ Background | Concept and Theory | Videos | Photos | Key Attributes | Publications & People | Press | History | User Reviews | Awards and Recognition | ShapeWriter Inc.]

Background

The ShapeWriter research project was founded in 1999 at the IBM Almaden Research Center against the backdrop of 1. User computing was moving beyond the desktop but user interaction, particularly text entry, remains a key challenge to the future of mobile computing. 2. Writing systems are rarely scientifically researched or systematically designed but new forms of computing bring forth opportunities to invent and research novel writing methods. 3. The tension between ease of use for novice users and high efficiency for expert users, critically raised and studied in the research of marking menus, should be a central and general research question in Human-Computer Interaction. Text entry, being an intense interaction process, is a critical test bed for such research.

Through over 10 years of research, publication, software release and finally a start-up company, the ShapeWriter project pioneered the gesture keyboard paradigm of information input on touch screens.

Gesture Keyboard Concept and Theory

Tapping on individual letters on touch screens is a tedious and redundant process from an information theory point of view (over specification). Instead, the stroke that swipes through all letters of a word on the keyboard can be viewed as a geometric code of the word. Each word in a lexicon (a list of words) has its own code on a given keyboard layout. There are two extremes in drawing such a geometric code: visual tracing, and gesturing. Visual tracing from letter to letter on the keyboard is a visually guided, closed-loop, recognition-based, and relatively slow but easy action. Humans are good at remembering stroke patterns. In use, these stroke patterns can be remembered by the user (mostly subconsciously) so the shape writing processing gradually shifts from visual tracing toward the gesturing extreme: a memory recall based, open-loop and relatively fast and efficient action. Users will treat the entire stroke as a single chunk of information to produce. In this sense, shape writing is a new shorthand writing method, one word per stroke. Due to the Zipf's law effect, a small number of words are used disproportionally frequently and their stroke patterns get memorized rather quickly. Longer and less common words are typically made of common fragments whose shapes can also be quickly remembered.

Photo Gallery

Videos

Google Tech Talk on ShapeWriter 2009

ShapeWriter Android 2009 Video Tutorial

ShapeWriter iPhone 2008

ShapeWriter WritingPad 2008 (Winner of ADC I)

ShapeWriter Introduction 2007

SHARK 2004 Video. Click here for a better resolution version

Key Attributes and Technologies

Intelligent pattern recognition: The trace of each word in a large lexicon (e.g. 50,000 words) forms a geometric pattern on a keyboard. Using computational intelligence ShapeWriter finds the closest word whose pattern matchs the input stroke gesture on keyboard in real time (with imperceptible time delay). Due to the large amount of white space in a lexicon, an intended word can still be recognized although irrelevant letters between intended letters are crossed or even if some of the letters in a word are missed in the stroke gesture. ShapeWriter is therefore fundamentally error tolerant. The best way to do shape writing recognition is to take multiple sources of information (shape, location, path, context etc) and integrate them in a probabilistic framework.

Ease of error correction: alternative words that closely matche the user gesture stroke are presented to the user so even if the user stroke is in fact closer to another word, the intended can still be easily selected.

Unified command and text input: the same mechanism for shape writing text is also used for issuing commands by Command Strokes. For example if the user swipes a stroke from a command key to c-o-p-y, ShapeWriter copies the selected text.

Dynamic and adaptive lexicon: ShapeWriter gives preference to the most common active vocabulary of 10,000 or so words in the 50,000 (or more) words lexicon. In fact only active words can come out by default. A passive word in the lexicon can only be presented as in a suggestion list. However if the suggestion is selected, it will instantly become an active word. This quick expansion scheme means that ShapeWriter can rapidly adapt to a individual user's writing vocabulary. For a name or a word not in the lexicon, the user can tap it once and add it to the lexicon for shape writing next time.

Keyboard layout optimization: The layout of ShapeWriter's soft keyboard defines the ideal trace of each word. The familiar Qwerty layout is currently the default ShapeWriter keyboard but it is also possibly the worse possible layout. Shape writing tends to zig-zag left and right in a similar fashion for many words. In the long run, an optimized layout has great potential in further improving shape writing efficiency and experience. Learning shape writing on a new layout is much easier than learning typing on a new layout because of human sensitivity to geometric patterns.

Authors & Publications

This list also includes papers on soft keyboard and gesture interface related to ShapeWriter. Recommended readings on ShapeWriter are marked with ****.

2009

o ** Zhai, S., Kristensson, P.O., Gong, P., Greiner, M., Peng, S., Liu, L. Dunnigan, A., Shapewriter on the iPhone: from the laboratory to the real world.ACM CHI 2009 Conference on Human Factors in Computing Systems Extended Abstracts. pp. 2667-2670.

o Lee, S., Zhai, S., The performance of touch screen soft buttons. Proc. of ACM CHI 2009 Conference on Human Factors in Computing Systems. pp. 309-318.

o Appert, C., Zhai, S., Using strokes as command shortcuts: cognitive benefits and toolkit supportProc. ACM CHI 2009 Conference on Human Factors in Computing Systems. pp. 2289-2298.

2008

o Zhai, S., On the ease and efficiency of human-computer interfaces. Proc ACM ETRA 2008: ACM Eye Tracking Research & Applications Symposium, pp 9-10 (keynote address abstract, slides)

o Zhai, S. Kristensson, P.O, Interlaced QWERTY: accommodating ease of visual search and input flexibility in shape writing. Proc of CHI 2008: ACM Conference on Human Factors in Computing Systems, pp 593-596.

o ** Per Ola Kristensson, Shumin Zhai: Improving word-recognizers using an interactive lexicon with active and passive words. Proc ACM Intelligent User Interfaces 2008, pp 353-356

2007

o **** Kristensson, P.O., Zhai, S. Command Strokes with and without Preview: Using Pen Gestures on Keyboard for Command Selection,  Proc. CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.

o Cao. X, Zhai, S. Modeling Human Performance of Pen Stroke Gestures, Proc. CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.

o Cockburn, A. Kristensson, P.O., Alexander, J. Zhai, S Hard Lessons: Effort-Inducing Interfaces Benefit Spatial Learning, Proc. CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.

o Kristensson, P.O., Zhai, S. Learning Shape Writing by Game Playing (Interactivity Paper), CHI 2007: ACM Conference on Human Factors in Computing Systems, 28 April - 3 May, 2007, San Jose, California.

o Moran, T. P., Zhai, S. Beyond the Desktop Metaphor in Seven Dimensions, closing chapter in V. Kaptelinin and M. Czerwinski (Eds), Designing Integrated Digital Work Environments: Beyond the Desktop Metaphor.

o ** Kristensson, Per Ola, Discrete and Continuous Shape Writing for Text Entry and Control, Linkoping University, Department of Computer and Information Science, PhD thesis, 200 pages, Supervisor: Zhai, Shumin,  Opponent: Buxton, William, 2007.  ISBN: 978-91-85831-77-7

2006

o **** Zhai, S., Kristensson, P.O., Introduction to Shape Writing, IBM Research Report RJ10393 (A0611-006), November1, 2006 (also as a Chapter 7 of by I. S. MacKenzie and K. Tanaka-Ishii (eds), Text Entry Systems: Mobility, Accessibility, Universality, Morgan Kaufmann Publishers, pp 139, 158)

2005

o *** Kristensson, P-O., Zhai, S. Relaxing Stylus Typing Precision by Geometric Pattern Matching. Proc. IUI 2005, ACM Conference on Intelligent User Interfaces, 2005, pp. 151 - 158, ACM Press.

o Zhai, S., Kristensson, P-O, Smith, B.A., In search of effective text interfaces for off the desktop computing, Interacting with Computers, 17(3):229-250, 2005.

2004

o **** Kristensson, P.O., Zhai, S., SHARK2: A Large Vocabulary Shorthand Writing System for Pen-based Computers, Proc. ACM Symposium on User Interface Software and Technology (UIST 2004), Oct 24-27, Santa Fe, New Mexico, CHI Letters 6(2), pp. 43 - 52, ACM Press.

2003

o **** Zhai, S., Kristensson, P.O., Shorthand Writing on Stylus Keyboard, in Proceedings of CHI 2003, ACM Conference on Human Factors in Computing Systems, Fort Lauderdale, Florida, April 5-10, 2003. pp 97-104.

2002

o Zhai, S., Sue, A., Accot, J., Movement Model, Hits Distribution and Learning in Virtual Keyboarding, in Proc. of CHI'2002: ACM Conference on Human Factors in Computing Systems, Minneapolis, Minnesota, April 2002.  pp 17-24.

o Zhai, S., Hunter, M., Smith, B.A., Performance Optimization of Virtual Keyboards,  Human-Computer Interaction. Vol. 17, No2&3. 2002.  pp 229-269.

2001

o Zhai, S., Smith. B. A. Alphabetically Biased Virtual Keyboards Are Easier to Use - Layout Does Matter,  in Extended Abstracts of CHI 2001, ACM Conference on Human Factors in Computing Systems, Seattle, Washington, 31 March-5 April 2001 (Short Talk), p321-322.

o Smith, B.A., Zhai, S. Optimised Virtual Keyboards with and without Alphabetical Ordering - A Novice User Study,  in Proc. of INTERACT 2001: Eight IFIP Conference On Human-Computer Interaction, Tokyo, Japan, July 9-13, 2001. p92-99.

2000

o Zhai, S., Hunter, M., Smith, B.A., The Metropolis Keyboard -- An Exploration of Quantitative Techniques for Virtual  Keyboard Design, in theProceeding of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST 2000), November 5-8, 2000, San Diego, California. pp 119-128.

o Hunter, M., Zhai, S., Smith, B.A., Physics-based Graphical Keyboard Design, CHI'2000. April 1-6, 2000, The Hague, The Netherlands. 157-158.

Press

A sample list of ShapeWriter's press coverage. A more complete list is here

	Top 11 iPhone Applications A hidden gem, this little app allows you to circumvent the iPhone's standard hunt-and-peck fingertip typing...	Time.com December 21, 2008
	Android "ShapeWriter": Ingenious New Touch Screen Keyboard One of the top 50 Winners for Google's first round of Android Development Grant money, ShapeWriter creates a brilliant new way to quickly and accurately make use of a touch screen keyboard...	Digg.com May 13, 2008
	Research focusing on handheld devices ...Shumin Zhai, from IBM's Almaden research lab in San Jose, Calif., and Per-Ola Kristensson, from Linkoping University in Sweden, have developed a way to write words via a few, linked strokes of a stylus...	San Jose Mercury News April 8, 2003
	New Way Of Writing Text entry software spell relief for weary BlackBerry thumbs...	San Jose Mercury News July 15, 2005
	Text Entry Epiphany for the Tablet PC- SHARK "...what I feel is a revolutionary breakthrough... best on-screen text entry system I have ever used. This method is so simple and accurate it amazes me every time I use it. It is a must try for every user of a Tablet PC or other touchscreen enabled computer."	James Kendrick's top ten tech blog November 03, 2004
	Total recall boosts PDA writing. The idea of remembering word patterns and connecting the dots might not sound like an easy way to write an e-mail...	BBC News World Edition August 15, 2007
	SHARK spells relief for tired thumbs "Zhai gave a presentation on SHARK earlier this month during a conference at IBM's Almaden Research Center in San Jose, Calif., where he works, and his ideas led me to one of those rare eye-opening moments when you suddenly see what could be the answer to a long-festering problem..."	The Seattle Times August 29, 2005
	Digitales Kalligrafie. Nur mühsam lassen sich Wörter in Handcomputer und Handys eintippen. Eine neue Technik beschleunigt die Eingabe enorm...	Die Zeit September 1, 2005
	Shark streamlines writing interfaces "Sitting at a table in a small conference room, research student Per-Ola Kristensson slides a stylus in a seemingly random zigzag across a grid of letters on a tablet computer...."	The Irish Times July 7, 2005
	Dra ett streck över din sms-tumme Snart kan du skriva sms dubbelt så snabbt. Tacka Per Ola Kristensson för det. 27-åringen har utvecklat ett nytt, revolutionerande skrivsystem för mobiltelefoner.	Aftonbladet February 8, 2007
	New-age keyboard: Trace, don't write Shark (Shorthand-Aided Rapid Keyboarding) is an advanced pen-based shorthand method that allows users to input words into mobile devices by tracing them letter by letter on a virtual keyboard....	CNET July 11, 2005
	Kurzschrift für Mini-Monitore "Forscher des IBM Almaden Research Center haben eine neue Methode erfunden, Texte mit kleinen mobilen Geräten einzugeben. Dabei arbeiten sie mit einer virtuellen Tastatur, auf der sie rasch Zeichenfolgen markieren können...."	Handelsblatt July 25, 2005
	Do not ditch the stylus just yet ...IBM has developed a way to write with a stylus that lets users reach 80 words per minute with minimal training...	New Scientist July 30, 2005
	Best IPhone Apps: Office and Personal Productivity These iPhone applications help extend your business desktop to the phone...	PCWorld January 06, 2009
	SHARK spells relief for tired thumbs "...a radically different approach that is easy to learn and fast."	Mike Langberg's tech column August 29, 2005
	Innovation: The mobile future of the keyboard Cellphone users don't have it easy - many enter far more letters than numbers into their gadgets, but most phones still make you do so using a number pad. Meanwhile, the designers of smart phones seem determined to make touchscreen keyboards the norm before they have been fully perfected.	NewScientist April 02, 2009

Notable Awards and Recognition

• Hailed as "A revolutionary breakthrough" in 2004 by James Kendrick's top ten tech blog, only days after the first public release of SHARK.
• From 2003 to 2005, the project was reported by major news organizations including The New York Times, Finantial Times, San Jose Mercury News, and the BBC.
• Google Android Developer's Challenge prize in 2008
• Top 11 iPhone Applications by Time.com
• 14 must-have iPhone apps by Examiner.com
• Top 10 mobile applications by 2009 Razorfish Digital Outlook
• Top 10 iPhone Apps for Girls by TotallyHer.com
• Best iPhone Apps: Office and Personal Productivity by PC World
• "IBM Research Division Accomplishment" recognition in 2010

ShapeWriter, Inc

Based on a technology licensing and service agreement with IBM, ShapeWriter Inc was founded in 2007 to further develop and commercialize ShapeWriter technology. The company's mission "is to make mobile phones, tablet computers, game pads, and all other touch-screen devices more useful - and user friendly - with software based upon shape writing technology". ShapeWriter Inc was acquired by the market leader in information input, Nuance Communications, in May 2010.

"ShapeWriter" and the ShapeWriter Logo are trademarks of ShapeWriter Inc.