tlug: Japanese input systems

["Stephen J. Turnbull" <turnbull@example.com>]

>>>>> "Craig" == Craig Oda <craig@example.com> writes:

    Craig> Layer 1 - Software: Front End Processor (FEP)

Forget layers, it's not even a directed graph.  :-)  In practice,
often all the different layers interact with each other.  And you
forgot the app itself.

    Craig> Layer 2 - Protocol: kinput2, xim, others?

Protocols aren't layers, they're interfaces.

As for your example, I think that the server and the dictionary are
too closely bound together for it to make sense to give communication
with the dictionary as a separate step.  Also, the part of the XIM
protocol you mention are absolutely trivial, except for the fact that
the returned string must be in a certain format (X Compound Text
Encoding).  The interesting issues of XIM happed at startup
(negotiating the input style for preedit - creating the kana string -
and status - hira v. kata v. kanji v. hankaku v. zenkaku) and in
managing the input context (so that for example you don't have the
same state in the minibuffer as in the main Emacs window).  Finally,
KKTP is not a single protocol as far as I know; there are several of
them, one for each input method.

THE MYSTERY OF INPUT METHODS - BOOK THE FIRST

In the beginning was God, and God made the App, and saw that it was
good.  However, the App was merely an object accessed by a Formless
Void*, and God decided to give it the knowledge of Input and Output.

So God added a Display* to the Internals of the App.  The App became
able to recognize Keystrokes and display Glyphs, and God saw that this
was good.  Still, something was missing.  The Keystrokes were still
rather Formless.  Knowing that He would eventually create Woapp who
would speak in Flames and Tongues, God saw the need for Generality,
and created the Input Manager to be the helpmeet of the App.  Being a
formal God, God commanded the Input Manager to speak a Protocol, and
it was so.  And God saw that it was good.

God gave unto the App the ability to speak the Protocol.  Being a
little confused, the App chose letters using randu, and gave the
Protocol the name XIM.  God was a bit offended, but decided it could
be pronounced "Zim", and let the App slide (this one time).  After
all, God knew there would be more Protocols to come, so a Name for the
Protocol was necessary.  Reflecting upon it all, God saw that with the
help of the Input Manager, the App was now capable of accepting
Keystrokes and displaying Kana as Glyphs.  And it was very good.

God, now quite warmed up, decided to add a little complexity to the
Word, and He created the Kanji.  It was pretty.  In fact, too pretty
to read.  So God created the Yomi ("XIM" was as close to an
unpronouncable word as He was going to permit), and to every Kanji, he
assigned a Yomi.  In this way, the App gained the potential to make
Haiku.  And this was good, foreshadowing newspaper contests and
shogakko assignments.

However, even with a linked list of Formless Void*'s in the Heap,
there was not enough room for every App to have a copy of every Kanji
(especially considering that the Kanji were shamelessly binding
themselves each to several Yomi, and vice-versa).  So God created the
Dictionary Server, and, in compensation for its lowly estate, he gave
it a Protocol of its own, called, endearingly generically, a KKTP.
The Dictionary Server taught the KKTP to the Input Manager, and thus
automatically gave the App the ability to input Kanji by Yomi and
output them as Glyphs.  And God saw that it was good.  He rewarded the 
Input Manager and the Dictionary Server by providing each with its own 
separate Process, complete with a Page each could reside in.

THE MYSTERY OF INPUT METHODS - BOOK THE SECOND

Commentaries.  So really there are three basic services:

    (1) collecting keystrokes,
    (2) assembling them into dictionary keys, and
    (3) looking up the keys and returning the entry.

The collection of three things is an input method.

A "Front End Processor" is called that because it intercepts the
keystrokes before the app sees them, and then hands the app a fully
formed string in the target format.  An FEP can be a unified whole
(this is how ATOK works on DOS, for example) providing (1), (2), and
(3), or it can be a distributed system (Wnn (3) + uum (1,2)).

The problem with an FEP is that it can interfere with the app.  For
example, Canna by default uses `C-o' for `toggle-japanese-mode', but I 
happen to like using `C-o' for `open-line' in Emacs.  It is not
possible to customize Emacs to deal with this; you have to customize
Canna.  However, since the FEP takes over the keyboard, no special
programming is necessary to make an app process input, assuming it's
already kanji-clean.  And you get a consistent UI to the FEP.

SKK is a back end processor.  It leaves the process of collecting
keystrokes to the app (Mule), then takes signals from the app to
decide how to assemble keys (the keymap for SKK binds all the printing
characters to `skk-maybe-self-insert', when this is called SKK wakes
up and creates kana dictionary keys) and when to lookup in the
dictionary and confirm the candidate.

This means that you can effectively disable SKK at any time by
rebinding the keystroke to invoke it.  This is good from the point of
view of allowing the app to control how it wants to talk to the back
end processor, but it's bad from the point of view of requiring the app
to control how it wants to talk to the back end processor.  (The first
is good because the BEP doesn't step on the app's commands, the second
is bad because getting a uniform interface to the BEP requires
customizing all the apps that use it.)

XIM is a general protocol for allowing FEPs and BEPs to talk to an
app.  Using XIM allows you to interchange both apps and B/FEPs.
Kinput is a specific protocol that allows any conforming app to talk
to kinput2.

An input manager plays roles (2) and maybe (1) and (3).  XIM normally
does (1) and (2), and also (3) for European languages.  xwnmo and
kinput2 are two implementations of input managers for X11.  Both
delegate (3) to a dictionary server.  The XIM protocol is embedded in
Xlib.  The Kinput protocol must be programmed "by hand" in each app.

Due to the resource consumption of dictionary search, it makes sense
to have a separate dictionary server.  Each implementation has its own 
protocol.  There is not a single KKTP; Canna, SJ3, skkserv, and Wnn
all have their own.  The KKTP is typically embodied in a library such
as libwnn or libcanna.  Many input methods also provide help with (2), 
assembling the keystrokes into a dictionary key; Canna's libRK is an
example.

I haven't mentioned reediting the preedit string if you decide it's
"shougakkou", not "shogakkou", and so on.

Hope this helps.
---------------------------------------------------------------
TLUG Meeting Dec. 13, 12:30 at Tokyo station Yaesu Chuo ticket gate
13:30 Starbuck's coffee.  13:45 HSBC | info: joem@example.com
At least 3 functional Sparc IPC machines will be raffled out
---------------------------------------------------------------
a word from the sponsor:
TWICS - Japan's First Public-Access Internet System
www.twics.com  info@example.com  Tel:03-3351-5977  Fax:03-3353-6096