Calliope 0.2.1: Moving to Mongo and Tomcat

This project is about Digital Scholarly Editions or DSEs. These are Web-based publications of cultural heritage texts. My design for DSEs calls for a REST Web Service, originally called hritserver, but now renamed Calliope, which provides the components for building a DSE within a Web authoring environment like a CMS. The infrastructure looks like this:

As you can see, at the bottom there is a database, including an images handler. Currently Calliope uses CouchDB and stores the images in the file system for speed. But I now have made it possible to define a single Java class to extend Calliope to handle other databases. MongoDB, a rival NoSql database to CouchDB, is supposed to be faster. Unlike Couch it handles images well, through the GridFS module, making replication of the entire database easier. In Couch you have to attach images to documents, and word has it that when those images are large it becomes sluggish. One advantage of Couch, though, is its revision control, which Mongo lacks. In Mongo, your latest copy of a document is your only copy.

The architecture has also a number of canonical views, which are only suggestions. They can be extended, customised and redecorated. They also know nothing about which database they are running on. They communicate with the Calliope service, which gives them partial Web-pages, which they use to compose the views. This is thus a pure example of the familiar Model-View-Controller paradigm.

Another improvement in version 0.2.1 is making it able to run equally well in Jetty embedded mode or Tomcat. The difference is that in Jetty the web application is part of a runnable application that makes it easy to debug and set up, as well as launch. Tomcat uses a plugin type of architecture where Web applications are stored in WAR files (really ZIP or JAR archives), which are really just directories with configuration information. Tomcat acts as a Web Server and hands off control of everything in the WebApp's context to the stuff unpacked from the WAR file. So one difference is that the URL for the Jetty test service is http://localhost/tests/ and for Tomcat at the moment it is http://localhost:8080/calliope/tests/ (Calliope is the name of the WAR file). So what I have now is a combination of two ways to launch Calliope: via Jetty or Tomcat, using either of two databases, Couch and Mongo, so there are a total of four possible configurations. This kind of flexibility helps others to consider adopting your software.

C versus Java speed and memory test

I have now translated the functions that load or save MVDs from disk from Java to C. They are both moderately complex and use several classes – a good real world test. Here is the preliminary speed test for the very similar functions loadMVD and saveMVD, when run on the file kinglear.mvd averaged over 10 iterations, discounting the first result, which is always slower:

Language	Time (microseconds)	Memory (bytes)
Java (load)	46127	1104072
C (load)	1608	364544
Java (save)	56294	348752
C (save)	4844	36864

This surprises me. I have been led to believe that Java was nearly as fast as C and sometimes faster. Or so the philosophy of the JIT compiler goes. And no, I didn't include the JIT compilation or JVM load or save time, but in both cases I took the time at the start and end of the relevant function. And no, I am very experienced in writing in both languages. The C version is just a translation of the Java program, but with this speed difference who wants to use Java?

The memory results were kind of expected, though. Everyone knows that Java is a memory hog.

The loadMVD function loads a binary file format on disk then parses it into an in-memory form. The saveMVD function reverses the process but is a bit more complicated because it must reconnect transposed segments.

Technique of measurement

In C I used getrusage (the ru_maxrss field) for memory and gettimeofday for timing. In Java I used System.nanoTime() and Runtime.getRuntime().freeMemory(); (after calling System.gc() the first time). The Java code was compiled with debugging off, the C code with -O3, no debugging symbols. The Java version was 1.6.0_35 on MacOSX 10.8.1, and the C compiler was gcc 4.2.1 on the same platform. Both programs were run on the commandline.

Comparison and alignment in literary texts

Humanists have always wanted to compare texts, but the computational techniques they are still using are based on a method devised in the 1960s for very early computers with severely limited RAM. They call it 'collation', because it reminds them of the manual process of comparing texts in the print era. Automatic collation examines the text word by word within a window of at most a page. The first word or line encountered closest to the expected position is taken as an alignment position and the window moved on by that amount. In this way it proceeds like a man trying to piece together a giant jigsaw puzzle using a torch in a field at night. It cannot see alignments outside of its window and mistakes once made lead to errors that must be manually corrected.

Suffix trees

But computer science has moved on a long way since 1962, when the method was first devised. The idea of aligning the entire text of two versions became possible with the invention of practical methods for constructing suffix trees in the 1990s. The principle of the Maximal Unique Match as used in bioinformatics is something that digital humanists should be more aware of. The text can be efficiently compared by aligning it greedily on the longest match shared by two versions that is unique, not word by word. This virtually eliminates misalignments or mistakes during collation. A 'window' is not needed because modern computers have enough memory to take in the whole text at a time. The suffix tree can quickly tell us whether any given substring is found in a text in a time proportional to the length of that substring. This represents such a big speedup over conventional techniques that I wish more people would sit up and pay attention.

Revising nmerge

When I wrote the nmerge program in 2009 it was nearly a first. Although MEDITE had used suffix trees for alignment on literary texts, it was limited to two versions at a time. Also, it had no format to store differences. nmerge could handle an arbitrary number of versions and store them in a Multi-Version-Document (MVD) format so they could be efficiently read back later. But I never claimed that nmerge was anything but a 'first cut at a difficult problem'. Since 2009 we have learned a lot about what it should do and how to improve it. And now, after several false starts, a realistic rewrite is in progress.

What it plans to do is:

1. Simplify the alignment process by splitting up the existing functionality of nmerge into a series of plugins. The only tasks that the core nmerge program will perform will be to load and save MVD files, and to manage the plugins, of course.
2. Rewrite the entire program from scratch in the C language. This should overcome memory problems with Java by dynamically allocating memory only as needed, instead of wastefully at present. Also C gives the program great longevity and portability as well as speed. Also provide language wrappers so it can be called natively in PHP (as an extension) and Java (via JNI).
3. Use multi-threading to improve performance. Individual sub-alignments and building of suffix trees can carry on simultaneously.
4. Transpositions can be computed using a technique that exploits adjacency of short transposed sections. In this way even transpositions containing minor revisions can be detected. This should improve alignment quality.
5. Alignment will be by shuffling the fragments of the MVD, not by pasting in differences into a explicit variant graph. This should greatly improve the program's simplicity.
6. Changing the MVD file format so that versions and groups are merged into version-IDs. This should make version specification simpler by using a hierarchical naming system based on paths like /folios/F1, or /C376/add0, rather than on tables of separate groups and versions.
7. Change the default text encoding from UTF-8 to UTF-16. This will allow easy comparison between Chinese and other languages like Bengali, which split almost all characters across byte-boundaries.
8. Provide a test-suite to verify every aspect of the program as it is being written and to insulate it from damage if any changes are made later.

I have already made a good start refactoring the tool into a series of plugins: 16 to be precise. There is even one for adding a version to an existing MVD. Since it is proceeding by plugins rather than a single monolithic block of code I anticipate early results in a few weeks at most. I have high hopes for this new tool, as it will lift MVDs into areas where it has not gone before, into general use.

Last piece in the import puzzle

People wanting to import XML into the HRIT system probably have XSLT scripts that transform the files into some other form and then format it into HTML. Perhaps the two steps are not even separated. If they try to use the current HRIT import system then TEI constructs like the following (taken from the TEI-Lite manual) will fail:

<list>
 <head>A short list</head>
 <label>1</label>
 <item>First item in list.</item>
 <label>2</label>
 <item>Second item in list.</item>
 <label>3</label>
 <item>Third item in list.</item>
</list>

The reason is that the <head> element is inside the <list> element. If we translate one-for-one the elements of XML into elements in HTML we will have to delete the <head> element, because none of the <h1>, <h2>, <h3> elements in HTML can appear inside <ul> or <ol>. But what we really want is to do is move it outside <list> and give it an attribute like type="list". But that's manipulation of the XML DOM, which neither stripper nor formatter (my two import tools) currently perform.

Brainwave

Whatever new facilities I add to stripper to allow such transforms you can bet that someone will need something in XSLT that isn't supported. My stripper program would just keep on getting more and more complex. And I would waste more and more time. So I had the simple idea not to modify stripper or formatter at all, but just to add a further step into the import process. All we have to do is allow XSLT transforms to take place on the imported XML files as a first step in the importation process, through the use of a tool like Xalan. By default a TEI-Lite stylesheet could perform the necessary transforms on the XML to turn it into sane XML for easy conversion into standoff properties. Not only is this a trivial change to implement, it is also extremely powerful. Although existing stylesheets may have to be modified for this to work, no loss of functionality can any longer be claimed for the HRIT system over existing XML based digital editions. A neat result, indeed.

It seems that the only XSLT processor that works on MacOSX and Linux any more is libxslt. XML may not be dead yet but its tools at least are dying. A sign of the times?

D'Oh!

I forgot that Java has an XSLT processor built in, so after getting libxslt to work via JNI I had to scrap it and redo it more simply. Which just goes to show that having a coffee and a walk in the garden before you code something is often time well spent, even though it looks and feels like you're loafing.

Running couchdb as a daemon on OSX

When you install couchdb on OSX using homebrew it doesn't set it up for running continuously. Whenever you launch it using sudo couchdb -b and your Mac goes to sleep it will kill the process, even if you used nohup. To make it run as long as you don't shut down your computer, and launch whenever you start it up you have to run it as a "daemon". Here's how I did it.

1. Locate the launch daemon file "org.apache.couchdb.plist" located in /usr/local/Cellar/couchdb/1.2.0/Library/LaunchDaemons or some similar place.
2. Now edit the file. I used:

   sudo nano org.apache.couchdb.plist

   and change the contents of the XML <string> element immediately following the <UserName> element to "root" instead of "couchdb".
3. Now copy the file to the right place:

   /Library/LaunchDaemons

4. Finally use the launchctl command to load it:

   sudo launchctl load /Library/LaunchDaemons/org.apache.couchdb.plist

   (You can also unload it using the same command but with "unload" instead of "load".) I've modified the hritserver-start.sh script so it no longer kills and relaunches couchdb. If you are using hritserver make sure you have an up-to-date copy of that.

Installation of hritserver

I though I would post some instructions for installing hritserver, since we don't yet have an installer. I should probably write one.

Installation on MacOSX

1. Homebrew

Homebrew can be fetched from http://mxcl.github.com/homebrew/. Follow the instructions there.

2. gcc

To build the C libraries used by hritserver you will first have to install gcc:

brew tap homebrew/dupes
brew install apple-gcc42

3. Install couchdb

The command is

brew install couchdb

4. Set up databases

You need to first download the hritserver package, available from www.github.com/HRIT-Infrastructure. Click on the hritserver repository, then the Downloads tab. Select "hritserver-0.1.3.tar.gz" and download it (Don't click on "Download as .tar.gz" or .zip). When it is on you hard disk it will unpack automatically or by double-clicking on it or on the commandline (OSX removes the .gz automatically):

tar xvf hritserver-0.1.3.tar

Now move the folder to a convenient location:

cd ~/Downloads
mv hritserver-0.1.3 ~
cd ~/hritserver-0.1.3

First set up the database for admin access. In the hritserver-0.1.3 folder are two scripts add-user.sh and couchdb.sh Run this command first:

sudo ./couchdb.sh

After running this you have to press return to get back the prompt. Check that couchdb is running:

ps aux | grep couchdb

You should get two process numbers, one 1-line long (that's the command you just ran) and a longer one about 6 lines long. That's couch running.

Now run:

./add-user.sh

It should respond with "-hashed-9222..." and a lot of hex numbers.

Now test that couch is running by typing the url in a browser: http://localhost:5984/_utils

There should be two entries in red: _replicator and _users.

Now install the test databases:

cd backup
./upload-all.sh

The script asks for a password. Type in jabberw0cky (with a zero). This is a master script that calls all the other upload-*.sh files. (So you can upload them individually.) Finally run hritserver from the command line:

5. Run the installer

cd ..
sudo ./install.sh
sudo ./hritserver-start.sh

Hit return. The service will run even when you log off. To stop the service, log in as the same user who launched it initially and type:

sudo ./hritserver-stop.sh

Access the service on http://localhost:8080/tests/ (trailing slash is significant). To make it visible on port 80 add the following lines to the end of /etc/apache2/httpd.conf,and restart apache

On OSX:

ProxyPass /tests/ http://localhost:8080/tests/ retry=0
ProxyPass /corpix/ http://localhost:8080/corpix/ retry=0
ProxyPass /import/ http://localhost:8080/import/ retry=0
ProxyPass /upload/ http://localhost:8080/upload/ retry=0

Now restart apache: sudo apachectl restart

Ubuntu

Add the above lines to /etc/apache2/mods-available/proxy.conf.

Restart apache2: sudo service apache2 restart

ADDING OTHER TEXTS

The mmpupload tool can be used to upload files in XML or plain text in batches. See www.github.com/HRIT-Infrastructure.

Consolidation of code on Github

I've put all my free software offerings on Github. This service has the advantage of unlimited repositories and collaborators. So I created an "organization" called Hrit-Infrastructure into which I put all the general software components. Outside under my own account I'm putting more specific tools that I use for ingestion of external formats etc. HritServer needs a better installer but I've put a basic set of instructions onto the wiki.

At the moment Hritserver uses couchdb, but my colleagues are porting it for me to Mongodb, which is a lot faster. We want speed, because this is where digital humanities software often falls down. I notice that a lot of new projects and old ones too use interpreted languages like ruby, php and python. Sure they're cool, but they are slooooow. One benchmark I saw tested ruby and found it to be at least 200 times slower than C and Java, and php is 500 times slower. That's why I'm happy that hritserver is written in a composite of those two languages. We will do a lot more than the competition and we will do it blindingly fast.