Adding a directory to java.library.path

In addition to what I said before, there is actually a cool way to add a directory to java.library.path. If you launch your application via a script you can execute a commandline Java tool to output the current library path. For example, this simple Java program adds "/usr/local/lib/" to that path and prints it without a trailing CR to the console:

public class LibPath
{
    public static void main( String[] args )
    {
        System.out.print(System.getProperty(
            "java.library.path"));
    }
}

To use it just compile it, and then invoke it in the script:

LIBPATH=`java LibPath`:/usr/local/lib
java -Djava.library.path=$LIBPATH ....

And as if by magic the java library path acquires a new directory before your prorgam is run.

Posting multipart form data

For testing I needed to simulate a web-browser doing a file-upload. I tried to Google this but I couldn't find a suitable answer that worked. So I rolled my own. This might save other people some trouble too. First comes the MIMEMultipart object, which stores the body of a multipart post:

import java.io.FileInputStream;
import java.io.File;

public class MIMEMultipart 
{
    StringBuilder text;
    static String CRLF = "\r\n";
    String boundary;
    public MIMEMultipart()
    {
        text = new StringBuilder();
        boundary = Long.toHexString(
            System.currentTimeMillis()); 
    }
    public String getContent()
    {
        return text.toString();
    }
    public String getBoundary()
    {
        return boundary;
    }
    public int getLength()
    {
        return text.length();
    }
    public void putStandardParam( String name, 
        String value, String encoding )
    {
        StringBuilder sb = new StringBuilder();
        sb.append("--" + boundary).append(CRLF);
        sb.append("Content-Disposition: form-data; "
            +"name=\""+name+"\"");
        sb.append(CRLF);
        sb.append("Content-Type: text/plain; charset=" 
            + encoding );
        sb.append(CRLF);
        sb.append(CRLF);
        sb.append(value);
        sb.append(CRLF);
        text.append( sb.toString() );
    }
    public void putBinaryFileParam( String name, 
        String fileName, String mimeType, 
        String encoding ) throws Exception
    {
        // compose the header
        StringBuilder sb = new StringBuilder();
        sb.append( "--"+boundary );
        sb.append( CRLF );
        sb.append("content-disposition: form-data; "
            +"name=\"" );
        sb.append( name );
        sb.append( "\";  filename=\"");
        sb.append( fileName );
        sb.append( "\"" );
        sb.append( CRLF );
        sb.append("Content-Type: "+mimeType ); 
        sb.append( CRLF );
        sb.append("Content-Transfer-Encoding: binary");
        sb.append( CRLF ); // need two of these
        sb.append( CRLF );
        text.append( sb.toString() );
        // now for the file
        File input = new File( fileName );
        FileInputStream fis = new FileInputStream(input);
        byte[] data = new byte[(int)input.length()];
        fis.read( data );
        fis.close();
        text.append( new String(data,encoding) );
        text.append( CRLF );
    }
    public void finish()
    {
        text.append( "--" );
        text.append( boundary );
        text.append( "--" );
        text.append( CRLF );
    }
}

To call it, open a standard Java URLConnection:

private static void printResponse( 
    URLConnection conn )
{
    try
    {
        InputStream is = conn.getInputStream();
        while ( is.available() != 0 )
        {
            byte[] data = new byte[is.available()];
            is.read( data );
            System.out.println(new String(data,
               "UTF-8"));
        }
    }
    catch ( Exception e )
    {
        e.printStackTrace( System.out );
    }
}....
URL url2 = new URL("http://localhost:8080/strip");
URLConnection conn = url2.openConnection();
MIMEMultipart mmp = new MIMEMultipart();
mmp.putStandardParam( Params.FORMAT,Formats.STIL,
    "UTF-8" );
mmp.putStandardParam( Params.STYLE,"TEI/drama",
    "UTF-8" );
mmp.putBinaryFileParam( Params.RECIPE,"recipe.xml",
    "application/xml","UTF-8" );
mmp.putBinaryFileParam( Params.XML,
    "act1-scene2-F1.xml",
    "application/xml","UTF-8" );
mmp.finish();
conn.setDoOutput(true);
conn.setUseCaches(false);
((HttpURLConnection)conn).setRequestMethod("POST");
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.setRequestProperty("Content-Type", 
    "multipart/form-data, boundary="
    +mmp.getBoundary());
conn.setRequestProperty("Content-Length", 
    Integer.toString(mmp.getLength()));
OutputStream output = conn.getOutputStream();
output.write( mmp.getContent().getBytes() );
output.flush();
output.close();
// get response and print it
printResponse( conn );

Extend MIMEMultipart if you like by adding a method for plain text files and other types of part.

Loading native libraries in Java

This is an old problem, so I thought I'd write down my current experiences to save others, and myself, pain in future.

You write a native library libFoo.so or foo.dll or libFoo.dylib etc. for Java. And you store it in a convenient location, not a system location, because your software shouldn't meddle with that. To use it you need to call System.loadLibrary("foo");. This will probably give you: "Exception in thread "main" java.lang.UnsatisfiedLinkError: no foo in java.library.path". Where did I go wrong?

System.loadLibrary looks in the Java library path, "java.library.path". Cool, let's set that in the program, just before we call loadLibrary. We can get some platform-independence by passing in the library path on the commandline:

String old = System.getProperty("java.library.path");
System.setProperty("java.library.path",old+":/usr/local/lib");

It doesn't find the library because you can't change "java.library.path" after starting the JVM. It just ignores your additional directory.

Everyone says set the environment variable LD_LIBRARY_PATH to (in my case) /usr/local/lib. This doesn't work either. On Linux and OSX at least Java ignores that variable when setting up java.library.path. In any case setting LD_LIBRARY_PATH globally for your application will screw up something else on your system. Not cool.

Third attempt. Set java.library.path on the java commandline:

-Djava.library.path=/usr/local/lib

Now you've changed the JVM so things could go wrong. Instead of having the system default library path where everything is, you've redefined it to a custom location. Unfortunately there's no universal way to ADD /usr/local/lib to java.library.path. So the best you can do is find the java library path on your system (by writing a Java program that outputs System.getProperty("java.library.path")) and then add /usr/local/lib to that value and finally specify the entire string to java:

-Djava.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:/usr/local/lib

This is what I have to do on Mac OSX. Of course it's entirely platform-specific, which is stupid for a programming language that is supposed to be platform-independent. On the other hand this yucky solution is the best one on offer. Since when you've finished developing you'll be running it time and time again on the same platform it probably doesn't matter much.

Alternatively, you could just put your library in the current directory, which will work on Windows (reportedly) and Mac OSX but not Linux.

Installing couchdb on Mac OSX

Installing couchdb on Linux is a breeze. At least with the Debian package manager. But on OSX, where I was stuck over the Christmas break, installation is a pain. Of course if you believe the hype homebrew will save us. All you have to do is install it and type: brew install couchdb. Except that, it doesn't work. Packages have to be maintained, and unless the homebrew authors do all that work, their packeges will soon break. So I had to do it myself. This is the formula on 64 bit systems:

1. Download the latest Erlang source. Configure with --enable-darwin-64bit . Otherwise it compiles in 32 bit and it won't work with the other components, especially icu. Then make, make install as usual.
2. Download and install ICU (configure, make, make install)
3. Download and install couchdb. Configure, make, make install. And it all should work.

Now that Apple's great leader has passed on maybe someone in charge will see that a proper package manager would be a good idea for OSX.

A new kind of standoff markup

When markup was first invented it was always embedded in the text. The earliest forms were like formatting instructions to tell us how the text should look when it was printed:

.indent 5
blah blah blah
.indent 0

This meant that the text should be indented five spaces, say, for a bit of quoted text. This might be a problem if you later wanted all quotes indented 10 spaces. Then in the 1980s some folks had the bright idea of embedding only abstract information about textual structure rather than specific formats:

.q
blah blah blah
.endq

Then when it was printed a separate program would read a stylesheet and convert the .q and .endq instructions into their correct indents. This was called 'generalised markup'. For a while people thought this was wonderful. It was used in the first web-pages in the early 1990s to make HTML, the Hypertext Markup Language. Every web page you read today is still encoded this way - well, nearly. But the problems persisted. The tags were all embedded in the text, which made it difficult to read. As more and more information had to be added eventually there was so much markup that it was almost impossible to read the text without first formatting it. Which was a problem if you wanted to edit it. Then, in the mid-1990s some people invented standoff markup.

Standoff markup

Standoff markup removes the embedded tags and stores them separately from the text. Each of the externally stored tags is associated with:

• an offset into the text where it could be reinserted
• the distance between pairs of start and end-tags as measured in the underlying text
• any attributes that belonged to the original start-tag

Standoff markup doesn't change the markup structure; it just separates it from the text and makes it possible to combine different markup sets with the same underlying text. For the reader it also removes the confusion arising from the embedding of complex tags. But there were also drawbacks:

• If the text was altered then all the offsets in the standoff markup file would have to be updated.
• It was still not possible for the textual properties described by the tags to overlap.
• Markup sets could be exchanged but not combined. You couldn't, for example, add both metrical and formatting structure to the same document, or merge markup sets written by different people.

Standoff markup is suited to the corpus linguistics applications it was originally developed for, where various natural language tagging tools produced differently marked-up versions of the same underlying text. But in the humanities this method has proved less popular because of the need to edit the text, at least in the initial stages of preparing an electronic edition - which is what I am interested in.

Standoff properties

To distinguish our solution from conventional standoff markup we call it 'standoff properties'. It can also be used, as in our software, to allow the editing of either the text or the markup while keeping the other half automatically in sync.

Standoff properties extends standoff markup in two significant ways:

1. By allowing overlap between properties
2. By allowing sets of markup properties to be freely combined

Overlap

Software engineers use different kinds of data structures for various tasks, such as graphs, arrays, hash tables etc. Humanists and corpus linguists have unnecessarily limited themselves to embedded markup languages and thereby limited representations of their data to tree structures. Moving to a data structure that resides outside of the text liberates them from that restriction.

Mixing markup sets

A set of markup tags, stripped from a standard XML file is always properly nested because it is sorted by the order in which tags were encountered in the original XML file. This can be used to reconstruct the document tree in cases where several tags start at the same point in the underlying text. For example:

<book name="my book"><chapter n="1"><text>...

In standoff form these properties/tags might be represented as:

"book" start: 0 length: 12345
"chapter" start: 0 length: 234
"text" start: 0 length: 234

Here the three tags 'book', 'chapter' and 'text' all start at the same point in the text they enclose. The order can be used to decide that 'text' goes inside 'chapter' and not the other way around, even though they may describe equal ranges in the base text.

But if we were to allow different sets of markup tags to be combined, what would the order be? Since the two trees, which represent two sets of markup tags, can't be combined, this reliance on the order of the tags will have to be abandoned if we want truly mixable tagsets.

Lifting the restrictions

Digital humanists have been calling for these relaxations in the strictness of markup for many years. But the reality is that the external world of markup has not changed. Web pages are still represented as tree-structured HTML. So if 'standoff properties' are adopted as an alternative representation for markup, how is it possible to convert them into a well-formed and valid formatted HTML file? The conventional solution is to use the XML Transformation Language (XSLT), but that requires embedded XML as input. So a new technique is needed to perform the conversion. That's what this posting is all about.

Standoff properties to HTML

The reader may well be skeptical that this is possible at all. Anyone who has worked in this field over the past 20 years will know that there have been a large number of attempts, more or less unsuccessful, to allow overlap in marked-up texts. It may also seem impossible to convert sets of arbitrarily overlapping properties with no real structure into rigid tree-formatted HTML. But I am going to prove it, although to make the proof short enough to read I'm only going to outline it. For the more sceptical there is a program to back it up that has already been extensively tested. You can access it on the Digital Humanities Testbed website and try it out for yourself.

To prove that standoff properties can be converted to HTML we can omit certain confusing details. Firstly we can ignore attributes because a set of attributes is associated with each property or tag on a one-to-one basis. We can also forget about how properties are turned into HTML tags because this is likewise a one-for-one mapping. For example, we can just specify that the property 'stage' should always be converted into the HTML tag 'p'. Finally, since property sets can be freely combined, it suffices to prove that one set of standoff properties can be converted into a HTML document tree.

Deriving nesting information

In order to build a tree from data that has no such structure some information about which properties may be allowed to nest inside other properties is obviously required. This information can be derived from two sources: firstly by scanning the list of properties and their ranges in the text it is possible to compute how often a particular property is entirely 'inside' another property. So if the property-set was derived from stripping an XML file this nesting information could not violate any requirement of the XML file's original schema, if it already conformed to it. In fact XML can perfectly well do without any form of 'schema', or syntax recipe, and can derive it from its 'well-formedness', that is the strict nesting of its paired tags. We will be going one step beyond that by not even mandating well-formedness.

First of all some defintions:

• An 'element' in XML means a pair of start and end-tags and their intervening content. The XML element '<em>really true</em>' projects the property 'em' over the text-range 'really true'. An element can also contain other elements.
• A 'property' on the other hand is just a name for a range in the text. Unlike elements, properties don't have any intrinsic ability to nest.

By scanning a file containing a set of property names and their ranges for a particular text – a property-set – a matrix of how often each property was observed to be inside itself and every other property can be derived. A table from a simple TEI-XML file for a play might look like this:

	head	stage	speech	speaker	line	para	italics	text
head	0	0	0	0	0	0	0	0
stage	0	0	0	0	0	0	0	7
speech	0	0	0	0	0	0	0	0
speaker	0	0	0	0	0	0	0	0
line	0	0	0	0	0	0	0	0
para	0	0	0	0	0	0	0	0
italics	0	0	0	0	0	0	0	0
text	0	0	0	0	0	0	0	0

Table 1

Reading from left to right the numbers means 'head', 'stage', 'speech' etc are found so many times inside the properties named in the various columns. So the property named 'stage' is found inside the property named 'text' a total of seven times.

A similar table can be drawn up for HTML because all the nesting rules can be found in its schema. So, for example, the tag <span> may appear inside <p>, but not vice-versa. So the corresponding fragment of the HTML matrix would look like this:

	span	p
span	1	1
p	0	1

Table 2

In reality the full HTML table has 107 rows and columns and is too big to show here. Binary values are used instead of the frequencies of Table 1 because this is general nesting information derived from a schema, not from actual texts. Reading the table from left to right, the binary value '1' means 'may nest inside' and '0' means 'may not nest inside'. Now by mapping each of the property names from Table 1 to their HTML tag equivalents, we can look up the corresponding values in a full-sized version of Table 2 with the 107 tags of HTML5. If there is a 1 in the corresponding location of the HTML table then the frequency count in Table 1 will be left as is, and if the HTML table has a 0, meaning that the HTML equivalents may not nest, then the frequency, however large, will be zeroed. This guarantees that the nesting information derived from the property set, when converted to HTML, specifies HTML-compatible nesting rules.

Enclosing versus nesting

Recall that from the definition above a 'property' consists of a name and a range in the underlying text. What Table 1 records are the nesting characteristics of the property-names. To say property-name A nests inside property-name B is a general statement about those two properties. On the other hand to say that property A encloses property B means that the range of property A is outside the range of property B by at least one character on the left or right. So, for example, the property 'speech' encloses the property 'speaker' even though they both start at the same point:

<sp><speaker>Bast.</speaker> Nothing my Lord.</sp>

Document Object Model

The idea of a document object model (DOM) is borrowed from SGML/XML. HTML is an instance of an SGML language, and a HTML document can be described by a tree-structure of 'nodes', each containing other descendant nodes called 'children', and nodes at the same level called 'siblings'. Figure 1 shows the structure of a basic DOM tree.

To be continued ...

Extensions to CorCode

I've been thinking about the exact role of CorCode and how we render it on screen as HTML. There are some problems with the simple CorTex, CorCode, CorPix model that need figuring out. Hopefully they're just details.

TEI and XSLT

For TEI embedded markup people write XSLT stylesheets to transform the text, typically into HTML. Since a particular encoding of a text is embedded in it, and is highly customised, the XSLT stylesheet that transforms it cannot be fully portable to other people's texts:

1. they may want to render the same data differently, or
2. they may have custom tags and attributes that need rendering according to their local GUI requirements

The reason that TEI tries to mandate particular names for tags and attributes is so that standard stylesheets can be used. In practice this does not work very well because just as the encoding is customised the stylesheets also need customisation as required by these two points. On the other hand, if we no longer mandate any standard encoding and just allow the user to specify fully the names of properties (aka tags) then it is up to the encoder to specify what is to be done with them.

How to do this in HRIT?

One objective of designing HRIT is to make things easy and to make them work smoothly. So the user should download Cortex and CorCode and they should automatically merge locally and produce output that looks good in their local GUI tool. But how?

Both functions of the transformation - the one into a visual appearance and its interpretation by the application are specific to the encoding itself. So I am thinking that the transforming css file that performs both duties in HRIT needs to be closely associated with the CorCode. It shouldn't need to be transformed locally for a particular application because how can that transformation do anything without knowledge of the format? Imagine I write a css file to render King Lear, and use it within single view. Cool. But what happens if I download Little Dorrit by Charles Dickens in the same application, and my css file no longer works? So the css file can't belong to the application, or at least most of it can't. It has to belong to the CorCode. This is more or less what TEI already does: an XSLT stylesheet is designed for a particular collection of customised text encodings and is useless outside of that domain.

Ergo: Corform

So maybe we need a fourth type of basic data: corform that is as tightly bound to corcode as corcode is itself bound to cortex. That sounds less evil to me than extending the corcode format to support formatting. The good news is that we wouldn't need as many corforms as corcodes, because they would be version- and even file-independent. But it would be both functionally and visually specific to a specific application. We might assign each corcode a "style" and then specify corforms like render that style. We'd want many corforms for each style but not the other way around. So a suitable rest resource url might look like: http://luc.edu/hrit/corform/style-name/app-name/render-name. "app-name" could be "default", meaning any application, or it could be "hritsingle" meaning the single view application; a suitable style-name might be "TEI"; "render-name" might be "freds-format". An individual corform resource would be in css.

So the way it would work in practice the user would specify a particular corform resource manually, and it would be saved until changed. But if none was specified, the application name combined with the corcode's style name should identify a default rendering in every case.

Writing a basic php5 extension

I wanted to put formatter into a php extension because the CorCodeDemo currently has to write the CorCode and CorTex returned by the server to temporary files because formatter is currently a C commandline tool, and the CorTex and CorCode responses are too long to be commandline arguments. Also making formatter directly accessible from within php will effectively make it a replacement for XSLT, which is my intention.

The trouble with "hello world" as a php extension

I followed the instructions at http://devzone.zend.com/article/1021 but there were several problems. Documenting them might help others.

The first gotcha I discovered was that you're supposed to build php from the latest sources and install that on your system. This is actually quite dangerous on a Linux system because doing so invalidates the version installed by the package manager. The default deb package on Ubuntu puts files here, there and everywhere, and making the source code build put everything in exactly the same places is rather difficult and would destroy the package manager information about php. On the other hand, putting it elsewhere gives you two versions and the actual one invoked depends on the order directories are searched. The main problem this leads to is that building an extension in any version other than the one it is run in will prevent the extension from loading. My original php version was 5.3.2, which is what my server is still using. The latest version is currently 5.3.6, which became my commandline version. I didn't want to delete the old version because I had spent some effort installing and configuring it with xdebug, mysql and xslt.

The second gotcha was that the commandline php.ini file is not the same as the one used by the web server. On my system the apache2 one is in /etc/php5/apache2 and the commandline one is in /etc/php5/cli. You really must check the following:

1. that the extension_dir used during execution is the one you put your extension in
2. that the php.ini file that loads your extension is the one actually being used by php on the commandline (or server)

You can check the php.ini file by invoking on the commandline:

php -i | grep php.ini

And the extension_dir similarly:

php -i | grep extension_dir

Like me you'll probably be surprised by what it says. But match them up, and match the compile and the execution versions and it will work. Remember though, that the server will use a different php.ini file, and a different extension_dir. The best way around this dilemma is to download the same version of the source as is currently installed on your system, and build with that.