Digital Variants

Friday, 30 March 2012

Bare bones multi-file upload

What I need for HritServer is an import facility that can handle multiple file uploads. I've found many fancy scripts for doing this that are way too complex and too specific. Sometimes all you need is a bare bones solution you can tailor to suit your needs rather than a fully fledged product you have to spend ages understanding and cutting down to size. Also I hate building in dependencies that only increase the tendency for code to break. So here's my simple contribution to the HTML multifile upload problem.

The basic idea is to have one <input type="file"> for every file you want to upload. But you hide all but the current empty one. So when the user selects the only visible input file element it sets itself to the chosen file, adds itself to an invisiible list of input file elements, and creates a fresh one for the next time. To make it easier to see what's already been selected I maintain a secondary list of paths in a table. Next to each entry is a remove button that let's you take out individual entries. That's it. Here is the php code to handle the upload on the server side. Replace this with something else if you like, such as Apache file upload in Java:

And here is the HTML, with self-contained javascript.

<html>
<head>
<style type="text/css">
input.invisible { display: none }
</style>
<script type="text/javascript">
function remove( value )
{
    var repo = document.getElementById("repository");
    var child = repo.firstChild;
    while ( child != null )
    {
        if ( child.nodeName=="INPUT"&& child.value==value )
        {
            child.parentNode.removeChild(child);
            break;
        }
        child = child.nextSibling;
    }
    // now look at the table
    var table = document.getElementById("listing");
    child = table.firstChild;
    var finished = false;
    while ( child != null && !finished )
    {
        if ( child.nodeName=="TR" )
        {
            var gchild = child.firstChild;
            while ( gchild != null )
            {
                if ( gchild.nodeName=="TD" )
                {
                    var ggchild = gchild.firstChild;
                    if ( ggchild!=null&&ggchild.nodeName=="SPAN" )
                    { 
                        if ( ggchild.textContent!=null&&ggchild.textContent==value )
                        {
                            finished = true;
                            child.parentNode.removeChild(child);
                            break;
                        }
                    }
                }
                gchild = gchild.nextSibling;
            }
        }
        child = child.nextSibling;
    }
}
function doaddfile()
{
    var input1 = document.getElementById("input1");
    var repo = document.getElementById("repository");
    var listing = document.getElementById("listing");
    var parent = input1.parentNode;
    var input2 = document.createElement('input');
    input2.setAttribute( "type", "file" );
    input2.setAttribute( "name", "uploadedfile[]" );
    input2.setAttribute( "onchange", "doaddfile()" );
    input1.setAttribute( "class", "invisible" );
    input1.removeAttribute("id");
    input2.setAttribute("id","input1");
    parent.removeChild( input1 );
    parent.appendChild( input2 );
    repo.appendChild( input1 );
    // now create the row in the listing table
    var text = document.createTextNode(input1.value);
    var span = document.createElement("span");
    span.appendChild(text);
    var row = document.createElement("tr");
    var cell1 = document.createElement("td");
    cell1.appendChild(span );
    row.appendChild( cell1 );
    // add a button to remove the entry
    var button = document.createElement("input");
    button.setAttribute("type","button");
    button.setAttribute("onclick","remove('"+input1.value.replace(/\\/g,"\\\\")+"')");
    button.setAttribute("class","remove");
    button.setAttribute("value","remove");
    var cell2 = document.createElement("td");
    row.appendChild( cell2 );
    cell2.appendChild( button );
    if ( listing.firstChild == null )
 listing.appendChild( row );
    else 
        listing.insertBefore( row, listing.firstChild );
}
</script>
</head>
<body>
<form enctype="multipart/form-data" action="/upload.php" method="POST">
<div>
Choose a file to upload: <input id="input1" onchange="doaddfile()" name="uploadedfile[]" type="file" /><br />
</div>
<table id="listing">
</table>
<div id="repository">
</div>
<input type="submit" value="Upload File" />
</form>
</body>
</html>

Call the first "upload.php" and the second "upload.html" then put them both in the root document directory of your web-server. If you have php installed navigate to /upload.html and take it from there. You can add styling and change the server script easily because it is simple.

Wednesday, 7 March 2012

Uploading a directory of images to couchdb

I wanted to have a general script to upload a set of images to couchdb. Couch can't store images as documents because it uses JSON for that. But you can still create a JSON document and attach a set of annotations to it in the form of images. But here's the catch: you have to specify the document's revision id, and that changes after every image you add.

But I had two problems. My directory of images contained sub-directories. I also wanted to access the images directly using a simple URL. So if I had a directory structure like:

    list
        file1.png
        one
            file2.png
            file3.png
        two
            file4.png

I would want the relative URLs to be: /list/file1.png and /list/one/file2.png etc. The trick in writing the script is to extract the revid from the server response. I used awk for that, then used the returned value to upload the next entry in that directory. The second trick is to use %2F not / as a directory separator when creating the docids. Couch doesn't allow nesting in the database structure but you can simulate it by creating documents called:

list
list%2Fone
line%2Ftwo

The first posting to each of those documents doesn't need a revid, but subsequent ones do. That's just a feature of couch. So here's the script. It's a bity sloppy because if couch responds with an error to any upload it will fall over. This bit of error-handling is currently left as an exercise to the reader. I'll put it in later and may update the post then. To use the script put it into a file called upload.sh and then invoke it thus: ./upload.sh images, where "images" is the master image directory.

Saturday, 25 February 2012

The importance of deletions

Standoff properties describe what happens to runs of text, but what about runs of text you don't want, or need to put into attributes when it is formatted as HTML? This problem was revealed when I tried to use formatter to generate a dropdown menu from a list of versions spat out by nmerge. What nmerge gave me was the description of the work on one line, then the name of the version-group, followed by short names and long names for each version:

King Lear
Base F1 Version F1
 F2 Version F2
 F3 Version F3
 F4 Version F4
 Q1 Version Q1
 Q2 Version Q2

But I needed to turn this into HTML that looked like this:

<select name="versions" class="list">
<option title="Version F1" class="version">F1</option> 
<option title="Version F2" class="version">F2</option> 
<option title="Version F3" class="version">F3</option> 
<option title="Version F4" class="version">F4</option> 
<option title="Version Q1" class="version">Q1</option> 
<option title="Version Q2" class="version">Q2</option> 
</select>

It's obvious that strings like "Version F1" need to be moved from the text into attributes, and strings like "Base" and "King Lear" need to be deleted outright. No doubt XSLT wizards are laughing at me, pointing out that if only I had embraced XML I would be able to transform XML into exactly this format easily. But choosing a simpler model for representing textual properties doesn't mean any loss of functionality. In fact "less is more". All it needs is for these problems to be thought through and solved.

All you have to do is allow each standoff property to be "removed". I already had this feature but the length had to be zero. I used this for removing the TEI header from TEI documents and saving the content in an annotation of an empty range. But the length had to be zero, and the text the elements formerly enclosed was removed permanently from the base text. Since I wanted to reuse the same text for other applications I didn't want to remove any text permanently. So I just allowed the length of a "removed" property to be > 0. A deleted property now removes the text it affects from the output, and also any ranges that it encloses, as well as parts of any ranges it overlaps with. So in the JSON all you do is define some "empty" ranges that have the "removed" property set to true:

{ "name": "list", "annotations": [ { "name": "versions" } ], "len": 94, "reloff": 10 },{ "name": "empty", "removed": true, "len": 5, "reloff": 0 },{ "name": "top-group", "annotations": [ { "name": "Base" } ], "len": 88, "reloff": 5 },{ "name": "version", "annotations": [ { "description": "Version F1" } ], "len": 2, "reloff": 0 },{ "name": "empty", "removed": true, "len": 10, "reloff": 3 }...

What this machine-generated JSON says is that the string "Base" should be deleted. Long version names like "Version F1" should be moved to attributes called "description" for the short-names. The resulting HTML given above is exactly that output by formatter. So now we can also pick and choose which text gets selected and where it appears.

Thursday, 29 December 2011

Adding a directory to java.library.path

In addition to what I said before, there is actually a cool way to add a directory to java.library.path. If you launch your application via a script you can execute a commandline Java tool to output the current library path. For example, this simple Java program adds "/usr/local/lib/" to that path and prints it without a trailing CR to the console:

public class LibPath
{
    public static void main( String[] args )
    {
        System.out.print(System.getProperty(
            "java.library.path"));
    }
}

To use it just compile it, and then invoke it in the script:

LIBPATH=`java LibPath`:/usr/local/lib
java -Djava.library.path=$LIBPATH ....

And as if by magic the java library path acquires a new directory before your prorgam is run.

Wednesday, 28 December 2011

Posting multipart form data

For testing I needed to simulate a web-browser doing a file-upload. I tried to Google this but I couldn't find a suitable answer that worked. So I rolled my own. This might save other people some trouble too. First comes the MIMEMultipart object, which stores the body of a multipart post:

import java.io.FileInputStream;
import java.io.File;

public class MIMEMultipart 
{
    StringBuilder text;
    static String CRLF = "\r\n";
    String boundary;
    public MIMEMultipart()
    {
        text = new StringBuilder();
        boundary = Long.toHexString(
            System.currentTimeMillis()); 
    }
    public String getContent()
    {
        return text.toString();
    }
    public String getBoundary()
    {
        return boundary;
    }
    public int getLength()
    {
        return text.length();
    }
    public void putStandardParam( String name, 
        String value, String encoding )
    {
        StringBuilder sb = new StringBuilder();
        sb.append("--" + boundary).append(CRLF);
        sb.append("Content-Disposition: form-data; "
            +"name=\""+name+"\"");
        sb.append(CRLF);
        sb.append("Content-Type: text/plain; charset=" 
            + encoding );
        sb.append(CRLF);
        sb.append(CRLF);
        sb.append(value);
        sb.append(CRLF);
        text.append( sb.toString() );
    }
    public void putBinaryFileParam( String name, 
        String fileName, String mimeType, 
        String encoding ) throws Exception
    {
        // compose the header
        StringBuilder sb = new StringBuilder();
        sb.append( "--"+boundary );
        sb.append( CRLF );
        sb.append("content-disposition: form-data; "
            +"name=\"" );
        sb.append( name );
        sb.append( "\";  filename=\"");
        sb.append( fileName );
        sb.append( "\"" );
        sb.append( CRLF );
        sb.append("Content-Type: "+mimeType ); 
        sb.append( CRLF );
        sb.append("Content-Transfer-Encoding: binary");
        sb.append( CRLF ); // need two of these
        sb.append( CRLF );
        text.append( sb.toString() );
        // now for the file
        File input = new File( fileName );
        FileInputStream fis = new FileInputStream(input);
        byte[] data = new byte[(int)input.length()];
        fis.read( data );
        fis.close();
        text.append( new String(data,encoding) );
        text.append( CRLF );
    }
    public void finish()
    {
        text.append( "--" );
        text.append( boundary );
        text.append( "--" );
        text.append( CRLF );
    }
}

To call it, open a standard Java URLConnection:

private static void printResponse( 
    URLConnection conn )
{
    try
    {
        InputStream is = conn.getInputStream();
        while ( is.available() != 0 )
        {
            byte[] data = new byte[is.available()];
            is.read( data );
            System.out.println(new String(data,
               "UTF-8"));
        }
    }
    catch ( Exception e )
    {
        e.printStackTrace( System.out );
    }
}....
URL url2 = new URL("http://localhost:8080/strip");
URLConnection conn = url2.openConnection();
MIMEMultipart mmp = new MIMEMultipart();
mmp.putStandardParam( Params.FORMAT,Formats.STIL,
    "UTF-8" );
mmp.putStandardParam( Params.STYLE,"TEI/drama",
    "UTF-8" );
mmp.putBinaryFileParam( Params.RECIPE,"recipe.xml",
    "application/xml","UTF-8" );
mmp.putBinaryFileParam( Params.XML,
    "act1-scene2-F1.xml",
    "application/xml","UTF-8" );
mmp.finish();
conn.setDoOutput(true);
conn.setUseCaches(false);
((HttpURLConnection)conn).setRequestMethod("POST");
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.setRequestProperty("Content-Type", 
    "multipart/form-data, boundary="
    +mmp.getBoundary());
conn.setRequestProperty("Content-Length", 
    Integer.toString(mmp.getLength()));
OutputStream output = conn.getOutputStream();
output.write( mmp.getContent().getBytes() );
output.flush();
output.close();
// get response and print it
printResponse( conn );

Extend MIMEMultipart if you like by adding a method for plain text files and other types of part.

Sunday, 25 December 2011

Loading native libraries in Java

This is an old problem, so I thought I'd write down my current experiences to save others, and myself, pain in future.

You write a native library libFoo.so or foo.dll or libFoo.dylib etc. for Java. And you store it in a convenient location, not a system location, because your software shouldn't meddle with that. To use it you need to call System.loadLibrary("foo");. This will probably give you: "Exception in thread "main" java.lang.UnsatisfiedLinkError: no foo in java.library.path". Where did I go wrong?

System.loadLibrary looks in the Java library path, "java.library.path". Cool, let's set that in the program, just before we call loadLibrary. We can get some platform-independence by passing in the library path on the commandline:

String old = System.getProperty("java.library.path"); System.setProperty("java.library.path",old+":/usr/local/lib");

It doesn't find the library because you can't change "java.library.path" after starting the JVM. It just ignores your additional directory.

Everyone says set the environment variable LD_LIBRARY_PATH to (in my case) /usr/local/lib. This doesn't work either. On Linux and OSX at least Java ignores that variable when setting up java.library.path. In any case setting LD_LIBRARY_PATH globally for your application will screw up something else on your system. Not cool.

Third attempt. Set java.library.path on the java commandline:

-Djava.library.path=/usr/local/lib

Now you've changed the JVM so things could go wrong. Instead of having the system default library path where everything is, you've redefined it to a custom location. Unfortunately there's no universal way to ADD /usr/local/lib to java.library.path. So the best you can do is find the java library path on your system (by writing a Java program that outputs System.getProperty("java.library.path")) and then add /usr/local/lib to that value and finally specify the entire string to java:

-Djava.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:/usr/local/lib

This is what I have to do on Mac OSX. Of course it's entirely platform-specific, which is stupid for a programming language that is supposed to be platform-independent. On the other hand this yucky solution is the best one on offer. Since when you've finished developing you'll be running it time and time again on the same platform it probably doesn't matter much.

Alternatively, you could just put your library in the current directory, which will work on Windows (reportedly) and Mac OSX but not Linux.

Saturday, 24 December 2011

Installing couchdb on Mac OSX

Installing couchdb on Linux is a breeze. At least with the Debian package manager. But on OSX, where I was stuck over the Christmas break, installation is a pain. Of course if you believe the hype homebrew will save us. All you have to do is install it and type: brew install couchdb. Except that, it doesn't work. Packages have to be maintained, and unless the homebrew authors do all that work, their packeges will soon break. So I had to do it myself. This is the formula on 64 bit systems:

Download the latest Erlang source. Configure with --enable-darwin-64bit. Otherwise it compiles in 32 bit and it won't work with the other components, especially icu. Then make, make install as usual.
Download and install ICU (configure, make, make install)
Download and install couchdb. Configure, make, make install. And it all should work.

Now that Apple's great leader has passed on maybe someone in charge will see that a proper package manager would be a good idea for OSX.

About this blog

This blog is a technical record of my attempts to create a first class website for ecdosis.net. This will be a revision of www.digitalvariants.org and is intended to incorporate genetic texts in the MVD (Multi-Version Document) format. It will be the first website to allow the user to view and edit original texts with all their raw corrections, revisions, and variant versions as they were truly meant to be: as multi-version texts. A lot of people have talked about the theoretical possibility of doing this but the tools they choose are not up to the task. In fact the history of Digital Humanities is all about shoehorning humanistic problems into off-the-shelf technical solutions that don't fit. This project, on the other hand, is about breaking free from the limitations of mere markup and database structures to represent the true nature of originally analog documents.