Managing sites with automatic upload: mmpupload

An archive site obviously needs to be maintained. There are various aspects to that, such as checking that it is still up, backing up the data and introducing new data. Towards solving the backup problem I have been writing some scripts for the Harpur archive that will automatically upload the entire site. My idea eventually is to manage several sites, each consisting of a folder of files, and uploading and downloading (for backup) so that it will be easy to change the content without having to go through the tedious GUI-based editing tools such as form-based file import. It is this last problem that I have been trying to automate.

The situation with Harpur is common to many archives: a set of works collected from various physical texts. Each is a folder containing files to be merged. In my case they are all XML or text files, and the import service in hritserver can handle both formats, even within the one work. Also there are plain files such as anthologies, and biographies, which only exist in one version. I thought it would be easy to script the actual upload using a tool like curl. However, curl presents problems in emulating mime/multipart upload such as that generated by a HTML form. I wanted to emulate that precisely because I wanted the user also to use a form on an 'import' tab to upload individual works. Such added works would then get backed up and eventually join the main/upload download cycle. Also curl is a complex tool that supports many features that get in the way of simple uploads. It is sensitive to spaces and brackets in filename paths, and I wanted my folders to have natural names like 'GENIUS LOST. THE SORROWS PART EIGHT WHITHER' and 'FRAGMENT (1)'. You have no idea how hard it was to write a script that builds a set of path names like that. After trying for a week I decided to write my own upload tool, which I called mmpupload.

It is a simple tool that takes a list of file names or paths, and a set of name-value pairs, wraps them up into a mime multipart document, sends it to a named service and reports the response to the console, and also whether it worked or not. With Harpur I already have 259 folders and there are rumoured to be around 700, so this tool is a key component of the automated archive solution. Each archive has to be a virtual machine that supports ssh and Java, so I can run hritserver. rsync can then be used in combination with ssh to update the site on a regular basis. I already have three planned: the Harpur archive, possibly a Conrad archive, Tagore and Digital Variants. If I can manage all these - and it's just a question of managing the scale - I can add others in time without going mad. That's the beauty of a software solution that is mostly shared by different projects and entirely automatable.

Here's an example of mmpupload:

mmpupload -f AUTHOR=harpur  -f "WORK=Eden Lost" -u http://localhost/upload.php "EDEN LOST/C376.xml" "EDEN LOST/A88.xml"

Simple, huh? There's even a man page and it makes correctly on OSX and Linux.

couchdb: delete document if it exists

I am uploading some files via a script to a couchdb database. In many cases the files are already there, but out of date. So I want to replace them. However, couch complains that there is an 'update conflict', even if I explicitly try to delete the resource without supplying its revision id. So I followed the instructions to just test that a document is present, extracted the rev-id and then composed a delete operation. In bash scripting language, and using curl as the http transmitter it looks like this:

erase_if_exists()
{
    RESULT=`curl -s --head $1`
    if [ ${RESULT:9:3} == "200" ]; then
      REVID=$(expr "$RESULT" : '.*\"\(.*\)\"')
      curl -X DELETE $1?rev=$REVID
    fi
}

Why HEAD? Because HEAD retrieves only the HTTP document headers but not the document, which could be large. Curl can send a HEAD request using curl -X HEAD but it hangs on reading the response, so the above formulation is an alternative that works. The -s tells it not to print out any download progress, and in this function $1 is the URL. The HTTP response code is extracted from the RESULT string using indexing. If it is 200, the document is present. Here's a sample response:

HTTP/1.1 200 OK
Server: CouchDB/1.1.1 (Erlang OTP/R15B)
Etag: "3-04bacb883737ffed82700eacdf4e74f2"
Date: Fri, 18 May 2012 21:17:04 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 8077
Cache-Control: must-revalidate

The REVID gets extracted from the Etag property via a regular expression, and in the example is set to 3-04bacb883737ffed82700eacdf4e74f2. This is passed in another call to curl as a URL parameter. You also have to embed the username and password into the URL for this to work, or you can use the -u option to specify a username.

Bare bones multi-file upload

What I need for HritServer is an import facility that can handle multiple file uploads. I've found many fancy scripts for doing this that are way too complex and too specific. Sometimes all you need is a bare bones solution you can tailor to suit your needs rather than a fully fledged product you have to spend ages understanding and cutting down to size. Also I hate building in dependencies that only increase the tendency for code to break. So here's my simple contribution to the HTML multifile upload problem.

The basic idea is to have one <input type="file"> for every file you want to upload. But you hide all but the current empty one. So when the user selects the only visible input file element it sets itself to the chosen file, adds itself to an invisiible list of input file elements, and creates a fresh one for the next time. To make it easier to see what's already been selected I maintain a secondary list of paths in a table. Next to each entry is a remove button that let's you take out individual entries. That's it. Here is the php code to handle the upload on the server side. Replace this with something else if you like, such as Apache file upload in Java:

<?php $num = count($_FILES['uploadedfile']['name']); for ( $i=0;$i<$num;$i++ ) { $name = $_FILES['uploadedfile']['name'][$i]; $type = $_FILES['uploadedfile']['type'][$i]; $size = $_FILES['uploadedfile']['size'][$i]; $tmpname = $_FILES['uploadedfile']['tmp_name'][$i]; if ( strlen($name)>0 ) print "<p>uploaded name: $name size: $size type: $type temp name: $tmpname</p>"; } ?>

And here is the HTML, with self-contained javascript.

<html> <head> <style type="text/css"> input.invisible { display: none } </style> <script type="text/javascript"> function remove( value ) { var repo = document.getElementById("repository"); var child = repo.firstChild; while ( child != null ) { if ( child.nodeName=="INPUT"&& child.value==value ) { child.parentNode.removeChild(child); break; } child = child.nextSibling; } // now look at the table var table = document.getElementById("listing"); child = table.firstChild; var finished = false; while ( child != null && !finished ) { if ( child.nodeName=="TR" ) { var gchild = child.firstChild; while ( gchild != null ) { if ( gchild.nodeName=="TD" ) { var ggchild = gchild.firstChild; if ( ggchild!=null&&ggchild.nodeName=="SPAN" ) { if ( ggchild.textContent!=null&&ggchild.textContent==value ) { finished = true; child.parentNode.removeChild(child); break; } } } gchild = gchild.nextSibling; } } child = child.nextSibling; } } function doaddfile() { var input1 = document.getElementById("input1"); var repo = document.getElementById("repository"); var listing = document.getElementById("listing"); var parent = input1.parentNode; var input2 = document.createElement('input'); input2.setAttribute( "type", "file" ); input2.setAttribute( "name", "uploadedfile[]" ); input2.setAttribute( "onchange", "doaddfile()" ); input1.setAttribute( "class", "invisible" ); input1.removeAttribute("id"); input2.setAttribute("id","input1"); parent.removeChild( input1 ); parent.appendChild( input2 ); repo.appendChild( input1 ); // now create the row in the listing table var text = document.createTextNode(input1.value); var span = document.createElement("span"); span.appendChild(text); var row = document.createElement("tr"); var cell1 = document.createElement("td"); cell1.appendChild(span ); row.appendChild( cell1 ); // add a button to remove the entry var button = document.createElement("input"); button.setAttribute("type","button"); button.setAttribute("onclick","remove('"+input1.value.replace(/\\/g,"\\\\")+"')"); button.setAttribute("class","remove"); button.setAttribute("value","remove"); var cell2 = document.createElement("td"); row.appendChild( cell2 ); cell2.appendChild( button ); if ( listing.firstChild == null ) listing.appendChild( row ); else listing.insertBefore( row, listing.firstChild ); } </script> </head> <body> <form enctype="multipart/form-data" action="/upload.php" method="POST"> <div> Choose a file to upload: <input id="input1" onchange="doaddfile()" name="uploadedfile[]" type="file" /><br /> </div> <table id="listing"> </table> <div id="repository"> </div> <input type="submit" value="Upload File" /> </form> </body> </html>

Call the first "upload.php" and the second "upload.html" then put them both in the root document directory of your web-server. If you have php installed navigate to /upload.html and take it from there. You can add styling and change the server script easily because it is simple.

Uploading a directory of images to couchdb

I wanted to have a general script to upload a set of images to couchdb. Couch can't store images as documents because it uses JSON for that. But you can still create a JSON document and attach a set of annotations to it in the form of images. But here's the catch: you have to specify the document's revision id, and that changes after every image you add.

But I had two problems. My directory of images contained sub-directories. I also wanted to access the images directly using a simple URL. So if I had a directory structure like:

    list
        file1.png
        one
            file2.png
            file3.png
        two
            file4.png

I would want the relative URLs to be: /list/file1.png and /list/one/file2.png etc. The trick in writing the script is to extract the revid from the server response. I used awk for that, then used the returned value to upload the next entry in that directory. The second trick is to use %2F not / as a directory separator when creating the docids. Couch doesn't allow nesting in the database structure but you can simulate it by creating documents called:

list
list%2Fone
line%2Ftwo

The first posting to each of those documents doesn't need a revid, but subsequent ones do. That's just a feature of couch. So here's the script. It's a bity sloppy because if couch responds with an error to any upload it will fall over. This bit of error-handling is currently left as an exercise to the reader. I'll put it in later and may update the post then. To use the script put it into a file called upload.sh and then invoke it thus: ./upload.sh images, where "images" is the master image directory.

#!/bin/bash contains() { echo `echo $1 | grep -c $2` } docid() { echo `echo $1 | awk '{gsub(/\//,"%2F",$1);print $1;}'` } filename() { echo `echo $1 | awk '{l=split($1,a,"/");print a[l];}'` } upload() { local REVEXPR="" if [ $3 ]; then REVEXPR="?rev=$3" fi local FILENAME=`filename $f` local RESPONSE=`curl -X PUT http://user:password@localhost:5984/corpix/$1/$FILENAME$REVEXPR --data-binary @$2 -H "Content-Type: image/png"` echo `echo $RESPONSE | awk '{split($1,a,"\"");print a[10]}'` } descend() { local DIR=$1 local REVID="" local DOCID=`docid $DIR` for f in $DIR/* do if [ -d $f ]; then descend $f elif [ -f $f ]; then local CONTAINS=`contains $f ".png"` if [ $CONTAINS = 1 ]; then REVID=`upload $DOCID $f $REVID` fi fi done } descend $1

The importance of deletions

Standoff properties describe what happens to runs of text, but what about runs of text you don't want, or need to put into attributes when it is formatted as HTML? This problem was revealed when I tried to use formatter to generate a dropdown menu from a list of versions spat out by nmerge. What nmerge gave me was the description of the work on one line, then the name of the version-group, followed by short names and long names for each version:

King Lear
Base F1 Version F1
 F2 Version F2
 F3 Version F3
 F4 Version F4
 Q1 Version Q1
 Q2 Version Q2

But I needed to turn this into HTML that looked like this:

<select name="versions" class="list">
<option title="Version F1" class="version">F1</option> 
<option title="Version F2" class="version">F2</option> 
<option title="Version F3" class="version">F3</option> 
<option title="Version F4" class="version">F4</option> 
<option title="Version Q1" class="version">Q1</option> 
<option title="Version Q2" class="version">Q2</option> 
</select>

It's obvious that strings like "Version F1" need to be moved from the text into attributes, and strings like "Base" and "King Lear" need to be deleted outright. No doubt XSLT wizards are laughing at me, pointing out that if only I had embraced XML I would be able to transform XML into exactly this format easily. But choosing a simpler model for representing textual properties doesn't mean any loss of functionality. In fact "less is more". All it needs is for these problems to be thought through and solved.

All you have to do is allow each standoff property to be "removed". I already had this feature but the length had to be zero. I used this for removing the TEI header from TEI documents and saving the content in an annotation of an empty range. But the length had to be zero, and the text the elements formerly enclosed was removed permanently from the base text. Since I wanted to reuse the same text for other applications I didn't want to remove any text permanently. So I just allowed the length of a "removed" property to be > 0. A deleted property now removes the text it affects from the output, and also any ranges that it encloses, as well as parts of any ranges it overlaps with. So in the JSON all you do is define some "empty" ranges that have the "removed" property set to true:

{ "name": "list", "annotations": [ { "name": "versions" } ], "len": 94, "reloff": 10 },{ "name": "empty", "removed": true, "len": 5, "reloff": 0 },{ "name": "top-group", "annotations": [ { "name": "Base" } ], "len": 88, "reloff": 5 },{ "name": "version", "annotations": [ { "description": "Version F1" } ], "len": 2, "reloff": 0 },{ "name": "empty", "removed": true, "len": 10, "reloff": 3 }...

What this machine-generated JSON says is that the string "Base" should be deleted. Long version names like "Version F1" should be moved to attributes called "description" for the short-names. The resulting HTML given above is exactly that output by formatter. So now we can also pick and choose which text gets selected and where it appears.

Adding a directory to java.library.path

In addition to what I said before, there is actually a cool way to add a directory to java.library.path. If you launch your application via a script you can execute a commandline Java tool to output the current library path. For example, this simple Java program adds "/usr/local/lib/" to that path and prints it without a trailing CR to the console:

public class LibPath
{
    public static void main( String[] args )
    {
        System.out.print(System.getProperty(
            "java.library.path"));
    }
}

To use it just compile it, and then invoke it in the script:

LIBPATH=`java LibPath`:/usr/local/lib
java -Djava.library.path=$LIBPATH ....

And as if by magic the java library path acquires a new directory before your prorgam is run.

Posting multipart form data

For testing I needed to simulate a web-browser doing a file-upload. I tried to Google this but I couldn't find a suitable answer that worked. So I rolled my own. This might save other people some trouble too. First comes the MIMEMultipart object, which stores the body of a multipart post:

import java.io.FileInputStream;
import java.io.File;

public class MIMEMultipart 
{
    StringBuilder text;
    static String CRLF = "\r\n";
    String boundary;
    public MIMEMultipart()
    {
        text = new StringBuilder();
        boundary = Long.toHexString(
            System.currentTimeMillis()); 
    }
    public String getContent()
    {
        return text.toString();
    }
    public String getBoundary()
    {
        return boundary;
    }
    public int getLength()
    {
        return text.length();
    }
    public void putStandardParam( String name, 
        String value, String encoding )
    {
        StringBuilder sb = new StringBuilder();
        sb.append("--" + boundary).append(CRLF);
        sb.append("Content-Disposition: form-data; "
            +"name=\""+name+"\"");
        sb.append(CRLF);
        sb.append("Content-Type: text/plain; charset=" 
            + encoding );
        sb.append(CRLF);
        sb.append(CRLF);
        sb.append(value);
        sb.append(CRLF);
        text.append( sb.toString() );
    }
    public void putBinaryFileParam( String name, 
        String fileName, String mimeType, 
        String encoding ) throws Exception
    {
        // compose the header
        StringBuilder sb = new StringBuilder();
        sb.append( "--"+boundary );
        sb.append( CRLF );
        sb.append("content-disposition: form-data; "
            +"name=\"" );
        sb.append( name );
        sb.append( "\";  filename=\"");
        sb.append( fileName );
        sb.append( "\"" );
        sb.append( CRLF );
        sb.append("Content-Type: "+mimeType ); 
        sb.append( CRLF );
        sb.append("Content-Transfer-Encoding: binary");
        sb.append( CRLF ); // need two of these
        sb.append( CRLF );
        text.append( sb.toString() );
        // now for the file
        File input = new File( fileName );
        FileInputStream fis = new FileInputStream(input);
        byte[] data = new byte[(int)input.length()];
        fis.read( data );
        fis.close();
        text.append( new String(data,encoding) );
        text.append( CRLF );
    }
    public void finish()
    {
        text.append( "--" );
        text.append( boundary );
        text.append( "--" );
        text.append( CRLF );
    }
}

To call it, open a standard Java URLConnection:

private static void printResponse( 
    URLConnection conn )
{
    try
    {
        InputStream is = conn.getInputStream();
        while ( is.available() != 0 )
        {
            byte[] data = new byte[is.available()];
            is.read( data );
            System.out.println(new String(data,
               "UTF-8"));
        }
    }
    catch ( Exception e )
    {
        e.printStackTrace( System.out );
    }
}....
URL url2 = new URL("http://localhost:8080/strip");
URLConnection conn = url2.openConnection();
MIMEMultipart mmp = new MIMEMultipart();
mmp.putStandardParam( Params.FORMAT,Formats.STIL,
    "UTF-8" );
mmp.putStandardParam( Params.STYLE,"TEI/drama",
    "UTF-8" );
mmp.putBinaryFileParam( Params.RECIPE,"recipe.xml",
    "application/xml","UTF-8" );
mmp.putBinaryFileParam( Params.XML,
    "act1-scene2-F1.xml",
    "application/xml","UTF-8" );
mmp.finish();
conn.setDoOutput(true);
conn.setUseCaches(false);
((HttpURLConnection)conn).setRequestMethod("POST");
conn.setRequestProperty("Accept-Charset", "UTF-8");
conn.setRequestProperty("Content-Type", 
    "multipart/form-data, boundary="
    +mmp.getBoundary());
conn.setRequestProperty("Content-Length", 
    Integer.toString(mmp.getLength()));
OutputStream output = conn.getOutputStream();
output.write( mmp.getContent().getBytes() );
output.flush();
output.close();
// get response and print it
printResponse( conn );

Extend MIMEMultipart if you like by adding a method for plain text files and other types of part.