Paratexts in PSEF format

What my reading about digital scholarly editions has revealed is that most people think of the scholarly edition as not only composed of a multi-version core-text, that is the thing being studied, but also a collection of single version paratexts. An example is the WoolfOnline site, where the distinction is made explicitly by the category 'contextual' as distinct from texts, images and bibliography. Contextual here is a collection of postcards and biographical information, but it could be anything: articles about the core-texts, photographs that are not facsimilies of the texts, a formal biography etc. There needs to be some way to wrap up this data into the digital blob that travels with the main edition. Otherwise, there is the danger that paratexts will become detached from their roots, since they may be explicitly linked to files in the archive/online edition. The updated top-level structure of the current PSEF (portable scholarly edition format) looks roughly like this:

• cortexs and corcodes: Each of these is identified by a docID, which is a relative path such as english/shakespeare/kinglear/act1/scene1". Each version within a document is given a version ID as well, expressed the same way, such as /folios/F1, which is the F1 version of the folios of that play. Put them together and you get a complete path that uniquely identifies that text and its markup. Documents are themselves split into two parts: cortexs and corcodes:
  • Cortexs are the basic plain text files. For each physical version there are potentially many sub-files, each representing a layer such as a normalised spelling layer, a version with abbreviations expanded, the base text after the first level of corrections etc. Each layer is a coherent plain text.
  • CorCodes are the standoff markup files, potentially many per plain text file. If you want to represent, say, structural markup and a set of references to locations in the text, then a set of links to an external image, each can be stored in separate files, and later merged to produce a single HTML file for viewing in a browser. This also keeps the text cleanly separated from the markup.
• CorPix is a collection of facsimile images of pages of specific texts. So several layers within a Cortex/Corcode document might refer to the same set of facsimile images, or to different images. Corpix files are referred in to the same way, via docIDs.
• Corform is a collection of formats in some stylesheet language such as CSS or XSLT. They also have docIDs, and are referred to by the CorCodes, which need a default rendering format (although you can substitute another or add an extra one).
• Config is a collection of JSON files with configuration parameters for any part of the edition, usually for importing. These files can be used to specify import filters, or to provide long version names for the short versions used in docIDs.
• Misc: this stores the paratextual files, again using the simple docID scheme, but without the complication of multiple versions.

In addition, the Cortexs and Corcodes can be entirely replaced by the format of your choice. Currently the psef-tool supports HTML, TEXT (described above), XML and MVD. So that, using the plain text and HTML formats alone, any ordinary tool can be used to read a psef-archive, so it can claim to be interoperable - usable in many programs. No special tools are needed.

Distributed Digital Scholarly Editions

A certain scholar in the field has been talking about distributed DSEs for years. I make no claim as to the originality of the basic idea, but the implementation is a custom extension to AustESE. The basic problem is this. When you create a DSE you rarely have ALL the data in one place. If I give you a DSE of Charles Harpur, do I include all the images of all of his manuscripts? That would be several gigabytes. You probably wouldn't like that. But let's say I gave you 3 megabytes instead, being all the text and commentary only. The images could just be stored via links or pointers to the real data. This is the case with manuscripts in Europe now, because they are all stored on Europaeana.

The specific situation I am thinking of is a biography of an author that is stored on a different server, and making a copy of it is simply out of the question, because it is subject to constant editing and we want just one copy to exist. But what if on MY server I stored a link that pointed to it. I could treat the file just the same as the real resources that reside there. In other words I could have a virtual file system that covered both real files, such as "english/harpur/The nevers of poetry", which would contain an MVD and the text of the poem, but also a file called "english/harpur/biography", which would be a LINK to "http://austese.net/sites/all/modules/austese_repository/api/events/", with a parameter of project=21 and a username and password needed to get it. But if I put all that information into the LINK then I could treat the LINK as if IT was the biography and forget about the details of how to fetch it.

I realise this sounds a lot like handles - intermediary URLS like doi://, or like a file system on disk with soft and hard links. The difference is that it is implemented as a virtual file system as part of a digital scholarly edition. The advantage is then that I can annotate those virtual files even if their address changes, and I can have all the files that make up the 'distributed' edition in one place - the real and the virtual. The real advantage is then that you can treat local and remote files as if they were the same thing for the purposes of editing, updating, annotating, reading, etc.

Java stack size and the Nmerge memory limitation

My Nmerge program runs within a JVM whose maximum memory is set to 1GB. That seemed to be plenty but it still ran out of something whenever I gave it more than about 50K of text to digest. I always thought this was because Java was such a memory-hog. And it is. But that wasn't the reason it was failing for years. The simple answer was that increasing memory space in Java via the -Xmx option doesn't increase the stack space in proportion. Stack space is controlled by the -Xss option, and the default stack size is just 512K. Now that's plenty for most applications but nmerge was no ordinary program. At its heart were two mutually recursive procedures. I proved mathematically that they would terminate and produce a valid result, but what I couldn't estimate was the maximum stack size they would require. It turns out that the stack size needed is proportional to the length of the file. Or more precisely it was the number of 'nodes' in the graph. A short file with a complex graph structure (and lots of variation) might need more stack space than a longer file with relatively little variation. Whatever the reason, increasing the stack space to 4MB allowed the program to deal with much larger files. And I had a Homer Simpson D'oh! moment.

Splitting TEI files into coherent layers

Embedded markup prevents transcriptions from being interoperable, because the tags are chosen on the basis of human interpretation, not predictable machine instructions. Removing embedded markup from text, however, is fraught with problems, particularly if tags are used to record variant versions. Recently I have had to revisit this problem that I thought was already solved. It wasn't. I needed to rethink it and hammer out a reliable algorithm for the mountains of very variable data it was likely to deal with. It's important because without it we'll never be able to build interoperable tools in the digital humanities because transcriptions deal with multiple versions all of the time.

Taking the markup out of a text that contains alternatives naïvely produces garbage. A simple example: <del>dog</del><add>cat</add> would produce 'dogcat'. It is much better to store markup and text separately, and to separate out the versions into complete, readable texts. If desired, differences between versions can then be computed, and merged, so that a compact representation of a document can be created to act as a focus for queries. For example: 'what are the changes carried out by the author as the first layer of correction, or the second layer?' 'What are the differences between the first and second complete drafts, the differences between MS A and MS B?' 'Markup the text of the Bloggs edition using this markup set and that markup set combined and send it to a book format'. All these operations are at best inefficient using embedded versions, and at worst impossible. But it is not actually necessary to merge the variant texts. Just storing them as separate versions, although less efficient than the merged representation, still allows applications to access coherent layers of a multi-version document using simple applications.

Taking versions out of the text

One of the main functions of the Versioning Machine is to do exactly this. But it's hard to do well. What I am going to describe is a fourth attempt to do it properly, using only clearly defined properties of a TEI-encoded text. So here's my main concept of version 4 of the splitting algorithm:

Clusters and siblings

Clusters

A cluster is a fragment of a DOM tree containing tags that describe embedded versions of a text. In the TEI schema the tags <mod>, <subst>, <choice>, <sic> and <corr>, <add> and <del>, <abbr> and <expan>, <orig> and <reg>, <app>, <lem> and <rdg> describe such textual phenomena. Of these <mod>, <subst>, <choice> and <app> can be more or less ignored, because all they do is bracket the other tags. <subst> is used, for example, to group <add>s and <del>s. I'm going to call the other tags siblings, even though the term may also describe ordinary siblings of unrelated elements in a DOM tree. In my case a sibling means an element that is one of <sic> and <corr>, <add> and <del>, <abbr> and <expan>, <orig> and <reg>, <lem> or <rdg>. The set is not fixed, however, and further elements that behave in the same way could be defined, or these could be renamed. A sibling can either be adjacent to another sibling or be its descendant or ancestor, because that's all you can do in a tree. (I'm discounting interlinking elements via attributes because this is non-computable in the general case when used to record versions.)

Identifying a cluster in a tree is easy. A sibling, once encountered, is constructed depth-first (that is, children first, then siblings at the same level). It doesn't matter if a direct child is not a member of the sibling set. That doesn't break the cluster, because even if no children are themselves siblings, they are still affected by the choice instigated by their parents. They are still conditional text. What breaks a cluster and sets it off from the rest of the text is the occurrence of a non-sibling element next to a sibling at the top-level of a cluster. Or the end of the list of siblings itself. So when, for example, a <rdg> is the last in an <app> series. A cluster may also be a single element, such as a lone <del>. It's still a cluster of size one.

Siblings are also numbered using the depth-first ordering. The first <rdg> encountered can be called 'rdg0', and the second 'rdg1' etc. But unconnected siblings at the same level (such as two independent deletions in the same line) are given the same version-tag. All non-sibling elements outside of any cluster automatically receive the 'base' version to indicate that they belong together implicitly to 'all' versions.

Four rules are needed to separate versions for display or extraction.

Rule 1

All the ancestors of a sibling belonging to version v must also belong to version v, because otherwise ancestors would not be visible if that version was selected.

Rule 2

In any cluster the first alternative at the top level, except if it is added, inherits the 'base' version. If the first alternative has child elements that are not siblings then those elements inherit the base version likewise. If they are sibling elements then Rule 2 applies to them recursively.

Rule 3

In any cluster the final uncancelled or uncorrected alternative at the top level inherits all uncancelled versions in the document that do not already occur in the cluster itself. Like Rule 2 this applies recursively to child elements.

Rule 3 does not apply if a set of alternatives does not have any uncancelled alternative at the top level or if all its alternatives use the wit attribute to manually specify versions.

Rule 4

The first sibling in each sibling-sequence inherits the versions of its parents within its own cluster.

Sustainability of software in the digital humanities

Everyone knows that it is very difficult to maintain whatever you create in this field, let alone make any progress. And once progress has been made, the result breaks very quickly. We have seen this scenario played out countless times but I'm not going to point fingers. That's just how it is. I am reminded very much of Alice through the looking glass where the Red Queen takes Alice by the hand and runs with her for some time as fast as she can go. But when they finally stop Alice realises the horrible truth:

'Why, I do believe we've been under this tree the whole time! Everything's just as it was!'

'Of course it is,' said the Queen, 'what would you have it?'

'Well, in OUR country,' said Alice, still panting a little, 'you'd generally get to somewhere else—if you ran very fast for a long time, as we've been doing.'

'A slow sort of country!' said the Queen. 'Now, HERE, you see, it takes all the running YOU can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!'

The answer to this dilemma is not to complain that you don't have enough resources, but simply to program economically. First put everything you can into the most stable software platform you know: in my case the C language. Much of the computation you need to do can be expressed this way. Programs that compute phylogenetic trees, compare and format texts, load and save formats, in fact anything you need to do that performs basic operations that you can code once, debug and forget. I have C programs from the 1980s that still compile and run flawlessly. The modern programmer seems to have forgotten that C is the bedrock of everything they do: every scripting language they use, every operating system (OK, except Windows, which is C++ I believe) is written in this. It cannot go away or the entire world would cease to be. And it doesn't change. If you don't have any dependencies, if your code just computes something then you can write, debug and forget. That saves a lot of maintenance work.

Secondly, leverage existing open source code. Write your extra stuff to use existing CMSes and other platforms, with as little glue code as you can manage. Let the wider community maintain that platform for you. Don't expect that your service will last. It won't. But you can call the C-code from the service; in fact you can call C-code from any language and compile it for any platform, so that the service need only be a wrapper around core functionality. Be ready for change: expect it. Then when you have to rewrite your code it will just be a matter of rewrapping it all in the latest style.

Thirdly, don't expect that the open source community is going to either fix your code or maintain it for you, because they won't. You have to be prepared to do it all yourself, at least until you have a great product that everyone is using, like Linux. So far no one in the digital humanities has got that far. And maybe they never will. You have to believe in what you do, and not be dependent on only working when you have grant money.

Exporting a digital scholarly edition

One of the most often requested features in a digital scholarly edition service is the ability to export data produced within it. This is a more complex problem than importing, which typically happens in stages: one adds some source texts, then marks them up, then adds annotations, images and formats to change its appearance and function over time. Getting all that data back out again into one archive that can be transported to a new site, or transferred to a different program can be a bit messy. Also adopting a radically new way to store texts for editing, viewing and comparing makes it harder to just "export" the stuff.

So to overcome this I have added a PDEF (portable digital edition format) export function to calliope. This queries the server for all the files, markup views, annotations, images and formats, and downloads them all in a single .zip or .tar.gz archive. At the moment this is just a service in Calliope: the user invokes the calliope service e.g. in curl:

curl -X GET http://localhost:8080/pdef/?DOC_ID=english%2Fshakespeare%2Fkinglear.*\&FORMAT=TEXT\&FORMAT=MVD -o archive.tar.gz

This downloads a file, whose structure is: archive/+english/shakespeare/kinglear/act1/%scene1/MVD/corcode/default archive/+english/shakespeare/kinglear/act1/%scene1/MVD/cortex.mvd archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/Base/F1 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/Base/F2 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/Base/F3 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/Base/F4 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/Base/Q1 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/Base/Q2 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/corcode/default/versions.conf archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/Base/F1 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/Base/F2 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/Base/F3 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/Base/F4 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/Base/Q1 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/Base/Q2 archive/+english/shakespeare/kinglear/act1/%scene1/TEXT/cortex/versions.conf archive/@corform/TEI/default archive/archive.conf

This archive can then be uploaded using the mmpupload tool (or will be when I have modified it) to a new repository, effectively installing the DSE in a new location. All the user has to provide is a wildcard document identifier -- in this case "english/shakespeare/kinglear.*" and it will download all the files with that docid prefix. (Because this is a URL the "/"s have to be escaped as "%2F".) Upload supports two other formats: XML and MIXED. MIXED allows text and XML files to be freely intermingled.

Calliope 0.2.1: Moving to Mongo and Tomcat

This project is about Digital Scholarly Editions or DSEs. These are Web-based publications of cultural heritage texts. My design for DSEs calls for a REST Web Service, originally called hritserver, but now renamed Calliope, which provides the components for building a DSE within a Web authoring environment like a CMS. The infrastructure looks like this:

As you can see, at the bottom there is a database, including an images handler. Currently Calliope uses CouchDB and stores the images in the file system for speed. But I now have made it possible to define a single Java class to extend Calliope to handle other databases. MongoDB, a rival NoSql database to CouchDB, is supposed to be faster. Unlike Couch it handles images well, through the GridFS module, making replication of the entire database easier. In Couch you have to attach images to documents, and word has it that when those images are large it becomes sluggish. One advantage of Couch, though, is its revision control, which Mongo lacks. In Mongo, your latest copy of a document is your only copy.

The architecture has also a number of canonical views, which are only suggestions. They can be extended, customised and redecorated. They also know nothing about which database they are running on. They communicate with the Calliope service, which gives them partial Web-pages, which they use to compose the views. This is thus a pure example of the familiar Model-View-Controller paradigm.

Another improvement in version 0.2.1 is making it able to run equally well in Jetty embedded mode or Tomcat. The difference is that in Jetty the web application is part of a runnable application that makes it easy to debug and set up, as well as launch. Tomcat uses a plugin type of architecture where Web applications are stored in WAR files (really ZIP or JAR archives), which are really just directories with configuration information. Tomcat acts as a Web Server and hands off control of everything in the WebApp's context to the stuff unpacked from the WAR file. So one difference is that the URL for the Jetty test service is http://localhost/tests/ and for Tomcat at the moment it is http://localhost:8080/calliope/tests/ (Calliope is the name of the WAR file). So what I have now is a combination of two ways to launch Calliope: via Jetty or Tomcat, using either of two databases, Couch and Mongo, so there are a total of four possible configurations. This kind of flexibility helps others to consider adopting your software.