View source for TheHague112012Minutes

== Morning of 21/11/2012 ==

For names, refer to the [[TheHague112012Participants|list of participants]].

=== Discussion on text modelling ===

Two main alternatives to XML:
* range based model (for annotations)
* variant graph (for collation)
** Tara is using (with [http://collatex.sourceforge.net/ CollateX]) a graph-based model more complex sthan Schmidt's one.

Paolo is interested in a text model that can represent textual layers (graphical, alphabetical, linguistic; see Orlandi, [http://books.google.it/books?id=GLzFSAAACAAJ Informatica testuale])
* Tara: we can stretch Schmidt's graph model to represent the different Orlandi's layers. They're still graphs
* Gregor: variant graphs are not suitable for this; they're not variants. LMNL (range-based model), instead, seems to be fit for [http://books.google.it/books?id=GLzFSAAACAAJ Orlandi]'s "textual layers".

=== Gregor's presentation ===

Gregor describes his implementation of a range-based (LMNL-based) textual model, used so far for the back-end of the [https://faustedition.uni-wuerzburg.de/dev/project/about Faust Project].
* the implementation is in Java
* annotations that have a name and are namespaced (this comes from XML)
* text is a sequence of events
* layers have names like "TEIW" or "Europeana" and contain texts
* layers have anchors, so one text can point to another or to ranges of another text
* one pointer can point to multiple anchors. E. g. layer 'alignment' points to two different anchors (and aligns them)
* layers can include whatever data (Json, an XML file etc.)
* a TextRepository is just a collection of those layers. It's something I can query
* you can create a graph of the layers existing in a text repository
* TextStream
** Gregor's model is differnt that XML
*** the SAX API works with XML trees
*** you can walk through the tree there
*** a range-based model, isntead does not have such easy stacks
*** How do you transform XML into range-model? Any element (with opening and closing tags) becomes a range

=== Each participant's agenda ===

A round-up on the interests of each of the participants:
* practicality: what can we build on top of e.g. a range-based model (from the datastore to the presentation layer)
* query/search functions on top of a text model
* variant graph vs. range-based models
* processing (equivalent to XSLT?), querying (equivalent to XPath/XQuery)
* variant graph: traversal patterns?
* interfaces, APIs, JS libraries
* problem of variation and how it is handled on different (conceptual) layers of a text
* common model? can we find a generalized model incorporating features from the variant graph and a range-based model
* bridge the gap!
* integration scenarios; import of existing data, multiple use cases on top of those (what is the smallest thing that could possibly work? -- how do we get there)

=== Division of labour ===

See [[TheHague112012DivisionOfLabour]]

== Afternoon of 21/11/2012==

=== Bootcamp's agenda and goals ===

Interoperability is best achieved through web services. The same does Gregor's Java system (that he's showcased this morning). Also [http://collatex.sourceforge.net/ CollateX] does so now.

[http://www.juxtasoftware.org/ Juxta] today shows its underlying range-based model only if you use the API. For the standard user using juxtacommons.org, it's just XML-in/XML-out.

Moritz and Gregor discuss the JavaScript querying prototype they already built for Java. Client-side or server-side?

Ronald: is a DSL (domain specific language) the way for us to go?

'''Bootcamp's goals:'''
* We want to use the TEI palette (vocabulary) to build a non-XML range-model-based annotation of plain text
* We want to build tools (a Doman Specific Languge) for texts annotated this way, so we foster the development of this kind of texts encoding/annotation model (alternative to XML)
* Our goal is to make a console on a website, rather than simply an API
** so the user can test the potential of a querying language that queries texts marked with a range-based model

The point of doing of this is:
* overcoming overlapping
* exploiting the recursive potential lf LMNL (annotating annotations)
** there's a prototype (with a console) in http://www.piez.org/

We should create a web interface. Gregor sets out an example of its workflow:
* the text repository: textrepo.net
** POST
** <xml/>
* creation of a new text: textrepo.net/12
** GET
** PUT → 2201 creator
* creation of an annotation to text 12: textrepo.net/13
** (but bear in mind that  annotations are texts in their own respect)
* a query on text 13: textrepo.net/13?q=(and...())
* a query on the whole repository (all texts in the repository)
** textrepo.net/?q=...

=== Textual layers at Faust project ===

(This is not directly related with this Bootcamp's project)

At Faust's project they're keying/transcribing a MS twice: one transcription for the diplomatic layer, one transcription for the 'linguistic' (regularised) layer.
* ''Issue'': how do you align the two texts? They do it now by collating them.
* ''Open issue'': how will they eventually store (e. g. in XML) this alignment? They've been collaborating with TEI SIG on genetic editions for this, but they have not yet reached a solution for such granular alignment (word-level granularity), so there is no solution yet as to how to store the alignment in XML/TEI. Paolo proposes to create one XML file for each layer (one for the diplomatic layer, one for the 'linguistic' layer, both texts being encoded at 'w'/word level), and a third XML file including only the linking of single words in XML transcription file 1 (diplomatic) with words in XML transcription file 2 ('linguistic').



== Morning of 22/11/2012 ==

=== Recap on yesterday's discussion ===

* abstract text model
** why we need a new one (other than XML)
* two main issues
** text variation
*** graph-based model
** annotation
*** a range-based model
*** overlapping is not a problem any longer
* goal: we want to build a web service, a text repository implementing a LMNL textual model

Since today, we're splitting in two groups (see [[TheHague112012DivisionOfLabour]]):
* Back-end (Java)
* Front-end (Javascript)

=== "Immutable texts" open issue ===

''Issue'':
* Gregor's model is based on the assumption that texts in the text repository are immutable. Any time the editor adds a word, the system stores a new text (with a new ID, a new URI)
* It is hard to migrate annotations to text 1 into corresponding annotations to text 2
* But editors normally edit one sentence (text 1), then annotate that sentence. Then, they add another sentence (which creates text 2). In this case, what happens to annotations to annotations to text 2?

=== Front-end: search for related projects ===

We look for other (open source) DH projects possibly doing what we want to do, so we can build on what they already have done.
* [http://textusproject.org/ Textus]
** connected with Europeana (RTF for cultural artifacts)
** they make you do arbitrary annotation (through XPath-like-identified ranges) to any webpage
* Is [http://standoffmarkup.org/ Standoff Markup Editor] doing what we want to do?
* Marco is working on [http://www.catma.de/ CATMA]
* Textus (usability issues)
* AnnotateIt (compilation issues) http://okfnlabs.org/annotator/demo/
* PundIt http://thepund.it/
* http://www.awesomehighlighter.com/
* http://brat.nlplab.org/

And the winner is... [http://okfnlabs.org/annotator/demo/ Annotator]!
Here is an installation guide written by Ronald: [[InstallAnnotateIt]]

Arash installed Annotator into his server. This is what he emailed us:
"Here is my annotator page running:
    http://www.arsa-apps.net/annotator/
ElastichSearch is running and accessible through port 9200:
     http://www.arsa-apps.net:9200
Annotator-Store is also running and accessible through port 5000:
    http://www.arsa-apps.net:5000
I couldn't bind my HTML-Page to the Store yet..."

''Issue: how do we markup ranges in the DOM?''
1. By inserting spans in the DOM via JavaScript? But when the user inserts a span (annotation) within another span (annotation), then JavaScript will count the characters offset from the closest parent (the older annotation/span). A solution might be http://stackoverflow.com/questions/4811822/get-a-ranges-start-and-end-offsets-relative-to-its-parent-container
2. Better to mark the span through milestones, and let JS visualise the annotation?

''How should JS visualise the annotation?''
1. highlight? What if 2 annotations overlap?
2. by inserting parentheses? But they count as characters in the DOM (so they mess up the offset calculation);
3. by inserting IMGs (no characters)! Only problem: the page layout (interlinear space etc.): if the user scales up the page font size, the IMG should scale along properly

Solution 3 seeems OK. These SVG IMGs should look like brackets: e. g. "{", with numbers on top of it to differentiate the start of range 1 from the start of range 2, which otherwise would just look the same. But these numbers must be a part of the SVG image (not text: we don't want them to mess up the offset).

When the user hovers span 1, he (JS) gets a box with the content of the annotation, which can be either a string or a JSon structure.

''Only empty elements''
The HTML that a browser should manage (to display the plain text coming from the back-end) should not include any non-empty element, like <tag>...</tag> (to avoid interfering with the offset calculation). So the best solution seems to be:
* one big <pre> wrapping the whole text, so we keep the txt file line breaks;
* using CSS like in http://www.longren.org/wrapping-text-inside-pre-tags/ to make sure that the text wraps.