Rebooting TEI Pointers

Editor's Notes

• #2: I am painfully aware that this is the last paper of the last session of the day, and that I’m all that stands between you and beer. And, moreover, that the topic I’m presenting on isn’t the most accessible one ever. I hope the dry topic will make the beer that much more refreshing afterwards\n\n
• #3: \n
• #4: XPointers are an extension to URLs. What's a URL? A URL is an address. It tells you where to go, but (I must emphasize) not what to do when you get there. XPointers are tacked onto a URL and address a location inside the document referenced by the URL. This is an important point, because although XPointers look very much like programming language functions, they are not functions. They don't actually dictate what happens to the things they address, they just tell you how to get there.\n\nTEI has a set of XPointer schemes, detailed at http://www.tei-c.org/release/doc/tei-p5-doc/en/html/SA.html#SATS \nWarning: this section is so abstract, you may temporarily lose the will to live while reading it. It has been the victim of a Catch-22. It is difficult enough to grasp that most people never bother to try using it, and so few people have tried using it that it has never accumulated a good set of implementations and use-cases.\n
• #5: \n
• #6: Well, this is a good question. If nobody ever uses them, should they even exist? I've come to the conclusion that they do actually cover a set of edge cases that are worth dealing with and that more people are likely to run into as TEI adoption spreads.\n\nThe TEI has satisfactorily proved by example that trees make for a very useful (even though not perfect) data structure for documents of many kinds. Trees have the advantage over simpler constructs of being easy to navigate. The XML tree-structure even has a path language which can be used to address nodes in a document or retrieve values from it. With XML and XPath, we can get at pretty much any part of a TEI document that we might want. So what is the point of Pointers?\n\nWell, XPointers allow you to address unmarked pieces of a document. They let you address things like a name embedded in a chunk of text, or an apparatus lemma spanning a series of elements and partial text nodes. You may object: "well, you can do that with XPath too!" But in fact, technically, you can't. XPath does have a number of functions that allow you to get at bits of text (substring, string-before, string-after, and so-on), but these functions return atomic values rather than nodes in the document tree. In other words, they return the text but discard the context.  This is ok for certain modes of work, but if what you actually want to do is address the text in its context, then XPath may not work for you, because it can only address whole nodes in their context.\n
• #7: Now, I mentioned earlier that these are edge cases, and they are, because if you want to say something about a chunk of text in an XML document, what you typically do is put some markup around it with an id attribute that you can point to. Easy. Done.\n\nThe XPointer use case comes in when for some reason you can't insert new elements into the target document or when you really don't want to.  This might happen if you wanted to have markup that annotates a document published by someone else, and you don't want to or can't re-host a derivative version with your own markup added. We have another use case at papyri.info, a project I work on which provides online editing of editions of ancient documents. Our markup editor only deals with marking up aspects of the text, not things like names, organizations, dates, etc. To add more types of markup to the mix would be to raise the complexity bar too high. So if we want to add that kind of markup, we'll need to do it in a standoff fashion, using something like TEI Pointers.\n\nSo the answer is, actually, that you probably don't need to care until or unless you run smack into one of these edge cases. Then you may find that you suddenly start to care a lot.\n
• #8: Now, I mentioned earlier that these are edge cases, and they are, because if you want to say something about a chunk of text in an XML document, what you typically do is put some markup around it with an id attribute that you can point to. Easy. Done.\n\nThe XPointer use case comes in when for some reason you can't insert new elements into the target document or when you really don't want to.  This might happen if you wanted to have markup that annotates a document published by someone else, and you don't want to or can't re-host a derivative version with your own markup added. We have another use case at papyri.info, a project I work on which provides online editing of editions of ancient documents. Our markup editor only deals with marking up aspects of the text, not things like names, organizations, dates, etc. To add more types of markup to the mix would be to raise the complexity bar too high. So if we want to add that kind of markup, we'll need to do it in a standoff fashion, using something like TEI Pointers.\n\nSo the answer is, actually, that you probably don't need to care until or unless you run smack into one of these edge cases. Then you may find that you suddenly start to care a lot.\n
• #9: Now, I mentioned earlier that these are edge cases, and they are, because if you want to say something about a chunk of text in an XML document, what you typically do is put some markup around it with an id attribute that you can point to. Easy. Done.\n\nThe XPointer use case comes in when for some reason you can't insert new elements into the target document or when you really don't want to.  This might happen if you wanted to have markup that annotates a document published by someone else, and you don't want to or can't re-host a derivative version with your own markup added. We have another use case at papyri.info, a project I work on which provides online editing of editions of ancient documents. Our markup editor only deals with marking up aspects of the text, not things like names, organizations, dates, etc. To add more types of markup to the mix would be to raise the complexity bar too high. So if we want to add that kind of markup, we'll need to do it in a standoff fashion, using something like TEI Pointers.\n\nSo the answer is, actually, that you probably don't need to care until or unless you run smack into one of these edge cases. Then you may find that you suddenly start to care a lot.\n
• #10: Earlier this year, a working group on TEI Pointers was convened. The members include myself, Piotr Bański, Syd Bauman, Gabriel Bodard, Martin Holmes, and Laurent Romary. We've been discussing what to do about the Pointers section in the guidelines, starting with the development of a set of use cases, and moving on to the development of a draft update to the section, which I've been editing (with feedback from the other members). Discussions are still ongoing, and the following represents my own views, not those of the working group as a whole, though it has certainly been informed and I hope improved by their input..\n
• #11: There are a variety of problems with the TEI Pointer schemes as currently defined. The first being the negative feedback loop that I identified earlier, where lack of implementations mean lack of use, which means lack of use cases, which means lack of work on implementations. We need to break that cycle by providing a reference implementation for whatever we end up recommending.\n\nSecond, the specification needs to be made more explicit and less abstract, so that it is actually possible to develop implementations. This includes detailing what exactly a TEI Pointer addresses when it indicates a point or a range, tightening up the syntax, and modifying or adding to the definitions in the specification.\n
• #12: So, what is a point? This is tricky, because they don’t exist. They are a useful construct for saying where to start or stop when grabbing a chunk of document, or maybe for describing where to insert a bookmark, but they don’t correspond to anything actually in an XML document, and particularly not to anything in the XML Infoset. So points are a bit theoretical. The old specification defines them as the positions next to elements or between characters in a text node. The old specification does not make it entirely clear, but there have to be strict limitations on where points can be. They can’t be in the middle of an element (i.e. between the brackets), for example. Though perhaps they could be inside an attribute.\n
• #13: Points can define the start and end of a range. So a range is the piece of document between two points. But again, we have to be careful about what that means. In the draft specification, I’ve taken a hard line that ranges can’t contain partial elements, so if you do something silly like [range-1.graffle], you don’t get the elements foo, bar, and baz at all. Instead you get a sequence that looks like ("\\n  ", "\\n    ", "Here be"). \n
• #14: You only get elements when the whole element is part of the sequence. This solves a big implementation roadblock, because it’s very difficult (maybe impossible) to know how to deal with a range containing what would be not-well-formed markup. There’s no such thing as half an element node in the XML DOM, for example. By defining exactly what a range can be—basically what’s called a Sequence in XSLT 2.0—we can eliminate that obstacle to implementation.\n\nSo this is all very boring. Thanks to those of you who’ve managed to stay awake this far! It’s also a bit mad—Nobody in their right mind is going to type most of these things. We might find ways of generating them though.\n
• #15: \n\nLet’s see if we can figure out how pointers might actually be useful. I noted earlier that the set of use cases I see this technology targeting are cases where you want to layer new markup on top of existing markup that you either can’t, or don’t want to change. There’s a caveat built in here: as with any URL, you are taking a risk that the thing you point at won’t always be there. If you’re using pointers with documents that change regularly, you’ll need to come up with strategies for mitigating the risk of broken links.\n\nSome usage scenarios I can think of are:\n\nIncluding and transcluding text: I want to import chunks of document from outside, or from another part of my document.\nLeveraging annotations or standoff markup during transformations\n\n
• #16: I'll conclude with a couple of examples of ways I might use TEI pointers in my own work. In Papyri.info, we take a very opinionated view of how texts should be structured. These are editions of texts where the object is to present the nature of the text as observed and interpreted by its editors. So we take care to mark things like gaps in the text, unclear letters, aspects of how the scribe wrote the text, editorial supplements, etc. We do not mark names of places, people, organizations, dates, events, and so on. If we did this inline it would be too much of a mess. Nevertheless, this is something we'll want to do. We might do something like slide 13: leverage the annotation in the HTML output by using it to supply a callout with further information on mouseover.\n
• #17: I'll conclude with a couple of examples of ways I might use TEI pointers in my own work. In Papyri.info, we take a very opinionated view of how texts should be structured. These are editions of texts where the object is to present the nature of the text as observed and interpreted by its editors. So we take care to mark things like gaps in the text, unclear letters, aspects of how the scribe wrote the text, editorial supplements, etc. We do not mark names of places, people, organizations, dates, events, and so on. If we did this inline it would be too much of a mess. Nevertheless, this is something we'll want to do. We might do something like slide 13: leverage the annotation in the HTML output by using it to supply a callout with further information on mouseover.\n
• #18: Second, I've started as an experiment to see how you might take the apparatus from one edition and apply it to an earlier edition. Perseus has an edition of Ovid's Ibis from 1879 that they publish, and they make the XML available. What if I wanted to take their text, but use S. G. Owen's apparatus from his 1915 Oxford Classical Text to "enhance" it?\n
• #19: Second, I've started as an experiment to see how you might take the apparatus from one edition and apply it to an earlier edition. Perseus has an edition of Ovid's Ibis from 1879 that they publish, and they make the XML available. What if I wanted to take their text, but use S. G. Owen's apparatus from his 1915 Oxford Classical Text to "enhance" it?\n
• #20: Second, I've started as an experiment to see how you might take the apparatus from one edition and apply it to an earlier edition. Perseus has an edition of Ovid's Ibis from 1879 that they publish, and they make the XML available. What if I wanted to take their text, but use S. G. Owen's apparatus from his 1915 Oxford Classical Text to "enhance" it?\n
• #21: Second, I've started as an experiment to see how you might take the apparatus from one edition and apply it to an earlier edition. Perseus has an edition of Ovid's Ibis from 1879 that they publish, and they make the XML available. What if I wanted to take their text, but use S. G. Owen's apparatus from his 1915 Oxford Classical Text to "enhance" it?\n
• #22: Second, I've started as an experiment to see how you might take the apparatus from one edition and apply it to an earlier edition. Perseus has an edition of Ovid's Ibis from 1879 that they publish, and they make the XML available. What if I wanted to take their text, but use S. G. Owen's apparatus from his 1915 Oxford Classical Text to "enhance" it?\n
• #23: Second, I've started as an experiment to see how you might take the apparatus from one edition and apply it to an earlier edition. Perseus has an edition of Ovid's Ibis from 1879 that they publish, and they make the XML available. What if I wanted to take their text, but use S. G. Owen's apparatus from his 1915 Oxford Classical Text to "enhance" it?\n
• #24: As an aside, this isn't necessarily something you'd want to do, except as a way of examining the scholarly tradition. A.E. Housman had nothing good to say about Owen's edition.\n
• #25: So the current state of play is this: I've been working on a draft specification, which is open for comment on Google Drive at https://docs.google.com/document/d/1JsMA-gOGrevyY-crzHGiC7eZ8XdV5H_wFTlUGzrf20w/edit# — this is under revision in my copious spare time, and I've started (only started, mind!) building a reference implementation at https://github.com/hcayless/tei-pointers. There is really nothing to look at there yet, but I'm trying to make progress on it, so watch this space!\n
• #26: So the current state of play is this: I've been working on a draft specification, which is open for comment on Google Drive at https://docs.google.com/document/d/1JsMA-gOGrevyY-crzHGiC7eZ8XdV5H_wFTlUGzrf20w/edit# — this is under revision in my copious spare time, and I've started (only started, mind!) building a reference implementation at https://github.com/hcayless/tei-pointers. There is really nothing to look at there yet, but I'm trying to make progress on it, so watch this space!\n
• #27: So the current state of play is this: I've been working on a draft specification, which is open for comment on Google Drive at https://docs.google.com/document/d/1JsMA-gOGrevyY-crzHGiC7eZ8XdV5H_wFTlUGzrf20w/edit# — this is under revision in my copious spare time, and I've started (only started, mind!) building a reference implementation at https://github.com/hcayless/tei-pointers. There is really nothing to look at there yet, but I'm trying to make progress on it, so watch this space!\n
• #28: So the current state of play is this: I've been working on a draft specification, which is open for comment on Google Drive at https://docs.google.com/document/d/1JsMA-gOGrevyY-crzHGiC7eZ8XdV5H_wFTlUGzrf20w/edit# — this is under revision in my copious spare time, and I've started (only started, mind!) building a reference implementation at https://github.com/hcayless/tei-pointers. There is really nothing to look at there yet, but I'm trying to make progress on it, so watch this space!\n