Automating book covers w/ XML

Editor's Notes

• #2: \n
• #3: \n
• #4: Point 3 is of course the meat of this presentation.\n
• #5: Fiction, non-fiction, audio books, Children’s books, ebooks\nRoughly 2000 titles/year\n\n
• #6: What it means to be a trade book publisher. We pay authors to write books--venture capitalist. We don’t create our own content; that’s for the authors to do. We do prepare content for publication: copyedit, and design, create marketing material--and a book cover these days is marketing, like packaging for any product. We arrange for manufacturing. We distribute, market and sell books.\n\n
• #7: In terms of technology, the infrastructure is mostly Microsoft, with a .NET development environment. We use SQL Server for our databases, including business-critical systems. And of course Mark Logic for content management.\nSo Why MarkLogic?\n
• #8: XML important: Six or more years ago, tasked to identify a way to future-proof our content. We were embarking on a project that involved digitizing thousands of books for which we had no files, the Digital Warehouse project. Not to be confused with a Data Warehouse. Three-pronged approach: structured data (SQL Server); binary data (DAM); XML data (MarkLogic Server). We knew we would have to transform the content into various forms, so MarkLogic. Now book XML is used not just for future-proofing, but also for search and for extracting sample chapters.\n
• #9: Why search is important. 19% of all online retail is done through Amazon.com.\nEven having a “placeholder image” for a book cover gets us a 10% bump in pre-publication sales.\n
• #10: \n
• #11: \n
• #12: Here is the jacket for a pretty successful hardcover book. Let’s look at the problems involved in developing book covers like this one. \nLet me take a moment to talk about book covers in today’s marketplace. They are extremely important to the marketing of our products. They act as packaging, but even more than that they are the representation to the world of that product. A couple of slides ago you saw a bunch of images representing the products that S&S publishes. You probably didn’t even give it a thought that these were pictures of book covers. As a result, a lot of effort goes into getting just the right look, and just the right words on a cover. As a result they go through many rounds of approvals and revisions before they’re considered ready to be printed.\nOne impetus to developing the system I’m about to describe to you is the demand on the part of Sales and Marketing to have exactly the words we use on the cover of a book represent it online. Another revolves around two issues . . . <click>\n
• #13: One problem facing our designers is that for a publisher the size of Simon & Schuster, the books come in a lot of different sizes, various formats, and from various imprints. Imprints are profit units each of which has a slightly different outlook on the world, and likely to be run by someone with a healthy ego and strong opinions, resulting in each having a slightly different layout. All these different layouts and sizes poses a problem for designers and for the workflow--getting clear, definitive and accurate information to the designers about what they’re supposed to create.\nAdded to this was the fact that information about what is supposed to go on the cover was coming from multiple and sometimes contradictory inputs—emails, word files that were dropped in one or another folder on a file server, or printed or even written notes. Also, these weren’t coming to the designer in any particular order or even following a schedule. Editors with clout left off entire sections of copy until the last minute.\nYou’re going to hear me use the term “copy” here, which comes from the advertising and publishing industries. It means “text.” \n
• #14: <click>The thing is, we have all the pertinent format information about book covers in our production database. That’s one of the SQL databases I alluded to earlier. I figured we should use it. \n<click>To help avoid confusion, I also got buy-in from the art directors for them to develop a definitive, concise library of templates and, though it might not seem important, a vocabulary of terms by which to call the various sizes and formats (jacket, cover, trade paper, French flap, step-back and so on). <click>We also have a system for editors to write descriptive copy about books for internal use, the tipsheet system. So they’re used to writing copy in a web-based system. Writing cover copy in a related system isn’t that much of a stretch.\n
• #15: What we established with the aid of Cover Automation is a clear workflow for book covers, from the creation of cover copy through to the design of the cover itself. This is a somewhat simplified view. 1. Acquisition Editor writes his or her Tipsheet. 2. Once Editorial and Marketing agree on a format for the book and some initial specifications (format, size, approximate length) this information is shared with Production. 3a. The Editor or sometimes a Copywriter writes the cover copy, usually based on what was written for the Tipsheet. 3b. Production finalizes the specifications. 4. Cover copy review and revision process, digitally. 5. Cover Automation system combines the finished copy, spec’s, metadata to create White Layout\n
• #16: We have a vocabulary for describing the elements of a cover, and it’s got sections that we have names for, but you can’t say that covers are really structured. The elements appear in different places, in a different order; not all covers have all elements. Still, we can use the information we have to our advantage, especially because we have a tool like Mark Logic to work with.\n
• #17: Here is the Cover Editor. It’s a Web page generated out of our product database, Title Management. The Cover Editor provides keys to solving the problem. <click> Notice that it contains important metadata, especially the definitive type (hardcover, so a dust jacket) and size of the cover. Again, this is taken from the database that we use to actually order manufacturing, so it has to be dead accurate.\nThe Cover Editor is also the single source for cover copy: It allows Editorial to write and edit the copy for the various sections of a cover. <click> It allows them to place items in the order they want them to appear in (although here the author’s name isn’t on the top of the front cover, where it should be). Of course it also allows them to add elements, such as praise quotes.\n
• #18: Let’s scroll down the Web page and take a closer look at the book’s description, and follow that through. The Cover Editor is a pretty full-featured program. It It allows for basic formatting--bold and italics--and keeps track of all text changes and records when they happened and who worked on it. I noticed that Sarah Branham added most of the elements to this cover, and that the copyeditor Carole Schwindeller worked on it as well.\nThe right-hand column is for notes--italicize this, make that prominent, are you sure you want to use this adjective three times in the same paragraph? For this title there didn’t seem to be any need for that.\n\n
• #19: The system also provides routing of the cover copy through various departments. When the user <click> “sends” the cover copy to someone, the system creates an email that identifies the sender. They can write a message to the recipient, and this email contains a link to this Web page. \nOnce all the approvals are received, Editorial sends the cover copy to the designer. <click> She clicks the Build Cover button. \nShe then chooses <click> the imprint, format and size, and tells the system to build a cover for her. We debated creating the cover automatically because the system has all that information, but the art directors felt that there enough departures from the standards involved that they were more comfortable having the designers go through this step and choose manually.\n
• #20: The result that is downloaded to the designer’s desk is what I call the White Layout. <click> Here you can see the Description that we have been following. \nLet’s take a look at what is happening behind the scenes.\n
• #21: Behind the layout--and what the designer doesn’t see--is an XML structure. Here’s the Description again <click>, with its corresponding element in the structure. So how do we get to this point? \n
• #22: We took a look at the Cover Editor, which contains the metadata and the cover copy. The SQL database which provides the Cover Editor generates an XML version of the data to go on the cover. \nAnother key element is the template, which we’ll examine in a moment. The template corresponding to the format and size of the book is exported to a version of XML which the MarkLogic server can read, and the server combines the two to create the White Layout. We’ve all heard about Babbage’s Difference Engine. What we have here is a Replacement Engine. Replacement is a key concept to understanding how this system works.\nLet’s take a look at that process step-by-step.\n
• #23: We took a look at the Cover Editor, which contains the metadata and the cover copy. The SQL database which provides the Cover Editor generates an XML version of the data to go on the cover. \nAnother key element is the template, which we’ll examine in a moment. The template corresponding to the format and size of the book is exported to a version of XML which the MarkLogic server can read, and the server combines the two to create the White Layout. We’ve all heard about Babbage’s Difference Engine. What we have here is a Replacement Engine. Replacement is a key concept to understanding how this system works.\nLet’s take a look at that process step-by-step.\n
• #24: We took a look at the Cover Editor, which contains the metadata and the cover copy. The SQL database which provides the Cover Editor generates an XML version of the data to go on the cover. \nAnother key element is the template, which we’ll examine in a moment. The template corresponding to the format and size of the book is exported to a version of XML which the MarkLogic server can read, and the server combines the two to create the White Layout. We’ve all heard about Babbage’s Difference Engine. What we have here is a Replacement Engine. Replacement is a key concept to understanding how this system works.\nLet’s take a look at that process step-by-step.\n
• #25: We took a look at the Cover Editor, which contains the metadata and the cover copy. The SQL database which provides the Cover Editor generates an XML version of the data to go on the cover. \nAnother key element is the template, which we’ll examine in a moment. The template corresponding to the format and size of the book is exported to a version of XML which the MarkLogic server can read, and the server combines the two to create the White Layout. We’ve all heard about Babbage’s Difference Engine. What we have here is a Replacement Engine. Replacement is a key concept to understanding how this system works.\nLet’s take a look at that process step-by-step.\n
• #26: Here it is in Indesign format. Once the designer has the layout set, we identify the elements that are to be populated with cover copy, using InDesign’s capability to introduce structure into their layouts. Notice that this one contains 14 structural elements. So in addition to the cover copy that Editorial writes, it allows us to populate items like prices and ISBN, and what you might call “boilerplate,” like the line that tells buyers that the book is also available as an ebook or an audio. This is a great timesaver, and eliminates the possibility of of a lot of errors, as it used to be up to the designers (notoriously bad typists) to type in the ISBN, prices, and tag lines like “Also available as and ebook.”\nI want to bring to your attention that there is just one placeholder frame in the flap section <click>. Yet, as we’ll see, there will be more than one element that is to be placed in the flap. We could never predict how many elements there might be in a section, and we’ll discuss how we solved that particular problem a little later on.\n
• #27: Once the template is set, it is saved in idml format. IDML is an XML format, consisting of XML files describing each aspect and piece in the layout, then placed in a ZIP wrapper, similar to what Microsoft is doing with its Office documents. What is important about the fact that Adobe has done this, is that the IDML format is both complete and native. That is key for us, because it allows us to edit or even create InDesign documents through XML.\nHere is the template opened in an XML editor. Notice all the folders listed down the left side of the window.\n
• #28: Here I’ve opened the “Stories” folder, and chosen one of the “Stories.” I’ve highlighted the tag that identifies this Story as the “placeholder” frame for copy that is to be placed on the flap <click>. A “placeholder” frame is the first in a column of one or more frames in that Section. \n
• #29: The other key building block is the XML output by the SQL database. We refer to it as “cover-info.” Cover-info XML is not valid, but it is well-formed and Mark Logic is perfectly fine with it. It is a collection of the elements that are to appear on the cover, both those that were written by Editorial and data generated from the product database.\nWe can see the first section, frontcover, and the first three and part of a fourth element, identified as TextFrames: the TITLE, SUBTITLE, AUTHOR, and part of the HEADLINE, “#1 New York Times bestselling author” The words “New York Times” are in a SPAN identified as “emphasis.”\n
• #30: If we scroll down through this file, we come to the Description, which we were following. It’s part of the Section called “flap.” There is a quote from Stephen King followed by the Description we were looking at before. So, as I pointed out before, here are two text frames that are to be positioned in the same section.\nHere’s how we solve that problem: Notice the InDesign-specific information. There is an argument in the section tag called y-offset. This indicates that each TextFrame for that section is to start 54 points--or 3/4 inch--below the previous one. \nAlso notice that the ParagraphStyle argument. It corresponds with InDesign paragraph styles, and the CharacterStyle corresponds with character styles set up in the template.\nWhere Mark Logic comes into the equation, as you might have guessed, is putting these two XML streams--the IDML of the template and the XML of the cover-info--together to generate the White Layout.\n
• #31: Let’s look behind the scenes at what happens when the Designer pushes the “Generate Layout” and then “Start” buttons. In this administrator’s view we can see the templates we’ve saved in IDML format. These correspond to the choices made by the designer in terms of division, format and size. Clicking on one of these listings we can see the ISBNs which were processed with that template.<click> Clicking on that displays the date and time that it was processed, along with a string that shows us what is happening: <click> At the bottom of the browser we see a string that calls a module called Controller, an XQuery, and passes to it two arguments, the ISBN and the “Job Ticket” that corresponds to the template.\n
• #32: Let’s look behind the scenes at what happens when the Designer pushes the “Generate Layout” and then “Start” buttons. In this administrator’s view we can see the templates we’ve saved in IDML format. These correspond to the choices made by the designer in terms of division, format and size. Clicking on one of these listings we can see the ISBNs which were processed with that template.<click> Clicking on that displays the date and time that it was processed, along with a string that shows us what is happening: <click> At the bottom of the browser we see a string that calls a module called Controller, an XQuery, and passes to it two arguments, the ISBN and the “Job Ticket” that corresponds to the template.\n
• #33: Let’s look behind the scenes at what happens when the Designer pushes the “Generate Layout” and then “Start” buttons. In this administrator’s view we can see the templates we’ve saved in IDML format. These correspond to the choices made by the designer in terms of division, format and size. Clicking on one of these listings we can see the ISBNs which were processed with that template.<click> Clicking on that displays the date and time that it was processed, along with a string that shows us what is happening: <click> At the bottom of the browser we see a string that calls a module called Controller, an XQuery, and passes to it two arguments, the ISBN and the “Job Ticket” that corresponds to the template.\n
• #34: Let’s look behind the scenes at what happens when the Designer pushes the “Generate Layout” and then “Start” buttons. In this administrator’s view we can see the templates we’ve saved in IDML format. These correspond to the choices made by the designer in terms of division, format and size. Clicking on one of these listings we can see the ISBNs which were processed with that template.<click> Clicking on that displays the date and time that it was processed, along with a string that shows us what is happening: <click> At the bottom of the browser we see a string that calls a module called Controller, an XQuery, and passes to it two arguments, the ISBN and the “Job Ticket” that corresponds to the template.\n
• #35: I’m not going to go through the Xqueries line by line. I’m not an Xquery expert by any stretch, but that would take us the better part of a month to do. \n
• #36: Suffice to say that Controller does what its name implies. It runs the show. The heavy lifting is done by another Xquery, lib-ss. That’s where “run-layout” resides.\n
• #37: Lib-ss is 14 screens long, but the operative phrase appears in this snippet: “replace all the replaceable content.” A few slides back we looked at the XML contained in one of the IDML templates. It has placeholder-type tags, and values for them. We also looked at the cover-info XML delivered by the SQL database. It has tags in it called placeholder-type, pageitem type, and pageitem-number.\n
• #38: The cover-info is placed into a corresponding folder in one part of the MarkLogic system <click>. What lib-ss and its minions do is open the cover-info XML and for each XML element <click> it looks into the content folder <click> for the relevant JobTicket and finds the Story with the relevant placeholder-type <click>--the “flap” in our case. It then replaces the contents of that story with the contents from cover-info <click>. It stores the result in the staging folder <click>.\nIf there is more than one pageitem for that placeholder-type--the Description in “flap” in our case--it generates another Story, identical in type and size, and drops it in, 3/4” lower on the layout. \nOnce it is through processing the contents of cover-info, it then combines--in memory--the Stories that are in Staging with the ones in the Job Ticket, replacing whatever stories there have the same ID. \n
• #39: It ZIPs this collection of files into an IDML, which the user downloads. Here is the IDML, opened in an XML editor, with the first Story of the flap displayed. If you look closely enough, <click> you’ll see that the local formatting is preserved from the Cover Editor.\n
• #40: Here is the Description that we were following. <click>It is tagged as item-content, and a description.\n
• #41: So the IDML that is delivered to the user--what I call the “white layout”--contains a wealth of structural information. We don’t turn on the structure view for the user, of course.\n
• #42: The designer then proceeds to add graphics, photos, color, and to set the type to create the finished layout.\n
• #43: The result still contains the structure of the template even at this stage. Here the book description is highlighted. The system is actually designed so that if this layout is saved as an IDML, it can be submitted to the system and the corresponding fields we saw in the Cover Editor could be updated with any changes made to the text in the InDesign file. The idea was that this finished cover would contain the final word on what editors, publishers and marketing types wanted to say about the book.\nHowever, we uncovered two flaws in our logic.\n
• #44: For one thing, many covers, like this one, have artwork instead of text. For Children's, this meant that our approach of having cover copy delivered from a database wasn't of much use. There wasn't much text on their covers anyway. But designers in the Adult Division wanted to be able to turn some of the text into art, too. And they were apt to break up or combine frames of text, resulting in the IDML Stories getting out of synch.\nMore importantly, waiting for the final, approved text on the cover has become too much of a luxury. While at the time we began this project, total, letter-perfect correspondence between the cover and online copy was mandatory. Online marketing demands that this information be timely. That means it must be out before the cover is even close to being ready for the printer. We’re getting to the point that what goes out o the Net does not have to batch the cover character for character.\n
• #45: We are thinking of other applications for this technology, though. Here is the page from our HTML-based digital catalog for Sing You Home, but we’re exploring the idea of using a similar approach to building print catalogs, as well.\n
• #46: Our Digital Group, with the help of MarkLogic Professional Services and some third-party tools, has built a content enrichment and search tool which will be going live shortly. Here I’ve typed in the phrase “civil rights.” It has turned up a list of relevant titles and extracted appropriate sections of text. The tool also allows the user to drill down into the title. At a demonstration of the tool, someone typed in Quaddafi and turned up a book from years ago, that everyone had forgotten about, written by a personal friend of his. I think this will become a valuable tool, and it shows why saving content in XML is important, even though we don’t have an XML-based workflow.\n
• #47: So we’ve covered (read) <click> <click> <click> <click> <click>\n
• #48: So we’ve covered (read) <click> <click> <click> <click> <click>\n
• #49: \n