p>by John W. Maxwell, Simon Fraser University (jmax@sfu.ca)
with Meghan MacDonald, Travis Nicholson, Jan Halpape, Sarah Taggart, and Heiko Binder
May 2009
Check out the rendered PDF version produced by the method described herein.
XMLProduction [PDF]
This paper makes an argument for, and describes a method for integrating web-based content management with high-quality print production. Using a simple web-based content management system (in this case, a wiki), treating the HTML-based content as “real” XML, and using a relatively simple XSLT transformation, web-based content can be directly and confidently moved into Adobe InDesign CS4 (using Adobe’s new IDML file format), producing automated layout and formatting. The case is made that content management and editorial process are best handled on the web, and that print output ought to be derived from web content, rather than the more typical other way around.
This spring, as part of his now-viral “Six Projects that Could Change Publishing for the Better,” Michael Tamblyn called for “an XML workflow that doesn’t suck.” At SFU’s Master of Publishing program, we have the beginning of one of those projects. At the least, we have a considered approach and a functional set of tools: something good enough to critique.
This is NOT an argument for why you should have an XML workflow. Those exist, as well as excellent arguments for why you might not want one. One of the points Tamblyn made in his coverage of the issue is that XML editorial and production workflows do exist—as SGML, they have done since the 1980s—but that they tend to be technically or operationally complex and require a considerable amount of programmer support.
O’Reilly, the world’s leading technical publisher, has made books out of SGML and XML for decades. Their famous colophons, which also provide interesting tidbits about the animals that grace their covers, usually give some detail about the toolchains employed to produce the current volume. Here are two of my favourites:
The inside layout was designed by Mary Jane Walsh. The text was prepared by Jeffrey Friedl in a hybrid markup of his own design, mixing SGML, raw troff, raw PostScript, and his own markup. A home-grown filter translated the latter to the other, lower-level markups, the result of which was processed by a locally-modified version of O’Reilly’s SGML tools (this step requiring upwards of an hour of raw processing time, and over 75 megabytes of process space, just for Chapter 7!). That result was then processed by a locally-modified version of James Clark’s gtroff, producing camera-ready PostScript for O’Reilly. (from Mastering Regular Expressions)
Alicia Cech and David Futato designed the interior layout based on a series design by Nancy Priest. The authors’ text in POD was converted by Lenny Muellner into DocBook 3.1 SGML. The print version of this book was created by translating the SGML source into a set of gtroff macros using a Perl filter developed at O’Reilly & Associates by Norman Walsh. Steve Talbott designed and wrote the underlying macro set on the basis of the GNU gtroff -ms macros; Lenny Muellner adapted them to SGML and implemented the book design. The GNU groff text formatter version 1.11 was used to generate PostScript output. Mike Sierra provided crucial help with the Chinese and Japanese Unicode characters in Chapter 15, Unicode. The text and heading fonts are ITC Garamond Light and Garamond Book. The illustrations that appear in the book were produced by Robert Romano and Rhon Porter using Macromedia FreeHand 8 and Adobe Photoshop 5. (from Programming Perl, 2nd ed.)
Big textbook publishers like Wiley and Pearson have XML toolchains that allow them to manage complex, multi-participant workflows too. They don’t usually publish the details on the last page.
Interestingly, SGML and XML production workflows were pioneered right here in Toronto at Coach House Press in the 1980s—the O’Reilly colophon above draws directly on the Coach House work. But that’s another story.
The challenges of building—and living with—an XML workflow are clear enough. The return on investment is a long-term proposition.
Regardless of the benefits XML may provide, the starting reality is that it represents a very different way of doing things than the one we are familiar with. The Word Processing and Desktop Publishing paradigm, based on the promise of onscreen, WYSIWYG layout, is so dominant as to be practically inescapable. It has proven really hard to get from here to there, no matter how attractive XML might be on paper.
A considerable amount of organizational effort and labour must be expended up front in order to realize the benefits. This is why XML is often referred to as an “investment”: you sink a bunch of time and money up front, and realize the benefits—greater flexibility, multiple output options, searching and indexing, and general futureproofing—later, over the long haul. It is not a short-term return proposition.
And, of course, the returns you are able to realize from your XML investment are commensurate with what you put in up front: fine-grained, semantically rich tagging is going to give you more potential for searchability and recombination than a looser, more general-purpose approach, but it sure costs more. For instance, the Text Encoding Initiative (TEI) is the grand example of pouring enormous amounts of energy into the up-front tagging, with a very open-ended set of possibilities down the line. TEI helpfully defines a level to which most of us do not have to aspire.
But understanding this on a theoretical level is only part of the challenge. There are many practical issues that must be addressed. Software and labour are two of the most critical. How do you get the content into XML in the first place? Unfortunately, despite two decades of people doing SGML and XML, this remains an ugly question.
In 2009, there is still no truly likeable—let alone standard—editing and authoring software for XML. For many (myself included), the high-water mark here was Adobe’s FrameMaker, substantially developed in the late 1990s. With no substantial market for it, it is relegated today mostly to the tech writing industry, unavailable for the Mac, and just far enough afield from the kinds of tools we’re used to that its adoption represents a significant hurdle. And Frame was the best of the breed; most of the other software in decent circulation are programmers’ tools—the sort of thing that, as Michael Tamblyn pointed out, encourages editors to drink at their desks.
The labour question represents a stumbling block as well. The skill- and mind-sets that make effective XML editors have limited overlap with those shared by literary and more traditional production editors. The need to think of documents as machine-readable databases is not something that comes naturally to folks steeped in literary culture. In combination with the sheer time and effort that rich tagging requires, many publishers simply outsource the tagging to India, drawing a division of labour that spans oceans, to put it mildly.
Once you have XML content, then what do you do with it? How do you produce books from it? Presumably, you need to be able to produce print output as well as digital formats. But while the latter are new enough to be generally XML-friendly (e-book formats being largely XML based, for instance), there aren’t any straightforward, standard ways of moving XML content into the kind of print production environments we are used to seeing. This isn’t to say that there aren’t ways of getting print—even very high-quality print—output from XML, just that most of them involve replacing your prepress staff with Java programmers.
It’s not that XML is new, or immature, or untested. Remember that the basics have been around, and in production, since the early 1980s at least. But we have to take account of a substantial and long-running cultural disconnect between traditional editorial and production processes (the ones most of us know intimately) and the ways computing people have approached things.
Interestingly, this cultural divide looked rather different in the 1970s, when publishers were looking at how to move to digital typesetting. Back then, printers and software developers could speak the same language. But that was before the ascendancy of the Desktop Publishing paradigm, which computerized the publishing industry while at the same time isolating it culturally. Those of us who learned how to do things the Quark way or the Adobe way had little in common with people who programmed databases or document management systems. Desktop Publishing technology isolated us in a smooth, self-contained universe of toolbars, grid lines, and laser proofs.
So, now that the reasons to get with this program loom large, how can we bridge this long-standing divide?
The answer, I think, is staring us in the face. The bridge is the Web, a technology and platform that is fundamentally based on XML, and which many publishers are by now comfortably familiar with. Okay, maybe not comfortably, but at least the web is already something that you’re working with; you already either know or have on staff people who understand it and can work with it.
The foundation of our argument is this: rather than looking at jumping to XML in its full, industrial complexity—which seems to be what the StartWithXML initiative is suggesting, publishers instead leverage existing tools and technologies—starting with the web—as a means of getting XML workflows in place. This means making small investments and working with what you know rather than spending tens of thousands of dollars on XML software and rarefied consultants. It means re-thinking how the existing pieces of your toolchain fit together; re-thinking the existing roles of software components you’re already using. It means, fundamentally, taking the web seriously as a content platform, rather than thinking of it as something you need to get content out to, somehow.
If nothing else, the web represents an opportunity to think about editorial and production from outside the shrinkwrapped Desktop Publishing paradigm for a change.
At this point some of you may be thinking: wait a moment, the web isn’t really made out of XML; the HTML that makes up most of the web is at best the bastard child of SGML and it is far too flaky/unstructured/underpowered to be taken seriously.
I will counter by arguing that although HTML on the web exists in a staggering array of different incarnations, and that the majority of it is an unstructured mess, this does not undermine the general principle that basic, ubiquitous web technologies can make a solid platform for content management, editorial process, and production workflow.
With the advent of a published XML standard in the late 1990s came the W3C’s adoption of XHTML: the realization of the web’s native content markup as a proper XML document type. Today, its acceptance is almost ubiquitous, even while the majority of actual content out there may not be strictly conforming. The more important point is that most contemporary web software, from browsers to authoring tools to content management systems (from blogs to enterprise systems), are capable of working with clean, valid XHTML. Or, to put the argument the other way around, clean, valid XHTML content plays absolutely seamlessly with everything else on the web.
The objection which follows, then, will be that even if we grant that XHTML is a real XML document type, that it is underpowered for “serious” content because it is almost entirely presentation (formatting) oriented; it lacks any semantic depth. In XHTML, a paragraph is a paragraph is a paragraph, as opposed to a section or an epigraph or a summary. In contrast, more “serious” XML document types like DocBook or DITA-derived schemas are capable of making semantic distinctions about content chunks at a fine level of granularity and with a high degree of specificity.
This may be true, but this is precisely where we should be questioning the cost-benefit relationship. More semantic richness is of course a good thing, but in order to achieve it you must make that much more of an investment up front. XHTML is a mature and fairly comprehensive system for representing the basic building blocks of prose. So there is an argument for recalling the 80:20 rule here. If XHTML can provide 80% of the value with 20% of the investment, then what exactly is the business case for spending the other 80% to achieve that last 20% of value? I suspect the ratio is actually quite a bit steeper than 80:20 for most publishers.
Furthermore, just to get technical for a moment, XHTML is extensible in a fairly straightforward way, through the common “class” attribute on each element. Web developers have long leveraged this kind of extensibility in the elaboration of “microformats” for semantic-web applications. There is no reason why publishers shouldn’t think to use XHTML’s simple extensibility in a similar way for their own ends.
Finally, the best reason of all to take XHTML seriously is its unparalleled ubiquity. My quick glance at Wikipedia shows more than 180 different XML markup languages, about 175 of which you’ve never heard of—and neither has the software you use. XHTML, on the other hand, is supported by a vast array of software that you use everyday, starting with your web browser. For this very reason, XHTML is in fact employed as a component part of several more specialized document types (ONIX and ePub among them). Why re-invent a general-purpose prose representation when XHTML already does the job?
It is worth pausing for a moment to consider the role of XHTML in the ePub standard for ebook content. An ePub file is, anatomically, a simply disguised zip archive. Inside the zip archive are a few standard component parts: there are specialized files that declare metadata about the book, and about the format of the book. And then there is the book’s content, represented in XHTML. An ePub book is a web page in a wrapper. This simple fact needs to be recognized, if only for the sake of demystification. This is not rocket science.
To sum up the general argument: the web as it already exists presents incredible value to publishers, as a platform for doing XML content management with existing (and often free) tools, and without having to go blindly into the unknown. At this point, we can offer a few design guidelines:
- Prefer existing and/or ubiquitous tools over specialized ones wherever possible;
- Prefer free software over proprietary systems where possible;
- Prefer simple tools controlled and coordinated by human beings over fully automated (and therefore complex) systems;
- Play to our strengths: use web software for storing and managing content, use layout software for layout, and keep editors and production people in charge of their own domains.
At the SFU Master of Publishing Program, we have been chipping away at this general line of thinking for a few years. Over that time, web content management systems (NB: if you have a blog, you have a web content management system) have been getting more and more sophisticated, all the while getting more streamlined and easier to use. The web is beginning to be recognized as a writing and editing environment used by millions of people. And the ways in which content is represented, stored, and exchanged online have become increasingly robust and standardized.
The missing piece of the puzzle has been print production: how do we move content from its malleable, fluid form online into the kind of high-quality print production environments we’ve come to expect after two decades of Desktop Publishing? Anyone who has tried to print web content knows that the existing methods leave much to be desired (hyphenation and justification, anyone?). In the absence of decent tools for this, most publishers quite naturally think of producing the print content first, and then think about how to get material onto the web for various purposes. So we tend to export from Word, or from Adobe, as something of an afterthought.
While this sort of works, it isn’t elegant, and it completely ignores the considerable advantages of web-based content management. Content managed online is stored in one central location, accessible simultaneously to everyone in your firm, available anywhere you have an Internet connection, and usually exists in a much more fluid format than Word files. If only we could manage the editorial flow online, and then go to print formats at the end, instead of the other way around. At SFU, we made several attempts to make this work by way of the supposed “XML import” capabilities of various Desktop Publishing tools, without much success.
This year, Adobe solved this part of the problem for us with the introduction of its Creative Suite 4. What CS4 offers is the option of a complete XML representation of an InDesign document: what Adobe calls IDML (InDesign Markup Language). It is available as both an export and an import in the CS4 version of InDesign. This means that content can be worked on in InDesign and/or in the XML-aware tools of your choice and moved seamlessly back and forth. The IDML file format is—like ePub—a simply disguised zip archive that, when unpacked, reveals a cluster of XML files that represent all the different facets of an InDesign document: layout spreads, master pages, defined styles, colours, and of course, the content. IDML is a well thought-out XML standard that achieves two very different goals simultaneously: it preserves all of the information that InDesign needs to do what it does; and it is broken up in a way that makes it possible for mere mortals (or at least Master of Publishing students) to work with it.
What this represented to us in concrete terms was the ability to take web-based content and move it into InDesign in a straightforward way, thus bridging web and print production environments in a way that used existing tools and skillsets, with a little added help from free software. We would take clean XHTML content, transform it to IDML-marked content, and merge that with nicely designed templates in InDesign. The result is an almost pushbutton publication workflow, which results in a nice, familiar InDesign document that fits straight into the way publishers actually do production.
The Toggle Systems project group (the people listed at the beginning of this paper) worked to demonstrate a proof of concept, using a real book as fodder: the CCSP’s anthology of our graduate student research, Book Publishing 1. This book was produced in the normal way back in 2005, laid out using InDesign CS2. The book designer, Carol Aitken (herself an MPub alum), did us the favour of consistently applying a stylesheet throughout, so the formatting of the original was nice and consistent.
To begin with, we worked backwards, moving the book content back to clean XHTML. The simplest method for this conversion—and if you want to create web content, this is an excellent route—was to use Adobe’s “Export to Digital Editions” option, which creates an ePub file. Recall that ePub is just XHTML in a wrapper, so within the ePub file was a relatively clean XHTML document. It was notably cleaner (that is, the XHTML tagging was simpler and less cluttered) than InDesign’s other web-oriented exports, possibly because Digital Editions is a well understood target, compared with somebody’s website.
In order to achieve our target of clean XHTML, we needed to do some editing; the XHTML produced by InDesign’s “Digital Editions” export was presentation-oriented. For instance, bulleted list items were tagged as paragraphs, with a class attribute identifying them as list items. Using simple search-and-replace, we converted such structures to proper XHTML list and list-item elements. Our guiding principle was to make the XHTML as straightforward as possible, not dependent on any particular software to interpret it.
We broke the book’s content into individual chapter files; each chapter could then carry its own basic metadata, and the pages conveniently fit our web content management system (which is a wiki). We assembled a dynamically generated table of contents for the twelve chapters, and created a cover page. Essentially, the book was entirely web-based at this point.
When the book chapters are viewed online, they are formatted via a CSS2 stylesheet that defines a main column for content as well as dedicating screen real estate for navigational elements. We then created a second template to render the content for exporting; this was essentially a bare-bones version with no navigation and minimal styling. Pages (or even the entire book) can be exported (simply via the “Save As…” function in a web browser) for use in either print production or ebook conversion.
Adobe’s IDML language defines elements specific to InDesign; there is nothing in the language which looks remotely like XHTML. So a mechanical transformation step is needed to convert the XHTML content into something InDesign can use. But this is not as hard as it might seem. Both XHTML and IDML are composed of straightforward, well-documented structures, and so transformation from one to the other is, as they say, “trivial.” We chose to use XSLT (Extensible Stylesheet Language Transforms) to do the work. XSLT is part of the overall XML specification, and thus is very well supported in a wide variety of tools. Our prototype used a scripting engine called xsltproc, a nearly ubiquitous piece of software that we found already installed as part of Mac OS X (contemporary Linux distributions also have this as a standard tool), though any XSLT processor would work.
Rather than create an entire IDML document, which specifies every facet of an InDesign file (spreads, masters, styles, etc.), we opted for a somewhat simpler route. InCopy, Adobe’s distributed editing extension for InDesign, allows a number of editors to feed stories into a single InDesign layout simultaneously (think of a newspaper production workflow, with different editors working on copy right up to the press deadline). Prior to the release of CS4, working this way would require purchasing separate licences for the InCopy client software. But CS4 allows the creation of InCopy files as XML—in fact, InCopy CS4’s native file format is a subset of IDML called ICML, including only a story’s basic content and stylesheet components. In other words, we don’t need to buy InCopy, because we just replaced it with the web. Our wiki is now plugged directly into our InDesign layout. It even automatically updates the InDesign document when the content changes. Thanks, Adobe!
We wrote an XSLT transformation script that converted the XHTML content from the web into an InCopy ICML file. The script itself is less than 500 lines long, as was written and debugged over a period of about a week. The script runs in a matter of seconds, and the resulting .icml file can be “placed” directly into an InDesign template. The ICML file references an InDesign stylesheet, so the template file can be set up with a house-styled layout, master pages, and stylesheet definitions for paragraphs and character ranges.
The result is very simple and easy to use. Our demonstration requires that a production editor run the XSLT transformation script manually, but there is no reason why this couldn’t be built directly into the Web content management system, so that exporting the content to print ran the transformation automatically. The resulting file would then be “placed” in InDesign and proofed.
Our project aimed to be a proof of concept, but is robust enough to handle embedded images and reasonably complex tables; in short, we were able to essentially duplicate the production of the entire Book Publishing 1 volume.
Now, the Book Publishing 1 proof-of-concept was artificially complex; we began with a book laid out in InDesign and ended up with a look-alike book laid out in InDesign. Next time, however—for instance, when we publish Book Publishing 2 —we can begin the process with the content on the web, and keep it there throughout the editorial process. The book’s content could potentially be written and edited entirely online, as web content, and then automatically poured into an InDesign template at press time. “Just in time,” as they say.
This represents an entirely new way of thinking of book production. With a web-first orientation, it makes little sense to think of the book as “in print” or “out of print”—the book is simply available, in the first place online; in the second place in derivative digital formats; and third, but really not much more difficult, in print-ready format, via the usual InDesign/CS print production system publishers are already familiar with.
Creating electronic versions from XHTML source is vastly simpler than trying to generate these out of the existing print process. The ePub version is extremely easy to generate; so is online marketing copy or excerpts for the web, since the content begins life web-native.
Since an ePub file is essentially XHTML content in a special wrapper, all that is required is that we properly “wrap” our XHTML content. Ideally, the content in an ePub file is broken into chapters (as ours was) and a table of contents file is generated in order to allow easy navigation within an ebook reader. We used Julian Smart’s free tool eCub to simply and automatically generate the ePub wrapper and the table of contents. The only custom development we did was to create a CSS stylesheet for the ebook so that headings and paragraph indents looked the way we wanted. If you start with XHTML content, creating ePub is almost too easy.
Ironically, this production workflow is essentially the same as the one developed at Coach House Press in the mid-1980s by Kate Hamilton and Nelson Adams. Then, content was developed in SGML (the Coach House had early involvement in SGML editing software); a set of custom transformation scripts converted the content into a format usable directly by the QuarkXPress of the day. Hamilton and Adams remarked that their goal was to keep as much of the editorial process in the SGML format as possible, rather than making changes in the resulting Quark documents; this ensures that the source files, rather than the layout files, are the final, archival versions.
The basic components of the process are the same, but there is one profound difference: today, we are able to put the process together using nothing but standard, relatively ubiquitous web tools: the web itself as an editing and content management environment; standard web scripting tools for the conversion process, and the well-documented IDML file format in order to integrate the layout tool. Hamilton and Adams were out on a limb, in comparison, building it entirely themselves, and having little or no support from the outside world. Still, they were able to make this work then; how much simpler and easier it is to accomplish today!
| Michael Tamblyn, “Six Projects That Could Change Publishing for the Better,” Booknet Canada Tech Forum. March 19, 2009. See http://www.quillandquire.com/blog/index.php/2009/03/20/michael-tamblyns-six-good-ideas-for-the-future-of-publishing/. |
| Text Encoding Initiative. See http://www.tei-c.org/index.xml. An example document is at http://www.sbl-site.org/assets/U16/U16.xml or http://www.perseus.tufts.edu/hopper/xmlchunk.jsp?doc=Perseus%3Atext%3A2001.05.0168%3Achapter%3D11%3Apage%3D78. |
| See http://toc.oreilly.com/startwithxml/. |
| XHTML is a “reformulation of HTML 4 in XML 1.0.” See http://www.w3.org/TR/xhtml1/. |
| DocBook is an XML schema and document type for technical manuals, though it is often used for a much wider variety of publications. See http://www.docbook.org/whatis. |
| DITA is the “Darwin Information Typing Architecture, … an XML-based, end-to-end architecture for authoring, producing, and delivering technical information.” See http://www.ibm.com/developerworks/xml/library/x-dita1/. |
| Microformats “are a set of simple, open data formats built upon existing and widely adopted standards.” Microformats are based on existing web technologies. See http://microformats.org/about/. |
| Wikipedia’s “List of Markup Languages” page: http://en.wikipedia.org/wiki/List_of_XML_markup_languages. |
| In 2006, the Master of Publishing project group “Beehive” prototyped a system for semi-automatically producing publishers’ catalogues directly from ONIX metadata. The system transformed ONIX to a much simpler data set, then used the rudimentary XML import in Adobe InDesign CS2 to flow the content into templated catalogue pages. See http://thinkubator.ccsp.sfu.ca/TechProjects2006XMLCatalogProduction. In 2007, the “FunnelWeb” project group prototyped a system to import XHTML web content into InDesign CS2. Both projects were limited by the constraints imposed by InDesign’s XML import feature; the catalogue production idea seems to have been a better fit with what Adobe had in mind here. See http://thinkubator.ccsp.sfu.ca/FunnelWeb. |
| For a gentle introduction to CS4’s IDML language, see “Tim Cole’s InDesign BackChannel.” October 14, 2008. See http://blogs.adobe.com/indesignchannel/2008/10/the_most_important_new_feature.html. |
| A more open ended discussion of the programmability of the CS4 suite is provided by Adobe’s Gary Cosimini in a presentation titled “Dynamic Publishing Solutions” from the Booknet Canada Tech Forum. March 2009. See http://blip.tv/file/2042790. |
| The “Toggle Systems” project culminated April 3, 2009. Project documentation can be found at http://thinkubator.ccsp.sfu.ca/TechProjects2009StartWithWWWXMLWorkflow. |
| Rowland Lorimer, John W. Maxwell, and Gillian Shoichet, eds. 2005. Book Publishing 1. Vancouver: CCSP Press. See http://www.ccsppress.org/Titles/BookPublishing1 and, for the XHTML versions, http://thinkubator.ccsp.sfu.ca/wikis/bp1. |
| For more information about Adobe’s InCopy software, see http://www.adobe.com/products/incopy/. |
| The XHTML-to-IDML script can be found at http://thinkubator.ccsp.sfu.ca/wikis/xmlProduction/tkbr2icml.xsl. It will run in any standard xslt processor. Note that the script has not been extensively tested with content from different sources. Nevertheless, it is a working prototype that can serve as a first step toward developing production-ready systems. |
| Julian Smart’s eCub software is available here: http://www.juliansmart.com/ecub. |
| Liora Alschuler. 1995. ABCD…SGML. A Users’ Guide to Structured Information. Boston: International Thomson Computer Press. |
