Chapter 5

Working with multiple files.

[back to index]

Creating an external DTD

Towards the end of the previous chapter, our test document failed validation because the DTD contained a parameter entity reference, and the parser returned this error message:

As we have already mentioned, parameter entity references only work when you use them in DTD's that are separated from the document content. So, in order for our entity references to work we need to remove the DTD from the document prolog. One word of caution before we start, however: the parser we have been using in the examples so far has one important restriction. If we wish to validate XML documents that use an external DTD, all of the files (both the XML document and the DTD) must reside on a server that is accessible through the World Wide Web. This means that we can no longer paste the document instance into the text field, but we must type the full URL of the file. As I mentioned in one of the first chapters, it might be a good idea to install a web server on your local machine, even if you do have access to a larger server. This is not as frightening as it may sound. There are a number of freeware servers available, and most of them are relatively easy to install and operate. For the examples in this booklet I have used Apache, since this is both freeware and available for all major operating systems. Apache can be downloaded at: http://www.apache.org/dist/

If you do not want to go to the trouble of installing a server on your machine, there are several XML parsers that run locally. I have already mentioned nsgmls, which is a part of the SP Suite by James Clark. This is one of the best parsers on the market, but there are some problems with it. It was originally written for SGML and not XML, so it does take some time to modify it to accept XML without problems. In addition to this, it runs from an MS-Dos command line, which immediately makes it an unattractive alternative to many people. A good alternative solution to the two mentioned above lies in one of the specialised XML editors named XML Spy (version 2.5 or higher). This is a specialised XML editor that allows you to check both well-formedness and validity with files that are stored locally. The program has a good graphical interface as well as a "source mode" that allows you to edit documents as if you were using a text pad. The only requirement for the program is that you have Internet Explorer 5 installed (which you need to have installed anyway if you work with XML). A major drawback with this software is that the error messages returned by the parser are not good enough compared to the one at Brown University. A trial version of this program can be found at: http://www.icon-is.com/

With the software problems taken care of, we can now turn to what this chapter is really about: external files. As we saw at the end of the previous chapter, the parser refused to accept our document because of the parameter entity reference. To make this document valid again, we have to create a Document Type Definition that is separate from the XML Document instance itself. This is really quite simple. If we look at the example from the end of chapter four, we already have all information that we need to do this, we just have to rearrange it a little bit. Over the next couple of pages we will see how this is done.

Up until now we have written all the parts of our DTD inside the document declaration in the XML document itself. The first thing we need to do when we create our external DTD is to remove all element, attribute and entity declarations from the document. Just select everything between the two square brackets in the Document Type Declaration and use the 'cut' function of the editor you are using. When this is done, you need to create a new document and paste the lines you cut into this blank document. If you do this, you should have a document that looks like this:

What we have just done, is to copy all of the rules regarding the structural information of our CD example into a document of its own. This is all the information we need in the DTD, so if we just save this file under the name "collection.dtd" in the same directory as the original XML file, we have just created an external DTD for our CD collection. What we actually call the file is not important, but it must obey the rules regarding XML names described earlier, and it should have the ".dtd" file extension.

In order for our example to work together with this DTD, we need to make a few modifications to what is left of it. After cutting the lines with all the declarations, we are left with the following in the prolog of our document (the document element and its content is unchanged, so I have left it out):

Since we now are going to use an external DTD for this XML document, it is no longer a standalone document. We must therefore change this value in the XML declaration to "no". The rest of the XML declaration should remain unchanged.

When it comes to the <!DOCTYPE> declaration, this must also be changed to fit the new DTD. The Document Type is still COLLECTION, but since the rules for this Document Type is no longer included we must somehow specify where the external DTD can be found. First of all, we can delete the two square brackets, since they were used as containers for the different declarations in the DTD. To make the connection between the XML document instance and the external DTD, we need to have a <!DOCTYPE> declaration that looks like this:

What we have done in this line is to add the SYSTEM keyword and an address that points to the external DTD. We will get back to what this means very soon, but first we can see if it works. Our prolog should now look like this:

The rest of the document should be left unchanged. If we run this example through a parser, it should now be valid again. Not only have we created an XML file that successfully makes use of an external DTD, but our parameter entity reference should also be valid now.

[back to index]

Using External DTDs

At the beginning of chapter three we briefly discussed the usefulness of Document Type Definitions that are separate from the XML document itself. The main benefit is that a single DTD can be used by an indefinite number of documents, instead of just the one document that actually contains the DTD (For more information on external DTD's, see Appendix I). Even if using external DTD's is generally recommended, there are some very important points that need to be observed regarding this. The most important one deals with changes made to the DTD. If one makes a change in an external DTD, all the documents that uses this DTD are affected by the change. This means that relatively minor changes can make a vast number of XML documents invalid. Changes in external DTD's should therefore be thought through very carefully.

In the example above, the XML document and the DTD are stored in the same directory on the web server. We therefore only specified the filename of the DTD inside the quotes in the Document Type Declaration. If you have a DTD that applies to a large number of documents, it is not very likely that all of these XML documents are contained in the same directory, some may even reside on a different computer altogether. In such cases you are faced with two options:

1. Place a copy of the DTD in each directory that contains an XML document that makes use of the DTD in question.
2. Point to the location of the external DTD through an absolute or relative address (URL) in the Document declaration of each individual XML document.

The first alternative here is not really an option, so we are left with the address. If you have some experience with HTML and web servers, the concepts of relative and absolute addresses are probably familiar already so feel free to skip the next two paragraphs.

Relative URL's

A relative URL gives the address or location of a document relative to its own position. In the example we have used the address "collection.dtd" is a relative URL. It simply means that the DTD we are using is contained in the same directory as the XML document. If the DTD is placed in a directory above the XML document itself the URL would have been:

Absolute URL's

Unlike relative URL's, absolute URL's use some sort of fixed location as starting point for the address of the external file. This is usually either the root directory of the server or a Web address. Instead of using a relative address in the example above, I could have written it like this:

Public DTD's

The only thing we haven't discussed yet in the example above, is the use of the SYSTEM keyword. This is not an optional part, but it is used to indicate that we are using a private DTD. By the term 'private' in this context, we mean a DTD that is used by a single author or a relatively limited group of people. We mentioned earlier that the main benefits of using external DTD's in XML (and SGML) is that this allows encoding standards to be shared by a large number of users. This again means that large organisations or interest groups can create standardised markup schemes for their particular field of use. These kind of DTD's that are intended for broad and public usage use the PUBLIC keyword instead of the more limited SYSTEM keyword. As an example we can have a look at the Document declaration for a document that uses the EAD DTD described at the beginning of chapter three.

Except for the PUBLIC identifier, this looks fairly similar our test example. But there is one major difference: PUBLIC DTD's have a unique name. This name consists of four parts separated by double slashes. The first part tells us whether or not this DTD has been approved by a standards organisation. If it starts with ISO or a plus sign ( + ), the DTD has been approved as a standard. Since our example starts with a hyphen sign ( - ), it is not considered to be an international standard. The next section deals with the name of the DTD's owner, which in this case is the Society of American Archivists. This is followed by a section stating the type of document the DTD describes. In our example this is the so-called Encoded Archival Description, EAD. The final part is a two-letter ISO 639 language identifier, which tells us the language of the DTD. In our example the DTD is written in English.

So far, we have learned how to create our own DTDs that are separate from the XML document itself. Furthermore, we have seen how we can connect our XML documents to external DTDs located on our own local machine and on other machines on the World Wide Web. One thing we have not discussed, or even mentioned, is the possibility to use more than one DTD to describe a single document. This is not something that we will go into much detail about, but the next section of this chapter will summarise a few basic rules concerning this.

[back to index]

Using multiple DTD's to describe one document

In our example so far, we have not felt any particular need to work with more than one DTD for our collection document. The reason for this is quite simply that this example is neither complex enough nor huge enough to justify all the work this would cause us. In certain cases, however, there might be good reason to use multiple DTD's for a particular document. As I said above, this is not meant to be a step-by-step guide to how this is done since this is in no way central to our understanding of how XML works. It is, on the other hand, a fairly good illustration of the flexibility and usefulness of XML.

Up until now, we have not bothered to give anything else than a superficial description of a CD collection. There are so many more items of information on a single CD that could have been described. On each individual track, we could have added information on musicians used, recording location, recording date and maybe a transcription of the lyrics. If we go on like this, we will probably find that what seems like a straightforward item, can be analysed down to the last detail, which will give us a very detailed and complex DTD indeed. In such cases it might be a good idea to split one very complex DTD into several smaller ones. This can help us to get a clearer view of what might have seen like an overly complicated structure.

An additional point to this kind of structure, is that it increases the "usability" of your DTD. By this I mean that it becomes easier for people to borrow and use only parts of a DTD. If someone decides to create a DTD for their own CD collection, they may find parts of the DTD we have created to be OK, and other parts to be complete rubbish. If we have broken our complex DTD into smaller pieces, it will be easy for these persons to "mix and match", from different DTD's and they don't have to make mistakes by editing from one huge DTD.

To illustrate this by using our existing example, we can try to change our TRACK element a little bit. We stated above that a track can be described in much greater detail than we have done so far. The level of detail can of course vary depending on what kind of CD one is describing. It goes without saying that an opera lends itself to a different kind of description than a rock album. But, as I said - let's stick to the example we have become familiar with. The Album we have described is a 4-CD collection of demo recordings and unreleased tracks from Bruce Springsteen's entire career. It seems unlikely that an artist have used the same musicians for all his recordings through a 30-year period, so this is something that we can enter information about in the individual TRACK elements.

Except for the NAME element, we have not really entered any information that describes the content of the TRACK element. Let's see if we can't do something about this. To keep things relatively simple, we can add two new sub elements to TRACK: <MUSICIAN> and <LOCATION>. MUSICIAN describes the name of the individual musicians and what type of instrument they are playing, while LOCATION describes the name of the recording studio and the date of the recording. If we try to build a new DTD for the TRACK element based on this information, it could look something like this:

As we have learned in this chapter, this is a fairly standard DTD. Let's save it in the same directory as the XML document itself and the original DTD. What you call it is not important, as long as you follow the XML name restrictions. If you have followed the example so far there should now be three files in our little test directory:

• The XML document (testing.xml)
• The original DTD (collection.dtd)
• The new DTD for the TRACK element (track.dtd)

If we want to use the newer, more complex structure for the TRACK element with our XML document, we need to make a modification to the original DTD which contains the somewhat simpler TRACK element. The simplest way to this, is to remove all element, attribute and entity declarations dealing with the TRACK element in this DTD. We will, in other words, have to remove the following lines:

When this is done, we need to find a way to link the new TRACK DTD to the original DTD (remember that the XML document draws all its information from this original DTD). This link to the second DTD is made by using a parameter entity reference like we described towards the end of the previous chapter. To make our new DTD work, we will have to add the following two lines where we removed the element declarations for TRACK:

The first line is the entity declaration itself. If we compare it to the parameter entity reference from chapter 4, it looks fairly similar:

The main difference between them is that while one of them actually contain all of the replacement text within the entity declaration itself, the other one collects the replacement text from an external file (the 'track.dtd'). With the entity declaration in place, we only need to use the parameter entity reference inside the DTD to connect the 'track.dtd' like we have illustrated in the line below the new entity declaration. Remember that the new declaration must come before you actually use it in the DTD.

Now, try to run the XML document with the new DTD's without making any changes to the content of the file. If you used the example as it is described above, the new elements should be optional. Once we have made sure that our new structure actually works with the existing document, we can try to add some information to the TRACK element. As an example, we can replace the first TRACK element with these lines:

Our example is still valid XML, but now we are using two DTD's to describe a single document instead of one. As we have mentioned earlier, this might not be a very sensible thing to do for a document which is not very complex, but there are other situations where it will make more sense. One type of situation where we can make good use of this structure is when we want to make a large number of documents with a standard header (or footer). This brings us to the final point about external DTD's:

[back to index]

Using both internal and external DTDs

Part of the argument for mixing internal and external DTDs have been made implicitly in the sections above. Internal DTDs are fine if you have a small number of documents without a similar structure, whereas external DTDs are better suited for large numbers of documents with a reasonably similar structure. So, what if you have a fairly large number of documents with some similar structure, but also great individual differences? My point here is that in these cases it could be a good idea to use both internal and external DTDs to describe your documents.

As an example, let us imagine a series of documents that are to be marked up with one standard header, but where the content of the documents themselves vary a lot. To describe these documents in XML, we decide to use one standard header and a more individual description of the different documents. We create one DTD for the header and store it as 'header.dtd', and then we start to describe the documents in XML. The first document in our stack is one that is entirely made out of text, so we decide that for this document we need tags for the body of text and a number of paragraphs in addition to the standard header. A DTD for this kind of document could look something like this:

We have just created a prolog for a document that uses a standard, external header and allows paragraphs of text inside a body. As a final point regarding this kind of structure we will have a short look at something that may cause some problems. As we have just seen, it is possible to use both internal and external DTDs for the same document. A logical question would then be: can we use internal DTDs to modify larger, external DTDs? The theoretical answer is yes, but on a more practical level: no.

As an example: let's say we are using a large, external DTD like the EAD described in Appendix I. For most purposes this suits our needs very well, but in a few cases we would like to make some minor modifications to it. Making these changes internally in the few documents concerned would then be the preferred solution, because this will not affect the other documents in any way. Normally, most computer languages allow something called inheritance. This basically means that rules declared internally in a document take precedence over external rules. This is true for HTML to certain extent, and according to most manuals, in XML also. As an example, consider the following line from 'XML: Extensible Markup Language' by Elliotte Rusty Harold: " In the event of a conflict with elements of the same name in external DTDs, the elements declared internally take precedence."

Since this book was published after the official W3C recommendation for XML 1.0 (10-February-1998 : http://www.w3.org/TR/REC-xml) , we must assume that this will actually work in XML. If try to make some modifications to one of the elements in our external DTD in the fashion described above, like this:

the parser will return this result:

So, what's wrong? The problem here is that some parsers accept this as legal XML and some don't. The two parser I have been using for this manual (Brown University's online validator and the built-in parser of XML Spy 2.5) do not accept this as legal XML. In fact, XML Spy refuses to save this document altogether. Other parsers accept it, but return a warning that it contains a duplicate element declaration. So, the lessons to be learned here is that it might be a good idea not to rely entirely on a single parser, since there seems to be significant differences between them in regard to what they consider to be valid XML.

[back to index]

More about attribute types

In chapter four we discussed attributes and saw that there are ten different attribute types in XML. In our discussion of these attribute types, we did not say anything about the final four types because some knowledge about linking to external files was required. Now that we have attained that knowledge, it is time to continue with the final four attribute types:

• ENTITY
• ENTITIES
• NOTATION
• Enumerated NOTATION

We have already learned that XML consists of several entities put together. Furthermore, we have seen that these entities can be divided into two main groups: internal and external, where we can have both parsed and unparsed entities. Typical examples of unparsed external entities are binary data, like an image file for instance. It is referred to as unparsable because an XML parser will not be able to be able to validate the content of such a file. The final four attribute types are used, mainly, in a document's DTD to link such external files to the XML document itself. Like the example above with internal and external DTDs, this is not something that is supported by all browsers and validators, so trying to use this is not absolutely certain to work. The examples below must therefore be seen as a more theoretical template for how it is supposed to work, rather then a blueprint for how things actually work in the real world.

[back to index]

ENTITY

In order to properly use this attribute type we really also need to use the NOTATION type, since they complement each other in many ways. But let's deal with one thing at the time. As we mentioned in the paragraph above, the ENTITY attribute type is used to link external data, like image files or sound clips, to the XML document instance. To use our collection example again: let's say that we want to add a scanned version of the Compact
Disc cover to document. Here is a step-by-step guide to how can be done by using the ENTITY (and NOTATION) attribute type.

We decide that the image should be included as part of the general information of the CD, so we have to add an element called IMAGE to the parent element. Like this:

So far, so good. Now we have to make sure that this new element can contain the external, binary data in the image file. First of all we have to create an declaration for the new element. This element will not actually contain any data between the opening and closing tag, but a reference to the external file in an attribute inside the opening tag. We will therefore declare this element as empty:

Since we have already stated that the element will contain an attribute that will point to the external file, we will have to create a declaration for this attribute also. The attribute that we are about to declare will not contain textual information, like the ones we have been using so far. This attribute will contain an external entity, so it must be of the ENTITY type. A declaration for the attribute (which we decide to call 'source') would look like this:

At this stage, we have declared an element that allows one the attributes to reference an entity. The final part that is missing, is a declaration of the entity itself. Our entity in this case is an image file called "tracks.jpg" which is located in the same directory as the rest of our files. If we decide to call our entity 'cover', this is what a valid declaration would look like:

As you have probably noticed, there are certain similarities between this declaration and the document declaration in the prolog. An entity is subject to more or less the same rules regarding the use of the SYSTEM and PUBLIC keywords, as well as the use of relative and absolute filenames. If you add the three lines we have just created to the existing DTD and run it through the parser, you should be able to validate it.

Even if we have got a valid DTD that tells us how this external entity is to be handled, there is nothing in the XML document itself that makes the actual reference to the entity. To do this, we need to insert an IMAGE element in the document. From what we have learned so far, and based on how we have declared the element in the DTD, the extra line should look something like this:

In principle this seems to be fine, but if you run it through a parser you will get various error massages. I have this one returned to me:

The reason for this is really quite simple. The official XML 1.0 specification specifically states that entity references are not allow as attribute values. This means that we will have to use just the name of the entity, like this:

If we make this little change and try to parse the document again, we will still get an error message. This time you will get a complaint about the entity itself. The reason for this is that, unless otherwise specified, the parser treats the external data as text. Which brings us over to the next point on the agenda.

[back to index]

NOTATION

We have just learned that unless the parser is told different, the external entity will be treated as text. The NOTATION attribute type is used to tell the parser what kind of data we are dealing with, and also to specify the program used to open these external files. The first step on the way to making this work is to add a few things to the ENTITY declaration in the DTD:

What we have added at the end here is the actual NOTATION. This notation tells the XML processor that the data we are referencing is of the type JPEG. If we run our file through the parser at this stage, we will get an error message like this:

This means that even if our notation has explained to the parser that we are dealing with a JPEG file, we must declare what this means. This is done through a NOTATION declaration in the DTD: