• | Table of Contents
• | Index
XQuery Kick Start
By James McGovern, Per Bothner, Kurt Cagle, James Linn, Vaidyanathan Nagarajan
Publisher | : Sams Publishing
Pub Date | : September 25, 2003
ISBN | : 0-672-32479-2
Pages | : 384

XQuery Kick Start delivers a concise introduction to the XQuery standard, and useful implementation advice for developers needing to put it into practice. The book starts by explaining the role of XQuery in the XML family of specifications, and its relationship with XPath. The authors then explain the specification in detail, describing the semantics and data model, before moving to examples using XQuery to manipulate XML databases and document storage systems. Later chapters discuss Java implementations of XQuery and development tools that facilitate the development of Web sites with XQuery. This book is up to date with the latest XQuery specifications, and includes coverage of new features for extending the XQuery language.


• | Table of Contents
• | Index
XQuery Kick Start
By James McGovern, Per Bothner, Kurt Cagle, James Linn, Vaidyanathan Nagarajan
Publisher | : Sams Publishing
Pub Date | : September 25, 2003
ISBN | : 0-672-32479-2
Pages | : 384

Copyright
About the Authors
 | James McGovern
 | Kurt Cagle
 | Per Bothner
 | James Linn
 | Vaidyanathan Nagarajan
Acknowledgments
 | The Author Team
 | James McGovern
 | Kurt Cagle
 | Per Bothner
 | James Linn
 | Vaidyanathan Nagarajan
We Want to Hear from You!
Introduction
 | Audience
 | Contents of this Book
 | What this Book Is Not
 | Source Code
Part I. XML Concepts
 | Chapter 1. XML: The Foundation of XQuery
 |  | Background
 |  | What XML Offers
 |  | Portable Data
 |  | How XML Works
 |  | The Evolution of XML
 |  | The XML Family
 |  | In Brief
 | Chapter 2. Understanding the XPath 2.0 Specification
 |  | In Search of Context: XQuery and XPath
 |  | The Need for XPath
 |  | Directories and Nodes
 |  | XPath 1.0
 |  | XPath 2.0's New Features
 |  | Understanding Sequences
 |  | Handling Aggregate Functions
 |  | The tokenize() Function
 |  | Working with External Resources
 |  | Cutting Verbosity with XPath Command Structures
 |  | Regular Expressions
 |  | Of Data Types and Dates
 |  | In Brief
Part II. The XQuery Language
 | Chapter 3. XQuery Language Expressions
 |  | The Need for XQuery
 |  | Why Not SQL?
 |  | XSLT and XPath as Query Languages
 |  | The Structure of XQuery
 |  | Assignments with let
 |  | Iterations with for
 |  | Returning Results
 |  | where Oh where?
 |  | Ordering Up Some XQuery
 |  | Conditional Logic
 |  | Defining Functions
 |  | In Brief
 | Chapter 4. Extensions and Modular XQuery
 |  | Extending XQuery with Libraries
 |  | Enhancing XQuery with Pragmas and Extensions
 |  | Importing Modules
 |  | Invoking External Functions
 |  | Using Library Modules Effectively
 |  | In Brief
 | Chapter 5. XQuery Applications: Presentations, Graphics, and Syndication Feeds
 |  | Viewing XML
 |  | Creating an XQuery Pipeline
 |  | HTML: A Regional Weather Report
 |  | Making the Table More Sophisticated
 |  | Parameterized XQuery
 |  | SVG: Graphing from XQuery
 |  | Syndication with RSS
 |  | Building an XQuery Syndication Engine
 |  | In Brief
 | Chapter 6. Working with the XPath 2.0 Data Model
 |  | XML Infosets
 |  | Node Values and Types
 |  | Types and Type Systems
 |  | Schemas and Complex Types
 |  | Validation Produces Type Annotations
 |  | Sequence Types
 |  | Static Typing
 |  | In Brief
 | Chapter 7. Understanding XQuery Semantics
 |  | XQuery Semantics Introduction
 |  | Processing Model for XQuery
 |  | Differences Between XQuery and XQuery Core
 |  | XQuery Data Model
 |  | Schemas and Types
 |  | Functions
 |  | Notation
 |  | Expression of Semantics in XQuery
 |  | Type Level
 |  | Function Semantics
 |  | In Brief
 | Chapter 8. XQuery and Databases
 |  | Relational Database Concepts
 |  | A Relational-to-Object Database: Concept Mapping
 |  | SQL Concepts
 |  | Relational Technology
 |  | Oracle 9i XQuery Support
 |  | IBM DB2 XQuery Support
 |  | GoXML DB XQuery Support
 |  | In Brief
 | Chapter 9. Advanced XQuery Topics
 |  | XML and Documents
 |  | XML Design for Data Storage
 |  | XQuery Limitations
 |  | In Brief
Part III. Using XQuery
 | Chapter 10. Using Microsoft XQuery Services
 |  | Microsoft XQuery Services
 |  | Example
 |  | Additional Examples
 |  | XQuery Classes
 |  | In Brief
 | Chapter 11. Generating XML and HTML Using XQuery
 |  | Tools for Generating Web Pages
 |  | A Photo Album Application
 |  | Generating the Overview Page
 |  | Generating the Picture Pages
 |  | Running XQuery Programs as Servlets
 |  | Running XQuery Programs from CGI Scripts
 |  | Comparing XSLT and XQuery
 |  | In Brief
 | Chapter 12. Working with Java and XQuery
 |  | Qexo: An XQuery-to-Java Compiler
 |  | Running Qexo
 |  | XQEngine
 |  | In Brief
Part IV. Appendixes
 | Appendix A. XML Fundamentals
 |  | Processing Models for XML Data
 |  | XML Documents
 |  | XML Documents and DTDs
 |  | Well-Formedness and Validity of XML Documents
 |  | XML Syntax and Semantics
 |  | Entity References
 |  | Rules for Writing XML
 |  | CDATA Section
 | Appendix B. Other Sources of Information
 |  | Specifications
 |  | Books
 |  | List Servers
 |  | Implementations
Index


Copyright

Copyright © 2004 by Sams Publishing

All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, photocopying, recording, or otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions. Nor is any liability assumed for damages resulting from the use of the information contained herein.

Library of Congress Catalog Card Number: 2002109777

Printed in the United States of America

First Printing: August 2003

06 05 04 03 4 3 2 1

Portions of this book previously appeared on the Web at http://www.gnu.org.

Trademarks

All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Sams Publishing cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Warning and Disclaimer

Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information provided is on an "as is" basis. The authors and the publisher shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book or from the use of the CD or programs accompanying it.

Bulk Sales

Sams Publishing offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales. For more information, please contact

U.S. Corporate and Government Sales
1-800-382-3419
corpsales@pearsontechgroup.com

For sales outside of the U.S., please contact

International Sales
1-317-428-3341
international@pearsontechgroup.com

Credits

Associate Publisher

Michael Stephens

Acquisitions Editor

Todd Green

Development Editor

Tiffany Taylor

Managing Editor

Charlotte Clapp

Project Editor

Andy Beaster

Copy Editor

Seth Kerney

Indexer

Larry Sweazy

Proofreader

Tracy Donhardt

Technical Editors

Irsan Widarto

Alan Wexelblatt

Technical Reviewer

Ed Peters

Team Coordinator

Cindy Teeters

Interior Designer

Gary Adair

Cover Designer

Aren Howell

Page Layout

Eric S. Miller

Dedication

In memory of Edgar Codd (1923-2003), father of the modern relational database and an early supporter of XQuery. We would not be here without you.


About the Authors

    James McGovern

    Kurt Cagle

    Per Bothner

    James Linn

    Vaidyanathan Nagarajan


James McGovern

James is currently employed as an Enterprise Architect for The Hartford Financial Services Group, Inc. He is the coauthor of the best-selling books Java Web Services Architecture (Morgan Kaufmann), Practical Guide to Enterprise Architecture (Prentice Hall), and J2EE Bible (Wiley). He has sixteen years of experience in information technology. James is a member of the Java Community Process, and is currently working on Performance Metric Instrumentation (JSR 138) specification. James is a member of the Worldwide Institute of Software Architects, and holds industry certifications from Microsoft, Cisco, and Sun.

James can be reached at james@xquerykickstart.com.


Kurt Cagle

Kurt was a founding writer and frequent contributor for Fawcette's XML and Web Services Magazine, and has spoken on XSLT, SVG, and XSL-FO issues at more than a dozen conferences in the last five years. Kurt wrote his first book on XML in 1998, was one of the first authors to concentrate on the XSLT 1.0 specification for the Microsoft environment in Sybex's XML Developer's Handbook, and was a contributing author to the best selling Beginning XML for Wrox Press, among thirteen other books. Most recently, he wrote SVG Programming for Apress. He lives in Kirkland, Washington with his wife and two daughters, and writes novels when he isn't writing computer books.

Kurt can be reached at kurt@kurtcagle.net.


Per Bothner

Per has a computer science degree from Oslo, Norway and a Ph.D. from Stanford (1988). After graduating, Per worked for Digital and the University of Wisconsin before joining Cygnus in 1991. At Cygnus, he worked on program development tools, including library, compiler and debugger implementation of C++, and Chill. He was the technical leader of Cygnus's Java group from its formation, as well as GCJ (the "GNU Compiler for the Java platform," a Java ahead-of-time compiler based on the GCC suite and runtime environment). GCJ is based on his ideas and includes much of his code. Per is a founding member of the GCC Steering Committee.

Per implemented the original STDIO implementation used in Linux, the Emacs terminal emulator term, the original C Preprocessor library, has fixed bugs in XTERM and bash, and has worked on many GNU and other widely used programs. Per also developed Kawa, which compiles Scheme functions on the fly into Java classes. The Kawa framework is also being used to implement other languages, including JEmacs (a Java re-implementation of the Emacs text editor, including support for the Emacs Lisp extension Language) and XQuery.

Per can be reached at per@bothner.com.


James Linn

James is a coauthor of the book Practical Guide to Enterprise Architecture (Prentice Hall) and has seven years of experience in software development. He has worked with Java, C++, and MFC. He has a masters degree in biochemistry from the University of California—Riverside, and a masters degree in software engineering from Fairfield University. He is a consultant with Hartford Technology Services Company.

James can be reached at jim@raptureart.com.


Vaidyanathan Nagarajan

Vaidyanathan is the coauthor of two recent books on Java Technology: Professional EJB (Wrox), and J2EE Bible (Wiley). He has more than six years of experience in information technology. Prior to joining The Hartford Financial Services Group, Inc. as an enterprise developer, he worked as a consultant to Netscape Professional Services. He holds an MBA from a leading New England business school. He already has a masters degree in mathematics and computer science from the Indian Institute of Technology. His main interests include programming in Java and robotics.

Vaidyanathan can be reached at vnathan@msn.com.


Acknowledgments

    The Author Team

    James McGovern

    Kurt Cagle

    Per Bothner

    James Linn

    Vaidyanathan Nagarajan


The Author Team

We want to say thank you to all the people who contributed to this book and made our dreams become reality. This team was magical in that we found the elusive groove that our mentors have so often described.

Special thanks goes to the entire team of Sams Publishing for the support and professionalism given to this undertaking. We would like to thank our acquisitions editor Todd Green for giving us the chance to write the best-selling book on XQuery. Even though the team has never met him, it feels as if we have known him for years.

We would also like to thank Development Editor Tiffany Taylor and all of the other people required to make this book a success.


James McGovern

God has blessed my soul with the world's best wife, Sherry, and my son, Little James, to whom I owe my life and happiness. To my mother Mattie Lee and my father James, I would not be the person I am today without your encouragement and persistence. To my mother-in-law Soogia and late father-in-law Sylvester, thank you for including me as a member of the family.

I would like to say hello to my niece and nephews Karina (Alex), Nicholas, Ian, Kelon, Keifer, my aunt Daisy, my cousins Pamela, Annika, and Demesha.

To my peers at The Hartford Financial Services Group, Inc. (especially the EBC, ETSD, TAG, HTSC and PCAC), I thank you for providing a great place to moonlight and a challenge in many aspects.


Kurt Cagle

I want to thank and acknowledge my grandparents, Barney, Edna, Jo, Earl and Lylburn. As we go through the press of life with work and family, we do not always recognize the salient wisdom of experience that our families provide to us until it is too late. May I use their insights well, and remember and emulate their love, that I can pass them on to my own children. Thank you.


Per Bothner

I'd like to thank my partner, Nathan Williams, for love for more than nine years, and his support and patience while I worked on fun but not always profitable projects. Also thanks to my parents, my brothers Tor and John, my co-workers and management at Apple (my "day job"), the users of Kawa and Qexo who always challenge me to do better, and the general GNU and Free Software community.


James Linn

I thank my dog Trabbie, my cat Harlie (a.k.a. Wanda Widebody) and most especially, my wife Terria for her acceptance of this project and her forbearance in not hurting James McGovern like she wanted to.


Vaidyanathan Nagarajan

I would like to dedicate this book to my wife Padma, my parents, and my in-laws for giving me the wonderful opportunity to achieve my ambitions and goals through moral support and willingness.

I would like to extend my special thanks to James McGovern for giving me this wonderful opportunity to work with him on this book. He introduced me to the world of writing technology books last summer through Professional EJB.

Cheers to my team, who breathe technology at work: Andy, Cliff, Jess, Scott, Tom, and my manager Scott, who gave me the exceptional opportunity to work with this unique group.

MAY THE FORCE BE WITH TECHNOLOGY! Happy XQuerying!


We Want to Hear from You!

As the reader of this book, you are our most important critic and commentator. We value your opinion and want to know what we're doing right, what we could do better, what areas you'd like to see us publish in, and any other words of wisdom you're willing to pass our way.

As an Associate Publisher for Sams Publishing, I welcome your comments. You can email or write me directly to let me know what you did or didn't like about this book—as well as what we can do to make our books better.

Please note that I cannot help you with technical problems related to the topic of this book. We do have a User Services group, however, where I will forward specific technical questions related to the book.

When you write, please be sure to include this book's title and author as well as your name, email address, and phone number. I will carefully review your comments and share them with the author and editors who worked on the book.

Email: | feedback@samspublishing.com
Mail: | Michael Stephens Associate Publisher Sams Publishing 800 East 96th Street Indianapolis, IN 46240 USA

For more information about this book or another Sams title, visit our Web site at www.samspublishing.com. Type the ISBN (excluding hyphens) or the title of a book in the Search field to find the page you're looking for.


Introduction

by James McGovern

XML is an extremely versatile markup language, capable of labeling the information content of diverse data sources, including structured and semi-structured documents, relational databases, and object repositories. A query language that uses the structure of XML can intelligently express queries across all of these kinds of data, whether physically stored in XML, or viewed as XML via middleware. Query languages have traditionally been designed for specific kinds of data. Existing proposals for XML query languages are robust for particular types of data sources, but weak for other types. The XQuery specification is designed to be broadly applicable across all types of XML data sources.

The XQuery specification is one of the largest working groups of the W3C, and is composed of more than 30 member companies. The key component of XQuery that will be familiar to current XML developers is XPath. An XPath example such as //book/editor is also an XQuery syntax that indicates "find all book editors in the current collection." XQuery also has SQL-like capabilities, and might be used in the near future to query relational databases.

XQuery is a large part of the XML specification, and will be pervasive in future implementations of databases, search engines, and registries. There is no single book that addresses the thought process, architecture, and best practices for implementation that guide a user through the process. Our goal is to create such a guide. The market is looking for clear, detailed, factual (nonspeculative) information that will help them make informed technology decisions based on technical excellence and demonstrated ability.

The author team has never met a technologist that doesn't have a personal technology "religion" that features the best way to explain a given topic. In this book, the numerous examples will not contain a blow-by-blow narrative of what the code is doing. This is intentional. Publishers of big red books use this technique of printing out code to make the book thicker. This approach makes it hard for the reader to follow the topic, and provides no long-term benefit. In this scenario, the reader spends significant time understanding the variable-naming conventions, and other aspects that distract them from learning about the topic at hand.

The authors have also decided against printing out specifications, Javadocs, or similar abstracts. Again, many publishers use this tactic to make their books thicker and appear more valuable. Specifications are freely available for download from the various organizations, and therefore would not be of great value here.

The author team also wanted to avoid writing a book that contained highly speculative information on specifications that are not even close to release. Although we are early adopters of many of the technologies contained within this book, we will not create a book that has a limited shelf life.


Audience

Software vendors, magazines, and industry analysts will mention XQuery more and more in their commentary of the XML industry, as it is one of the latest standards. No doubt many book publishers will also jump on the bandwagon to create their own version of an XQuery book, but they will fail. The authors of this book set out on a journey to create a book we wanted to purchase ourselves, a book that is filled with practical architecture advice and solid implementation details. Our mission was to provide an overview of the implementation of the XQuery language, because no one book can cover everything. The focus was to enlighten our readers to this standard and guide you to creating the perfect solution for your own organization's needs. With this thought in mind, we thought it would be useful to provide examples for both the Java and .NET communities.

If you are the CTO of an Internet startup, architect of a large corporation, database administrator of a multi-terabyte data warehouse, or even a project manager, and appreciate a no-frills introduction to XQuery, this is the book for you. We have diligently tried to create the ultimate guide that explains both the architecture and implementation of XQuery. This book strives to provide enough detail to satisfy our audience. This book provides leading-edge yet practical examples to illustrate how Web services can be applied to existing business and technology initiatives. Where additional detail is required, this book will provide the stepping stone and point out additional sources of information.

Sometimes an author's anecdotal experiences might appear. Rather than edit it out for aesthetic reasons, we have decided to keep it in place, as it might provide insight and additional guidance to our readers. This book should be considered as the authoritative source and reference for XQuery.


Contents of this Book

XQuery, in its implementation form, is a powerful technology that will contribute to the successful development of many large mission-critical enterprise applications, yet on its own, it's relatively powerless. XQuery will become a core part of databases, document and content management systems, search engines, Web services, portals, and service-oriented architectures. When XQuery is combined with powerful languages such as Java or .NET, in conjunction with UML, design patterns, and a good software development methodology, the power of an organization to create meaningful solutions with strong value propositions is the key to XQuery and this book.

The authors recommend that you read each chapter in order, as each chapter builds on the previous chapter. Of course, as a reader, you have the option of reading the chapters in any order you desire, but we recommend at least reading a particular section in order.

This book is divided into three parts, plus appendixes:

  • Part I provides a brief introduction into XML, and serves as an introduction for those who have no prior experience with XML. It is limited to a cursory overview, and goes into enough detail for you to understand the concepts behind XQuery. Throughout the text, we will also provide additional sources of reference for XML where appropriate.

  • Part II introduces the XQuery specification and its atomic breakdown. This will help you form a mental model of how a programming language such as XQuery works, including when things go wrong and determining performance-related issues. Some of the topics that will be covered are optimization and representation issues.

  • Part III focuses on several implementations of the XQuery language and provides you with insights into both Microsoft and Java specifics. You will learn about Microsoft's .NET strategies and their incorporation of XQuery into their product suite. You will also work through detailed examples using Java to build solutions.

  • The first appendix provides a basic overview of XML, for those who are unfamiliar with it or need a refresher. The second appendix lists additional sources of information that can help ensure the success of your system.


What this Book Is Not

This book assumes from the beginning that readers understand the concepts behind XML in general, and have some experience with it. (If you aren't, Appendix A can help you get up to speed.) The coverage of any Java, PHP, or .NET topics will be limited to the APIs needed to develop an XQuery solution. All of the programming within this book assumes a basic understanding of object-oriented principles.

The author team has intentionally avoided recommendations of any particular technologies and/or methodologies used to deploy them. Methodologists are better suited to defining and selecting the appropriate methodology for an organization. Products change their feature sets at a dramatic rate, constantly leapfrogging each other and making any recommendation outdated.

For many of the coding examples, UML notation is used. This book will limit its UML examples to use cases and class diagrams. Within each of the diagrams, we will limit the notational complexity to the level typically covered in an introductory book.

Coding examples might also periodically refer to design patterns for which you should have some familiarity. A basic understanding of the most frequently used patterns, such as adapter, composite, chain of responsibility, observer, and factory should be sufficient. If you do not have familiarity with design patterns, we suggest the immediate purchase of Analysis Patterns: Reusable Object Models by Martin Fowler (Addison-Wesley), and Design Patterns by Erich Gamma (Addison-Wesley).


Source Code

Source code and updates for this book can be found at http://www.samspublishing.com/kickstart. This book will appear in the list of titles on the Kick Start page. You may also type the book's ISBN (0672324792) into the Search window without dashes, click Search, and you'll be taken to a page with links to the source code.

The source code used in this book is also available at http://www.xquerykickstart.com. The code contained within this book is freely available for download and may be used as your heart desires, with the sole restriction that you may not claim that you are the author. The publisher, authors, or their employees do not provide any form of warranty on the code or guarantee its usefulness for any particular purpose.

This book is 100% error-free! Not! If you believed us even for a second, I know of a bridge in Brooklyn that is for sale cheap. The author team and editors have worked laboriously to bring you an easy-to-understand, accurate guide on XQuery. The author team's motto is "eschew obfuscation." If you find mistakes, we would appreciate you contacting us at our respective email addresses. We equally appreciate any comments, suggestions, and praise you have for this book.


Part I: XML Concepts

1 XML: The Foundation of XQuery

2 Understanding the XPath 2.0 Specification


Chapter 1. XML: The Foundation of XQuery

by James Linn

IN THIS CHAPTER

  • Background

  • What XML Offers

  • Portable Data

  • How XML Works

  • The Evolution of XML

  • The XML Family

  • In Brief


Background

Since its first introduction, the Extensible Markup Language (XML) has generated extraordinary interest and support in the information technology world. It is a key part of Web Services and has become a major player in the Enterprise Application Integration (EAI) sphere. It is used by Scalable Vector Graphics (SVG) programs to render scalable graphics in an application- and platform-independent manner. It is used by advanced browsers to produce Web pages. XML databases have been created specifically to store XML, and existing relational databases have been modified to store and work with XML.

XML can be used in conjunction with largely unstructured documents such as Web pages, where it enables the separation of data from presentation. This is extremely important when the same content is to be displayed in different presentation devices, such as browsers and wireless devices.

XML can also be used as a pure data vehicle, allowing Java applications running on Unix boxes to communicate with COBOL programs running on IBM mainframes, as one example. These XML data objects can be stored in a database or filesystem and be queried for their information.

These are just a few examples of where XML has gained importance. For a full list of all current XML topics, go to the World Wide Web Consortium (W3C) Web page at http://www.W3C.org, which is the authoritative center for all XML initiatives. It is not an exaggeration to say that XML has become the language for data interchange.


What XML Offers

XML provides a platform- and language-independent method to mark up data. XML packages the description of what the data means with the data. This is the crux of why XML has become so important: It offers a means of packaging metadata (data about data) with the data.

In addition, XML is standards-based. The W3C is the governing body, and it acts as the source for all the specifications that govern the various XML initiatives. No major XML initiatives are based on proprietary protocols. All of the major players—Microsoft, Sun, IBM, HP and so on—adhere to the standards established by the W3C. The adherence to open standards is one of the major reasons that XML has gained so much acceptance so quickly.

XML is important for both content-intensive conventional Web pages and as a data transportation medium. For content-intensive applications such as Web pages, XML allows the separation of data from the formatting of that data. This makes it relatively easy to present the same content via many different vehicles and to dynamically alter Web pages. In addition, by attaching tags that describe the meaning of the data to data elements in a Web page, it is much easier to conduct meaningful searches across the World Wide Web (WWW).

Currently the World Wide Web is the largest repository of data in the world. But mining that data is very difficult. The more that XML becomes embedded in the WWW, the easier it will be to search.

However, much of the current activity in XML involves the use of XML purely as a data vehicle. Once again, XML's importance lies in the fact that XML packages the description of what the data means (the metadata) with the data itself. This is in sharp contrast to all other messaging formats, where the metadata lies in sources external to the message itself.


Portable Data

Because all the information needed to interpret an XML document file is included with the data itself, XML provides a totally portable means to move data. It is the perfect vehicle for carrying data such as is involved in a function call, from one application to another, one language to another, and from one operating system to another. This is why XML has created a great deal of interest in the EAI space.

Enterprise application integration can use many paradigms to enable applications running in different spaces to communicate with each other. Remote procedure calls (RPC) was one of the first developed. RPC can work well when all communicating programs are written in the same language (say C), and they are all running on the same platform (say Unix). Java Remote Method Invocation (RMI) is an example of RPC in the Java world.

The Common Object Request Broker Architecture (CORBA) was developed to enable the passing of data in a world of heterogeneous objects. A variety of factors has limited its penetration into the business world.

One of the major impetuses for the development of messaging applications such as IBM's MQ Series and messaging frameworks such as Sun's JMS (Java Messaging Service) was to aid in the enormous task of integrating the large number of disparate applications that exist in most corporations. It is interesting to note that as integration engines such as IBM's MQ Series Integrator have matured, XML is becoming more and more common as the format for the MQ or JMS messages sent between applications.

Using XML to carry the RPC parameters from application to application is the basis of the Simple Operating Application Protocol (SOAP). SOAP is the basis of Web Services, which enables the integration of applications from anywhere and anyone on the planet via the WWW. SOAP was the first protocol developed for RPC via XML. After SOAP, a more general specification for RPC using XML called XML-RPC was developed by the W3C.

The Business-to-Business (B2B) sector of the e-commerce space has also moved almost exclusively to using XML for data exchange. Electronic Data Interchange (EDI) was the original method of B2B data interchange. It works well, but is complex and expensive to set up and run. The key to EDI is the exhaustive Trading Partner Agreements that have to be negotiated between the two parties. This is where the metadata for the exchanged data streams is specified, among many other things. Although there are data format standards in some industries, most Trading Partner Agreements involve a great deal of reinventing the wheel. The expense and complexity of this effort puts EDI beyond the reach of most small and medium-sized businesses.

EAI technologies such as Web Services (which is XML-based) or messaging (which often uses XML as the message format) are much simpler and quicker to implement, and have put such B2B projects such as supply chain integration within the reach of most businesses.


How XML Works

As we have indicated, the essence of XML is in a deceptively simple idea: The descriptive content of the message is carried in tags attached to the data. The tags describe the data, and they indicate where the data begins and where it ends.

Here's an example of an XML data element:

<patient number>AD2002302</patient number>

This fragment demonstrates the use of tags to couple the description of the data with the actual data value itself.

MORE XML RESOURCES

This is an extremely abbreviated look at XML. It only gives the barest outline of the language. We provide a longer overview in Appendix A, "XML Fundamentals." For an in-depth look at XML and how to use it, I suggest a good XML book. Here are a few suggestions:

Essential XML Quick Reference by Aaron Skonnard and Martin Gudgin (Addison-Wesley), Sams Teach Yourself XML in 24 Hours by Michael Morrison (Sams), Professional XML by Mark Birbeck et al (Wrox), or the big kahuna itself, XML Bible by Elliotte Rusty Harold (Wiley).

The data can be further qualified by attaching attributes to the tag. Using the previous example, a patient visiting a hospital can have several identifiers associated with him or her—a number that uniquely identifies that person across all hospital visits, and an account number that is specific for this particular visit. In this example, the patient number is the account number, and it is indicated as follows:

<patient number type="account number">AD2002302</patient number>

Now let's take a more detailed look at the structure of XML documents, demonstrating all the necessary constructs.

Rules of XML

XML documents must be well-formed to be processed by an XML processor. The following basic rules must be followed for an XML document to be considered well-formed:

  • The entire document must be encompassed by a single element. The first element in a document is called the root element, and its tags must encompass the entire document. It is important to grasp the idea of the root element as wrapping all the data in an XML document. Therefore, if you have hold of the root element, you have all the document in your grasp.

  • When nesting elements, the internal element must be entirely contained by the external element. This flows from the first rule and applies the idea of nesting throughout the document.

  • All attributes must be quoted. Either single or double quotes are acceptable.

  • Tags are case-sensitive.

  • You must use end tags. If you forget an end tag, the XML parser will throw an exception. There is one exception to this—empty tags can carry the end marker as follows:

    <title/>

    which is the same as

    <title></title>

  • An XML declaration should begin your document. Here is an example of an XML declaration:

    <?xml version="1.0" encoding="UTF-8">

    Developers and even software vendors will often omit this step when they have created the applications that are the only consumer of the XML document (see the XML files that BEA WebLogic uses for an example). It is strongly suggested that developers acquire the habit of starting their XML documents with an XML declaration. The default encoding for XML files is UTF-8, so, strictly speaking, the encoding declaration in the previous sample is unnecessary. However, a number of XML renderers will not make this default assumption, so it is good practice to include an encoding directive as in this example.

If you are using Microsoft's Internet Explorer (IE) 5.5 or later as your browser, you can use it to check to make sure that your XML is well-formed. If IE successfully opens the file and displays the XML, it is well-formed. Notwithstanding the fact that IE will open an XML file that doesn't include a declaration without complaint, use XML declarations. Listing 1.1 shows an example of a well-formed XML document.

Listing 1.1 A Well-Formed XML Document
<?xml version="1.0" encoding="UTF-8"?>
<PatientRecord>
    <PersonalData>
        <FirstName>Sam</FirstName>
        <MI/>
        <LastName>Spade</LastName>
    </PersonalData>
    <BillingData>
        <PreviousAccNum/>
        <patientNumber type="account number">AD2002302</patientNumber>
        <patientNumber type="visitId">A20020712-5432</patientNumber>
    </BillingData>
    <HospitalData>
        <RoomNum location="ER">LH25</RoomNum>
        <RoomNum location="EastWing">D14</RoomNum>
        <RoomNum location="Morgue">Drawer3</RoomNum>
        <Physician type="Attending">Dr. Faustus</Physician>
    </HospitalData>
</PatientRecord>

The things to note from this example are the following:

  • The entire document is enclosed by the <PatientRecord></PatientRecord> tags. In this case, the root element or document element is the PatientRecord element.

  • The MI and PreviousAccountNum elements are blank, so the shortcut for a blank element (for example, <title></title>) is used.

  • Elements are nested entirely within other elements. In other words, the start tag and the end tag of the PersonalData element are both within the PatientRecord start and end tags. There is no limit to the nesting in an XML file.

  • The HospitalData element contains multiple other elements.

SHOP TALK: WHAT SHOULD BE AN ATTRIBUTE AND WHAT SHOULD BE AN ELEMENT

There is some debate about when to use an attribute and when to use child elements. For example, in Listing 1.1, the RoomNum element could have been recast as

<HospitalData>
    <Locations>
        <Room>
            <Type>ER</Type>
            <Num>LH25</Num>
        </Room>
            ... for the other rooms.
    </Locations>
</HospitalData>

My approach is that if the datum applies to or modifies another datum, it should be an attribute. If the two data are independent of each other, make them elements. This is offered as a suggestion; you will have to chart your own course on this.

As you have probably noted, XML messages are verbose. An XML message will be 2–4 or more times larger than a fixed-length record. In some applications, the extra size of the message makes using XML impractical.

Although it is true that the tags that wrap the data provide a description of the data, it is usually necessary for an external authority to specify exactly what name should be associated with a particular datum. For example, one company might use PartNum, the next company uses partNum, and the third company uses partnum. These differ only in case, but are completely different names as far as an XML parser is concerned. Several industries have established agencies to codify the XML messages for their industries. For example, the insurance industry has ACORD (http://acord.com), and the financial industry has IFX (http://www.ifxforum.org, choose IFX). In addition, two competing standards for business messages have been established: ebXML (http://www.ebxml.org) and Biztalk (http://www.biztalk.org). Both standards aim to provide message specifications to govern all business messages.

PARSING XML

Just as you need an application such as a Web browser to read HTML, an XML parser is required to work with XML files. For those of you working with Microsoft tools, you can use the MSXML tool from Microsoft (download it at http://msdn.microsoft.com/downloads/default.asp; go to Web Development, then XML Tools). Java aficionados will most likely look to the Xerces parser from the Apache project (download it at http://xml.apache.org/xerces2-j/index.html).

A standard such as Biztalk allows for rapid data exchange development. Tools exist that will quickly generate the code necessary to map existing messages into Biztalk-compliant XML. Suddenly, B2B projects such as supply chain integration become within reach of most businesses.

XML Validation

To be considered valid, an XML document must be well-formed, plus satisfy a set of user-defined rules. The capability for the developer to define the validation rules is what puts the extensible in XML. With HTML, users are unable to add elements and definitions. They have to work with what is defined by the W3C for the particular version of HTML that they are targeting. The need to give users the power to define elements was the initial impetus behind the development of XML.

The user-defined rules can take two forms: Document Type Definition (DTD) or XML Schema. DTDs are taken from SGML, and are written in their own language, which is very different from XML. DTDs come from the content delivery arena; hence the only data type that they recognize is strings. Because everything presented in a browser is a string, DTDs are perfect to define HTML versions. When HTML went from version 1.0 to 2.0, what changed was the DTD. Browser vendors then modified their browsers to work with the new DTD. If you want to use a DTD with an XML document, a DOCTYPE line such as the following must be in the document after the XML Declaration:

[View full width]

<!DOCTYPE ejb-jar PUBLIC '-//Sun Microsystems, Inc.//DTD Enterprise JavaBeans 1.1//EN'
   'http://java.sun.com/j2ee/dtds/ejb-jar_1_1.dtd'>

When a validating XML parser encounters this line, it will attempt to locate this DTD. It will then compare the XML document to the rules contained in the DTD. If it finds no errors, it will continue; otherwise, it will throw an exception.

XML Schema are much newer, and the concept is borrowed from the database world. XML Schema are written in XML. This means that XML Schema can be consumed by XML tools and modified if needed. It also means that there is one fewer language for you to master. In addition, XML Schema enable the user to specify the type of a data member. A full discussion of XML Schema is beyond the scope of this book, but an excellent introduction to the use of XML Schema appears at http://www.w3.org/TR/xmlschema-0/.


The Evolution of XML

Both Hypertext Markup Language (HTML) and XML have their roots in Standard Generalized Markup Language (SGML). This language was developed by Dr. Charles Goldfarb in the 1970s as an application-independent method of describing and presenting data in documents. Markup is a method of adding structural, presentational, and semantic information to a document alongside its content. SGML used the idea of encapsulating all its markup instructions in tags. The meanings of these tags were explained in a document called a DTD. SGML DTDs enabled users to incorporate changes in the markup and have them understood.

XML is a subset of SGML in that it only uses a subset of SGML's capabilities. HTML is an application of SGML. It uses tags such as <h1>Name</h1>, which declares Name to be the content of an <h1> element, a top-level heading that is usually rendered in large-sized font, and is usually visualized in Web browser applications. It also uses a DTD. This DTD is set by the W3C, and there are DTDs for each HTML version. For example, HTML Version 4.0 can be described by any of these six DTDs:

[View full width]

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0//EN" "http://www.w3.org/TR/REC-html 40/strict
  .dtd">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR
  /REC-html40/loose.dtd">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Frameset//EN">
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Frameset//EN" "http://www.w3.org/TR/REC-html40
  /frameset.dtd">


The XML Family

The following is a list of the protocols that are members of the XML family and that are important to the understanding of XQuery. Each protocol is undergoing continuous improvement, and is governed by a W3C working group. The most up-to-date and authoritative source of information on the members of the XML family can be found at the W3C's Web site: http://www.w3.org.

XSL

Extensible Stylesheet Language (XSL) is an XML language that allows for the description of style sheets for the presentation of XML. The XSL style sheet contains the information needed to present the data in an XML document. Using the XML-XSL paradigm completely separates data from presentation.

XSL has three sub-protocols:

  • XSL Transformations (XSLT)

  • XML Path Language (XPath)

  • XSL Formatting Objects (XSL-FO)

XSL-FO is extraneous to XQuery, and will not be discussed in any detail.

XSL has generated a great deal of interest in the Web development world because it can greatly speed not only the creation of Web pages, but also make their enhancement and maintenance much easier.

XPath

XML Path Language enables access to individual parts of data elements in the XML document by the use of statements that express the path to the object desired. It is used by XSLT and XQuery to select out parts of an XML document that the style sheet wants. The XPath protocol is central to XQuery—so central that XQuery 1.0 was developed in conjunction with the XPath 2.0 specification.

XPath works with an XML document as a tree of nodes. The current XPath specification details seven kinds of nodes:

  • Root

  • Element

  • Attribute

  • Text

  • Comment

  • Processing instruction

  • Namespace

XPath Expressions

XPath statements can best be thought of as expressions. XPath expressions can return a node, a Boolean, or a number. (Numbers in XPath are IEEE 754 floating-point numbers or integers.)

Given Listing 1.1, the sample XML document given previously, you could get the text node that contained the value of a patient's name with the XPath expression /PatientRecord/PersonalData/FirstName. The expression /PatientRecord/HospitalData/RoomNum would return a set of three text nodes.

The XPath expression / returns the root node of the document.

To select attribute nodes, use the @ sign. The XPath expression /PatientRecord/HospitalData/RoomNum/@location would return a set of three attribute nodes.

To use an expression, you pass a path into the expression. /PatientRecord/PersonalData/FirstName is an example of a path. XPath has relative and absolute paths. Relative paths are processed from the current context. Absolute paths are processed from the root node. So, /PatientRecord/PersonalData/FirstName is an absolute path. It will yield the same results no matter what the context is. FirstName is a relative path. It yields results identical to /PatientRecord/PersonalData/FirstName, if the current context is the /PatientRecord/PersonalData node. It yields an empty node set if the current context is some other node, such as the /PatientRecord/HospitalData node.

XPath incorporates the notion of location in a tree. The current location is referred to as the context. The context is noted by (.). So name(".") would return the name of the current node.

Actually, the context contains much more than just the current node. In addition to the current node, it contains:

  • The number of nodes in the list, and the position of the current node as numbers (floating-point numbers, not integers). The current position is guaranteed to be a nonzero positive integer that is less than or equal to the number of nodes in the list.

  • A set containing all the variables that are in scope, their names and the data associated with them.

  • A set containing the namespace declarations that are in scope for the declaration.

  • A list of all the XSLT and XPath functions available.

Navigation in XPath

To navigate the tree of nodes in an XML document, XPath uses the concept of axes. An axis is a collection of nodes that satisfy the current navigation criteria.

There are 13 axes in all. Here is the collection:

  • The child axis contains all the child nodes of the present context. This is the default axis, so all the XPath expressions that we have used heretofore resolve to the child::xpath expression in the unabbreviated syntax. Attribute and namespace nodes that are children of the context will not appear in the child::node collection.

  • The parent axis contains the node that is the parent of the current context. parent axis can be abbreviated with a double period (..).

  • The self axis contains the context node itself, and can be abbreviated with a period (.).

  • The attribute axis contains all the attributes of the context node. If the context is not an element node or if it has no attributes, an empty node set will be returned. The attribute axis can be abbreviated with the at sign (@).

  • The ancestor axis contains all the nodes from the parent of the context, up to and including the root node itself.

  • The ancestor-or-self axis returns the same thing as ancestor axis plus the context node.

  • The descendant axis is the same as the child axis with the addition of all the comment nodes. Attribute and namespace nodes are not returned in the descendant axis.

  • The descendant-or-self axis is the descendant axis plus the context node.

  • The preceding-sibling axis contains all the sibling nodes that precede the context node. Note that, if the context is an attribute or namespace node, the preceding sibling axis will be an empty set.

  • The following-sibling axis contains all the sibling nodes that follow the current node.

  • The preceding axis contains all the element, text and comment nodes of all the preceding siblings of the context. This axis will not return any ancestor or namespace nodes.

  • The following axis contains all the element, text and comment nodes of all the following siblings of the context. As with the preceding axis, the following axis will not return any ancestor or namespace nodes.

  • The namespace axis will contain all the namespace nodes of the context node if it is an element node. If it is not an element node, the axis will be empty.

XPath Functions

In addition to path expressions, XPath has functions. The following is a few of the functions from the core XPath functions that all XPath implementations must support. The list of functions available to XPath is quite extensive. This list is only partial coverage of what is available:

  • name()— This function takes a path expression and returns the name of the element, including the namespace governing that element. name("/") returns the name of the root of the XML document. The name() function is invaluable to establish where you are when moving through a set of nodes.

  • last()— The last() function gives you the size of the current context.

  • position()— This function returns the position within the context.

  • count(node-set)— When given a node set, this function will return the size of the set as a number.

  • string name(node-set)— This returns the extended or qualified name of the first node in the node set. The extended name is the namespace URI plus the node name in the following format: namespace-URI:local-name. The nodes are searched in document order.

  • string local-name(node-set)— This function returns the local part of the extended name of the first node in the node set that was passed in.

  • string string(object)— The string function converts an object to a string. If the object is an element node, the extended form of its name is returned. If the object is a number, its string value is returned. Booleans are converted to true or false as appropriate. If a node set is passed in, the first node in document order is processed. The XPath specification warns that string(object) should not be used to format a number into a string for presentation to users. XSLT function format-number and xsl:number element are intended for this use.

  • number string-length(string)— Gives you the length of the input string.

  • string normalize-space(string)— This is a very useful XPath function. It strips the whitespace from the beginning and end of the input string and reduces all of the interior whitespace to a single space. Whitespace in XPath is defined as ASCII x20 (space), x9 (horizontal tab), xD (carriage return), and xA (linefeed).

  • boolean boolean(object)— Analogous to string. Returns the Boolean value of an object.

  • boolean not(boolean)— Returns the obverse of the input Boolean.

  • boolean true() and boolean false()— Used when you need a Boolean true and false, respectively. Note that the strings true and false have no intrinsic meaning in XPath.

  • number number(object)— Converts the input object to a number. The rules are that numbers are converted to their string value, and Booleans are converted FOR BOTH to 0 or 1. You should only use the number function on objects that confirm to the XPath syntax for a number (digits[.digits]).

  • number sum(node-set)— Sums all of the numbers in the node set.

Node Tests

In addition to functions, XPath includes a set of node tests. Several of these node tests have a functionality similar to filters, in that they enable you to select out certain node types. These node tests are as follows:

  • text()— path-expression/text() selects out all of the test nodes from the path expression. The XPath expression /PatientRecord/HospitalData/*/text() will return a concatenation of all the text nodes in the HospitalData element, which will be LH25D14Drawer3Dr.Faustus.

  • processing-instructions() and comment()— These perform in a manner analogous to text() for processing instruction and comment nodes, respectively.

XPath Wildcards

XPath includes a set of wildcards. The asterisk (*) selects all of the element nodes in the current context. The * can be used with the attribute node indicator @. @* selects all the attribute nodes in the current context. When used with a namespace name, the * returns all the nodes in that namespace. When used with Listing 1.1, the sample XML document, hosp* returns all the nodes in the hosp namespace. @hosp* returns all the attributes in the hosp namespace. Because we didn't define a namespace for this XML sample, in this case @hosp* returns an empty set.

The node() function acts like a wildcard in that it returns all the nodes in the current context.

In addition, XPath has the // notation to flatten out the document tree. It indicates that zero or more elements may occur between the double slashes. So, when used on the sample XML document, the XPath expression //RoomNum will return the same node set as PatientRecord/HospitalData/RoomNum. Use the // wildcard sparingly. It slows the processor down enormously, as it is forced to traverse all the nodes in the document. In addition, if the document changes and another element is added with RoomNum elements, XPath statements that used to work could possibly break. Having said that, using the // wildcard in certain cases can make your XPointer statements more portable between XML documents. Any changes to the PatientRecord/HospitalData path will break the XPath expression PatientRecord/HospitalData/RoomNum, whereas //RoomNum might possibly still return what you desire.

Unabbreviated XPath Syntax

Up until now, we have been working with XPath's abbreviated syntax. PatientRecord/HospitalData/RoomNum is an example of abbreviated syntax. To work with axes, you will have to learn to use XPath's unabbreviated syntax. If the current context is the root node, the unabbreviated version of the preceding XPath expression is child::PatientRecord/child::HospitalData/child::RoomNum. Note the syntax: axis-name::xpath-expression.

The underlying concept of XPath is that of the node. It is important to keep that in mind as you study this section. XPath works with a node set to select subsets of nodes. It doesn't care about the data in the attributes or elements. In addition to selecting nodes, XPath provides a set of filters or predicates to winnow the node(s) of interest from the larger set returned in one of the preceding axis calls.

XPath Selection Syntax

The square bracket ([]) syntax allows for the selection of a node at a particular position in a node set. You can use a number or a function that returns a number or position inside the []. //RoomNum[2] selects the second RoomNum element node from the set returned by the XPath expression //RoomNum. Note that this is an array numbering system that starts with 1, not 0. //RoomNum[last()] will take you to the last node in the set. position() returns a number equal to the position of the context in the node set. Therefore, //RoomNum[position()+1] will select the next RoomNum node in the set.

MORE ON XPATH FUNCTIONS AND EXPRESSIONS

The first place that you should look for more information is the XPath section of the W3C site (http://www.w3.org/TR/xpath). You can also refer to a text dedicated to XPath and XSLT, such as Essential XML Quick Reference by Skonnard and Gudgin or XSLT and XPATH: A Guide to XML Transformations by John Gardner (Prentice Hall). To fully appreciate the power and versatility of XPath expressions, I refer you to a text that enumerates these functions in detail, such as XSLT by Doug Tidwell (O'Reilly) or XSLT Programmer's Reference by Michael Kay (Wrox).

Although the most common result of an XPath expression is a node set, XPath can return a Boolean, a number (this is a floating-point number), or a string.

XPath is critical for working both with XSLT and with XQuery. XPath is your path into an XML document.

XSLT

XSLT provides a mechanism to dynamically transform XML documents. XSLT can transform the content or the format of an XML document, or both. Because XSLT style sheets are written in XML, they inherit all of XML's benefits: platform and language independence, and the capability to be further modified by standard XML mechanisms.

XSLT, when combined with XSL, provides the presentation for an XML document. The XSL/XSLT combination is used when the same data is served to different clients, such as mobile phones using WML or browsers using HTML. Each client will have its own XSL style sheet. The different style sheets will have different output types. The style sheet for the browser will specify that the output be HTML, while the style sheet for the mobile phone will specify Wireless Markup Language (WML). It is important to note that the style sheets may or may not specify different elements for the page. In other words, in many cases, the only change needed to present data from an XML document in different clients is the choice of the output type. Other output types are text and XML. The XML output can be fed to a Formatting Object (FO) such as the PDF FO, and a PDF file will be produced. Or the input XML can be processed with a style sheet containing SVG elements to produce a file that will display graphics when viewed in a viewer.

XSLT can be used by an XSL style sheet to combine several XML documents. The combined dataset can then be formatted and presented to the client.

XSLT has pretty much everything that you would expect from a programming language:

  • Branching and looping capabilities

  • Conditional logic

  • Variables

XSLT uses a template-matching paradigm to separate out the nodes upon which it wants to act. The nodes upon which to act are located using XPath statements, functions, and filters. The output of the transformation is a modified or transformed node tree. The nodes of this transformed tree are inserted into the XSL style sheet, and the result presented to the client.

As an example of XPath modifying the node tree, if we start with the XML in Listing 1.1 and apply the simplest XSL style sheet, the style sheet will look like this:

<?xml version='1.0' ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/" />
    <xsl:apply-templates/>
</xsl:template>

All the text nodes are returned when this is applied to the XML. If we modify the xsl:apply-templates by adding a select, as in the following:

<?xml version='1.0' ?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/" />
    <xsl:apply-templates select"PatientRecord/PersonalData"/>
</xsl:template>

we will just get back SamSpade.

There are two methods of displaying HTML from the XML data that has been transformed by XSL. Netscape 6.1+ and IE 5.5+ have the engines necessary to turn an XSL–XML for all combos into HTML. If you must deal with older browsers, you can use the MSXML tools (download at http://msdn.microsoft.com/downloads/default.asp, go to Web Development then XML Tools) or Cocoon (download at http://xml.apache.org/cocoon/index.html) to generate the HTML on the server and send it to the browser.

This is a very general look at XSLT. XSLT is a complex subject, and far beyond the scope of this introduction. If you want to explore XSLT in any depth, we refer you to one of the excellent books that I have previously mentioned.

In this modern world of increased software development, expectations coupled with decreased budgets, XSL combined with XSLT makes the maintenance and enhancement of Web sites much easier and cheaper.

XPointer

XML Pointer Language (XPointer) is based on XPath, and it provides a mechanism to address (point to) structures that are internal to an XML document. It differs from XPath in that it can be used to point to objects whether or not they have an ID associated with them. XPointer provides broader capabilities to probe the interior of an XML document, looking for properties such as element types, attribute values, character content, and/or relative position.

Uses for XPointer include the following:

  • Including a fragment of a resource in a URI

  • Limit addressing to a point or a range in an element

  • The location of information by string matching

XPointer is another protocol that is strongly linked to XPath and is evolving with it.

XLink

XML Linking Language (XLink) is an XML language for describing links between documents. It is an enhancement of the existing HTML hyperlink mechanism.

With HTML, the only way to set a hyperlink to another place in a document or to another document is to use the <a> or anchor tag. The HTML linking mechanism has the following limitations:

  • Uses Uniform Resource Identifiers (URIs) to specify locations.

  • Is unidirectional in that the <a> tag indicates the start of the navigation, and the URI indicates the end point.

  • Both the start and the end of the link are statically expressed. The start is always the present document and the end is the URI.

  • The effect of clicking on a hyperlink is determined by the browser. There is no mechanism in an HTML hyperlink to specify what should happen when the hyperlink is clicked.

XLink provides greatly increased functionality to the hyperlink. Using XLink, it is possible to

  • Describe a linking relationship between more than two resources.

  • Specify metadata that applies to the link.

  • Describe links between two resources in documents other than the linked resources.


In Brief

  • XML is the most important information technology innovation to come out in the last five years. It is used in an enormous range of places and has an almost inexhaustible number of uses.

  • XML is language- and platform-independent and based on open standards.

  • XML is a powerful and versatile means for handling data because it carries all of its metadata with it in the tags around the data elements and in the structure of the XML document.

  • The construction of XML documents follows a set of rules. There is a set of rules that all XML documents must adhere to. Other rules can be generated by the user. These rules are contained in DTDs or XML Schema.

  • XSL, XSLT, XPointer, XLink and XPath are the other members of the XML family. XPath is of particular importance because XQuery uses XPath expressions and functions to locate specific data elements within XML documents.


Chapter 2. Understanding the XPath 2.0 Specification

by Kurt Cagle

IN THIS CHAPTER

  • In Search of Context: XQuery and XPath

  • The Need for XPath

  • Directories and Nodes

  • XPath 1.0

  • XPath 2.0's New Features

  • Understanding Sequences

  • Handling Aggregate Functions

  • The tokenize() Function

  • Working with External Resources

  • Cutting Verbosity with XPath Command Structures

  • Regular Expressions

  • Of Data Types and Dates

  • In Brief


In Search of Context: XQuery and XPath

XPath originally emerged at the intersection of XLink (which defines a method of using URLs to describe locations within a document) and XSLT, which provides XML transformations based upon some demonstrable set of patterns. As will be explained later in this chapter, XPath provided a way of extracting pieces of the tree and passing them on as a sequence of nodes to other processes.

XPath is powerful, but much like regular expressions, XPath can be cryptic and difficult for non-XML specialists to write. A note submitted to the W3C by AT&T, the University of Washington, and the University of Pennsylvania in 1998 (http://www.w3.org/TR/NOTE-xml-ql/) proposed a simplified query language that could be used as a "SQL" for XML. This proposal in time became XQuery.

What is the specific need for yet another query language? XPath, as previously mentioned, can be cryptic. It also doesn't readily handle the presentation of information into alternative forms. XSLT can do the latter quite well, but XSLT's template-based design, recursive structures, and mixed namespaces can be daunting, especially to new users. XQuery occupies a middle ground—sufficiently powerful that it can handle formatting without a lot of complications, and sufficiently simple that anyone with familiarity in languages such as SQL can quickly learn and use XQuery.

XQuery has the greatest support among database vendors, who see the language as providing a consistent and easy-to-use interface between their existing offerings and the increasingly sophisticated requirements of XML-based Web services.

The intermediate role of XQuery cannot be overemphasized. XQuery is often part of a pipeline with other tools—a Web services interface, an XSLT transformation, DOM code, or some other XML-related technology operation.

Currently XQuery, as defined by the W3C, is strictly a query language. Unlike SQL, XQuery does not let users create new tables (or XML structures), update the contents of a given element or elements, or identify legitimate users. Although this asymmetry might seem odd, it makes sense when you consider that XQuery is a much more generalized language for handling data access than SQL is.

After XQuery stabilizes and enough reference implementations emerge to show how viable the language really is, it's likely that the W3C XQuery working group will set up a specific task force to abstract the update scenario. After this happens, database programming (and programming in general) will probably change dramatically.

XML has three principal attributes that make XQuery such an important language to follow:

  • Abstraction— XML, being metadata, can create abstractions of nearly any data structure or state. Because of this, the state of nearly all types of processes can be rendered as an XML infoset and queried.

  • Transportability— Because it employs universal character encoding (Unicode), XML can be expressed as text that can be interpreted the same regardless of the platforms or languages of the producing and consuming endpoints. This makes it extraordinarily useful as a transport language.

  • Openness— XML exists under a royalty-free license, which means it is a specification that cannot be changed except by consensus of the W3C. This stability and the lack of licensing costs make XML an increasingly common solution in almost any environment.

These same principles also apply to XQuery itself. As a common, consistent vehicle for performing abstract data queries on any data source, not just relational databases, XQuery might soon be in a position to provide a common syntax for nearly any type of information query.


The Need for XPath

A database is useless if there is no easy way to access the data within it. The structure (or more properly, the meta-structure) of the data is in many respects the most significant aspect of any database. The capability to address the data that you are seeking is thus one of the very first aspects of building any kind of data store.

In a relational database, the records created are essentially flat—there is only one field of information for any given type of information in the record. XML does not have this distinction. An XML document can have one or many duplicate columns, with the relationships between the columns having arbitrary complexity, depending upon the schema that describes the document. A typical Web page, for instance, can be arbitrarily deep, with <div> elements holding other <div> elements that might in turn contain links or emphasis tags (that might in turn contain other elements). It is precisely for this reason—the lack of a regular structure—that XML can provide such headaches to indexers or database developers.

The solution to the dilemma of addressing a specific set of elements, attributes, text, or other (potentially more exotic) pieces in an XML document, was to come up with a formal language specifically designed for XML, a language called XML Path, or XPath. XPath can be thought of as holding a similar role for XML that SQL holds for relational databases—it provides a consistent means of describing the location of zero or more nodes in an XML document, the intersections of two or more branches.


Directories and Nodes

To understand XPath, it's worth looking at XML and its similarities to the folder/file directory systems that are so pervasive in contemporary operating systems.

In most operating systems, a directory is simply a resource that contains references to child resources. A resource doesn't have to be a file—it can be a real-time summary of information about a specific device, or can even be a reference into a database that retrieves data records.

These files or directories or resources are all examples of nodes. Nodes do not need to be of the same type—a folder and file, for instance, represent two somewhat different types, and the contents of a file could conceivably be thought of as simply being a node of another type.

Nodes can be identified by a basic set of characteristics—usually, a name that identifies the node (not necessarily uniquely) and a type that provides a clue to the underlying environment about what exactly the role of that node is in the environment. This type might determine whether the node is a branch node (can potentially have children) or a leaf node (can't have children within the context of the graph), and might also tell other things (this node is a symbolic link to another resource, for instance, or it exists solely to provide contextual information to the system).

These are the kinds of nodes used by the XPath data model:

  • Document nodes represent a complete XML document.

  • Element nodes represent XML elements.

  • Attribute nodes represent the attributes of an element. Note that namespace definitions are represented by namespace nodes instead.

  • Namespace nodes represent the in-scope attributes of an element.

  • Processing instruction nodes represent embedded XML processing instructions.

  • Comment nodes represent XML comments.

  • Text nodes represent character data. Note that an infoset consists of single-character items, but in the node representation, multiple contiguous character items are represented in a single text node.

A node may have children, which are the nodes below it in the tree hierarchy. For example, the children of an element node are the text nodes and nested elements (and occasionally other nodes) that are its contents.

Addressing

If you follow a specific path in a certain direction, no node will ever be traversed more than once, and each node has a specific (albeit non-unique) name. In general, in such a directory, the child nodes of a given node are uniquely named, although there's no guarantee that a child node won't have the same name as its parent.

This makes it easy to create a path to a specific point in the filesystem. On the Web, for instance, you might have a location given as http://www.foobar.com, which provides the root, then a subordinate directory structure /resources/XMLQuery/Chapter2/chapter.doc that points to a specific file relative to the root node. Together, this address should uniquely define a file location:

http://www.foobar.com/resources/XMLQuery/Chapter2/chapter.doc

An XML structure bears a great deal of similarity to a directory structure. There is typically a location, given in coordinates of the local environment, that identifies the source of the XML, and which is also assumed to be the location of that document's root node. However, more than one node can have the same label. This simple difference unfortunately complicates the process of being able to locate a specific node. For instance, the path

/AddressBook/Address/PhoneNumber

does not uniquely define a node, because there might be more than one address in the address book, and each address might be associated with more than one phone number. The best you can do is to create some form of constraints on this data. For instance, you could locate an item by its position in the tree, such as:

/AddressBook/Address[3]/PhoneNumber[2]

This would identify that you need the third Address child of the Address node, and from there you want the second phone number. The earliest XML parsers were built around this method of access, but the problem here is that this requires knowing the XML contents intimately to know whether this is the node you actually want.


XPath 1.0

The first major solution to the addressing problem was XPath 1.0. This version is specifically designed to work with collections of nodes—also know as node-sets. A node-set makes no distinction between the types of nodes; a node-set could be made up of elements, attributes, text-nodes, comments, processing-instructions (an XML node useful for passing information to a processor or parser), CData-Sections (a node type used to encode literals, even when they contain XML syntactical characters), or any combination of these elements. What's more, there is no real requirement that such node-sets be directly related to one another, except that they should be part of the same document.

This last point might be subtle, but it is a constraint that has caused problems in the interim, and one of the main driving factors for reform in XPath 2.0. In XPath 1.0, the document was, in many ways, the foundation on which all locations were measured, but an XML document represents only one way of describing information, and not one that necessarily fits in with a query language.

For instance, suppose that you have a database consisting of employee records as documents, which includes for each employee the branch location where she is employed. It might be more convenient to retain each employee as a specific document (ease of retrieval, a generalized indexing mechanism, and so on). However, if you want to get a collection of all the employees that belong to a specific branch office, you would have to break the rules of XPath to do this, because the node-set would have to consist of nodes from each document.

XSLT 1.0 cheated a bit to do this, because the transformation language was specifically designed to be able to integrate multiple XML documents together. It did this through the use of the document() function, which can specify multiple document arguments (through a node-set), and then join them into an ersatz node-set that can be manipulated directly.

This approach has several problems, however. First, there is a difference between the formal XPath definition of a node-set and XSLT's version. This means that if you want to use XPath outside of the context of XSLT, you can only deal with one document's nodes (for example, through the use of the W3C's XML DOM).

A second problem comes into play with the use of XML Query, which is, in some respects, an extension of XPath to provide a more database-friendly way of interrogating data. XML Query takes as a base operating principle the anonymity of data sources. From the standpoint of XML Query, there is no way of knowing whether any given node returned from the query is returned from the same file as any other node.

A third limitation that XML node-sets face comes from the fact that they operate explicitly on XML nodes, but they also have to provide support for the manipulations of strings and numbers. The question can then be raised: Is a string a node, especially if it is provided in the context of an expression such as concat("Address:",//Address)? In the fairly limited case of XPath 1.0, these exceptions are manageable, but you are dependent upon having a very small set of primitive data types, a real problem if you are attempting to do such things as manipulate date information (such as trying to find the date three weeks past a certain date).

One requirement that XPath 1.0 has no significant support for is this manipulation of more complex data types. Because of this, both the functions that XPath 1.0 provides and the kind of operations that can be performed with native XPath 1.0 can be very limited. Similarly, there is no real way to work with data structures (such as collections of strings, like the names of months) short of building temporary XML structures—an approach that works in XSLT (with some significant caveats), but doesn't work with XPath.

These aren't major limitations and problems, but especially because both XSLT 2.0 and XML Query needed to have some form of directory-like navigational structure, the W3C decided to create a new version of XPath, version 2.0, that would address these issues.


XPath 2.0's New Features

XPath 2.0 is functionally backward-compatible with XPath 1.0 (in most respects), although it utilizes a somewhat different programming model for handling groups of items. This insures that if you have previously used XPath 1.0 within XSLT, for instance, the XPath 2.0 parsers and processors should be able to work with that 1.0 version just fine.

However, XPath 2.0 also offers a fairly large set of new features that significantly expand the scope and capability of the language, including the following:

  • Replacing the XML node-set concept with the more generalized notion of a sequence

  • Including support for both the data types supported in the W3C XML Simple Data-Types Schema Recommendation and the more general XML Schema Structure Recommendation for handling complex data types

  • The introduction of data type constructors and the capability to type function parameters

  • More extensive number, string, and date functions

  • The introduction of regular expression match-and-replace capabilities, including the (truly awesome) tokenize() and string-join() functions

  • Moving the XSLT document() function into the domain of XPath (through the slightly different doc() function), along with the introduction of a function enabling you to read and manipulate unparsed text

  • A formal mechanism for incorporating extension functions within XPath

  • Techniques for specifying the ordering of specific data types, useful for both ordering data and filtering out unique items

Individually, none of these are earth-shattering changes, and many already exist in one form or another as extensions in languages such as XSLT. However, by moving these into the standard, it makes it much easier to migrate code from one system or language to another, and reduces the dependencies on any programming environment.

This chapter will examine these features in greater detail, starting with the power of sequences.


Understanding Sequences

A sequence can be thought of as a linear list of zero or more items. Those items can be nodes from an XML tree (an element, an attribute, a text node, processing instructions, and so forth) but they can also be strings, numbers, or other simple data types as defined by the XML Schema data type language. This differs considerably from the closest analog in XPath 1.0, the node-set. A node-set has to be made up of nodes and only of nodes, making it difficult to perform group operations on any other type of collection.

A sequence can be formed by explicitly enumerating the items in the sequence between braces:

(1,2,3,4,5)

or

('one','two','three','four','five')

where string expressions need to be placed in either single or double quotes.

Sequences can also contain mixed types of data. For instance:

(1,2,3,"one","two","three")

Not only are all sequences values, but all values are also sequences, because a sequence of just a single value is in all respects the same as the single value. It follows from this that you cannot nest sequences—you cannot have sequences of sequences, only flat single-level sequences. So, if you attempt to include a list as an element, the XPath processor will automatically concatenate the new list into the old at the position indicated. Thus,

(1,2,("a","b","c"),3,4)

will "flatten" to

(1,2,"a","b","c",3,4)

If you have experience with arrays or lists in other programming languages, you might think the inability to nest sequences is a strange and limiting restriction. Actually, it isn't really a limitation, because you can always use nested elements if you need nested data. For example, to represent a two-dimensional array, you can use nested elements like this:

<list>
  <list>11 12</list>
  <list>21 22</list>
</list>

Sequences Versus Node-sets

A major difference between XPath 1.0 and XPath 2.0 is that the latter has sequences, whereas the former does not. Instead, XPath 1.0 has node-sets. For instance, consider the following list of color definitions:

<colors>
       <color>
              <name>Red</name>
              <hex>#FF0000</hex>
       </color>
       <color>
              <name>Green</name>
              <hex>#00FF00</hex>
       </color>
       <color>
              <name>Blue</name>
              <hex>#0000FF</hex>
       </color>

       <color>
              <name>Yellow</name>
              <hex>#FFFF00</hex>
       </color>
</colors>

In XPath 1.0, the expression colors/color/name would retrieve a node-set consisting of four nodes:

<name>Red</name>
<name>Green</name>
<name>Blue</name>
<name>Yellow</name>

In XPath 2.0, the same expression retrieves a sequence of nodes:

(<name>Red</name>,<name>Green</name>,<name>Blue</name>,<name>Yellow</name>)

One of the more subtle differences between node-sets and sequences comes from the fact that in XPath 1.0, node-sets are unordered—you cannot rely upon the ordering of the nodes within a node-set to be the same from one implementation of XPath to the next. In XPath 2.0, however, a sequence is implicitly ordered; if one item in a sequence is found before another item, that order is retained upon any iteration. (These are actually equivalent, in that you can map a set into a sequence that is ordered and without duplicates, and back again, without information loss. Furthermore, any valid XPath 1.0 expression will behave the same under either model.) This difference comes into play when setting up potential rules for sorting.

Enumerating with the to Operator

One of the common complaints leveled against XSLT comes from the language's inability to create an iterated for loop. As an example, the structure

<xsl:for name="i" from="1" to="10">

doesn't (and can't) exist in XSLT, because it violates the restriction that a variable cannot be redefined in the same scope or context. It is possible to get around this restriction by creating a recursive function call that passes (and increments) a parameter to a child function, but this process can be dangerous—each time you increment your variable, you are adding to the stack, and eventually you can exceed the amount of available stack memory.

Sequences were created in part to solve this problem. With a sequence, you can list all the numbers to be enumerated, then iterate through them. For instance, the following fragment of XSLT (2.0) will print a table of squares, from 1 to 10:

<table>
<xsl:for-each select="(1,2,3,4,5,6,7,8,9,10)">
       <xsl:variable name="num" select="."/>
       <tr>
              <td><xsl:value-of select="$num"/></td>
              <td><xsl:value-of select="$num * $num"/></td>
       </tr>
</xsl:for-each>
</table>

This will create the result:

<table>
       <tr><td>1</td><td>1</td></tr>
       <tr><td>2</td><td>4</td></tr>
       <tr><td>3</td><td>9</td></tr>
       <tr><td>4</td><td>16</td></tr>
       <tr><td>5</td><td>25</td></tr>
       <tr><td>6</td><td>36</td></tr>
       <tr><td>7</td><td>49</td></tr>
       <tr><td>8</td><td>64</td></tr>
       <tr><td>9</td><td>81</td></tr>
       <tr><td>10</td><td>100</td></tr>
</table>

You probably don't want to enumerate every single number in the sequence. It would be far more convenient to use an expression such as

(1 to 10)

The creators of XPath 2.0 thought the same thing, and so created the to operator. This operator compares the first and second values, and if the second is larger than the first, it will create a sequence starting from the first value, incrementing by one until the second is reached. This can be expressed in more traditional languages as

for (i=a;i<=b;i=i+1)(add(list,i))

For integers, this will produce a list from the value a to the value b inclusive. However, suppose a is 1.5 and b is 4.2. The sequence

(1.5 to 4.2)

would generate the values

(1.5,2.5,3.5)

The value 4.2 is not included in the list, because the to operator iterates by one, and 4.2 is both greater than 3.5 and less than 4.5, which would be the next valid value in the to list.

If the initial value in a to expression is greater than the final value, the iterator becomes -1, rather than 1. The expression

(4.2 to 1.5)

would thus produce the sequence

(4.2,3.2,2.2)

You can also have a to expression as a single element in a sequence. For instance, you could set up the following sequence:

(1 to 3,6 to 10, 12.2 to 16.1, 5, 14 to 11)

This is equivalent to the expanded sequence:

(1,2,3,6,7,8,9,10,12.2,13.2,14.2,15.2,5,14,13,12,11)

Thus, to create the table of squares using this same notation, you could simplify it as

<table>
<xsl:for-each select="(1 to 10)">
       <xsl:variable name="num" select="."/>
       <tr>
              <td><xsl:value-of select="$num"/></td>
              <td><xsl:value-of select="$num * $num"/></td>
       </tr>
</xsl:for-each>
</table>

Additionally, you can use parameters (at least with XSLT2 and XML Query 1.0) to turn such an expression into a parametric function:

<xsl:function name="math:drawSquares" xmlns:math="http://metaphoricalweb/ns/math">
       <xsl:param name="a"/>
       <xsl:param name="b"/>
       <table>
       <xsl:for-each select="($a to $b)">
       <xsl:variable name="num" select="."/>
       <tr>
              <td><xsl:value-of select="$num"/></td>
              <td><xsl:value-of select="$num * $num"/></td>
       </tr>
       </xsl:for-each>
       </table>
</xsl:function>

You would then call this function as

<xsl:copy-of select="math:drawSquares(1,10)"/>

to get the same effect as provided by hard coding the lower and upper limits of the for-each enumeration.

A WORD ABOUT NAMESPACES

The capability to define functions in XSLT or XQuery (or any other language that supports XPath 2.0) opens up an interesting discussion about the nature of such namespaces. Namespaces in XML serve a lot of purposes. One of the most important is providing a way to group related methods or actions together. For instance, you might find it useful to collect a number of mathematical functions (trigonometric and exponential functions, for instance) in a common library. By establishing a namespace (such as http://metaphoricalweb/ns/math), you can bind these functions together and address them using an appropriate namespace prefix (see Chapter 3, "XML Query Language Expressions," for details on how to do this with XQuery). One of the more interesting implications is that namespaces used this way bear more than a passing resemblance to classes in more traditional languages.

Functions Acting On Sequences

Sequences work best when there is a rich panoply of functions for manipulating them, and XPath 2.0 is supplied with just such a set. These functions, listed in Table 2.1, make it possible to identify an item from a position in the sequence, determine whether a sequence is empty, retrieve the location of the first occurrence of an item, and much more. Note that a number of other functions also work on sequences in some manner, but they are typically tied more to another data type.

Table 2.1. Sequence Functions

FUNCTION

PROTOTYPE

MEANING

concatenate

concatenate($seq,$seq or $item)

Joins two sequences together. Equivalent to the ',' operator, but makes it easier to build sequences programmatically.

item-at

item-at($seq,$pos)

Retrieves the item at the indicated position within a sequence.

index-of

index-of($seq,$item)

Retrieves the index (1-based) of the first instance of a given item in a sequence.

empty

empty($seq)

Returns true if the sequence is empty.

exists

exists($seq)

Determines whether the given sequence is a non-empty sequence.

distinct-nodes

distinct-nodes($seq)

Returns only the distinct nodes in a sequence, by identity.

distinct-values

distinct-values($seq)

Removes only the distinct atomic values in a sequence.

insert

insert($seq,$pos,$item)

Inserts the item into the sequence at the given position.

remove

remove($seq,$pos)

Removes the item at the given position.

subsequence

subsequence($seq1,$pos,$length?)

Returns the sequence starting from the given position to the end. If the $length attribute is included, the sequence returned includes only $length items starting from $pos.

Operators

sequence-deep-equal

sequence-deep-equal($seq1,$seq2,$uri)

Compares each item in a sequence, using child nodes of nodes to perform comparisons as well.

sequence-node-equal

sequence-node-equal($seq1,$seq2,$uri)

Compares each item in a sequence, though nodes are only compared based upon their text value rendering.

union

union($seq1,$seq2)

Joins two sequences together, eliminating any duplicates.

intersect

intersect($seq1,$seq2)

Returns a sequence of only those items that are in both initial sequences.

except

except($seq1,$seq2)

Returns a sequence of only those that are in one or the other sequence, but not both.

Aggregate Functions

count

count($seq)

Returns the number of items in the sequence.

avg

avg($seq)

If all items in the sequence are numbers, finds the average of the items.

sum

sum($seq)

If all items in the sequence are numbers, finds the sum of the items.

min

min($seq)

Returns the minimum value item in the sequence if the items can be rendered as numbers or dates, and the first collated item if all items are strings.

max

max($seq)

Returns the maximum value item in the sequence if the items can be rendered as numbers or dates, and the last collated item if all items are strings.

Understanding the Core Sequences Functions

Some of these functions perform actions that can be abbreviated. For instance, consider the colors.xml file from the previous section:

<colors>
       <color>
              <name>Red</name>
              <hex>#FF0000</hex>
       </color>
       <color>
              <name>Green</name>
              <hex>#00FF00</hex>
       </color>
       <color>
              <name>Blue</name>
              <hex>#0000FF</hex>
       </color>

       <color>
              <name>Yellow</name>
              <hex>#FFFF00</hex>
       </color>
</colors>

The position() function within XPath 2.0 (and for that matter, XPath 1.0) retrieves the position of a given node within a sequence. Thus,

document("colors.xml")//color[position()=2]

will retrieve the color element at the second position (Green). Because this is a very common operation, however, this has also been collapsed so that the position() function is implied. The same statement could thus be written as

document("colors.xml")//color[2]

giving XPath more of an array-like characteristic. Note that this holds for sequences of simple data types as well. For instance, the expression

(3,6,9,12)[2]

retrieves the second value of the sequence, or the number 6. You can use this to good effect in creating lookup tables with sequences. For instance, you could assign in XPath or XML Query the names of the months into a sequence:

[View full width]

<xsl:variable name="months" select="('January','February','March','April','May','June'
  ,'July','September','October','November','December')"/>

Then later you can access any month by its "month-number":

<xsl:value-of select="$months[3]"/>

This expression retrieves the value March.

The position() function is contextual, relative to the current sequence. With a number of operations (either with pure XPath, or with XPath in conjunction with XML Query or XSLT), the system sets up a specific sequence that it treats as the current sequence. For instance, in XSLT, the <xsl:for-each select="(1,2,3)"> expression defines the sequence (1,2,3) as the current sequence, and maintains a pointer to the current item in the sequence being processed.

This current sequence can be accessed through a number of functions indirectly, though in general it can't be accessed directly from a function. The position() function, for instance, points to the index of the current item being processed within the sequence. The last() function, which also takes no arguments, retrieves the number of items in the current sequence, and is similar to the count() function, except that it refers only to the current context. The context-item() function will return the current item itself. Note that there is a distinction between the current sequence of an item as an entity and the parent of a (node) item.

In addition to the position() function, you can also use the item-at() function to retrieve the item from a given position in a passed sequence. This expression retrieves the value March as well:

item-at($months,3)

The index-of() function performs the inverse operation. For a sequence of simple data types, the index-of() function retrieves the location of the value being passed, or the value 0 if the item does not exist in the sequence. This expression retrieves the value 12, for instance:

index-of($months,'December')

In some cases, you'll be working with sequences of nodes (such as the colors given earlier). In that case, index-of() evaluates the text content of each node and compares it to the search criterion, given as a straight text match. Thus, to retrieve the location of the node with the name Blue, you'd use the index-of() function on a set of nodes:

index-of(document("colors.xml")//color/name,'Blue')

The expression document("colors.xml")//color/name creates a sequence of name elements—in this case, one per color node. The index-of function then iterates through each of these nodes and determines the text associated with those nodes (by concatenating each text node for that element together). The first node that has text matching the expression Blue will be identified—in this case, the third node, which means the index-of() function returns the value 3.

The concatenate() function similarly echoes another operator, in this case the , operator, which indicates the item after the comma should be joined to the sequence before the comma. If you have two sequences A = ("a","b","c") and B = ("d","e","f"), the expressions

concatenate($a,$b)

and

($a,$b)

will both create the new sequence

("a","b","c","d","e","f")

For all XPath sequence functions that return a resulting sequence, the sequence being processed is passed by value rather than reference. This means XPath functions create a copy of the sequence, operate on this copy, and then pass the result back as a new sequence. Although this process avoids possible side effects, it also means (as long as you are following the declarative model of XPath) sequence manipulation can be fairly expensive, memory-wise, because every sequence you create is a new copy of an existing sequence.

The insert() and remove() functions create new sequences from old ones. For instance, you could remove the Blue item from the colors list just with the sequence functions. If $colors has been previously defined to be document("colors.xml")//color, the expression

remove($colors,index-of($colors/name,'Blue'))

removes the Blue <color> entry from the sequence and passes the resulting set back as a new sequence. The lack of any reference passing assures that no unexpected side effects will happen on the initial data, even though it is not as efficient memory-wise as referential parameters.

The subsequence function retrieves part of a sequence, based upon position and (possibly) length. For instance, given

$alpha=("a","b","c","d","e","f")

then

subsequence($alpha,3,2)

returns the first two items starting from position 3, or the sequence ("c","d"). If the length parameter (the third parameter) exceeds the number of items in the sequence, the remainder of the sequence is returned:

subsequence($alpha,3,8) => ("c","d","e","f")

If no length parameter is provided, then similarly the rest of the sequence is returned (and not just the item at that position):

subsequence($alpha,3) => ("c","d","e","f")

If the length is negative or 0, the sequence returned will be the empty sequence. Note that you can test this with the empty() function:

empty(subsequence($alpha,3,-2))=> "true"

You can also test to determine whether the sequence in question exists:

exists(subsequence($alpha,3,-2))=> "false"

This says that a sequence is considered to exist only if it has some items within it:

exists(subsequence($alpha,3, +2))=> "true"

Manipulating Set Functions

One of the new facets of sequences is the fact that, unlike node-sets, it is possible to do standard set manipulation on them. For instance, if you have two sequences $a and $b, such that

$a=(1,2,3,4,5)

and

$b=(3,4,5,6,7)

then

union($a,$b) => (1,2,3,4,5,6,7),

returning all elements that are in both sequences while eliminating duplicates. Similarly,

intersect($a,$b) => (3,4,5)

retrieves only those elements that are common to both sets. Finally,

except($a,$b) => (1,2,6,7)

returns those elements that are in either set but not in both. This collection of set utilities makes it easier to make sophisticated filters. For instance, if you have one sequence that consists of managerial employees in all departments:

$managers = ("Jones","Shelby","Ryan","Cartwell","Mace")

and a second sequence that consists of all employees in the Marsten Building:

$marsten = ("Ryan","Halloway","Cartwell","Carter","Bebo")

then you can find all managers that are located in Marsten using this expression:

intersect($managers,$marsten)=>("Ryan","Cartwell")

This kind of analysis is not easily accomplished in XPath 1.0, but is trivial in XPath 2.0. Similarly, you can find all non-managerial employees in Marsten with the following expression:

except($marsten,intersect($managers,$marsten)) =>
("Halloway","Carter","Bebo")


Handling Aggregate Functions

Most of the aggregate functions defined in XPath are straightforward. For instance, the count() function counts the number of items in a sequence (or returns 0 if the sequence is empty). There is a distinction, carried over from XPath 1.0, between count() and last(). count(), which takes a sequence as an argument, retrieves the current number of items in that sequence. On the other hand, last() examines the current context—the sequence currently being acted upon in the XPath expression—and returns the position on the last item in this context. Consequently, last() does not take an argument.

The sum() and avg() functions perform as expected—summing up the content of numeric nodes or finding the average (taking this sum and dividing by the number of items), respectively. For instance,

avg((3,4,5,6,7)) => 5

returns the value 5 ((3+4+5+6+7)/5).

If non-numeric values are passed to either the sum() or avg() function, the functions return the default NaN message, indicating that the arguments are not numbers, and hence cannot be processed.

This is not the case with the min() and max() functions, two needed functions that weren't in the XPath 1.0 spec. These functions return the minimum and maximum values, respectively, from a sequence (converting the items from a node to a value if that's called for). This works as might be expected for numeric values:

max((3,7,8,2,4,5)) => 8

It also works with text values, returning the first or last item of a sequence, as would be expected alphabetically:

max(('beta','alpha','gamma','omega','mu')) => "omega"

This ordering invokes the concept of collation. Collation is a specific order that a set of data must subscribe to, such as the notion of alphabetical order. Because the rules of collation can be complex (especially with languages that don't have obvious ordering, such as Chinese), the XPath specification's authors decided to make collation an implementation-specific capability, tied into a specific namespace URL.

In practice, if you need a specific collation, you're at the mercy of the XPath processor vendor. The way such a collation would work, though, is universal. Suppose that you need a collation for ordering your documents in Greek. The vendor happens to have just such a collation built into the browser, and supplies the URL http://www.XSLTVendor.com/collations/greek to invoke this collation. To get the maximum value with this collation, you'd supply both the sequence and the collation:

max("Sparta, Aqhnoi, Qhbes","http://www.XSLTVendor.com/collations/greek") => "Sparta"

This needs to be clarified just a bit. Although it might vary based upon what the vendor chooses to do, in most cases, the URL given for the collation does not necessarily point to anything. Instead, it is just a unique sequence of characters that act as a key to the processor, which takes the key and uses it to match to a specific set of routines. For instance, a Java-based parser might actually use a reference to a Java class to achieve the same effect:

max("Sparta, Aqhnoi, Qhbes","java:sams.xpath.collations.greek") => "Sparta"

Because both the XPath specification and engines capable of handling XPath are still evolving at the time I write this, it's best to search the documentation for your particular processors to see what works there.


The tokenize() Function

Perhaps one of the most useful sequence functions in the XPath 2.0 arsenal, and one that has only been added as of the August 2002 working draft, is the tokenize() function. This function takes a source string and a delimiter string (such as a comma or tab character) and converts the results into a sequence.

As an example, the following converts an address with elements separated by commas into a sequence:

tokenize(("Morticia Adams,1313 Mockingbird Lane,Arkham, MA,01234"),",") =>
("Morticia Adams",
"1313 Mockingbird Lane",
"Arkham",
"MA",
"01234")

This capability can be accomplished in languages such as XSLT 1.0 with a great deal of work, but moving it into XPath 2.0 makes it far more accessible. The tokenize() function lets you take string text data (such as the contents of a tab-delimited file or a comma-delimited file output from Microsoft Excel or a simple database) and quickly and easily convert it into a form that can be manipulated in XPath.

For instance, suppose that you have a comma-delimited text file named addresses.txt with the contents:

name,street,city,state,zip
Morticia Adams,1313 Mockingbird Lane,Arkham, MA,01234
Buffy Summers,45 NW Boca Dell Inferno St.,Sunnydale, CA 71129
Willow Rosenberg,62 SE Circle Rd. Sunnydale, CA 71129
...

There are in fact two delimiters in the text file—a comma delimiter separating the fields, and a carriage-return/linefeed character set (depending upon the operating system). By applying the tokenize() function once to separate the text into distinct lines and again to convert each line into fields, you can do record-based manipulation. Although the full routine will be covered in Chapter 4, the basic routine (assuming the text is in a variable called $addressText) looks like this:

<!-- break the text into lines -->
<xsl:for-each select="tokenize($addressText,'&#13;&10;')">
       <xsl:variable name="record" select="."/>
     <!-- break each record into fields -->
       <xsl:for-each select="tokenize($record,',')">
              <!-- define each field -->
          <xsl:variable name="field" select="."/>
              <!-- process each field -->
       </xsl:for-each>
</xsl:for-each>

The first line uses the sequence &#13;&10;, corresponding to the carriage return/line feed that typically marks line breaks in text files. (The delimiter does not need to be a single character; it can be an extensive string.) This operation produces a sequence of text lines—records—which can in turn be processed individually.

The second tokenizing string iterates on the comma character (,) to break the line record into fields, which can then be manipulated one by one. By using tokenization, an operation that in XSLT 1.0 would have required extensive recursive function calls can be accomplished in a trivial manner.


Working with External Resources

One of the key concepts of XPath 2.0 is the fact that data can come from anywhere—passed as a parameter, defined explicitly as a sequence, or pulled in from an external data source. This last data provider illustrates something of the gulf that exists between XPath 1.0 and XPath 2.0. XPath 1.0 was intended primarily as a mechanism for dealing with a single, clearly defined XML document, which can be thought of conceptually as one explicit XML tree with a single node root. With XPath 2.0, on the other hand, data can come from a query that returns a sequence of nodes, including nodes that arise from imported XML documents.

SHOP TALK: DOCUMENT() AND DOC()

The document() function is probably familiar to you if you've worked with XSLT 1.0, but document(), which retrieves an XML document from an external source, could not be a part of XPath 1.0. The XSLT 1.0 document() function could take as its first argument one or more XML nodes, and each could contain a URL reference to an external document, meaning that the result could be a node-set with more than one root node. The or more part didn't fit into the data model of XPath 1.0, and consequently it needed to be added through the backdoor in the XSLT specification.

As useful as sequences are, the primary reason for their existence has to do explicitly with this requirement. By making sequences available as an intrinsic part of the data structure, the document() function could be moved into the XPath specification, where it properly belonged from the beginning. (There's a lesson here about being very careful about how you establish your ground rules when making standards.)

After some consideration, the W3C XPath Working Group decided not to explicitly copy the document() function directly. Instead, they created the simpler doc() function. The doc() function rakes a single argument representing the URL of the file to be retrieved. Thus doc ("myXMLDoc.xml") retrieves the content of the file myXMLDoc.xml. If the URL is relative (that is it does not begin with a protocol scheme such as http:), then it is resolved using to the current module's Base URI.

Although the doc() function is useful, it also still assumes that the documents being retrieved are valid XML documents with single nodes. However, suppose that rather than a specific document, you need to reference a Web service that returns a collection of XML nodes via some XML Query. Such queries do not have any requirement to return full XML documents, but could legitimately return just collections of nodes. The doc() function will fail in that case.

A second XPath 2.0 function called collection() handles precisely this situation. For instance, suppose that you have a Web service accessible via an HTTP GET command that returns the collection. You can retrieve it using something like the following expression:

collection("http://www.myserver.com/services?myService=abc")

The result will be the set of nodes returned by this service as a sequence.

XPath 2.0 is intended to work exclusively with XML data, so it doesn't have a facility to handle the incorporation of non-XML text into the parser. XSLT 2.0 does, however, have such a capability—the unparsed-text() function. This can retrieve non-XML text that can then be passed in to XPath via a variable. No such capability exists for XML Query, however, unless it is added through a custom extension.


Cutting Verbosity with XPath Command Structures

Anyone who has worked with XSLT 1.0 has likely encountered a very frustrating fact: XSLT is remarkably verbose, especially for things as simple as handling conditional logic. For instance, consider a fairly common requirement in producing output: generating a list of items separated by a comma, but only if the item in question is not the last item in the list.

For instance, consider the XML document colors.xml discussed earlier in this chapter:

<colors>
       <color>
              <name>Red</name>
              <hex>#FF0000</hex>
       </color>
       <color>
              <name>Green</name>
              <hex>#00FF00</hex>
       </color>
       <color>
              <name>Blue</name>
              <hex>#0000FF</hex>
       </color>
       <color>
              <name>Yellow</name>
              <hex>#FFFF00</hex>
       </color>
</colors>

To put together a list of comma-delimited items in XSLT, you'd need to do something like this:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:template match="/">
       <xsl:apply-templates select="//color"/>
</xsl:template>
<xsl:template match="color">
       <xsl:value-of select="name"/>
<xsl:choose>
<xsl:when test="position()!=last()">,</xsl:when>
<xsl:otherwise>&#13;&#10;</xsl:otherwise>
</xsl:choose>
</xsl:template>
</xsl:stylesheet>

The expression

<xsl:choose>
<xsl:when test="position()!=last()">,</xsl:when>
<xsl:otherwise>&#13;&#10;</xsl:otherwise>
</xsl:choose>

is remarkably unwieldy, especially with the caveat that you have to be extraordinarily careful not to introduce any additional unexpected white space into the mix. Moreover, such expressions are often difficult to follow, and add a lot of XSLT code that can be a pain to manage.

Using if-then-else

In XPath 2.0, it was recognized that on occasion XPath needs to be able to test conditions that can't easily be wrapped into the predicate expressions of normal XPath statements (the part of an XPath expression contained within square brackets []). To that end, the conditional statements, if, then, and else have been added to the XPath language.

For instance, the stylesheet in the previous section can be rewritten in XSLT 2.0 (with XPath 2.0) as:

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:template match="/">
       <xsl:apply-templates select="//color"/>
</xsl:template>
<xsl:template match="color">
       <xsl:value-of select="concat(name,if (postion()!=last()) then ',' else '&#13;&#10;')"/>
</xsl:template>
</xsl:stylesheet>

In addition to eliminating five lines of XSLT, this expression becomes a little easier to read, and the intent becomes much clearer. An added bonus here is that you don't need to worry about the unexpected addition of white space.

The else expression is not always required. For instance, if, after the last item is written to the output buffer, you want to close out a paragraph, you don't need to include the closing carriage return/line feed, and can consequently dispense with the else:

<xsl:template match="/">
       <para>
       <xsl:apply-templates select="//color"/>
       </para>
</xsl:template>
<xsl:template match="color">
       <xsl:value-of select="concat(name,if (postion()!=last()) then ',')"/>
</xsl:template>
</xsl:stylesheet>

The if expression must be enclosed in parentheses, but in this context the parentheses are functional, rather than indicating a sequence. The expression within the parentheses must either evaluate to a Boolean, or it must be possible for that expression to be cast to a Boolean. In this latter case, an empty string is considered equivalent to the Boolean function false(), as is the numeric value 0, whereas a non-empty string or nonzero value is considered to be equivalent to the Boolean function true(). Finally, an empty sequence (including an empty sequence from a node query) is also treated as the Boolean false().

Thus, the following expression:

[View full width]

if (document('colors.xml')//color[name='Orange']") then 'Color is in list' else 'Color is
   not in list'

returns the expression Color is not in list, because there are no nodes for which the expression is true (and hence no sequence).

It is possible to nest if statements. For instance, suppose you have a variable consisting of all the months. You could check to see whether a given month ends in the letter Y and is in the first half of the year with the following expression:

[View full width]

<xsl:variable name="months" select="tokenize('January,February,March,April,May,June,July
  ,August,September,October,November,December',',')"/>
<xsl:for-each select="$months">
       <xsl:value-of select="
if (ends-with(.,'y')) then
   (if (position() le 6) then
              concat(.,' ends in y in the first or second quarter.&#10;')
       else
              concat(.,' ends in y in the third or fourth quarter.&#10;'))
else
       concat(.,' does not end in y.&#10;')"/>
</xsl:for-each>

Several things become apparent in this sample. For starters, XPath does not make any distinctions with regard to white space; you can use carriage returns to make if/then/else statements more understandable without compromising the integrity of the expression.

Finally, the resulting operation after each then or else token is an XPath expression that can be converted to a string. This means that expressions such as

concat(.,' does not end in y.&#10;')"/>

are valid, with the concat() function joining the value of the current node with the expression ' does not end in y.&#10;'.

The XSLT, when evaluated, produces as a result:

January ends in y in the first or second quarter.
February ends in y in the first or second quarter.
March does not end in y.
April does not end in y.
May ends in y in the first or second quarter.
June does not end in y.
July ends in y in the third or fourth quarter.
August does not end in y.
September does not end in y.
October does not end in y.
November does not end in y.
December does not end in y.

Using for-return

I've cheated somewhat throughout this chapter, using variables without properly defining them. Part of the reason for holding off on this is the fact that, although they follow the same syntax, variables in XSLT and variables in XPath 2.0 are slightly different.

In XSLT (version 1.0 or 2.0), a variable is defined with either the variable or the parameter element:

<xsl:variable name="varName" select="XPathExpr"/>

Such variables are capable of holding both single values and node-sets/sequences. However, variables in XPath 2.0 are more restricted: A variable can only be defined to hold a single value, though that value can be a node with children. XPath, when embedded in XSLT or XML Query, can work with the more expansive variables of XSLT or XML Query—it just can't generate such variables on its own.

The new XPath 2.0 for keyword is one way in which such variables can be assigned. The full syntax for this command is

for $a in $b return c$

where $a is the name of a variable, $b is a sequence or node-set, and c$ is a formal XPath 2.0 expression. For instance, the expression

for $color in ("Red","Green","Blue") return concat(lower-case($color),":")

will iterate through the color list, assigning each item in turn to the variable $color. After each iteration, the statement after the return keyword is invoked with the $color value having the next value in the sequence, where it is converted to all lowercase. This will result in the output string:

red:green:blue

In XSLT you can define a variable that contains a sequence using the <xsl:variable> statement. For instance, in XSLT 2, the same result could be rendered as follows:

<xsl:variable name="colors" select="('Red','Green','Blue')"/>
<xsl:value-of select="for $color in $colors return concat(lower-case($color),':'"/>

In this case, $color is a pure XPath 2.0 variable, and $colors is an XSLT 2 variable. The same principle holds with XML Query:

LET $colors:= ("Red","Green","Blue")
FOR $color in $colors
RETURN concat(lower-case($color),":")

Here the FOR and RETURN keywords are those for XQuery, not XPath. The distinction between the two forms can sometimes be both subtle and maddening.

It is possible to mix for and if expressions. As an example, the Y test given earlier can be rendered as a pure XPath expression:

[View full width]

for $month in tokenize('January,February,March,April,May,June,July,August, September
  ,October,November,December',',')
return
if (ends-with($month,'y')) then
   (if (position() le 6) then
              concat($month,' ends in y in the first or second quarter.&#10;')
       else
              concat($month,' ends in y in the third or fourth quarter.&#10;'))
else
       concat($month,' does not end in y.&#10;')

Building a Custom Switch Structure

Finally, the one structure that doesn't have an exact analog in XPath is the switch statement, where a given operation is performed based upon a particular variable matching a given value from a range of values. You can, however, create a modified switch form by taking advantage of the properties of sequences. The secret is to invert the order you might expect for a sequence.

For instance, suppose that you have a variable $a that can match any of the expressions $expr1, $expr2, $expr3, and so forth. When a match is found, the result $r1, $r2, $r3, and so forth is returned, or $r0 if no match exists. You can then get the switch statement by using the form

($r0,$r1,$r2,$r3, ... $rN)[index-of (($expr1,$expr2,$expr3,...,$exprN),$a)+1]

As an example of this, the following will return the hex-value equivalent to a color, depending upon which color is specified:

("#000000","#FF0000","#00FF00","#0000FF")[index-of (("Red","Green","Blue"),$color)+1]

When $color is "Blue", the index of blue is 3. Add one to get the fourth item in the initial list, or #0000FF. If $color is "Yellow", the index is 0 (nothing matches), so the first indexed item (#000000, or black in this case) is returned. As each item in both the initial and matching sequences can be complex XPath expressions, you can generate potentially very sophisticated queries using this same structure.


Regular Expressions

XPath 2.0 makes a certain tacit assumption that was much less obvious in XPath 1.0—XML is still text, and the things that people need to do with XML still revolve around text processing. There are a host of new string-manipulation commands in XPath 2.0, some of which have been slipped in surreptitiously (such as the ends-with() function featured in the previous section). In most cases, these functions perform obvious tasks and are logical extensions to what already exists in XPath 1.0.

However, XPath 2.0 also incorporates for the first time the use of regular expressions to validate, match, replace, and tokenize text. Regular expressions (or regexes, as they are commonly abbreviated) provide a sophisticated way to match particular patterns in strings.

There are a number of subtly different regular expression languages in use, depending upon the host language and the platform, although most hew fairly closely to the regex language currently in use in the Perl language. The recommendation of the editors of the XPath specification currently is to assume that the Perl specification is the reference standard (although this might change by the time the Working Draft becomes a Recommendation).

Table 2.2 lists the more common regex character meanings. A comprehensive discussion of regular expressions and their use is beyond the scope of this book. For more information on their use, see Mastering Regular Expressions by Jeffrey E. F. Friedl (O'Reilly).

Table 2.2. Regular Expression Symbols

CHARACTER

DESCRIPTION

\

Marks the next character as special (if not already) or as given (if character has specialized meaning)

^

Matches the beginning of a line of text

$

Matches the end of a line of text

*

Matches zero or more instances of the previous character or expression

+

Matches one or more instances of the previous character or expression

?

Matches zero or one instance of the previous character or expression

{n}

Matches exactly n times

{n,}

Matches n or more times

{n,m}

Matches from n to m times

?

Following *, +, ?, {n}, {n,}, or {n,m}, signifies that a match is non-greedy (matches the least text it can)

.

Matches any single character except \n (line break)

(pattern)

Matches the pattern and captures the match

(?:pattern)

Matches the pattern but does not capture the match

(?=pattern)

Performs a positive look-ahead

(?|pattern)

Performs a negative look-ahead

x|y

Matches either x or y

[xyz]

Matches any of the enclosed characters

[^xyz]

Matches any character not enclosed

[a-z]

Matches a range of characters

[^a-z]

Matches any character not in the given range

\b

Matches a word boundary, between a word and a space

\B

Matches a non-word boundary

\cx

Matches the control character specified by x

\d

Matches a digit character

\D

Matches a non-digit character

\f

Matches a form-feed character

\n

Matches a newline character

\r

Matches a carriage return character

\s

Matches any white space character

\S

Matches any non-white space character

\t

Matches a tab character

\v

Matches a vertical tab character

\w

Matches any word character, including the underscore

\W

Matches any non-word character

\x n

Matches any hexadecimal character, where n is an integer

\num

Matches num, where num is a positive integer, a reference to previously captured matches

\u n

Matches a unicode character where n is a four-digit hexadecimal number

The advantage of using regular expressions comes from their capability to match the patterns of specific strings. One common instance of this is the need to match telephone numbers that can be entered as user input. The important facets of an American telephone number include the area code (aaa), the exchange (xxx) and the local code (cccc).

The most stringent test of a valid phone number would be to require that it be in the form

(aaa)xxx-cccc

A regular expression matching this format might look something like this:

^\(\d{3}\)\d{3}-\d{4}$

This can be read as follows:

^ | The start of the string must match the expression
\( | Explicitly match an opening parenthesis
\d{3} | Match three digits for the area code
\) | Explicitly match a closing parenthesis
\d{3} | Match three digits for the exchange
-\d{4} | Match a dash and four digits for the local code
$ | There can be no other characters after this match

The matches() Function

The XPath 2.0 matches()function takes a source string and a regular expression (along with an optional collation URL) and returns a sequence of integers indicating where in the source string the match occurred. For instance,

matches("aaabacad","a.")=>(1,2,3,5,7)

returns a sequence of integers for the first, second, third, fifth, and seventh positions in the string. Typically, you are probably less interested in the exact position of the matches than you are in the fact that the match occurred at all. Fortunately, you can use the same information, because an empty sequence is treated as a Boolean false() value, and a non-empty sequence is treated as a Boolean true(). For instance,

matches("(800)555-1212","^\(\d{3}\)\d{3}-\d{4}$"=>(1)

returns the sequence (1), indicating that the match occurs at the first character. You can use this in an if-then-else statement:

[View full width]

if (matches("(800)555-1212","^\(\d{3}\)\d{3}-\d{4}$") then "Phone number is valid" else
   "Phone number is invalid"

This will properly evaluate to "Phone number is valid". On the other hand, if the phone number given is of the slightly different format "800-555-1212", it will evaluate false because it does not match the given regex.

The replace() Function

You can extend the regular expression to match the more generalized case for phone numbers, but ideally, it would be useful to transform any "semi-valid" forms into a valid form. The XPath 2.0 replace() function can be used to accomplish this. The replace() function takes a source string, a matching regular expression, and a second expression indicating the replacement text. As with matches(), the replace() function can be used with general text in the way that you might expect,

replace("This is a test"," "," +") => "This+is+a+test"

with the string to be matched replaced with the new string. However, replace() really comes into its own when used with regular expressions. For instance, suppose that you create a more general-purpose phone number regex. You could then replace a number of different formatted phone numbers with one common format, as follows:

[View full width]

replace("(800) 555-1212", "^\(?(\d{3})\)?\.?\s*-?(\d{3})[.- ] (\d{4})$","$1-$2-$3") =>
   "800-555-1212"

The regular expression here matches a fairly broad number of formats:

(800)555-1212

800.555.1212

800-555-1212

(800) 555.1212

800 555 1212

...

The final expression "$1-$2-$3" might appear a little cryptic. In the initial regex, the area code, exchange and local numbers are each contained within parentheses, which indicate to the regex engine that these expressions are to be retained in the order they are found. Within the replacement string, the "$1-$2-$3" replaces the numbered expressions with the captured expressions in the order they were found in the source. The regex engine supports up to nine potential matches ($1 to $9).

Flags

Both the matches()and replace() functions can take another argument: a set of flags. Normally, the default mode for regex matches is to perform single line matches where the regex only searches to the end of the first line (dictated by the line break character). If you want to search through more than one line with the search, you need to append the m or multiline flag as an argument to either function:

matches("source","regex","m")
replace("source","regex","replaceStr","m")

Similarly, searches are usually case sensitive. If you want to find a match regardless of the case, you use the i flag. If you need both capabilities, you can combine the two:

matches("source","regex","im")

The tokenize() Function

The tokenize() function, introduced earlier, also tokenizes a string using regular expressions. This is an important point to consider when working with certain characters as tokenizers. For instance, the expression

tokenize("a1.b1.c1.d1.e1.",".") => {}

returns an empty sequence. This is not because there are problems with the function—it is behaving perfectly. Instead, because the delimiter string is a regular expression, the function is using each character encountered (the . match) as a delimiter. If you escape the character with a \., you will get the more expected result:

tokenize("a1.b1.c1.d1.e1.","\.") => {"a1","b1","c1","d1","e1"}

One useful consequence is that the tokenize() function parses a line of text that may have multiple possible delimiters. For instance,

tokenize("red;green,blue:orange-yellow","[;,:-]") =>
("red","green","blue","orange","yellow")

parses the string regardless of which delimiter is used, provided it is one of those in the regular expression [;,:-].


Of Data Types and Dates

Depending upon who you talk to, the most significant of the innovations in XPath 2.0 is either sequences, regular expressions, or the introduction of atomic types, especially date types.

Informally, a type is a set of values: those values that are instances of or belong to the type. The type system of a programming language is the collection (vocabulary) of types that the language definition distinguishes, including the rules for determining whether a value is an instance of a type, and for how to create complex types from simple types. (A simple type can only be expressed as character data, whereas a complex type can consist of sub-elements and attribute specifications.)

ATOMIC TYPES

Atomic values include numbers, values, and Booleans. There are two kinds of atomic types:

  • A primitive type is not defined in terms of some other type.

  • A derived type is based on some other type, its base type.

A derived atomic type is a restriction of its base type, because it is a restriction (sub-set) of the set of atomic values that belong to the base type.

A type error occurs when the operands of an operation have types that are not allowed for that operation. For example, in XQuery, you can add two numbers using the + operator, but you can't add two nodes, even if the nodes contain integer values. If your program tries to add two nodes, the XQuery processor should give you an error message instead.

It is useful to distinguish between the dynamic types and static types:

  • The type of a value is a dynamic type. Dynamic types exist during evaluation (at runtime). Dynamic types are sets of values that are instances of the type. A type specifies the meaning or interpretation of a value.

  • The type of an expression (a program fragment) is a static type. Static types are the types of declarations and program fragments as specified by the programmer or inferred by a compiler. If an expression has a (static) type and you evaluate the expression without a runtime error, the result is guaranteed to be an instance of the corresponding static type.

A dynamically typed language is one that doesn't have static types. Another way to say the same thing is that there is only a single type, which contains all values. In those languages, all type errors are runtime errors. The goal of a static type system is to detect type errors at compile-time, before actual execution. This is a process called type checking, and in some languages (including XQuery), it is a fairly complicated process.

Static type checking lets you detect and fix errors earlier. This is especially valuable for infrequently executed parts of a program, because they are less likely to get much testing. As a side benefit, if the compiler can determine the type of an expression, it might be able to generate more efficient code, and so the query might execute faster.

Types and XML

Type is a fairly strange concept within the world of XML. An XML document is, by its very nature, text. Unlike just about every other programming language or environment, the notion of type is abstract and separate from any explicit system characteristics.

To understand what this means, consider the notion of a floating-point number—a number with a decimal point that may or may not have digits in the mantissa (the part to the right of the decimal point). In most computer languages, you cannot explicitly represent most floating-point numbers exactly.

Moreover, in many of these languages, the internal representation of these floating-point numbers is determined by several factors, including a set number of bytes to represent the floating-point data type, the number of bits within those bytes that represent the mantissa (a power of two representation of a decimal value between 0 and 1), and the number of bits to represent the exponent (the power of two that gets multiplied by the mantissa).

Data types emerged as one way of dealing with these different internal representations, and until relatively recently, even these types differed from platform to platform.

Languages such as Java have standardized these concepts to a certain degree, so that there is a general uniformity for the representation of a Double type, for instance, but even so, it is a restriction that is imposed by the architecture of the machine.

Within XML, on the other hand, there is no real distinction between a string representation of a number and the number itself. The expression <temperature>98.2</temperature> is talking about a string "98.2" that just happens to have a semantic meaning of a floating-point number.

However, because XML currently needs to interface in a number of ways with other programming languages such as Java, C++, C#, or other statically typed languages, it is necessary to impose an abstract concept of type on XML. A number of different languages can do this, but the W3C has standardized this in the XML Schema Definition Language (XSD).

Types and XPath

XPath 1.0, which predates XSD by a couple of years, recognizes only four primitive types—XML nodes (in various flavors), strings, numbers, and Boolean values. This has not proved to be a major problem, but there are some limitations with a set of types this small. For example, there is no significant difference between integers and floating-point numbers, so certain operations that involve the accumulation of numbers are likely to introduce ever-increasing levels of error in the representations of those numbers.

As XPath, XSLT and XML Query are used more often for XML-to-XML transformations, the problem of the best way of representing numeric values becomes more pressing.

A second problem with XPath 1.0 was the lack of any date types. Most developers who needed date support within their XSLT or XPath expressions resorted to extension functions written in Java or JavaScript—solutions that limited the portability between platforms that XSLT promised.

XPath 2.0 incorporates a number of new data types, although it still does not include a ready extensibility mechanism for types. You can work with many (if not all) of the data types defined within the XSD-Instance Recommendation, but working with more complex types requires manipulating the schema associated with a given element (something discussed in greater detail in Chapter 4).

Dealing with Dates

A date can be an extraordinarily complicated thing to have to work with. XPath 1.0 adopted the convention of representing a date using the following notation:

YYYY-MM-DDTHH:NN:SS.DDDZzz

where

  • YYYY is a four-digit year

  • MM is a two-digit month

  • DD is a two-digit day

  • T is a literal T for Time

  • HH is an hour based on a 24-hour clock

  • NN is the two-digit minutes

  • SS is the two-digit seconds

  • DDD is the three-digit milliseconds

  • Z is the literal Z for Zone, and like T must be capitalized

  • zz is the time zone, with zz being a positive or negative offset from Greenwich Mean Time (if zz is left off, this indicates local time)

For instance, as I write this, the specific time is

2002-10-08T03:46:27.278Z

When two or more dates in this format are sorted alphabetically, they also end up being sorted chronologically. However, more sophisticated analyses of dates (such as the amount of time that passed from this time to 2003-05-21T05:17:23.225Z) cannot be handled with simple string and numeric manipulation functions.

For this reason, XPath 2.0 defines two distinct general date types—dateTime and duration—plus a number of inherited subtypes. These objects are binary entities, using the local computer's time system to internally represent the actual times as an offset from a given event. This has to do with what most Unix systems know as universal time, and relates to the first PDP-11 becoming operational in February of 1971. These systems keep internal time as a monotonically increasing unsigned integer that counts the number of seconds elapsed since 00:00 Universal time on January 1, 1970 in the Gregorian calendar (Julian day 2440587.5).

The dateTime() constructor converts a date given as a string to a binary dateTime object, which can then be manipulated by other functions. For instance, to find the duration between the two dates given previously, you'd use the verbosely named get-dayTimeDuration-from-dateTimes function:

[View full width]

get-dayTimeDuration-from-dateTimes(dateTime('2003-05-21T05:17:23.225Z'), dateTime
  ('2002-10-08T03:46:27.278Z'))

This, in theory, should give you a duration() object that can itself be turned into a string representation. In practice, many of the date functions are still very much in flux and not even supported in the most up-to-date XPath parsers, principally because there is a lot of debate about the number of functions needed to handle days, months, years, and so on as separate data types. I recommend that you read the relevant documentation for date manipulation when the specification is finalized.

There are a few useful functions that will almost certainly be included in the final draft: current-dateTime(), current-date(), and current-time(). The current-date() function returns the date portion from your computer's system clock; that is, 2002-10-08. The current-time() function returns the time stamp (05:03:32.529Z), and current-dateTime() concatenates the two:

2002-10-08T05:03:32.529Z

Even without the capability to manipulate times, being able to determine the current date or time makes it possible to schedule events, create timestamps in documents, and perform rough comparisons to see whether an event has passed.


In Brief

  • XPath provides a consistent means of describing the location of zero or more nodes in an XML document.

  • A node can be identified by characteristics including a name and a type.

  • A sequence is as a linear list of zero or more items. All sequences are values, and all values are sequences.

  • Unlike node-sets, you can do standard set manipulation on sequences.

  • Collation is a specific order that a set of data must subscribe to. It's an implementation-specific capability, tied into a specific namespace URL.

  • The tokenize() function takes a source string and a delimiter string and converts the results into a sequence.

  • In XPath 2.0, data can be passed as a parameter, defined explicitly as a sequence, or pulled in from an external data source.

  • XPath includes the conditional statements if, then, and else.

  • An XPath 2.0 variable can only be defined to hold a single value, although that value can be a node with children.

  • XPath 2.0 incorporates the use of regular expressions to validate, match, replace, and tokenize text.

  • XPath 2.0 introduces atomic types.


Part II: The XQuery Language

3 XQuery Language Expressions

4 Extensions and Modular XQuery

5 XQuery Applications: Presentations, Graphics, and Syndication Feeds

6 Working with the XPath 2.0 Data Model

7 Understanding XQuery Semantics

8 XQuery and Databases

9 Advanced XQuery Topics


Chapter 3. XQuery Language Expressions

by Kurt Cagle

IN THIS CHAPTER

  • The Need for XQuery

  • Why Not SQL?

  • XSLT and XPath As Query Languages

  • The Structure of XQuery

  • Assignments with let

  • Iterations with for

  • Returning Results

  • where Oh where?

  • Ordering Up Some XQuery

  • Conditional Logic

  • Defining Functions

  • In Brief


The Need for XQuery

When you combine XML, which is essentially an open data format independent of any specific data-formatting language, with a universal addressing mechanism such as URLs, great things happen. These two factors together make the endpoints of a transaction transparent to the kind of technologies that are on the other ends.

However, this transparency can break down in the presence of databases. Even if a database has an XML-based interface (either through SOAP or through HTTP POST or GET commands), querying that database to retrieve a dataset makes things more complicated. SQL is the standard way to make queries against most databases—but each database vendor utilizes slightly (or not so slightly) different implementations of the SQL standard. What's worse is that the resultsets that come back are also formatted according to the whims of the database provider.

XQuery was proposed as a solution to this conundrum. To make databases truly transparent (so that it doesn't matter whether there is a Microsoft SQL Server, IBM DB2, Oracle 9i, PostgreSQL, or any other vendor's database engine), it is necessary to make a query language that will provide the following:

  • Platform consistency— The language should be the same, regardless of which vendor or product is being used.

  • XML-centric— The language should treat non-XML databases as being equivalent to XML data stores. Note that this does not necessarily imply that the language has to be written in XML.

  • Set-capable— SQL differs from languages such as Java or Visual Basic because it implicitly works on sets of data all at once rather than one item of data at a time. A new query language would similarly need to be set-focused—or more properly, node-focused.

  • Ease of use— The language should be easy enough to enable someone with a minimal database background to create complex queries.

The goal of this chapter is to examine the language in XQuery that was designed to handle all these primary points. The examples in this chapter are deliberately on the simplistic side to better illustrate the role of the command structure, but in general, XQuery finds utility more in the creation of fairly complex scripts.


Why Not SQL?

One of the first questions you can ask about XQuery is "Why not just create an updated version of SQL and use it with XML?" Beyond the vendor implementation issue we mentioned, one of the biggest problems that occurs comes from the infosets of SQL versus XML.

When you make a SQL query, you create from several tables a single virtual table (in simplest terms), which consists of the intersections, unions, and other filtered operations of the source tables. This is one of the reasons why being able to factor a problem domain into a minimal set of tables is so important: The more overlap that exists in tables, the more work must be done to eliminate spurious data sets.

Relational databases can be extraordinarily complex. Moreover, they are not good with "semi-structured" data (documents). Even a simple HTML document can very quickly overwhelm a relational database with its apparent complexity, and one of the characteristics of XML is the fact that a given element can have a range of possible subelements (or might even have an open definition in which the element can have anything as a subelement).

SQL mirrors the tabular mindset of good database design, and has been optimized for it. This means that it doesn't handle unpredictability and irregularity in data structure well.

As more data moves into an XML form, SQL's limitations become more pressing. As code becomes more complex, the advantages of a common XML-oriented data query language will become more obvious.


XSLT and XPath as Query Languages

A well-defined query language already exists for XML: XPath, covered in detail in Chapter 2, "Understanding the XPath Specification." You can format that query output with XSLT to create any XML format, so you might wonder why we need a dedicated XML query language.

XPath is an integral part of XQuery, as will be discussed in this chapter. Thus the real question is, "Why use XQuery rather than XSLT?" There are a few answers, although how compelling they are depends on how experienced you are with using XML:

  • Easier to use— XQuery uses more procedurally-oriented code than XSLT, so it might be more familiar to use than the often paradigm-bending XSLT language.

  • Less verbose— This is a valid charge for XSLT 1.0, but less so for XSLT 2.0. XSLT is an XML-based language, and even simple routines can take up pages of code.

  • Less document-centric— XSLT assumes an input stream of an XML document, although with features such as unparsed-text() and collection(), this requirement is less stringent in XSLT2. XQuery works implicitly upon sets of nodes that don't necessarily have to be XML in origin, although in all likelihood any XQuery solution would do an implicit conversion to XML before processing it.


The Structure of XQuery

Having described why XQuery is more or less necessary, let's look at its general makeup. The language consists of these primary areas:

  • for/let— Analogous to the SQL SELECT and SET statements, the for and let clauses let you define variables or iterate across a range of sequence values that are in turn assigned to a variable.

  • where— Analogous to the SQL WHERE statement, the where clause provides a set of conditions that filter or limit the initial selection in a for statement.

  • order-by— Analogous to SORT BY in SQL, order-by provides the ordering constraints on a sequence.

  • return— Analogous to the SQL RETURN statement, the XQuery return clause uses a custom formatting language to create output. The output does not necessarily have to be XML, although it is optimized to produce XML.

  • XPath— Most XPath 2.0 functions are supported in XQuery, as is the axis model used to navigate over XML structures (although some data sources might not support all aspects of XPath because of the type of data involved).

  • Functions— Analogous to SQL stored procedures, you can define functions in XQuery using the XPath language that can be called inline in an XML query.

  • Namespaces— A feature of XML rather than SQL. The declare namespace function associates a namespace URI with a prefix, crucial for indicating functionality.

Any number of keywords are associated with XQuery, but these broad categories describe different syntaxes depending upon what specifically needs to be done. For instance, here's an example of a full XQuery expression that uses all five of these areas:

declare namespace xs = "http://www.w3.org/2001/XMLSchema"
define function summaryText($char) returns xs:string
{
      concat($char/name,' is a ',$char/gender,' ',$char/species)
}
<results>{
      let $chars := input()//character[gender = 'Female']
      for $char in $chars
            where $char/level gt 5
            return
                   <summary health="{$char/health}"> {
                   attribute level {$char/level},
                   attribute date {current-dateTime()},
                  summaryText($char)
                   }
                  </summary>
      }
 </results>

This is a query for pulling out a series of records from a game database. It assumes an initial XML data stream with records corresponding to different characters, as shown in the file characters.xml:

<characters>
<character>
    <name>Aleria</name>
    <gender>Female</gender>
    <species>Heroleim</species>
    <vocation>Bard</vocation>
    <level>5</level>
    <health>25</health>
</character>
<character>
    <name>Shar</name>
    <gender>Male</gender>
    <species>Human</species>
    <vocation>Merchant</vocation>
    <level>6</level>
    <health>28</health>
</character>
<character>
    <name>Gite</name>
    <gender>Female</gender>
    <species>Aelvar</species>
    <vocation>Mage</vocation>
    <level>7</level>
    <health>18</health>
</character>
<character>
    <name>Horukkan</name>
    <gender>Male</gender>
    <species>Udrecht</species>
    <vocation>Warrior</vocation>
    <level>5</level>
    <health>32</health>
</character>
<character>
    <name>Gounna</name>
    <gender>Female</gender>
    <species>Noleim</species>
    <vocation>Mage</vocation>
    <level>8</level>
    <health>31</health>
</character>
<character>
    <name>Sheira</name>
    <gender>Female</gender>
    <species>Human</species>
    <vocation>Cleric</vocation>
    <level>4</level>
    <health>9</health>
</character>
<character>
    <name>Drue</name>
    <gender>Female</gender>
    <species>Voleim</species>
    <vocation>Warrior</vocation>
    <level>6</level>
    <health>32</health>
</character>
<character>
    <name>Paccu</name>
    <gender>Male</gender>
    <species>Human</species>
    <vocation>Merchant</vocation>
    <level>5</level>
    <health>24</health>
</character>
</characters>

When the query is executed, it returns an XML document showing all female characters that are of greater than fifth level, with a text summary:

[View full width]

<results>
      <summary date="2002-10-24T14:38:48" health="18" level="7"> Gite is a Female Aelvar<
  /summary>
      <summary date="2002-10-24T14:38:48" health="31" level="8"> Gounna is a Female Noleim<
  /summary>
      <summary date="2002-10-24T14:38:48" health="32" level="6"> Drue is a Female Voleim<
  /summary>
 </results>

Note that this output has been reformatted somewhat to make it more legible. Whitespace is not usually significant in XQuery—without the formatting, the result of the previous query is as follows:

[View full width]

<results><summary date="2002-10-24T14:38:48" health="18" level="7"> Gite is a Female Aelvar
                  </summary><summary date="2002-10-24T14:38:48" health="31"
   level="8">Gounna is a Female Noleim
                  </summary><summary date="2002-10-24T14:38:48" health="32" level="6">Drue
   is a Female Voleim
                  </summary>

</results>

Because each of the major areas has its own language and syntax, the best way to understand the full XQuery language is to break it into the code for each type of expression.


Assignments with let

XQuery, like XPath and XSLT, is a declarative language. What that means in practice is a little more nebulous. One way of thinking about it is that in a declarative language, after an item is defined within a given scope (such as a program block or function), the item can't be redefined within that scope. For instance, the following code is illegal in a declarative language:

summa = 0;
while (a in (1,3,6,4,7)){
      summa=summa+a;
      }
print(summa);

What causes the problem is that the variable summa changes its value within the same context. For many programmers, the idea that you can't use an accumulator like this might seem counterintuitive, but it turns out that placing one restriction on your code can significantly reduce errors when both developing and deploying code.

A declarative language can perform the same type of operation, but it works on the assumption of a generalized context. In essence, you are creating a buffer to which you're adding content, and although you can control what goes into that buffer, after the buffered content is created, you can't go in and change that buffer—you can only create other buffers from that one. That's why certain operations, such as summation, require specialized functions:

let $summa := sum((1,3,6,4,7))

The let statement in XQuery defines a function that has a constant value. This might seem like a way of saying a variable, but in fact, after the function is defined, it can't be redefined. Thus,

let $summa := sum((1,3,6,4,7))
let $summa := $summa + 6

is illegal in XQuery, because you are attempting to redefine the variable $summa.

You can use the let operator in conjunction with XPath to create a reference to a sequence. For instance, in the characters.xml file, you could create a sequence of female characters and assign it to the variable $femaleChars as follows:

let $femaleChars := document('characters.xml')//character[@gender = 'Female']

You could then retrieve the second female character by using the sequence notations discussed in Chapter 2:

let $secondFemaleChar := $femaleChars(2)


Iterations with for

The let keyword by itself makes sense if you are dealing with one item from a sequence at any given time, but XPath is ultimately a set-manipulation language, and you need to have some way of dealing with the information as a set. This is the domain of the for keyword.

The principle purpose of for is to assign to a temporary variable successive values in a sequence. For instance, the following code line steps you through the first five (named) letters of the Greek alphabet:

for $grkLetter in ("alpha","beta","gamma","delta","epsilon")

This code first associates the value "alpha" with the variable $grkLetter to perform some processing, then sets the value to "beta", and so forth until it reaches "epsilon". You could also do this with a previously defined sequence stored in a variable:

let $femaleChars := document('characters.xml')//character[@gender = 'Female']
for $femaleCharacter in $femaleCharacters

Similarly, you can use the XPath to operator to iterate over numbers to do something analogous to the for statement in C++, Java, or Visual Basic. This example iterates over the first ten numbers:

for $index in (1 to 10)

After the discussion about XQuery being a functional language in which you are unable to assign multiple values to a single variable name, the use of the for statement might seem to be a clear violation. However, it isn't. Technically speaking, the restriction says you cannot create two variables with the same name in the same scope. This is somewhat analogous to a set of loops in a language such as JavaScript. This language has the var keyword, which indicates that the variable being defined is unique for this scope. For instance, consider the following JavaScript fragment:

for (var index=0; index!= 1; index++){
      write(index+":");
      for (var index=0;index!=2;index++){
            writeln(index);
            }
      }

You have two distinct scopes: the first belonging to the outer for loop, the second to the inner. This example, when run, prints a potentially counterintuitive result:

0:0
0:1
0:2
1:0
1:1
1:2

The outer scope is temporarily suspended when a variable with the same name is defined within the inner scope, as long as the inner variable is defined with the var keyword. This makes it possible to avoid the possibility of namespace collisions, where you end up naming a variable the same way someone else named it in some other piece of code.

In essence, the XQuery for operator acts the same way—the local variable (the variable before the in keyword) is defined within the scope of the internal block, something analogous to

for (var tempVar in mySequence){

in a language like JavaScript. $tempVar is instantiated, populated, used, and then destroyed, at which point a second (or third, or fifth, or whatever) $tempVar is created. Because the variable is never created when it already exists, it cannot violate the tenet of reassignment.

Both for and let can also work with full node trees that can be defined inline. For instance, you can create an XQuery that defines a set of regular expression filters, which can be accessed later:

let $filters := (
    <filter name="phone" regex="\(\d{3}\)\d{3}-\d{4}"/>,
    <filter name="zipcode" regex="\d{5}(-\d{4})?"/>,
    <filter name="email" regex="\w+@\w+\.\w"/>
    )
for $filter in $filters
return $filter

The output is as follows:

<filter name="phone" regex="\(\d{3}\)\d{3}-\d{4}"/>
<filter name="zipcode" regex="\d{5}(-\d{4})?"/>
<filter name="email" regex="\w+@\w+\.\w"/>

In this case, the sequence of elements is defined explicitly. Because whitespace is not (generally) significant within XML queries, you can create rich XML trees inline:

let $filters := (
    <filter>
        <name>phone</name>
        <regex>\(\d{3}\)\d{3}-\d{4}</regex>
    </filter>,
    <filter>
        <name>zipcode</name>
        <regex>\d{5}(-\d{4})?</regex>
    </filter>,
    <filter>
        <name>email</name>
        <regex>\w+@\w+\.\w </regex>
    </filter>
)
for $filter in $filters
return $filter

This code produces slightly more complex output:

    <filter>
        <name>phone</name>
        <regex>\(\d{3}\)\d{3}-\d{4}</regex>
    </filter>
    <filter>
        <name>zipcode</name>
        <regex>\d{5}(-\d{4})?</regex>
    </filter>
    <filter>
        <name>email</name>
        <regex>\w+@\w+\.\w </regex>
    </filter>


Returning Results

Neither assignment nor iteration by itself can produce output—they are used only to define variables or iterate through sets of variables, in a manner similar to the SQL SET and SELECT statements, respectively. The key to working with XQuery is to use these statements to choose the nodes with which you're going to work, and then pass those nodes onto the relevant output format. This is where the return keyword comes into play.

The purpose of return is to take the sets defined by the previous for and let statements and turn them into some form of result. It's important to realize that the result of an XQuery expression does not have to be XML. It could be a sequence of values, a single string expression, or a host of any other possible results, although the language is optimized to produce XML preferentially.

Notice, for instance, that the result in the previous sample is not, strictly speaking, an XML document:

<filter>
        <name>phone</name>
        <regex>\(\d{3}\)\d{3}-\d{4}</regex>
    </filter>
    <filter>
        <name>zipcode</name>
        <regex>\d{5}(-\d{4})?</regex>
    </filter>
    <filter>
        <name>email</name>
        <regex>\w+@\w+\.\w </regex>
    </filter>

Instead, it is a sequence of such documents. The output of an XQuery is a sequence of something, whether of XML document nodes, strings, numbers, or some combination. For instance,

for $a in (1 to 10) return $a

produces the following numeric sequence output

1,2,3,4,5,6,7,8,9,10

as distinct nodes.

This raises an interesting question. What is an output node? In essence, when an XQuery generates a result, the implementation of the result is application-specific. The result is, as mentioned, a sequence of items. Internally, what is returned usually is a DOM XMLNode object, although it might be subclassed as an element, attribute, text node, or some other resource. Typically, a non-XML result (anything that can't immediately be translated into an XML element or attribute) is returned as a text node, regardless of the data type of the variables being returned.

The expression after the return syntax can be a little confusing, especially if you are used to working with XSLT. You can introduce new elements into the output directly through the use of traditional XML-bracketed elements. For instance, you could in theory generate XML output from the list of numbers by placing an arbitrary XML element (such as a <number> tag) around the variable:

for $a in (1 to 3) return <number>$a</number>

Unfortunately, this will likely not give you the result you expect. The previous XQuery produces this result:

<number>$a</number>
<number>$a</number>
<number>$a</number>

Because any time you introduce an XML tag (opening and closing) into a result, the XQuery processor treats anything within those tags as being more markup and doesn't evaluate the result. Consequently, to do such an evaluation, you need to use the evaluation operators: {}. Such operators instruct the XQuery engine to treat the content within the brackets as XQuery expressions.

So, to get the expected result (a set of three numbers within tags), you change the XQuery to incorporate the evaluation operators:

for $a in (1 to 3)
return <number>{$a}</number>

This can lead to some interesting conditions. In any XQuery, there is an implicit assumption that the expression starts in evaluation mode—in other words, there is an implicit return statement at the highest level. That's why expressions such as

for $a in (1 to 3)
return <number>{$a}</number>

are evaluated in the first place. However, if you place arbitrary elements around the XQuery expression, the mode of operation switches into static mode:

<numbers><!-- now in static mode -->
for $a in (1 to 3)
return <number>{$a}</number>
</numbers>

In this case, the text is treated as if it is just that—text—until the evaluation brackets are reached. At that point, you ask the XQuery expression to evaluate a variable that has not been previously defined ($a), and it should fail. Indeed, using eXcelon Stylus Studio, the error received when this script ran was specifically "Variable a does not exist".

Consequently, to evaluate the text as if it were an XQuery expression, you must encompass the text within the <numbers> element with the evaluation operators {}:

<numbers>
{
for $a in (1 to 3)
return <number>{$a}</number>
}
</numbers>

This returns the expected results:

<numbers>
      <number>1</number>
      <number>2</number>
      <number>3</number>
</numbers>

This example also illustrates a second principal about evaluating XQuery expressions: You can have multiple nested {} elements, as long as they are used within elements in static context. For instance, in the example, the <numbers> tag puts the XQuery into static mode, and you have to place evaluation operators around the whole expression. Similarly, the <number> element puts the XQuery expression back into static mode, so you once again have to place the expression to be evaluated (in this case, the $a element) into the braces.

This can be seen in a slightly more sophisticated XQuery:

<numbers>
{for $a in (1 to 3) return
      <set>{for $b in (1 to $a)
            return <item>{$b}</item>
      }</set>
}
</numbers>

In this case, the $a variable iterates through the values from 1 to 3, producing <number> elements as a child. Each number element in turn evaluates from 1 to $a (whatever it happens to be for that loop) and performs its own internal return to produce <item> elements. This produces the following result:

<numbers>
      <set>
            <item>1</item>
      </set>
      <set>
            <item>1</item>
            <item>2</item>
      </set>
      <set>
            <item>1</item>
            <item>2</item>
            <item>3</item>
      </set>
</numbers>

These evaluated expressions can, of course, be more complex than simply returning the value of a variable. For instance, you can create a table in HTML that sums up sequences of varying lengths, as follows:

<html>
<head>
      <title>Summations</title>
</head>
<body>
<h1>Summations</h1>
<table>
      <tr>
            <th>Index</th>
            <th>Sum From 1 to Index</th>
      </tr>
{for $a in (1 to 10) return
      <tr>
            <td>{$a}</td>
            {
            let $b := (1 to $a)
            return
                   <td>{sum($b)}</td>
            }
      </tr>
}
</table>
</body>
</html>

This example points out several salient lessons. First, you can use XQuery to generate HTML, which makes it a particularly potent tool for creating reports—an avenue we'll explore in greater depth in Chapter 4, "XQuery and XSLT." Second, you can use XQuery functions in the result blocks, such as the use of the sum() function to add up each successive $b list (that is, the lists (1), (1,2), (1,2,3), (1,2,3,4), and so on). Finally, any variable that is defined in an outside expression (such as the $a variable) is available for use within the inside expression, such as

let $b := (1 to $a)

You can similarly perform such evaluated expressions within attributes. For instance, suppose you want to create a table of colors in HTML. To do so, you need both the name of the table and a rectangle of the appropriate color illustrating the shade, set using the Cascading Style Sheets background-color property, as follows (see Figure 3.1):

<html>
<head>
      <title>Summations</title>
</head>
<body>
<h1>Summations</h1>
{let $colors :=("white","red","blue","green","yellow","purple","orange","black")
return
<table border="1">
      <tr>
            <th>Color</th>
            <th>Example</th>
      </tr>
{for $color in $colors return
      <tr>
            <td>{$color}</td>
             <td style="background-color:{$color}">&#160;</td>
      </tr>
}
</table>
}
</body>
</html>
Figure 3.1. You can use XQuery to generate more than just textual data, as this color sample illustrates.

The entity &#160; is a nonbreaking space—within an HTML <td> element, it ensures that the background color will always be rendered. What's most important here is the use of the evaluated expression in the style attribute:

<td style="background-color:{$color}">&#160;</td>

This basically replaces the indicated expression {$color} with its associated values: "white", "red", "blue", and so on, and as with elements, the expression within the attribute block could be a full XQuery expression (whitespace, including carriage returns, doesn't count in the way the attribute is handled).

The tag notation is useful in certain circumstances, but sometimes it can get in the way. The element and attribute operators perform the same purpose, but they don't require the use of the closing tag. The previous XQuery could be rewritten using these operators as follows:

<html>
<head>
      <title>Summations</title>
</head>
<body>
<h1>Summations</h1>
{let $colors :=("white","red","blue","green","yellow","purple","orange","black")
return
<table border="1">
      <tr>
            <th>Color</th>
            <th>Example</th>
      </tr>
{for $color in $colors return
      element tr {
            element td {$color},
             element td {
                  attribute style {'background-color:',$color},
                  '&#160;'
                  }
            }
}
</table>
}
</body>
</html>

The non-XML usage for listing elements, attributes, and text content can make your code easier to read. The element constructor, for instance, takes the name of the element as the first parameter, and the value of the element (possible as an evaluated expression) as the second element. Thus,

element td {$color},

creates a new element <td> and places the text value of $color into it.

You can create sibling nodes (attribute, element, or text) with the comma separator (,) operator. Thus, in the definition of the second td element, the expression

element td {
      attribute style {'background-color:',$color},
      '&#160;'
      }

includes a new attribute node named style that in turn creates two child text nodes: the literal 'background-color' and the result of evaluating the $color variable. Because the content of an attribute must be a string, the XQuery engine concatenates these two values together into a single string value.

The same type of action is at work with the encompassing td element, which not only generates the style attribute, but also includes the literal '&#160;', the nonbreaking space character, as a text node. There is no direct concatenation here of the two nodes, by the way, because they are of a differing type—the attribute node attaches to an element as an attribute, whereas the text node is attached in a different way as part of the set of text nodes.

This can be given in a slightly simpler form. This expression

element a {
      element b {
            attribute b1 {t1},
            element c,
            'strD'
            }
      }

is the same as this tagged expression:

<a>
      <b b1="t1">
      <c/>
      strD
      </b>
</a>

The two formats are equivalent in their application, so you should use the format that works best for your needs.

Given the richness of possible output, there is always more to say about the return keyword, but the examples given here are sufficient to lay the groundwork for complex tasks.


where Oh where?

The where keyword serves as a means to define conditions about which data should be chosen, and serves a purpose roughly analogous to the SQL WHERE clause. In both cases, you are creating what is called a predicate cause, which is used to narrow a specific range of potential options.

For instance, suppose you want to display only those colors that contain the letter r in a table, as shown in the previous section. (Why you'd want this search criteria is a task better suited to psychoanalysts than programmers, but it's illustrative nonetheless.) Here you could use the where clause to quantify the search (see Figure 3.2):

<html>
<head>
      <title>Summations</title>
</head>
<body>
<h1>Summations</h1>
{let $colors :=("white","red","blue","green","yellow","purple","orange","black")
return
<table border="1">
      <tr>
            <th>Color</th>
            <th>Example</th>
      </tr>
{for $color in $colors where contains($color,'r')return
      element tr {
            element td {$color},
             element td {
                  attribute style {'background-color:',$color},
                  '&#160;'
                  }
            }
}
</table>
}
</body>
</html>
Figure 3.2. You can filter on colors that have the letter r using the where clause.

The contains() function is, of course, one of the functions that XQuery and XPath2 jointly share. It returns a Boolean value if the particular node is either a string or can be automatically converted to a string, and the second expression can be found in the first.

The where predicate should generally follow either the let or for keyword (usually for) but precede the statement's return clause. It should also return either a Boolean value or something that can be converted to a Boolean value, such as an empty node-list or sequence, the value 0 or an empty string ("").

The where expression does not hold quite the same level of importance in XQuery as does the analogous WHERE expression in SQL. For starters, it is possible to use XPath predicates in the for and let expressions to act as a filter for XPath (and it is more economical to do so). The previous example can be rewritten using an XPath predicate as follows:

{for $color in $colors[contains(.,'r')] return  . . .

Although this code might not be as readily understandable (you have to appreciate the rule of the context operator), it works just as well. In this case, the context operator (.) performs an automatic iteration over each node in $colors to perform the calculation to determine whether r is part of the color value. In the first case, the where expression examines each $color object in turn to determine which node to pass on; in the second, the XPath predicate winnows down the set of $colors to be passed to the iterator. It's the same result in both cases, but one occurs at a different point in the cycle than the other.


Ordering Up Some XQuery

For more than a year, XQuery had been inextricably tied to the acronym FLWR (presumably pronounced "flower") for For|Let|Where|Return. However, as of the November 15, 2002 draft, this acronym became FLWOR, an explicit recognition that ordering plays a fairly major part in database retrieval. The O stands for order by, a keyword that performs much the same actions as the SQL ORDER BY command.

The order by command follows for expressions (and where expressions where they exist), and indicates for a given set of data the order in which the data is output. When it is not explicitly specified with order by, the output order generally depends on the specific system architecture. With an XML file, for instance, the specific order is the order in which the XML parser walks the tree (usually the child of each node is processed, then the next sibling, and then the parent)—the so-called document order.

The order by keyword indicates that the content should be sorted in ascending order, unless otherwise indicated according to the data type of the expressions being evaluated. This also implicitly assumes that the data type has some internal notion of ordering, which in turn assumes that there is some sense of schema validation acting on the data. For instance, consider the characters.xml file defined earlier in the chapter. Assume that it has an XSD schema (characters.xsd), which looks something like this:

[View full width]

<?xml version="1.0"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

      <xsd:element name="characters" type="Characters"/>

      <xsd:complexType name="Character">
            <xsd:sequence>
                  <xsd:element name="name" type="xsd:string"/>
                  <xsd:element name="gender" type="Gender"/>
                  <xsd:element name="species" type="xsd:string"/>
                  <xsd:element name="vocation" type="xsd:string"/>
                  <xsd:element name="level" type="xsd:nonNegativeInteger"/>
                  <xsd:element name="health" type="xsd:int"/>
            </xsd:sequence>
      </xsd:complexType>

      <xsd:simpleType name="Gender">
            <xsd:restriction base="xsd:string">
                  <xsd:enumeration value="Female"/>
                  <xsd:enumeration value="Male"/>
                  <xsd:enumeration value="Other"/>
            </xsd:restriction>
      </xsd:simpleType>

      <xsd:complexType name="Characters">
            <xsd:sequence>
                  <xsd:element name="character" type="Character" minOccurs="0"
   maxOccurs="unbounded"/>
            </xsd:sequence>
      </xsd:complexType>

</xsd:schema>

The schema can then be associated with the characters.xml file by editing the enclosing <characters> element:

[View full width]

<characters xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi
  :noNamespaceSchemaLocation="http://www.kurtcagle.net/schemas/Characters.xsd">
<character>
    <name>Aleria Delamare</name>
    <gender>Female</gender>
    <species>Heroleim</species>
    <vocation>Bard</vocation>
    <level>5</level>
    <health>25</health>
</character>
<character>
    <name>Shar Denasthenes</name>
    <gender>Male</gender>
    <species>Human</species>
    <vocation>Merchant</vocation>
    <level>6</level>
    <health>28</health>
</character>
...
</characters>

The xsi:noNamespaceSchemaLocation gives the URL (here on a local machine) to the schema file. This schema thus indicates that <level> is a nonnegative integer, and <health> is an unbounded integer.

Given that, an XQuery can be written that orders the characters by health:

for $character in document('characters.xml')//character
order by health
return
      <healthReport>
            {$character/name}
            {$character/health}
</healthReport>

This will return the following collection:

<healthReport>
      <name>Sheira</name>
      <health>9</health>
</healthReport>
<healthReport>
      <name>Gite</name>
      <health>18</health>
</healthReport>
<healthReport>
      <name>Paccu</name>
      <health>24</health>
</healthReport>
<healthReport>
      <name>Aleria</name>
      <health>25</health>
</healthReport>
<healthReport>
      <name>Shar</name>
      <health>28</health>
</healthReport>
<healthReport>
      <name>Gounna</name>
      <health>31</health>
</healthReport>
<healthReport>
      <name>Horukkan</name>
      <health>32</health>
</healthReport>
<healthReport>
      <name>Drue</name>
      <health>32</health>
</healthReport>

Notice here that the order was done implicitly using integers. The first health report indicates a health of 9:

<healthReport>
      <name>Sheira</name>
      <health>9</health>
</healthReport>

Had the ordering been done without regard to the data type of the element, this entry would have been last, because 9 is alphanumerically higher in order than 32.

Suppose, however, that you didn't have a specific schema against which to validate the characters.xml file. Would this have made sorting numerically impossible? No, because you can also declare an element's data type inline:

for $character in document('characters.xml')//character
order by $character\health as xs:integer
return
      <healthReport>
            {$character/name}
            {$character/health}
</healthReport>

In this case, the contents of the <health> element are cast to an integer. The xs: namespace is implicitly defined within the XQuery schema, and indicates the XSD schema data types.

You can also determine the order of a query. The ascending and descending keywords determine the direction of the sort, and appear after the sort expression is given. For instance, suppose that each character element has an additional child element: <createdBy> of type xs:dateTime. To order the characters in descending order by the time they were created, you'd write this query:

for $character in document('characters.xml')//character
order by $character\dateCreated as xs:dateTime descending
return
      <dateReport>
            {$character/name}
            {$character/dateCreated}
      </dateReport>

Creating more sophisticated queries requires the intelligent use of FLWOR type expressions. For instance, if you want to list all entries in the characters.xml file by level, then sort all characters at that level by name, your query would need to use the distinct-values() function to retrieve the levels, then use that as the key for determining the set of each character within that level:

let $characters := document('characters.xml')//character
for $level in distinct-values($characters/level)
      order by $level
return
      <level value="{$level}">{
            for $character in $characters
            where $character/level = $level
                  order by $character/name
            return $character/name
            }
      </level>

This would create the following output:

<level value="4">
      <name>Sheira</name>
</level>
<level value="5">
      <name>Aleria</name>
      <name>Horukkan</name>
      <name>Paccu</name>
</level>
<level value="6">
      <name>Shar</name>
      <name>Drue</name>
</level>
<level value="7">
      <name>Gite</name>
</level>
<level value="8">
      <name>Gounna</name>
</level>

One of the problems with such sorting is that occasionally an empty order by expression will be used. For instance, suppose that a new character is added to the characters.xml list:

<character>
    <name>Yane Helavela</name>
    <gender>Female</gender>
    <species>Human</species>
    <vocation>Illusionist</vocation>
</character>

In this case, no level element exists. The question that arises is whether this particular element should appear at the beginning or end of a sequence when sorted. This problem is answered by the empty greatest and empty least keywords. The empty greatest command indicates that whenever an item to be sorted has an empty sort key (or doesn't have one at all), it will always be assumed to be at the end of the sort order. On the other hand, empty least will place the same element at the beginning of the sort order.

The stable keyword, when added to the list of order by qualifiers, indicates another condition: what happens when two identical sort keys are found for different elements. For instance, if two different entries have the same creation date, the order of the output for the items becomes more questionable. In that situation, the stable keyword is used to tell the processor to retain the initial order in which the elements are retrieved, whereas the processor is free to implement its own ordering scheme if the stable keyword is absent. In most cases, this shouldn't make a major difference in the ordering.

Finally, if you have a reference to a specific collation (such as "eng-us"), you can indicate this particular collation as the basis for sorting through the collation keyword on the order by expression. For instance,

for $character in $characters
      order by $character/name collation "eng-us"
return $character

will order the characters in ascending (the default) order using the U.S. English collation.


Conditional Logic

XQuery supports XPath 2.0, and consequently works with the conditional if/then/else keywords of XPath. This makes it possible to create fairly sophisticated logical expressions, depending on specific characteristics in the source data. For instance, suppose you want to list the characters from the characters.xml file in a table, with ledger printing (one row white, the next row light green, the next row white, and so forth). This could be incredibly difficult to do with FLWOR notation, but by incorporating the conditional if keyword (and a little CSS), it becomes much easier (see Figure 3.3):

[View full width]

<html>
      <head>
      <style>
.evenRow {{background-color:white;color:black;}}
.oddRow  {{background-color:lightGreen;color:black;}}
      </style>
      </head>
      <body>
            <h1>Characters</h1>
            <table cellspacing="0" cellpadding="3">
            <tr>
                  <th>Name</th>
                  <th>Gender</th>
                  <th>Species</th>
                  <th>Vocation</th>
                  <th>Level</th>
            </tr>
            {
      let $characters := input()//character
      for $character in $characters
      let $class := if (index-of($characters,$character) mod 2 = 0) then 'evenRow' else
   'oddRow'
            return
                  <tr class="{$class}">
                        <td>{string($character/name)}</td>
                        <td>{string($character/gender)}</td>
                        <td>{string($character/species)}</td>
                        <td>{string($character/vocation)}</td>
                        <td>{string($character/level)}</td>
                  </tr>
            }
            </table>
      </body>
</html>
Figure 3.3. Conditional logic can be used to create changes in both content and stylistic output.

This particular example works by applying CSS classes (evenRow and oddRow) to alternating lines in the output table. The conditional test relies upon the expression

let $class := if (index-of($characters,$character) mod 2 = 0) then 'evenRow' else 'oddRow'

where the index-of() function returns the position of the character relative to the $characters sequence. The mod keyword from XPath performs a modulus (or remainder) on the expression, returning 0 if the expression is divisible by two, or 1 if it is not. The then and else functions have implicit return elements associated with them, so they can include complex XQuery statements.

In addition to illustrating the use of the if/then/else statement (covered in greater detail in Chapter 2), this sample also illustrates another feature: escaping the bracket {} characters. CSS uses brackets to indicate the CSS rule definitions, but in the sample, the XQuery processor would attempt to interpret the contents as XQuery expressions. To escape this behavior, use double brackets rather than single brackets (that is, {{ and }} instead of { and }).

Note as well that, unlike other languages, you must include both a then and an else in an if expression in XQuery. Should you run into a situation where you don't need to return anything in the then or else block, return an empty string '' as the result:

If ($a) then 'b' else ''

I Ain't Got No…

The conditions that can be evaluated in the if block include anything that can be cast to a Boolean value. However, there are a few expressions that XQuery specifically provides that can significantly improve performance, namely some ... satisfies and every ... satisfies. These two expressions make it possible to determine whether there exists at least one item in a sequence that satisfies a given condition, and whether all items in a sequence satisfy a given condition, respectively, without having to create counting functions to perform the same tests.

As an example, for the set of $characters defined previously, you can test to see whether the group contains at least one mage (see Figure 3.4):

let $characters := input()//character
let $response := if (some $character in $characters satisfies $character/vocation="Mage") then
'Party has a mage' else 'Party does not have a mage.'
return
<html>
      <head>
      </head>
      <body>
            <h1>Mage Query</h1>
            <div>{$response}</div>
      </body>
</html>
Figure 3.4. This query determines whether a party of characters includes a mage.

The $response variable determines, for the set of $characters, whether at least one character has a vocation element of value "Mage", and returns the appropriate response string.

Similarly, the every keyword is used to perform a blanket test to determine whether all items in the set satisfy a condition, returning the value false() if even one item does not. For instance, assume that the cutoff level for a character to enter into a party is the fifth level. If even one character is below the fifth level, the party is underqualified:

let $characters := input()//character
let $response := if (every $character in $characters satisfies $character/level ge 5) then
'Party is qualified to depart' else 'Party is not experienced enough.'
return
<html>
      <head>
      </head>
      <body>
            <h1>Is Party Qualified?</h1>
            <div>{$response}</div>
      </body>
</html>

Once again, it's worth noting how much XQuery is geared toward set manipulation. The every and some functions are extremely efficient; for instance, some will stop evaluating the moment it discovers one item that satisfies the query. Because many databases also have the capability to do fast queries based upon generalized some or every queries, XQuery can leverage these to significantly speed up the evaluation of expressions.


Defining Functions

XPath has a fairly comprehensive set of functions for doing everything from performing date calculations to evaluating regular expressions. However, sometimes it's useful to be able to build more sophisticated functions out of this core set of basic functions for doing business logic-types of evaluations.

XQuery consequently also supports the capability to create user-definable functions. These functions are XQuery/XPath in origin, and are called in the same context as XPath expressions. For instance, suppose you want to take a date in the standard XSD notation (YYYY-MM-DD) and convert it into the American standard notation MM/DD/YYYY, and you want to do it for several different instances of data that have the following structure (directory.xml):

<directory>
      <file name="chapter3.xml" dateCreated="2002-11-25" dateModified="2002-11-28"/>
      <file name="chapter3app1.xml" dateCreated="2002-11-25" dateModified="2002-11-29"/>
      <file name="chapter3.toc" dateCreated="2002-11-25" dateModified="2002-11-30"/>
      <file name="chapter2.xml" dateCreated="2002-11-18" dateModified="2002-11-21"/>
</directory>

You can create an XQuery to provide a report:

<html>
      <body>
      <h1>File Report</h1>
      <table>
            <tr>
                  <th>File Name</th>
                  <th>Date Created</th>
                  <th>Date Last Modified</th>
            </tr>
{for $file in document('directory.xml')//file
return
      <tr>
            <td>{$file/@name}</td>
            <td>{let $refDateStr := string($file/@dateCreated)
      let $year := substring($refDateStr,1,4)
      let $month := substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)}</td>
      <td>{let $refDateStr := string($file/@dateModified)
      let $year := substring($refDateStr,1,4)
      let $month := substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)}</td></tr>
}
            </table>
      </body>
</html>

However, there are two difficulties. First, you have a certain degree of code duplication, with similar routines given for formatting the date in the new order. A second related problem is that it is difficult to ascertain exactly what the script is designed to do.

This is a case where working with functions can ameliorate your problems somewhat. You can define a new function called format-date() that takes an XSD type date string as a parameter and formats the date into an American Standard notation:

define function format-date($dt as xs:dateTime) as xs:String
{
let $refDateStr := string($dt)
      let $year := substring($refDateStr,1,4)
      let $month := substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)
}

First the code creates a new function called format-date in the immediate environment:

define function format-date

The name of the function can include alphanumeric characters along with the underscore (_) and dash (-) characters only.

You can set up zero or more parameters, separated by commas:

define function format-date($dt as xs:dateTime)

The parameters (such as $dt) must be preceded by a dollar sign. A parameter doesn't have to include a data type, as the data type can be inferred dynamically at runtime, but it's generally a good idea to include one if needed. Note, however, that this also precludes overloading (more than one function with the same name and different parametric signatures). The parameter-passing model is thus more akin to languages like JavaScript (or XSLT) than it is Java.

The xs: namespace prefix, discussed earlier, is necessary if you include schema types. Note that some XQuery parsers might support other schema languages, and as such will probably have different data type prefixes:

define function format-date($dt as xs:dateTime) as xs:string

The result type, similarly, need not be specified (as with parameters, it defaults to string and/or numeric general types depending upon the processor) but can be useful to ensure that the results are type-safe. The implication here, of course, is that the resulting output will be a string, rather than a specific nodal type.

The body of the function is itself an XQuery expression—here, a series of lets that breaks the initial text string into chronological pieces:

define function format-date($dt as xs:dateTime) as xs:string
{
      let $refDateStr := string($dt)
      let $year := substring($refDateStr,1,4)
      let $month:= substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)
}

Note that there are also date-specific functions that do the same thing; however, they are still being finalized in the XQuery working draft.

The outer return clause for the function returns the specific contents of the function to the outside world, and of course should have the data type specified by the return type in the function declaration (here, xs:string):

define function format-date($dt as xs:dateTime) as xs:string
{
      let $refDateStr := string($dt)
      let $year := substring($refDateStr,1,4)
      let $month:= substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)
}

You can have subordinate return values that create intermediate results, but you must have at least one final outer return value.

There is one implication of XQuery being a "side effect-free" language: You cannot, within XQuery, have a situation where the function changes some external, global variable. Anything passed into an XQuery function is passed by value, not reference. An immediate consequence is that a function must always return something of value—you cannot have a void function type (although you can have one with an empty string or sequence).

The functions are defined ahead of time within the XQuery command, and then are invoked as you would expect for functions. Taking the date-modifying report code mentioned earlier, the functional notation simplifies it considerably (see Figure 3.5):

define function format-date($dt as xs:dateTime) as xs:string
{
      let $refDateStr := string($dt)
      let $year := substring($refDateStr,1,4)
      let $month:= substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)
}

<html>
      <body>
      <h1>File Report</h1>
      <table>
            <tr>
                  <th>File Name</th>
                  <th>Date Created</th>
                  <th>Date Last Modified</th>
            </tr>
{for $file in document('directory.xml')//file
return
      <tr>
            <td>{string($file/@name)}</td>
            <td>{format-date($file/@dateCreated)}</td>
            <td>{format-date($file/@dateModified)}</td>
      </tr>
}
            </table>
      </body>
</html>
Figure 3.5. Consolidating reused code into functions can significantly simplify the source query, and encourages the development of code libraries.

The body of the code now contains much less code, and the intent of the programming becomes considerably clearer in this example. The expressions

<td>{format-date($file/@dateCreated)}</td>
<td>{format-date($file/@dateModified)}</td>

indicate that the dateCreated attribute and then the dateModified attribute of the file element (here contained in the $file variable) be changed into the MM/DD/YYYY notation, to produce the final output in HTML:

<html>
      <body>
            <h1>File Report</h1>
            <table>
                  <tr>
                        <th>File Name</th>
                        <th>Date Created</th>
                        <th>Date Last Modified</th>
                  </tr>
                  <tr>
                        <td>chapter3.xml</td>
                        <td>11/25/2002</td>
                        <td>11/28/2002</td>
                  </tr>
                  <tr>
                        <td>chapter3app1.xml</td>
                        <td>11/25/2002</td>
                        <td>11/29/2002</td>
                  </tr>
                  <tr>
                        <td>chapter3.toc</td>
                        <td>11/25/2002</td>
                        <td>11/30/2002</td>
                  </tr>
                  <tr>
                        <td>chapter2.xml</td>
                        <td>11/18/2002</td>
                        <td>11/21/2002</td>
                  </tr>
            </table>
      </body>
</html>

There are two issues to be aware of when dealing with functions: namespaces and libraries. The following sections describe these issues.

Namespaces

The XQuery language has its own specific (default) namespace that defines the functions that are commonly available. Although you can create functions into that namespace (which you are doing implicitly when you create a function without a namespace prefix), you stand the possibility of overwriting a system function with one of your own. In the current context, where all functions are local, this is not necessarily a bad thing, but in situations where more than one person is relying on these functions, that situation could prove disastrous.

SHOP TALK: NAMESPACES CAN BE USEFUL, NOT JUST A NUISANCE

People who work with XML only periodically sometimes see namespaces as something of a nuisance. Namespace identifiers are often long and unwieldy, and namespace prefixes can make a seemingly straightforward block of XML seem much more complex. However, namespaces can come in handy.

For instance, I've dealt with XML schemas describing such things as framework components (such as describing a form in Visual Basic via XML). A number of us were working on this application, and it was very important to ensure that the XML code one person wrote with samples wouldn't end up contaminating the base code until it had properly been approved.

To get around this, each developer was assigned their own namespace. The interpreter of the XML was programmed so that only certain namespaces would be enabled in each respective build (that is, you could run your own test-code, but other people running this code without the namespace enabled wouldn't have your functionality). After the XML code was deemed to be working correctly, a particular code's prefixes were swapped over to the formal implementation.

This system worked surprisingly well, especially as we moved more of our operant code into XML form. Without the namespaces, it would have been impossible to keep the coding straight; with them, not only could we tell at a glance whose code we were dealing with, but the software applications could use the same namespaces to assign functionality.

The declare namespace command lets you define other namespaces for use within function declarations. For instance, you might decide to create a package of date functions, and associate them with a given namespace URI (http://www.kurtcagle.net/schemas/xquery/date, for instance). This namespace would then be used to refer to all functions within that package as follows:

declare namespace dates = "http://www.kurtcagle.net/schemas/xquery/date"

define function dates:format-date($dt as xs:dateTime) as xs:string
{
      let $refDateStr := string($dt)
      let $year := substring($refDateStr,1,4)
      let $month:= substring($refDateStr,6,2)
      let $day  := substring($refDateStr,9,2)
      return concat($month,'/',$day,'/',$year)
}

<html>
      <body>
      <h1>File Report</h1>
      <table>
            <tr>
                  <th>File Name</th>
                  <th>Date Created</th>
                  <th>Date Last Modified</th>
            </tr>
{for $file in document('directory.xml')//file
return
      <tr>
            <td>{string($file/@name)}</td>
            <td>{dates:format-date($file/@dateCreated)}</td>
            <td>{dates:format-date($file/@dateModified)}</td>
      </tr>
}
            </table>
      </body>
</html>

By doing this, you avoid the problem of namespace collision, and not coincidentally, make it easier to organize your code.

Code Libraries

The second issue is a little more irksome, and has to do with the creation of code libraries. One of the principal reasons for working with functions is the capability to build function libraries that you (and others working in the same space) can use in your own code.

Currently (as of November 15, 2002) there is no provision within the XQuery specification for indicating code libraries, although there is an open-issue item concerning it. The primary difficulty in working with such external libraries revolves around the fact that such function libraries should realistically be in their own namespaces.

One speculative form for adding such functional libraries might look something like the following for the date library:

[View full width]

import "dateFunctions.xquery" in namespace dates = "http://www.kurtcagle.net/schemas
  /xquery/dates"

In this case, dateFunctions.xquery is a URL that contains all the custom functions associated with dates. Once declared in this manner, the functions require the namespace prefix to be invoked (for example, dates:format-date($myDate)). Note that by associating the namespace (and its prefix) with the function set, you can use a different prefix than any defined within the imported XQuery call.

Ultimately, functions in XQuery serve much the same purpose as stored procedures (SPROCS) within SQL: They simplify the coding involved within queries, and also form a mechanism for encapsulating business logic within queries (a topic to be covered in Chapter 4 and elsewhere). This becomes critically important in dealing with pipelined architectures, in which the XQuery acts as a filter on a dataset to be passed to another component (such as a Web service, or an XSLT transform). Expect to see more on function libraries in the final specification.


In Brief

  • XQuery is a declarative language. After an item is defined within a given scope (such as a program block or function), the item can't be redefined within that scope.

  • The let statement defines a function that has a constant value.

  • The for keyword assigns to a temporary variable successive values in a sequence.

  • The return keyword takes the sets defined by for and let statements and turns them into a result—XML, a sequence of values, a single string expression, and so on.

  • The where keyword lets you define conditions about which data should be chosen. It creates a predicate cause that narrows a range of potential options.

  • The order by command follows for expressions (and where expressions where they exist) and indicates for a given set of data the order in which the data is output. When it is not explicitly specified with order by, the output order depends on the system architecture.

  • XQuery works with XPath's conditional if/then/else keywords to create logical expressions.

  • XQuery provides many built-in functions and also lets you create user-definable functions. Anything passed into an XQuery function is passed by value, not reference, and a function must always return something of value.


Chapter 4. Extensions and Modular XQuery

by Kurt Cagle

IN THIS CHAPTER

  • Extending XQuery with Libraries

  • Enhancing XQuery with Pragmas and Extensions

  • Importing Modules

  • Invoking External Functions

  • Using Library Modules Effectively

  • In Brief


Extending XQuery with Libraries

If you could point to a single innovation within SQL that facilitated the use of the language at the enterprise level, it would probably be the capability to store and retrieve prebuilt scripts, passing in parametric information in the process. One of the later developments in the evolution of the XQuery Recommendation has been the recognition that it should be possible to build functions that encapsulate specific functionality in XQuery, and it should also be possible to store and retrieve these functions as external libraries, or modules, in which the functions are grouped by purpose. Additionally, XQuery makes it possible to define individual schema externally and import these as well, minimizing the requirements that such schema be contained as part of the main query. This provides an additional benefit: You can write extension functions into the XQuery engine for use by other languages.


Enhancing XQuery with Pragmas and Extensions

WATCH FOR CHANGES

The information in this chapter is based on the August 2003 XQuery working draft, and as such might have subtly changed by the time you read this.

The earliest form of XML was built on the foundations of SGML. It used processing instructions (PI) to provide processing information to the XML parser or SGML reader. For the most part, such processing instructions have faded from XML. One of the reasons has to do with a shift in thinking about XML documents—namely, that they should be seen as ways to contain specific data, and that it is the responsibility of the processor, not the data, to indicate how the data should be processed.

There are a number of advantages to this division of labor:

  • Intent— By removing PIs, you remove the author's intent about how the data should be consumed. If your application does something different with the data than the original PI indicated, you won't generate an error.

  • Portability— Every time you create a PI, you are making an assumption that the processor can understand this processing instruction—something that might be untrue in the platform-independent world of XML.

  • Logical flow— PIs are processed in the order in which they are encountered, and although most XML parsers do follow a fairly standard tree-walking algorithm, this cannot be assumed.

  • Specialized languages— SGML assumed the processing involved would be through a single specialized SGML renderer. However, XML can be processed by different applications in different ways.

Processing with Pragmas

Even though PIs are on the way out, there is value in passing instructions to the XQuery processor. For instance, suppose you want to ensure that a given operation doesn't hang because a reference to an external database points to a link that is intermittently slow. This falls outside the scope of the XQuery code. However, it might be worthwhile to signal an exception (raise an error) if the processing takes longer than ten seconds or so. This is the role of pragmas.

In practical terms, a pragma is an instruction to the XQuery implementation, and as such the pragmas available might be different from one implementation to the next. Typically, the pragmas are associated with a given namespace that identifies the implementation sets. For instance, a timeout pragma for the SuperXQuery engine (a hypothetical product) might be declared within your code as

declare namespace supxq = "http://www.superXQuery.com/SuperXQuery/pragmas"
... // Intermediate code
(:: pragma supxq:timeout 1000 ::)
count(input()//author)

The namespace declaration should look familiar from Chapter 3, "XML Query Language Expressions." The syntax

(:: pragma supxq:timeout 1000 ::)

identifies the expression as a pragma and notifies the processor that if the data hasn't loaded after 1,000 seconds, an error must be raised, using the mechanisms contained within the timeout() routine.

The specific scope of the pragma depends totally upon the pragma itself. This highlights the limitations of pragmas: They introduce functionality that is unique to the one processor, and users of such a pragma must have a much more thorough understanding of that functionality to use it than they would in writing basic queries.

If a pragma is defined in a module that is then used in a processor beyond what it was intended for, the pragma will be ignored. In some cases, this behavior might be desirable, but if it affects your programming, whether the pragma is evaluated can alter the outcome of the code, perhaps in unexpected ways. That's why you should use pragmas cautiously.

Understanding Must-Understand Extensions

The extension element is like a pragma that the processor must be able to understand (the process raises an exception otherwise). This stronger form of extension makes it far more useful in general processing. For instance, suppose you want to create the ext:outputTo extension, which appears after a return statement to indicate that the results should be sent to a file (or Web service) listed in the URL immediately after the expression. This might be used as follows:

[View full width]

declare namespace ext = "http://www.superXQuery.com/SuperXQuery/extensions"
... // Intermediate code
for $e in input()/employee[dateGreaterThan(@startDate,subtract-yearMonth
   Duration-from-date(current-date(),xdt:yearMonthDuration("P0Y1M")))]
   return <newEmployees>{$e}</newEmployees> (::extension ext:outputTo "newEmployees.xml" ::)

In this case, the XQuery retrieves all employees who have started within the last month as a collection wrapped within a <newEmployees> element, and outputs the result to the (local) URL newEmployees.xml.

Here you can see the difference between a pragma and an extension. The pragma would be ignored if the processor couldn't do something with it, which means the contents would be generated but sent to the default output stream, which might be undesirable. On the other hand, as an extension, the ext:outputTo must be understood; it will raise an error if it isn't, and the programmer can use the error as a notification to handle the processing in a different way.

XQUERY IMPLEMENTATION DIFFERENCES

It is possible for developers to customize extensions, but the specific mechanism for doing so may change from one XQuery implementation to the next. Please consult the API documentation for your implementation to see both what extensions exist and how to write extensions yourself, as this goes beyond the scope of this book.


Importing Modules

The pragmas and must-understand extensions are typically oriented toward optimization hints, debugging syntax, possibly external redirection of content, and so forth—facets that control the operation of the processor, but in all likelihood will have no direct impact upon the logic of XQuery expressions the user introduces. Modules provide a vehicle for you to extend the language in your own way.

As the extension mechanism has become more robust, the idea of a query as a standalone document has shifted to the concept of a query as being made up of a single main document (called the main module) along with zero or more imported extensions, called library modules. Figure 4.1 illustrates the relationship between the main module and the library modules.

Figure 4.1. The main module is constructed of both internally defined functions and externally referenced library modules.

Both main modules and library modules include a prolog, a block of code that defines the functions, schema, variables, and other imported resources to be used for generating the final query. Prologs are essentially resource or function libraries, akin to dynamic linked libraries (DLLs) in Windows. The query body contains the actual query that calls the functions and uses the schema defined within the prolog, and such a query body is only contained within the main module. (If query code is found outside of a function in a library module, the XQuery engine should raise an error.)

Borrowing Library Modules

A library module is defined using the module keyword. For instance, if library 1 has a namespace of http://www.metaphoricalweb.com/modules/module1, the module itself is as follows:

module http://www.metaphoricalweb.com/modules/module1
    define function precedes($a as node(), $b as node()) as boolean
    {
         $a << $b
         and
         empty($a//node() intersect $b)
    }
    define function follows($a as node(), $b as node()) as boolean
    {
         $a >> $b
         and
         empty($a//node() intersect $b)
    }

The main module then invokes this library (and calls it mod1):

import module namespace mod1 = "http://www.metaphoricalweb.com/modules/module1"

The functions defined within this library are referenced using the mod1 prefix:

let $employees := input()//employee
let $manager :=$employees[@manager='yes'][1]
for $e in $employees return
     if (mod1:precedes($e,$manager)) then $e else $manager

The syntax of a module definition precludes it from being included in the main module definition. The modules may be internally defined, so that you can reference them without having to specify a location. However, if the library module was an external file, you would also need to provide the location of the module, using the at keyword:

[View full width]

import module namespace mod1 = "http://www.metaphoricalweb.com/modules/module1" at "http:/
  /www.metaphoricalweb.com/modules/repository/xquery/module1.xquery"

Here, the library file containing the module is located at http://www.metaphoricalweb.com/modules/repository/xquery/module1.xquery on the Web.

In the prolog of module 2, module 3 is imported. The calling of that import is what you might expect:

[View full width]

module "http://www.metaphoricalweb.com/modules/module2"
import module namespace mod3 = "http://www.metaphoricalweb.com/modules/module3" at "http:/
  /www.metaphoricalweb.com/modules/repository/xquery/module3.xquery"

However, any functions or variables defined within module 3 are only accessible in module 2. The main module doesn't know anything about the third library module, and cannot access any functions within it. For instance, if module 2 defined functions A and B, and module 3 defined functions C and D, the main module would be able to call mod2:A() but not mod3:C().

This makes it possible to create an interesting model of inheritance. For instance, suppose that you had two modules modA and modB, defined as follows:

module "uri:modA"
       import module namespace modB = "uri:modB"
       define function fnA(){
              return "This is function A"
       }

       define function fnB(){
              return modB:fnC();
       }

module "uri:modB"
       define function fnC() {
               return "This is function C"
       }

       define function fnD(){
               return "This is function D"
       }

If the main module only imports modA,

import module namespace modA = "uri:modA"

it will only be able to call fnA and fnB:

(: valid :)
return modA:fnA()
(: returns "This is function A" :)

(: valid :)
return modA:fnB()
(: returns "This is function C" :)

(: invalid :)
return modB:fnC()

(: invalid :)
return modB:fnD()

However, the result of fnB is the routine called in modB:fnC(). In essence, module B is a private class that module A can reach, but the main module can't. On the other hand, if you defined an import path for module B as well in the main module, all of the functions would have been available to the main module.

Calling Imported Variables

Functions are not the only things that are imported from a library. It is possible for a library module to contain variables that can then be used from within the calling environment. As noted in Chapter 3, an XQuery variable is more like a constant in other languages—once defined, it cannot be redefined later. As such, XQuery variables can be used to store both commonly used constant values and frequently referenced sequences. For instance, the following module defines math constants:

module "http://www.metaphoricalweb.com/modules/math"
       define variable $pi as xs:double {3.141592653589793}
       define variable $e as xs:double {2.7182818281828}

In the main module, these can then be referenced using the appropriate namespace:

import module namespace math=" http://www.metaphoricalweb.com/modules/math"

function definition circle_area($r as xs:double) as xs:double {
       return $math:pi * $r * $r
       }

Here, the variable reference for p is inserted at $math:pi. In general, the format for referencing a variable defined in a library module is to use the dollar sign ($) followed by the qualified name (the namespace referenced name) of the variable.

The variable can also hold the result of an operation:

module "http://www.metaphoricalweb.com/modules/math"
       define variable $pi as xs:double {3.141592653589793}
       define variable $pi_invert as xs:double {1 div $pi}

In this case, $math:pi_invert will hold the value of 1/p.

Variables can be defined without having values assigned to them initially. When they appear in a let statement, the value assigned will then be the value that the variable holds:

define variable $x as xs:integer
let $x := 125

In this case, once the variable is defined, it will raise an error if it is assigned a value of a type that differs from what it was defined for. For instance, given the above,

let $x := 125.53

will raise an error, because the user is attempting to assign a floating point number to a declared integer. This will hold even for imported modules.

If two library modules have a function or variable with the same name, this "namespace" collision generally isn't a problem, and it illustrates the reason namespaces are used in the first place. If module A and module B both define a function foo(), the namespace designators will ensure that modA:foo() and modB:foo() are treated as different functions.

Importing XML Schema

A type-intensive XQuery document can quickly become a mess of xs:datatype elements, casts, and so forth. One way of ameliorating this is to import an XML schema that defines data types for individual elements. This is made possible in XQuery with the import schema command.

Suppose that you have a simple XML Schema (employee.xsd) describing an employee, as follows:

<!-- employee.xsd -->
<xsd:schema xmlns:xsd='http://www.w3.org/2001/XMLSchema'>

 <xsd:element name='name' type='xsd:string'>
 </xsd:element>

 <xsd:element name='title' type='xsd:string'>
 </xsd:element>

 <xsd:element name='salary' type='xsd:float'>
 </xsd:element>

 <xsd:element name='dateStarted' type='xsd:dateTime'>
 </xsd:element>

 <xsd:element name='employee' type=" Employee">
 </xsd:element>

  <xsd:complexType name=" Employee">
   <xsd:sequence>
    <xsd:element ref='name'/>
    <xsd:element ref='title'/>
    <xsd:element ref='salary'/>
    <xsd:element ref='dateStarted'/>
   </xsd:sequence>
  </xsd:complexType>

</xsd:schema>

This would be valid for an employee document, such as

<!-- employee1.xml -->
<?xml version=" 1.0" ?>
<employee xmlns=" http://www.metaphoricalweb.com/ns/employee">
    <name>Aleria Delamare</name>
       <title>Senior Developer</title>
       <salary>66000</salary>
       <dateStarted>2000-10-12T00:00:00</dateStarted>
</employee>

The import schema command can load in this XML Schema and ensure that elements are the proper type when being invoked in functions or other expressions. The syntax for import schema mirrors that of import module:

[View full width]

import schema namespace emp=" http://www.metaphoricalweb.com/ns/employee"  at " http://www
  .metaphoricalweb.com/schemas/business/employee"

This statement associates the namespace http://www.metaphoricalweb.com/ns/employee with the prefix emp:, then loads the XML Schema from its real location at http://www.metaphoricalweb.com/schemas/business/employee. Once the XML Schema is loaded, an instance of that XML Schema (one that has the given namespace) will be aware of its own instance data.

For example, consider the following XQuery, which retrieves an employee record and determines the period of time that the employee has been with the company:

[View full width]

import schema namespace empl=" http://www.metaphoricalweb.com/ns/employee"  at http://www
  .metaphoricalweb.com/schemas/business/employee

define function showTime($employee as empl:Employee) as xs:string
{
     let $interval := cast subtract-dates(current-date(),$employee/dateStarted) as xs
  :yearMonthDuration
     return concat($employee/name,' has been with the company for ',
   get-years-from-yearMonthDuration($interval),' years and ',
   get-months-from-yearMonthDuration($interval),' months.')
}

for $emp in doc("employee1.xml")//empl:employee
return showTime($emp)

This should return the string expression

Aleria Delamare has been with the company for 2 years and 9 months.

depending upon what the current date is. There are several things of note here. The showTime function takes as an argument an object of type emp:Employee:

define function showTime($employee as empl:Employee) as xs:string

This is made possible by the imported namespace definition—the XQuery parser now knows that this is a valid data type. If some other argument type is passed in, the processor will complain by raising an error.

Within the function, this auto-mapping of formats simplifies the date functions, because there is no reason to explicitly cast the result of $employee/dateStarted to an xsd:dateTime type. Similarly, because $employee/name has already been mapped to an xsd:string., there is no need to explicitly declare this in the concat() function as a string.

You can also define the XML Schema associated with the default elements used in a query by employing the default element import:

[View full width]

import schema default element namespace = "http://www.metaphoricalweb.com/ns/employee"
   at "http://www.metaphoricalweb.com/schemas/business/employee"

In this case, you wouldn't need to use a namespace prefix to identify elements.

With this notation, you can import XML Schema to make it easier to both validate incoming code and simplify the internal casting code. Doing so becomes even more important when you're dealing with functions that are defined outside the scope of XQuery, because they have much more strongly typed characteristics than is typical within an XQuery application.


Invoking External Functions

Suppose you want to retrieve a list of filenames from a given directory on your system or generate a reasonably random number for a simulation. It's possible to do both by feeding the appropriate XML into the query, but this approach might not be practical. The other alternative is to define external functions within XQuery. With an external function, you declare the signature of the function (the name, parameters and result type) but the specific implementation of this function is defined via some external mechanism. The exact mechanism for identifying these functions is still to be determined at the time of this writing, but it will likely use the same type of modular approach currently used for defining library modules.

For instance, Java defines a random number library in the class java.util.Random. It is probable that this library can be referenced as follows (if the XPath external function definition language is any indication):

import module namespace rand=" java:java.util.Random"
let $rndObj = rand:new()
let $rndNum = rand:nextDouble($rndObj)
return $rndNum

In this case, the namespace points to the java.util.random, with the java protocol prefix indicating that this should be interpreted as Java. The new() function provides a handle to the default constructor for that class, which creates a class instance of the Random class and assigns it to the variable $rndObj. Note that it is valid for an XQuery variable to hold a reference to an object in this manner.

Because XQuery (or more properly, XPath) does not have a formal object-oriented programming (OOP) syntax, methods to properties are invoked by identifying the method, then passing the class instance as the first argument into this method. For instance, in

let $rndNum = rand:nextDouble($rndObj)

the signature for the Java nextDouble method is

double nextDouble()

Within XPath, the instance is passed into the method to indicate that it should use that instance as its containing class instance. Having initialized $rndObj earlier, the XQuery syntax will then use it as its base object. In Java notation, this would be the same as

rndObj.nextDouble()

The instance will be freed up for garbage collection after the query completes, or it can be explicitly deallocated using the dispose() method:

rand:dispose($rndObj)

Working with the raw classes in this manner might not be completely effective. You might want to create wrapper classes that incorporate the initial invocations, exposing a custom library more amenable to being used by the XQuery. For instance, it might be useful to hide the creation of the random objector with a function that defines a sequence of random numbers. The library to do this would probably look like this:

(: random.xquery :)
module "http://www.metaphoricalweb.com/modules/random"
    import module namespace rand=" java:java.util.Random"
    define variable $rndObj as xs:double {rand:new()}

    define function random() as xs:double
    {
         return rand:nextDouble($rndObj)
    }

    define function random-sequence-double($numItems as xs:integer) as xs:double*
    {
       return
       for $index = (1 to $numItems)
            return
            rand:nextDouble($rndObj)
    }
    define function random-sequence-integer($numItems as xs:integer,$minValue as xs:integer,
       $maxValue as xs:integer) as xs:integer*
    {
       return
       for $index = (1 to $numItems)
            return
            rand:nextInteger($rndObj,$maxValue) + $minValue
    }

To create a sum of dice rolls (with each die having a value from one to six), you'd invoke this function as follows:

import namespace dice=" http://www.metaphoricalweb.com/modules/random"  at " random.xquery"

let $diceValue = sum(dice:random-sequence-integer(3,6))

This would generate a sequence of three random integers that vary from 1 to 6 (such as (1,5,6)), which the sum() function then sums to retrieve the result (12).

Function declarations built around external functions can significantly improve the performance of the query, but this can come at some cost to both portability and XQuery initialization. It's therefore worth looking at the larger issue of importing functions and XML Schema into XQuery from an organizational standpoint.


Using Library Modules Effectively

One of the biggest problems I have with stored procedures for a given database is that there is no standard way of breaking sets of sprocs into related units—you have to rely on an ad hoc naming convention to do so, and there are no real advantages in terms of inheritance or encapsulation that make such efforts worthwhile. The namespace mechanism, on the other hand, provides an interesting and potentially powerful way to create classes of XQuery modules, making it much easier to maintain the code and build a framework of APIs that can be reused again and again. This also has some advantages when dealing with compiled XQueries, because one of the other advantages of stored procedures is that they can be precompiled to be more efficient when invoked.

Organizing Libraries

The best way of dealing with an XQuery module is to envision it as either a static or dynamic class. A static class consists of a set of methods or variables that don't require an initial instance to work on. For instance, consider a library module that enhanced the date functions with named content for specifying months:

[View full width]

module "urn:months"
     declare variable $monthNames as xs:string* {("January","February","March","April"
  ,"May","June","July","August","September","October","November","December")}
     define function get-month-name($currentDate as xsd:dateTime) as xsd:string
     {
           let $monthNum := get-month-from-dateTime($currentDate)
           return $monthNames[$numthNum]
     }
     define function get-month-number($monthName as xsd:string) as xsd:integer
     {
           return index-of($monthNames,$monthName)
     }

The functions within the module will work on any dateTime object that's passed, and do not require the creation of any instance object:

[View full width]

import module namespace months=" urn:months"
import schema namespace empl=" http://www.metaphoricalweb.com/ns/employee"  at
http://www.metaphoricalweb.com/schemas/business/employee

define function showMonth($employee as empl:Employee) as xs:string
{
     return concat($employee/name,' started with the company in ', months:get-month-name
  ($employee/dateStarted),' of ', get-year-from-dateTime($employee/dateStarted))
}

for $emp in doc("employee1.xml")//empl:employee
return showTime($emp)

=> Aleria Delamare started in October of 2000.

In other cases, it's necessary to create routines that temporarily maintain state. In this case, the module should define a constructor that can then be passed as a token to maintain state. For instance, consider a situation where you want to build a string a bit at a time, rather than using the somewhat awkward concatenation feature. Because XQuery is a declarative language, you have to resort to an external language such as Java to build an alternative. Java's StringBuffer class fits this bill:

module "urn:stringBuffer"
     import module namespace sb=" java:java.lang.StringBuffer"
     define function new($str as xsd:string)
     {
          return sb:new("")
     }

     define function write($sbObj,$str as xsd:string) as xsd:string
     {
        sb:append($sbObj,$str)
        return ""
     }

     define function writeLine($sbObj,$str as xsd:string) as xsd:string
     {
        sb:append($sbObj,$str)
        sb:append($sbObj,"\n")
        return ""
     }

     define function toString($sbObj) as xsd:string
     {
       return sb:toString()
     }

With this function, when you invoke a new object, it creates a new string buffer, making it possible to have several string buffers maintained at any given time:

     import module namespace stringBuf=" urn:stringBuffer"

     let $buf1 := stringBuf:new("")
     let $buf2 := stringBuf:new("")
     return
     stringBuf:writeLine($buf1,"This is a test.")
     stringBuf:writeLine($buf2,"T'was brillig and the slithy toves")
     stringBuf:writeLine($buf1,"This is only a test.")
     stringBuf:writeLine($buf2,"Did gyre and gimbol in the wabes")
     stringBuf:writeLine($buf1,"If this had been an actual emergency")
     stringBuf:writeLine($buf2,"All mimsy were the borogroves")
     stringBuf:writeLine($buf1,"We'd be in serious trouble about now.")
     stringBuf:writeLine($buf2,"and the momraths outgrabe.")
     stringBuf:toString($buf1)
     stringBuf:toString($buf2)

=>
This is a test.
This is only a test.
If this had been an actual emergency
We'd be in serious trouble about now.
T'was billig and the slithy toves
Did Gyre and Gimbol in the wabes.
All mimsy were the borogroves
And the momraths outgrabe

The writeLine() functions generate output, but the output is an empty string. It's the only way to pass information into a stateful function within the stateless environment of XQuery, because in XPath, XQuery, and XSLT, an XPath function must produce something. There is no void type, because there should be no side effects.

SHOP TALK: SIDE EFFECTS

XQuery is designed to be side-effect free. However, in some cases introducing side effects can simplify your coding. A side effect is a change that is introduced in a variable through some internal machination. Weigh your use of side effects (such as maintaining a string buffer) carefully against making your code less portable and harder to maintain.

An example from a more traditional language (Visual Basic in this case) illustrates the dangers of such side effects. Global variables are discouraged in programming, but beginning programmers in particular like to use them because it's easier to do that than to build a complex set of objects.

I inherited some remarkably bad code from another developer on a particular commercial software product. One day, while working on a particularly odious problem, I kept getting apparently random results out of a routine that seemingly had no connection with the input. After more than two days of digging through source code and isolating problems, I discovered the problem was in a compiled module the other programmer had written, where he kept changing the value of two global variables as a side effect in a routine, which was then affecting a third global variable in another compiled routine. Needless to say, I ended up rewriting the entire module to root out these and other inconsistencies in the code.

While you can create code libraries that call on other code libraries that call on still other code libraries, keep in mind that every import adds considerably to the overhead of compiling the code, because these sub-libraries must be compiled and linked to the main applications.

Compiling XQuery

As mentioned earlier, stored procedures are extraordinarily fast, because they are compiled code. Although currently no implementations of XQuery support compilation, this has more to do with the immaturity of the market than it does with where XQuery is going. It is very likely that as implementations improve, compiled XQuery code will be the norm, rather than the exception.

In general, such an implementation will probably provide three alternatives:

  • Standalone compiled query applications or components

  • Compiled libraries coupled with a just-in-time compiled main module

  • A mix of compiled and noncompiled code, using pragmas in the code

The first case would work by following the import tree from within an XQuery application. Library modules that have no imports would be compiled first and loaded into the compilation for reference by modules that call these library modules. These in turn would be loaded into the main module. Finally, the main module would be compiled into a single standalone .EXE component (or JAR file). The data could then be passed to the JAR file via a command-line sequence. If the object is a JavaBean or COM component/.NET assembly, it would be invoked via methods on the component.

This approach works well when the XQuery object is part of a framework application, such as a JSP page or an ASP.NET process. In essence, this provides a vehicle for encapsulating business query functionality within a component with a well defined API.

The second approach makes each imported file its own specific compiled library, with importing modules referencing the imported library. This functionality works better when building a framework of query functions for use in a wide range of ad hoc queries, and in essence can be thought of as building extensions to the XQuery language. In this case, the compiled XQuery libraries would likely be invoked directly within the at attribute of the import module command.

The final approach is a mix of the first two. One advantage that XQuery has over SQL is that libraries can be imported from anywhere—both local systems and locations on the Web (as well as CAB or ZIP files, with the proper protocol). This also means that library modules can be cached, or even compiled and then cached. Pragmas can be used on module statements to determine whether a given module should be cached, and for how long—once a module is cached, it makes sense to compile it for maximum efficiency and performance.

This is all pointing to the likelihood that most XQuery compilers will be just-in-time compilers, using caching and linking to persist common queries, and periodically purging older queries when they cease being relevant, such that they will be recompiled again the next time they are referenced.


In Brief

  • Pragmas are instructions to the particular XQuery implementation. The scope of the pragma depends on the pragma.

  • The extension element is like a pragma, but the processor must be able to understand it.

  • Queries are made up of a single main module and zero or more imported extensions (library modules).

  • The prolog defines the functions, schema, variables, and other imported resources to be used to generate the final query.

  • You can import an XML Schema that defines data types for individual elements, using the import schema command.

  • An external function lets you declare the signature of a function. The specific implementation of the function is defined via an external mechanism.


Chapter 5. XQuery Applications: Presentations, Graphics, and Syndication Feeds

by Kurt Cagle

IN THIS CHAPTER

  • Viewing XML

  • Creating an XQuery Pipeline

  • HTML: A Regional Weather Report

  • Making the Table More Sophisticated

  • Parameterized XQuery

  • SVG: Graphing from XQuery

  • Syndication with RSS

  • Building an XQuery Syndication Engine

  • In Brief


Viewing XML

Raw XML is not a pretty sight. Even though XML's proponents have claimed that XML is human-readable, it would take someone with exceptional skill, patience, and visualization talent to look at a "naked" XML document and see it in their mind's eye. More typically, XML needs to be translated in some way to another format that can be rendered in a visualization tool, such as a Web browser, a text-to-speech processor, a PDF file, or any of the dozens of other presentation devices.

XQuery can be thought of as a tool to do two types of actions. For instance, one stream of data (say, from a file) might contain a template containing pointers to name and address blocks in a form letter, whereas a second stream of data (from one database) contains the name information, and a third stream of data (an XML-enabled LDAP directory) contains the address information, linked to the names. The role of XQuery is to bind these three streams of information into cohesive structures.

The second action is a specialized version of the first. XQuery is used in this context to create some form of presentation output—CSS-enabled XML, XHTML, SVG, XSL-FO, Xforms, and so on. Because most of these are likely also XML formats, the creation of such pages can be seen as another form of integration. However, in a different sense, XQuery does much the same thing as an ASP, JSP, or PHP page: It produces data-driven output to be served to a client.

RESOURCES FOR RELATED TECHNOLOGIES

This chapter covers a number of XML and related technologies, and unfortunately space does not permit a detailed elucidation of each one. If you want to know more about these technologies, the following links are good places to start:

  • SVG— http://www.w3.org/Graphics/SVG/Overview.htm8

  • XSLT and XSL-FO— http://www.w3.org/Style/XSL/

  • RSS 2.0— http://backend.userland.com/rss


Creating an XQuery Pipeline

First, we need to take a brief diversion into architecture to explore exactly how XQuery fits into the production process. To do this, let's consider the nature of XML itself. The initial view of XML saw XML as simply a subset of the SGML language for creating languages that best describe complex documents.

By early 2000, a second viewpoint about XML was beginning to emerge: XML became a means for representing varying types of data formats.

The third way of using XML has arisen with both the advent of SOAP-based Web services and the rise of Representative State Transfer (REST) architecture. In this view, XML can be thought of as streams of information that follow a unified grammar and structure, with the specific meaning of the streams contained within the schema of the XML. This view has the greatest relevance to XQuery.

XPath 2.0 defines a function called input() that contains the input stream of information—specifically, an input sequence of nodes. Currently this sequence is synchronous—it is provided all at once, although there is no reason to assume that it might not in the future be designed to be asynchronous as well. Regardless, the XQuery processes this information all at once as a filter in a pipeline. The result is almost certainly piped out to some other process: a Web service, an XSLT transformation, or the response object for a server to send to a Web client.

At this stage, relatively few XQuery processors are on the market, but most will work along the lines of the following pseudo-code, which passes formatting information to an output stream object:

public void InvokeQuery(InputStream is, URL queryURL, OutputStream os){

    XqueryProcessor xqProc = new XqueryProcessor();
    xqProc.setInputStream(is);
    xqProc.setOutputStream(os);
    xqProc.setQuery(queryURL);
    xqProc.query();
    return void;
    }

This routine creates a new XQuery processor and assigns the input stream and the location of the XQuery resource (a file, or perhaps an element in a database). The routine also sets the output stream for the processor—where the results will end up—and then calls the query() method to perform the actual query operation. In addition to performing the transformation, the query() method sends the result into the output stream.

This routine would likely be called as follows, pulling information from the Web service at http://www.mywebservices.com/basicService/data, using the query myQuery1.xquery and sending the results to the server's response object, and from there to the client:

URL sourceData = new URL("http://www.mywebservices.com/basicService/data");
URL querySource = new URL("myQuery1.xquery");
InvokeQuery(sourceData.getInputStream(), querySource, response);

The assumption throughout this chapter is that the input() function returns the XML resource described at the beginning of each example, which is passed through the is parameter.

You can think of input and output streams as female and male sockets, respectively—most output streams cannot be used (directly) as input streams, and vice versa. However, it is possible to indirectly transform one to the other via a PipedInputStream or PipedOutputStream object. These are analogous to female/female or male/male plug converters that are used for doing things like connecting two computers directly together.

You can use such filters to chain together multiple queries. For instance, to perform one query to generate an XML source from the input and another query to generate a report from this new data, you can use a structure something like this:

URL sourceData = new URL("http://www.mywebservices.com/basicService/data");
URL querySource = new URL("myQuery1.xquery");
URL reportQuerySource = new URL("Report.xquery");
OutputStream os = new OutputStream();
PipedInStream pIn= new PipedInputStream();
InvokeQuery(sourceData.getInputStream(), querySource, os);
InvokeQuery(pIn.setInputStream(os), reportQuerySource, response);

The os OutputStream is assigned the result of the first query, and is then used as the argument for the pipedInputStream, where it is essentially copied and made usable for reading rather than writing. Multiple chains can be set up this way, with each query performing a different task, a process known (not surprisingly) as pipelining. Note again that this is just pseudo-code—the actual code depends on the specific implementation, but will probably be similar.


HTML: A Regional Weather Report

Weather data provides an interesting set of information that works well either generated from SOAP messages or retrieved from tables, and it is always changing in a (more or less) rational way, making it useful for data mining to see whether specific trends are developing.

Suppose you are asked to prepare a Web page that reads the current weather conditions for a number of West Coast cities, retrieved from a collating Web service that reads reports from each city, and generates a generalized West Coast weather report in HTML. The file WeatherReport1.xml contains the data shown in Listing 5.1.

Listing 5.1 WeatherReport1.xml Contains a Sample of Weather Data on a Day in December
<reportRoot date="2002-12-15T05:40:52.706Z">
   <report>
      <city>Seattle</city>
      <state>Washington</state>
      <minTempF>23</minTempF>
      <maxTempF>41</maxTempF>
      <currentTempF>24</currentTempF>
      <condition>Clear</condition>
      <windDirection>SE</windDirection>
      <windVelocityMPH>3</windVelocityMPH>
   </report>
   <report>
      <city>Olympia</city>
      <state>Washington</state>
      <minTempF>30</minTempF>
      <maxTempF>31</maxTempF>
      <currentTempF>31</currentTempF>
      <condition>Cloudy</condition>
      <windDirection>W</windDirection>
      <windVelocityMPH>3</windVelocityMPH>
   </report>
   <report>
      <city>Tacoma</city>
      <state>Washington</state>
      <minTempF>23</minTempF>
      <maxTempF>38</maxTempF>
      <currentTempF>30</currentTempF>
      <condition>Clear</condition>
      <windDirection>N</windDirection>
      <windVelocityMPH>3</windVelocityMPH>
   </report>
   <report>
      <city>Yakima</city>
      <state>Washington</state>
      <minTempF>22</minTempF>
      <maxTempF>24</maxTempF>
      <currentTempF>23</currentTempF>
      <condition>Snow</condition>
      <windDirection>S</windDirection>
      <windVelocityMPH>12</windVelocityMPH>
   </report>
   <!-- Many more cities -->
   <report>
      <city>San Jose</city>
      <state>California</state>
      <minTempF>39</minTempF>
      <maxTempF>46</maxTempF>
      <currentTempF>43</currentTempF>
      <condition>Clear</condition>
      <windDirection>S</windDirection>
      <windVelocityMPH>0</windVelocityMPH>
   </report>
   <report>
      <city>San Diego</city>
      <state>California</state>
      <minTempF>39</minTempF>
      <maxTempF>52</maxTempF>
      <currentTempF>40</currentTempF>
      <condition>Cloudy</condition>
      <windDirection>W</windDirection>
      <windVelocityMPH>13</windVelocityMPH>
   </report>
</reportRoot>

The first task is to create a simple HTML document from this input. Although the temptation is to start with an HTML template, it is worth asking prior to building it whether any global variables need to be made available to the internal document. Realistically, you probably need to get the root <reports> node of the document, because you will need it to both display a table and to retrieve the date when the initial XML was generated, something along the lines of

let $reportRoot:=input()//reports
return ....

With this information, you have a consistent handle to the root of the document that can be accessed throughout the query. The next step is to begin the HTML template that will be used to get header information and titles, including the current date and time:

<html>
    <head>
        <title>Weather Report 1</title>
    </head>
    <body>
    <h1>Weather Report 1</h1>
    <h2>For {string($reportRoot/@date)}</h2>

The output of any {escaped} XQuery expressions will be nodes, not text, so you need to do an explicit conversion to a string format for element and attribute contents to end up in the output.

The next move is to create table output for the weather information. To build this, you first need to create the table headers, which turn out to be in the same order as the resources in the XML file. So, you can simplify your output of the table rows quite a bit by iterating over each column in presentation order with the $report/* XPath expression:

    <table border="1">
        <tr>
            <th>City</th>
            <th>State</th>
            <th>Min &#176;F</th>
            <th>Max &#176;F</th>
            <th>Current &#176;F</th>
            <th>Condition</th>
            <th>Wind Direction</th>
            <th>Wind Speed</th>
        </tr>
    {for $report in $reportRoot/report
    return
        <tr>
            {for $col in $report/*
            return    <td>{$col}</td>}
        </tr>
    }
    </table>
    </body>
</html>

The full query can then be written as GetWeatherReports1.xquery in Listing 5.2, which is illustrated in Figure 5.1.

Listing 5.2 The GetWeatherReports1.xquery Returns a Functional, Albeit Basic, Weather Report Query
let $reportRoot:=input()//reportRoot
return
<html>
    <head>
        <title>Weather Report 1</title>
    </head>
    <body>
    <h1>Weather Report 1</h1>
    <h2>For {string($reportRoot/@date)}</h2>
    <table border="1">
        <tr>
            <th>City</th>
            <th>State</th>
            <th>Min &#176;F</th>
            <th>Max &#176;F</th>
            <th>Current &#176;F</th>
            <th>Condition</th>
            <th>Wind Direction</th>
            <th>Wind Speed</th>
        </tr>
    {for $report in $reportRoot/report
    return
        <tr>
            {for $col in $report/*
            return    <td>{$col}</td>}
        </tr>
    }
    </table>
    </body>
</html>
Figure 5.1. The output from GetWeatherReports1.xquery is functional, but spare. Adding graphics is the next order of business.

    [ Team LiB ]


Making the Table More Sophisticated

This weather report gets the information across, but is probably only of real interest to a weatherman. Most people do not use ISO dateTime formats, for instance, so the date given in the output can appear bewildering to the average person checking the weather report. Additionally, the table should be formatted in an easier-to-read manner. Finally, and perhaps most importantly from a design standpoint, it would be nice to be able to change the sequence of the column headers, omit or add headers, and change the order of the rows without having to do a lot of recoding.

Because this information requires more sophisticated scripting, it is better to define a set of functions and temporary node collections that can then be referenced in the template. This approach keeps the intent of the template obvious and allows for the reuse of certain code.

For instance, consider the fact that each city's weather data gives the name of the state in full, rather than the shorter abbreviation. If you want to display the shorter form, you can include a number of if statements; but a better way is to create a lookup table that gives each state name and its corresponding value (in this case, the inverse of what you would expect—the shorthand abbreviation for the state name).

Creating such a lookup is not complex. Essentially, you create a collection of nodes (perhaps from an external source, although that technique isn't covered here) that has a structure such as this:

<elt name="key" value="data"/>

The name of the element is not important, but the name and value attributes are. For instance, in the weather example, the state mapping elements are as follows:

let $stateSymbols := (<state name="California" value="CA"/>,
                <state name="Washington" value="WA"/>,
                <state name="Oregon" value="OR"/>)

A lookup function can then rely on the uniformity of the attribute names to find the instance of the state element with the given name and return the associated value:

let $symbol := string($sequence[string(@name)=string($name)]/@value)

This can (and should) be wrapped into a function—lookup()—for ease of use:

define function lookup($sequence,$name){
    return string($sequence[string(@name)=string($name)]/@value)
    }

You then invoke the function to retrieve the symbol for Washington state as follows:

let $stateSymbol := lookup($stateSymbols,"Washington")

Note that this kind of facility is useful in other circumstances as well. For instance, in the weather table, it would be aesthetically pleasing to display the current weather condition as a graphic rather than as a word. Rather than rely on the source XML, you can create a map between the possible weather condition expressions and URLs pointing to the respective graphics:

let $weatherSymbols := (
    <condition name="Clear" value="sunny.gif"/>,
    <condition name="Partly Cloudy" value="partlyCloudy.gif"/>,
    <condition name="Cloudy" value="cloudy.gif"/>,
    <condition name="Rain" value="rain.gif"/>,
    <condition name="Snow" value="snow.gif"/>
    )

This map can then be used within an HTML image tag (<img>) to point to the appropriate image:

<img src='{lookup($weatherSymbols,$report/condition)}'/>

A similar technique lets you retrieve a month name based on the month number ("01" to "12"). The month number in this case is the result of extracting the sixth and seventh characters from a standard date string ("YYYY-MM-DD"). This number can be converted into a real number and used to extract the item from the list, but the lookup is probably as efficient. The lookup can be encapsulated into a function getMonthName():

define function getMonthName($monthNumStr){
let $monthNames := (
    <month name="01" value="January"/>,
    <month name="02" value="February"/>,
    <month name="03" value="March"/>,
    <month name="04" value="April"/>,
    <month name="05" value="May"/>,
    <month name="06" value="June"/>,
    <month name="07" value="July"/>,
    <month name="08" value="August"/>,
    <month name="09" value="September"/>,
    <month name="10" value="October"/>,
    <month name="11" value="November"/>,
    <month name="12" value="December"/>
    )
    return lookup($monthNames,$monthNumStr)
    }

You can compile all this information into an XQuery that presents a more readable view of the data, as shown in Listing 5.3, GetWeatherReports2.xquery, and Figure 5.2.

Listing 5.3 getWeatherReports2.xquery
[View full width]
define function tempF($temp){
    let $tempStr := concat(string($temp),'&#176;F')
    return $tempStr
    }

define function lookup($sequence,$name){
    string($sequence[string(@name)=string($name)]/@value)
    }

define function getMonthName($monthNumStr){
let $monthNames := (
    <month name="01" value="January"/>,
    <month name="02" value="February"/>,
    <month name="03" value="March"/>,
    <month name="04" value="April"/>,
    <month name="05" value="May"/>,
    <month name="06" value="June"/>,
    <month name="07" value="July"/>,
    <month name="08" value="August"/>,
    <month name="09" value="September"/>,
    <month name="10" value="October"/>,
    <month name="11" value="November"/>,
    <month name="12" value="December"/>
    )
    return lookup($monthNames,$monthNumStr)
    }

define function getDateTime($dateTime){
    let $dateStr :=string($dateTime[1])
    let $year := substring($dateStr,1,4)
    let $month := substring($dateStr,6,2)
    let $day :=substring($dateStr,9,2)
    let $hourMin :=substring($dateStr,12,5)
    let $hour24 :=if (substring($hourMin,1,1)='0') then substring($hourMin,2,1) else
   substring($hourMin,1,2)
    let $hour12 :=if ($hour24 eq '0') then '12' else if (xs:int($hour24) gt 12) then
   string(xs:int($hour24) - 12) else $hour24
    let $ampm := if (xs:int($hour24) lt 12) then 'AM' else 'PM'
    let $min :=substring($hourMin,4,2)
    let $monthName := getMonthName($month)
    return concat($monthName,' ',$day,', ',$year,' at ',string($hour12), ':',$min,$ampm)
    }

let $reportRoot:=input()//reportRoot
let $stateSymbols := (
    <state name="California" value="CA"/>,
    <state name="Washington" value="WA"/>,
    <state name="Oregon" value="OR"/>
    )

let $weatherSymbols := (
    <condition name="Clear" value="sunny.gif"/>,
    <condition name="Partly Cloudy" value="partlyCloudy.gif"/>,
    <condition name="Cloudy" value="cloudy.gif"/>,
    <condition name="Rain" value="rain.gif"/>,
    <condition name="Snow" value="snow.gif"/>
    )

return

<html>
    <head>
        <title>Weather Report 1</title>
    </head>
    <body>
    <base href="http://www.myWeatherSite.com"/>
    <h1>Weather Report 1</h1>
    <h2>For {getDateTime($reportRoot/@date)}</h2>
    <table border="1">
        <tr>
            <th>City</th>
            <th>Temp Range</th>
            <th>Current &#176;F</th>
            <th>Condition</th>
        </tr>
    {for $report in $reportRoot/report
    return
        <tr>
            <td>{string($report/city)}, {lookup($stateSymbols,$report/state)}</td>
            <td align="center">{tempF($report/minTempF)} - {tempF($report/maxTempF)}</td>
            <td align="center">{tempF($report/currentTempF)}</td>
            <td align="center"><img src='{lookup($weatherSymbols,$report/condition)}'/></td>
        </tr>
    }
    </table>
    </body>
</html>
Figure 5.2. This report provides more detailed information, and includes a graphic to better illustrate the weather.

It's worth examining the template portion of the XQuery to realize that it includes little in the way of functional code. Much of it has been removed to previously defined functions, making it easier to see the query author's intent and promoting extensive reuse of code.


Parameterized XQuery

With SQL, it's possible to write a few lines of SQL in a command-line interface that retrieve a basic view of data. However, in general, most reports and more complex SQL code are functional in nature, utilizing a stored procedure with dynamic parameters to make a query against the relevant data source to provide a richer form of output.

The same should hold for XQuery, although as it stands now, this is not quite the case. There is no formal mechanism for passing parametric information into an XQuery, as there is for XSLT. However, this is not as much of a limitation as it might seem, because you can use the document() function to preload an XML document that contains parametric information.

For instance, suppose you want to specify the sort keys of the weather listing, to order it by temperature or condition. You could create a specific query for ordering by one key or another, but doing so provides a lot of redundant code. However, if you set the parameters in an external XML document (here called parameters.xml), you can minimize the number of queries you need to write. The parameters file can be both simple and generic:

<parameters>
    <parameter name="sortKey" value="currentTempF"/>
    <parameter name="sortOrder" value="descending"/>
</parameters>

Note that the <parameter> element uses the same name/value attribute convention used earlier for the lookup function. This is not accidental, and illustrates how you can use conventions in your XML documents to simplify the amount of coding you need to do. In Listing 5.4, the weather table output in getWeatherReports2.xquery is made sortable, utilizing the document() function to retrieve the secondary XML.

Listing 5.4 getWeatherReports3.xquery
[View full width]
define function tempF($temp){
    let $tempStr := concat(string($temp),'&#176;F')
    return $tempStr
    }

define function lookup($sequence,$name){
    string($sequence[string(@name)=string($name)]/@value)
    }

define function getMonthName($monthNumStr){
let $monthNames := (
    <month name="01" value="January"/>,
    <month name="02" value="February"/>,
    <month name="03" value="March"/>,
    <month name="04" value="April"/>,
    <month name="05" value="May"/>,
    <month name="06" value="June"/>,
    <month name="07" value="July"/>,
    <month name="08" value="August"/>,
    <month name="09" value="September"/>,
    <month name="10" value="October"/>,
    <month name="11" value="November"/>,
    <month name="12" value="December"/>
    )
    return lookup($monthNames,$monthNumStr)
    }

define function getDateTime($dateTime){
    let $dateStr :=string($dateTime[1])
    let $year := substring($dateStr,1,4)
    let $month := substring($dateStr,6,2)
    let $day :=substring($dateStr,9,2)
    let $hourMin :=substring($dateStr,12,5)
    let $hour24 :=if (substring($hourMin,1,1)='0') then substring($hourMin,2,1) else
   substring($hourMin,1,2)
    let $hour12 :=if ($hour24 eq '0') then '12' else if (xs:int($hour24) gt 12) then
   string(xs:int($hour24) - 12) else $hour24
    let $ampm := if (xs:int($hour24) lt 12) then 'AM' else 'PM'
    let $min :=substring($hourMin,4,2)
    let $monthName := getMonthName($month)
    return concat($monthName,' ',$day,', ',$year,' at ',string($hour12) ,':',$min,$ampm)
    }

define function getParam($paramName){
    let $params := document('parameters.xml')//parameter
    return lookup($params,$paramName)
    }

let $reportRoot:=input()//reportRoot
let $stateSymbols := (
    <state name="California" value="CA"/>,
    <state name="Washington" value="WA"/>,
    <state name="Oregon" value="OR"/>
    )

let $weatherSymbols := (
    <condition name="Clear" value="sunny.gif"/>,
    <condition name="Partly Cloudy" value="partlyCloudy.gif" />,
    <condition name="Cloudy" value="cloudy.gif"/>,
    <condition name="Rain" value="rain.gif"/>,
    <condition name="Snow" value="snow.gif"/>
    )

let $sortName := getParam('sortKey')
let $sortOrder := getParam('sortOrder')

return

<html>
    <head>
        <title>Weather Report 1</title>
    </head>
    <body>
    <base href="http://www.myWeatherSite.com"/>
    <h1>Weather Report 1</h1>
    <h2>For {getDateTime($reportRoot/@date)}</h2>
    <table border="1">
        <tr>
            <th>City</th>
            <th>Temp Range</th>
            <th>Current &#176;F</th>
            <th>Condition</th>
        </tr>
    {for $report in $reportRoot/report
        order by {$sortName} {$sortOrder}
    return
        <tr>
            <td>{string($report/city)}, {lookup($stateSymbols,$report/state)}</td>
            <td align="center" >{tempF($report/minTempF)} - {tempF($report/maxTempF)}</td>
            <td align="center" >{tempF($report/currentTempF)}</td>
            <td align="center" ><img src='{lookup($weatherSymbols,$report/condition)}'/></td>
        </tr>
    }
    </table>
    </body>
</html>

The generation of such Web pages, using such techniques as building functions to simplify code generation or data retrieval, is straightforward. However, you can also take advantage of other XML-based languages such as SVG and XSL-FO to produce graphs and reports to other media.


SVG: Graphing from XQuery

Scalable Vector Graphics (SVG) provides a way to convert raw data into an XML-based visual representation. HTML served as a way of providing a uniform means to describe a document, one that could be rendered in more or less the same way, regardless of the platform the HTML renderer is on. This ubiquitous, open standard made much of the initial growth of the Internet possible.

However, documents are not, except in the most general sense, graphical entities. Typically, most Web designers utilize bitmap images (also known as raster images) to render the graphical portions of Web pages.

There are a number of problems with using such bitmaps in this context, however. Bitmaps cannot (generally) be changed dynamically. Even simple bitmaps can take up a significant amount of space, which in turn increases the bandwidth requirements and latency in displaying Web pages. Finally, bitmaps limit the ability to create localized interfaces.

One solution is to represent information about shapes using vectors to represent lines, points, curves, and shapes constructed from them. Such vectors can significantly reduce the amount of information needed to render an image and increase what you can do with the graphics, because you can decompose shapes into named elements that can be manipulated separately.

However, one final attribute of vector graphics makes them an ideal medium for graphics on the Web (and Web services). Vectors are scalable—if you change the dimensions of a vector, you do not lose information. Consequently, vector systems are more portable than bitmaps, are scalable to any dimension (within the limitations of system rounding error), and can often be rendered piecemeal.

Adoption of SVG has started off slowly, but is gaining considerable momentum. The Adobe SVG Viewer (ASV), available at http://www.adobe.com/svg, is one of the best viewers available. Additionally, a number of open-source SVG initiatives are under way, including the Apache project's Batik (http://xml.apache.org/batik/index.html), which is designed to produce a Java-based SVG renderer for client- or server-side work, and SVG Sharp (http://sourceforge.net/projects/svgdomcsharp/), which is attempting to create an SVG viewer for .NET using C#. The examples given here assume the use of the Adobe viewer.

Graphing Weather Trends

Weather is the epitome of chaos. This makes weather data particularly intriguing when viewed graphically, because trends that might be masked by absolute numbers are often obvious once graphed. This section looks at ways to create an SVG chart driven by XQuery for showing the minima, maxima, and averages of temperature changes over the course of a year.

The data has been simplified from previous sections, and rather than pulling information from live records, the actual weather data has been simulated. Therefore, in all likelihood, the shape of data in this exercise might end up looking different from a real weather pattern. Part of the initial data is as follows, reducing the reports to three important data points (a day's maximum [high], minimum [low], and average [mid] temperature):

<weatherReports>
  <report day="0" low="42" mid="43" high="51" />
  <report day="1" low="48" mid="50" high="56" />
  <report day="2" low="50" mid="50" high="54" />
  <report day="3" low="47" mid="51" high="54" />
  <report day="4" low="47" mid="50" high="55" />
  <!-- Many more days of this -->
  <report day="360" low="24" mid="37" high="46" />
  <report day="361" low="38" mid="44" high="47" />
  <report day="362" low="42" mid="42" high="45" />
  <report day="363" low="41" mid="41" high="42" />
  <report day="364" low="34" mid="35" high="48" />
</weatherReports>

SVG is a language for describing graphics primitives. In this case, the primitives can be broken down into a background element sized appropriately for the data points, and the three distinct line graphs indicating the low, mid, and high points of the day. In this example, each graph could have been treated as a series of line segments connecting the points, but this approach would create several hundred objects within the SVG document The better approach is to treat each graph as an irregular path, using a set of instructions for drawing each path as a distinct entity.

A sample SVG graphic for drawing the three graphs is as follows:

[View full width]

<svg>
<linearGradient gradientTransform="rotate(90)" id="heatSpectrum">
<stop offset="0%" stop-color="red"/>
<stop offset="10%" stop-color="orange"/>
<stop offset="20%" stop-color="yellow"/>
<stop offset="35%" stop-color="green"/>
<stop offset="50%" stop-color="blue"/>
<stop offset="75%" stop-color="aqua"/>
<stop offset="90%" stop-color="white"/>
</linearGradient>
<rect fill="url(#heatSpectrum)" height="220" width="720" x="0" y="0"/>
<g transform="translate(14,76)">
<path d="M0,-84 L2,-96 L4,-100 L6,-94 L8,-94 L10,-80 L12,-48 L14,-76 L16,-84 L18,-82 L20
  ,-68 ..." fill="none" stroke="blue" stroke-width="1"/>
<path d="M0,-102 L2,-112 L4,-108 L6,-108 L8,-11 L10,-132 L12,-92 L14,-94 L16,-90 L18,-84
   L20,-96 ..." fill="none" stroke="red" stroke-width="1"/>
<path d="M0,-86 L2,-10 L4,-10 L6,-102 L8,-10 L10,-11 L12,-74 L14,-88 L16,-84 L18,-82 L20
  ,-70 ..." fill="none" stroke="black" stroke-width="1"/>
</g>
</svg>
</g>
</svg>

Although a full description of SVG is well beyond the scope of this book, this particular script is fairly easy to follow. The <svg> element is a generalized container for any <svg>, serving much the same purpose as the <html> tag does for HTML.

Within this, you define a gradient (using <linearGradient>) called heatSpectrum that has colors arranged by their relative heat associations (blues and whites indicate cool temperatures, yellows and reds indicate hot temperatures). This serves as a background to the graphs, which not only provides a visual indicator of what any given point in the graph represents, but also reduces the requirement of drawing a graph scale in the background.

The <rect> element creates a rectangle <rect> that uses the gradient as a fill or paint. Its size is defined to accommodate a scale of a 110-degree range for up to one year (multiplied by a factor of two). This element uses the pixel coordinate system to position the rectangle, with the upper left-hand position corresponding to the origin (0,0); one unit point in the coordinate system corresponds to the width from one pixel to the next.

The <g> element serves as a grouping container (think graphic context or group) for other elements, and is notable as a way of creating a localized coordinate system transformation. Anything within the <g> element acts as if it is in its own local coordinate system relative to the <g> object. This can consequently be used as one (of several) ways of grouping objects together for movement or other processing.

The final element to be discussed, and the one most germane to XQuery, is the Path object. In a <path>, you use a specialized language for describing the connections between points, contained within the d attribute. For instance, you could describe a 100-pixel wide square centered at the origin as:

[View full width]

<path d="M-50,-50 L50,-50 L50,50 L-50,50 L-50,-50 M0,0z" fill="none" stroke="black"
   stroke-width="1"/>

The square is thus positioned so that it is moved 50 points to the left and above the origin. The pen, which indicates whether a line is drawn between points, is raised (via the M command) in the first pair of numbers, then is lowered and drawn (via the L command). Once lines are drawn, the pen moves 50 points to the right and above the origin, 50 points to the right and below the origin, and 50 points to the left and below the origin, before returning to upper left, and from there, back to the origin. After this, the z command indicates that this path forms a closed shape (which affects a number of other properties, such as the caps or bevels on paths and how fills occur).

The fill and stroke attributes determine the colors used for painting. Thus, a stroke of black, stroke-width of 1, and fill of none draw a one-unit-wide black line with no painted color within the interior regions of the graph.

A query language like XQuery is different in scope and effect than a transformation language such as XSLT, and ideally, it might be better to use XSLT to generate the SVG code. However, you can use XQuery, although in a slightly more limited fashion, as illustrated in the file BuildSVG.query (see Listing 5.5).

Listing 5.5 BuildSVG.query
[View full width]
define function graph($reportSet,$attributeName,$color){
element path {
    attribute fill{'none'},
    attribute stroke{$color},
    attribute stroke-width{1},
    attribute d{
        for $index in (0 to count($reportSet) - 1)

        let $report:=item-at($reportSet,cast as xs:integer($index + 1))

        return
            concat(if ($index=0) then 'M' else 'L',string(2 * $index),',', string(- 2 *
   $report/@*[name(.)=$attributeName]),' ')
        }
    }
    }

let $reports:=input()//report
let $minTemp:=min($reports/@low)
let $maxTemp:=max($reports/@high)
return
<svg>
<linearGradient gradientTransform="rotate(90)" id="heatSpectrum">
    <stop offset="0%" stop-color="red"/>
    <stop offset="10%" stop-color="orange"/>
    <stop offset="20%" stop-color="yellow"/>
    <stop offset="35%" stop-color="green"/>
    <stop offset="50%" stop-color="blue"/>
    <stop offset="75%" stop-color="aqua"/>
    <stop offset="90%" stop-color="white"/>

</linearGradient>
<rect x="0" y="0" width="720" height="{2 * (cast as xs:int($maxTemp) - cast as xs:int
  ($minTemp) + 10)}" fill="url(#heatSpectrum)"/>
<g transform="translate({cast as xs:int($minTemp) - 10},{cast as xs:int ($maxTemp) + 10})">
{
graph($reports,'low','blue'),
graph($reports,'high','red'),
graph($reports,'mid','black')}
</g>
</svg>

The core action in the XQuery is performed by the graph() user function. This function takes a node sequence of the data points as the initial argument. Because each report contains all three temperature data points (low, mid, and high) as attributes, a second argument is necessary to extract which data point applies. Finally, the third argument indicates which color is to be used to draw the path.

The graph() function illustrates the fact that you can use functions to create new XML elements:

[View full width]

define function graph($reportSet,$attributeName,$color){
element path {
    attribute fill{'none'},
    attribute stroke{$color},
    attribute stroke-width{1},
    attribute d{
        for $index in (0 to count($reportSet) - 1)

        let $report:=item-at($reportSet,cast as xs:integer($index + 1))

        return
            concat(if ($index=0) then 'M' else 'L',string(2 * $index),',', string(- 2 *
   $report/@*[name(.)=$attributeName]),' ')
        }
    }
}

Here, the alterative provides element/attribute notation is used to specify the given path element, so that the output for the d attribute (which specifies the path coordinates) can be given without having to create a temporary variable. It's notable that with such a variable, you can use the normal tag approach. By assigning the result of the path generator to the variable $d, you can simplify the code as follows:

[View full width]

define function graph($reportSet,$attributeName,$color){
    let $d:=for $index in (0 to count($reportSet) – 1) {
        let $report:=item-at($reportSet,cast as xs:integer($index + 1))
        return concat(if ($index=0) then 'M' else 'L',string(2 * $index),',', string(- 2 *
   $report/@*[name(.)=$attributeName]),' ')
        }
    return
    <path fill="none" stroke="{$color}" stroke-width="1" d="{$d}"/>
}

Either approach is, of course, valid.

The value assigned to $d in the second statement iterates through the list in a slightly nonstandard way. It is important to determine whether the item in question is the first one in the list (has an $index of 0), because this command is an initial move with the pen up (which thus implies the use of the M command). Every other item is a move with the pen down (an L command, for line).

The default coordinate system for SVG is screen-oriented, with the x-axis increasing to the right and the y-axis increasing toward the bottom of the screen. However, most graphs are designed with the y-axis increasing toward the top of the screen. For this reason, the y coordinates are flipped by being multiplied by -1. However, doing this makes all the coordinates negative, and consequently makes them appear above the top of the graph (presumably off the screen). Consequently, once the paths are drawn, the whole graph is pushed down by the height of the graph, so that the graph is visible.

There is much more to SVG than can be covered here. This sophisticated multimedia language is capable of animation, filter effects, and interactivity. You can find more information about SVG at the SVG W3C site (http://www.w3.org/Graphics/SVG/Overview.htm8). In addition, the SVG Wiki (http://www.protocol7.com/svg-wiki/default.asp) provides a number of helpful tips and techniques for enhancing your SVG.


Syndication with RSS

Syndication is a term that has recently gained currency within the Web publishing world. In 1999, Dave Winer of Userland created an application called Radio Userland that allowed people to easily publish to the Web. What was most significant about this application was that when a user published her blog, the Radio Userland application also sent an XML feed that followed the Really Simple Syndication (RSS) schema to a Web service. This service combined the feeds with other feeds and then broadcast the results to subscribers. In this way, people who were interested in the blog could link to it anytime a new story was posted.

RSS jumped into prominence later that year when Netscape adopted it (and adapted it, causing a rift in the specification) to identify channels in the company's own material. Like so many fundamental technologies, other people saw RSS and what it could do and started building their own services around it. Anyone who has used the news feed in Linux KDE or Gnome has used an RSS-based service.

RSS comes in a number of different flavors; most application developers have standardized on the RSS 2.0 specification (http://backend.userland.com/rss). The specification defines one or more channels (with descriptive content), which in turn define one or more news items. For instance, the following is a sample RSS 2.0 document for my own site, The Metaphorical Web (metaphorical.rss):

[View full width]

<rss version="2.0">
    <channel>
        <title>The MetaphoricalWeb</title>
        <link>http://www.metaphoricalweb.com</link>
        <description>A collection of essays and code resources for XML developers, .NET
   users, and people interested in distributed application development. </description>
        <docs>http://backend.userland.com/rss</docs>
        <managingEditor>kurt@metaphoricalweb.com</managingEditor>
        <webMaster>webmaster@metaphoricalweb.com (Kurt Cagle)</webMaster>
        <language>en-us</language>
        <copyright>Copyright 2003, Cagle Communications</copyright>
        <ttl>60</ttl>
        <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
        <lastBuildDate>Mon, 04 Feb 2003 0:00:01 GMT</lastBuildDate>
        <cloud domain="data.metaphoricalweb.com" port="80" path="/RPC2"
   registerProcedure="metaphorical.rssPleaseNotify" protocol="xml-rpc"/>
        <generator>Cagle Communications</generator>
        <guid>http://www.metaphoricalweb.com</guid>
        <image>
            <url>http://www.metaphoricalweb.com/images/metaweb.png</url>
            <title>The MetaphoricalWeb</title>
            <link>http://www.metaphoricalweb.com</link>
        </image>
        <item>
            <title>The Metaphorical Web #10: Mortality</title>
            <description>In which Kurt has a brush with death in many forms. </description>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm#death"</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/">Death </category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Space Exploration<
  /category>
            <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        <item>
            <title>The Metaphorical Web #10: Venality</title>
            <description>A look at the BSA's most recent campaign of FUD, and why it will
   only make the Open Source movement stronger.</description>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm#BSA"</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> BSA</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> intellectual
   property</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Open Source<
  /category>
            <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        <item>
            <title>The Metaphorical Web #9: Play</title>
            <description>Play forms an integral part of invention, and by extension of
   effective programming. We are doing a disservice to ourselves and the industry to strip
   out this essential quality. </description>
            <link >http://www.metaphoricalweb.com/xml/metaphor9.htm#play</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> play</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> programming<
  /category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> innovation</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> BSA</category>
            <pubDate>Fri, 10 Dec 2002 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        <item>
            <title>The Metaphorical Web #9: Microsoft's Move to SVG?</title>
            <description>While not yet an avalanche, SVG is beginning to make its way into
   the Redmond software giant in a number of subtle ways. </description>
            <link >http://www.metaphoricalweb.com/xml/metaphor9.htm#svgMicrosoft</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> SVG</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Microsoft</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Visio</category>
            <pubDate>Fri, 10 Dec 2002 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        <item>
            <title>The Metaphorical Web #9: The DNA of XSLT</title>
            <description>In which the author explores the link between genetics and
   template based programming with XSLT.</description>
            <link >http://www.metaphoricalweb.com/xml/metaphor9.htm#dnaXSLT</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/">XSLT </category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Functional
   Programming</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Procedural
   Programming</category>
            <pubDate>Fri, 10 Dec 2002 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        </channel>
    </rss>

Although much of this code is straightforward, a few elements should be defined in more depth. The <channel> element describes a particular venue and corresponds more or less to a Web site (or a television channel). An RSS document can contain more than one channel, particularly if the feed is an aggregate of several other such feeds.

The <link> element binds the channel to a specific Web page or similar resource—an RSS agent can use this information to provide a generalized link to the channel site, but it doesn't necessarily have anything to do with the items within the channel. The <image> element works in conjunction with this—a number of RSS browsers use the <image> and <title> elements to provide a representation of the channel in the GUI, and use the <link> as a hypertext link to the Web site when the image is clicked on.

The <ttl> (time to live) element is a nod to syndication. The RSS document is not a static entity—most servers update this file periodically to indicate new material coming into the system or old material moving out of it (yesterday's news). The <ttl> element contains the number of minutes that should elapse before an agent checks back to see whether something has changed. If the RSS document has been written for a fairly low-volume site (say, material is updated weekly), this number is the number of minutes in a week (10,080, for those keeping records). On the other hand, if material is coming in quickly (your news feed really is a news feed), updating once an hour (<ttl>60</ttl>) is recommended. By setting this number, you can keep external agents from trying to access your feed more often than you update it, thus reducing server load.

The <pubDate> and <lastBuildDate> provide time references using the IETF Date and Time Specification (RFC 822). The pubDate indicates the date and time in which the RSS document was regularly published. The lastBuildDate can override this value, indicating the last time the RSS document was published. The <pubDate> element is useful to indicate when an agent should check to see when the feed is usually published, and the lastBuildDate indicates whether there's been a revision since that time, within the <ttl> boundaries. Typically the two have the same value.

The <category> element raises some interesting possibilities. A category is an implicitly obvious thing that, upon examination, turns out to be less obvious, and in some cases, downright complex. A category should basically give you a key into a taxonomy, defining a term that can be used by other agents in a consistent fashion. At its simplest level, <category> could be a single expression—for instance, <category>web development</category>. To ensure that this term is meaningful beyond a simple description, though, it should be a term from a taxonomy. A taxonomy (or ontology) provides relationships between specific terms and provides a specific lexicon of acceptable terms. The domain attribute provides a link to a reference taxonomy for the terminology in question. For instance,

<category domain="http://www.metaphoricalweb.com/taxonomy/"> death</category>

points to a document (http://www.metaphoricalweb.com/taxonomy) that gives the various terms for discussion.

If everyone who utilizes the RSS feed agrees to the same set of categorical references, this can make for a much more robust distributed application because you can filter news content by category, regardless of who produces it.

The final element of note within a channel is <cloud>. A cloud is an aggregator of multiple RSS feeds from different sources. There are literally hundreds of thousands of RSS feeds in existence; the purpose of the aggregators is to publish the RSS feeds to any other agents that subscribe to the cloud (rather than subscribing to the RSS feed directly, although that also can be done). This is usually accomplished by calling an XML-RPC or SOAP method to notify, update, or delete an RSS entry from the repositories.

The individual <item> elements provide the resources to be syndicated. They use a number of elements in common with the <channel>, but also include a few that are specialized. The <link> element points to the resource, although this resource could of course be generated via a CGI script. The assumption with the <link> is that it is in a form that can be inferred from the link extension or from the CGI interface.

In some cases, however, the item itself might contain one or more pieces of multimedia (audio or video clips, for instance) that could take a long time to download. The <enclosure> element provides a way to speed up the apparent time of this transaction. Because the user agent often can subscribe to an RSS feed independent of human intervention, it can perform the downloading of the resource in the background while you sleep. That way, when a user clicks on the article, the multimedia is already there, and starts up automatically and transparently. When the RSS no longer carries the article, these resources can be released from the cache so as not to fill up your hard drive.

Finally, the <guid> element uniquely defines the <item> on the Web. This element can be a URL or URI, but it doesn't have to be—it can just as readily be a UID or GUI, a combination of a computer network's MAC number and a timestamp, or some other unique identifier. This makes it possible to archive the resource and reference it at some future time without having to use the RSS link that's given (which might point to a CGI script that only returns the most recent article for that link).

Not all elements are required for an RSS feed to work. For instance, the following is a significantly stripped-down version of the previous feed:

<rss version="2.0">
    <channel>
        <title>The MetaphoricalWeb</title>
        <link>http://www.metaphoricalweb.com</link>
        <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
        <item>
            <title>The Metaphorical Web #10: Mortality and Venality</title>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm"</link>
        </item>
        <item>
            <title>The Metaphorical Web #9: Play</title>
            <link >http://www.metaphoricalweb.com/xml/metaphor9.htm</link>
        </item>
        </channel>
    </rss>

Although the RSS 2.0 specification is relatively new, it is rapidly becoming adopted as the de facto standard for Web-based syndication efforts, which in the long run will probably translate into it finding its way into both open- and closed-source applications. XQuery is a useful mechanism to both query and shape these RSS feeds, and might play an increasingly important role in this syndication story.


Building an XQuery Syndication Engine

Syndication schemas such as RSS serve a number of purposes. They correlate links with titles for those links. They (usually) provide a thematic classification system that can be queried (and the data itself can be queried, albeit much less efficiently). As I write this, support for the most recent version of XQuery is still lacking in most databases. To get around this limitation, I looked at one of the databases to provide a means of tying the publication of an article or resource to a specific schedule, both in terms of publishing a resource and of moving that resource to an archive when it reaches a certain age.

The challenge is to look at points where XQuery could make a difference, given that it is currently not supported natively by PostgreSQL. An obvious area comes in providing an interface that generates SQL statements to update the contents of a database. In essence, the XQuery engine will use the RSS feed sent to it to write SQL, which is then passed to PostgreSQL via a pipeline.

The XQuery file updateRSS.xquery takes as input an RSS file and produces SQL output that updates two SQL tables—channels and items:

[View full width]

define function outputNull($seq){
    let $output := if (count($seq) ne 0) then string($seq) else ''
    return $output
    }

let $rss:=input()
let $channels:=$rss//channel
for $channel in $channels
return
(concat("INSERT INTO 'channels' (title,guid,link,description,docs,editor,webmaster
  ,language,copyright,ttl,pubdate,lastbuilddate,generator)
VALUES ('",
        string($channel/title),"',
    '", outputNull($channel/guid),"',
    '", string($channel/link),"',
    '", outputNull($channel/description),"',
    '", outputNull($channel/docs),"',
    '", outputNull($channel/managingEditor),"',
    '", outputNull($channel/webMaster),"',
    '", outputNull($channel/language),"',
    '", outputNull($channel/copyright),"',
    '", outputNull($channel/ttl),"',
    '", outputNull($channel/pubDate),"',
    '", outputNull($channel/lastBuildDate),"',
    '", outputNull($channel/generator),"'
);")
,
for $item in $channel/item
return
concat("INSERT INTO 'item' (title,guid,link,description,category,pubDate,author,channelGuid)
VALUES ('",
        string($item/title),"',
    '", outputNull($item/guid),"',
    '", string($item/link),"',
    '", outputNull($item/description),"',
    '", outputNull($item/category),"',
    '", outputNull($item/pubDate),"',
    '", outputNull($item/author),"',
    '", outputNull($channel/guid),"'
);"));",
"GO")

The first XQuery function defined, outputNull, generates the text contents of the XML node as a string if the element exists in the RSS document, or returns an empty string if it does not. This ensures that empty content is still passed into the table at the specific position even if there is no value (alternatively, a NULL element could be added to the output to do the same thing).

The $rss variable holds the contents of the input stream, which should be the RSS document. This variable in turn is used to create SQL INSERT statements. The code to do this is unusual in that the output is not XML, but rather formatted text. This raises some problems, not least of which being that XQuery assumes the tokenized output will be interpreted as an XML document. Consequently, the purpose of the routine is to generate one very large string, using the concat() function. Note that a concat() function can contain other concat() functions to build substrings, a principle that is used to good effect here.

For a typical RSS report, there might be one or more channels. Each channel contains one or more items, which means the items are nested with the channels. The XQuery unrolls them, converting the result into a linear list of INSERT commands. When run against the Metaphorical Web feed (metaphorical.rss), this produces the following result:

[View full width]

INSERT INTO 'channels' (title,guid,link,description,docs,editor,webmaster,language
  ,copyright,ttl,pubdate,lastbuilddate,generator)
VALUES ('The MetaphoricalWeb',
    'http://www.metaphoricalweb.com',
    'http://www.metaphoricalweb.com',
    'A collection of essays and code resources for XML developers, .NET users, and people
   interested in distributed application development.',
    'http://backend.userland.com/rss',
    'kurt@metaphoricalweb.com',
    'webmaster@metaphoricalweb.com (Kurt Cagle)',
    'en-us',
    'Copyright 2003, Cagle Communications',
    '60',
    'Mon, 04 Feb 2003 0:00:01 GMT',
    'Mon, 04 Feb 2003 0:00:01 GMT',
    'Cagle Communications'
);
INSERT INTO 'item' (title,guid,link,description,pubDate,author,channelGuid)
VALUES ('The Metaphorical Web #10: Mortality',
    ' http://www.metaphoricalweb.com/xml/metaphor11.xml',
    'http://www.metaphoricalweb.com/xml/metaphor10.htm#death" ',
    'In which Kurt has a brush with death in many forms.',
    'Mon, 04 Feb 2003 0:00:01 GMT',
    'kurt@kurtcagle.net (Kurt Cagle)',
    'http://www.metaphoricalweb.com'
);
INSERT INTO 'item' (title,guid,link,description,pubDate,author,channelGuid)
VALUES ('The Metaphorical Web #10: Venality',
    ' http://www.metaphoricalweb.com/xml/metaphor11.xml',
    'http://www.metaphoricalweb.com/xml/metaphor10.htm#BSA" ',
    'A look at the BSA's most recent campaign of FUD, and why it will only make the Open
   Source movement stronger.',
    'Mon, 04 Feb 2003 0:00:01 GMT',
    'kurt@kurtcagle.net (Kurt Cagle)',
    'http://www.metaphoricalweb.com'
);
INSERT INTO 'item' (title,guid,link,description,pubDate,author,channelGuid)
VALUES ('The Metaphorical Web #9: Play',
    ' http://www.metaphoricalweb.com/xml/metaphor11.xml',
    'http://www.metaphoricalweb.com/xml/metaphor9.htm#play',
    'Play forms an integral part of invention, and by extension of effective programming.
   We are doing a disservice to ourselves and the industry to strip out this essential quality.',
    'Fri, 10 Dec 2002 0:00:01 GMT',
    'kurt@kurtcagle.net (Kurt Cagle)',
    'http://www.metaphoricalweb.com'
);
INSERT INTO 'item' (title,guid,link,description,pubDate,author,channelGuid)
VALUES ('The Metaphorical Web #9: Microsoft's Move to SVG?',
    ' http://www.metaphoricalweb.com/xml/metaphor11.xml',
    'http://www.metaphoricalweb.com/xml/metaphor9.htm#svgMicrosoft',
    'While not yet an avalanche, SVG is beginning to make its way into the Redmond
   software giant in a number of subtle ways.',
    'Fri, 10 Dec 2002 0:00:01 GMT',
    'kurt@kurtcagle.net (Kurt Cagle)',
    'http://www.metaphoricalweb.com'
);
INSERT INTO 'item' (title,guid,link,description,pubDate,author,channelGuid)
VALUES ('The Metaphorical Web #9: The DNA of XSLT',
    ' http://www.metaphoricalweb.com/xml/metaphor11.xml',
    'http://www.metaphoricalweb.com/xml/metaphor9.htm#dnaXSLT',
    'In which the author explores the link between genetics and template based programming
   with XSLT.',
    'Fri, 10 Dec 2002 0:00:01 GMT',
    'kurt@kurtcagle.net (Kurt Cagle)',
    'http://www.metaphoricalweb.com'
);

GO

This result is piped out to the file insertIntoDatabase.SQL. The SQL code assumes a channel SQL table that's configured as

[View full width]

(title,guid,link,description,docs,editor,webmaster,language,copyright,ttl,pubdate
  ,lastbuilddate,generator)

and an item table, given as

(title,guid,link,description,pubDate,author,category, channelGuid)

This SQL file is in turn passed into the PostgreSQL database system (assuming a database named newsfeeds) to update the relevant tables:

%bash$psql -f insertIntoDatabase.SQL newsfeeds

You can use similar techniques to perform any other type of database modification. It is likely, given the role the database creation and update code makes in turning SQL into more than just a query language, that in time XQuery will also have a database modification component. Until then, to use XQuery to handle updating the database, you must use indirect methods of creating SQL output.

Querying the RSS Feed

PostgreSQL does not have XQuery support at the time of this writing. However, the PostgreSQL development team has announced that it will be possible to do XQuery calls once the XQuery recommendation is finalized, probably along the lines of the following. (Please consult the documentation that ships with that version of PostgreSQL for the eventual syntax.)

Assuming the PostgreSQL postmaster (the database server) is running, the PostgreSQL command-line client is invoked via the psql command. You can run this command in a number of modes, depending on certain command-line flags. For instance, suppose the --xquery flag indicates that queries default to the XQuery engine. The news feed database can then be invoked as follows:

%bash$psql --xquery -X newsfeeds

This mode does two things. It indicates that the language for any queries should default to XQuery, and it indicates (through the -X flag) that the output will be XML.

Once started, the PostgreSQL client moves into a different command prompt mode:

newsfeeds=#

Individual lines of XML Query language can be used, terminated with semicolons to indicate the end of a line:

newsfeeds=# let $itemCount := count(//item) return $itemCount;
<?xml version='1.0'?>
<root>5</root>

Because the output is given as XML, it always specifies a <root> element, even if the result is a scalar. If the resultset is made up of a sequence, it will also be contained in this same fashion:

[View full width]

newsfeeds=# let $items := //item return $items;
<?xml version='1.0'?>
<root>
        <item>
            <title>The Metaphorical Web #10: Mortality</title>
            <description>In which Kurt has a brush with death in many forms. </description>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm#death"</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Space Exploration<
  /category>
            <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        <item>
            <title>The Metaphorical Web #10: Venality</title>
            <description>A look at the BSA's most recent campaign of FUD, and why it will
   only make the Open Source movement stronger.</description>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm#BSA"</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/">Open Source <
  /category>
            <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>

</root>

More complex queries could span multiple lines simply by excluding the semicolon terminator. An incomplete line is indicated by a dash in the prompt, rather than an equal sign:

newsfeeds-#

Thus, the following XQuery retrieves all items for which the category is Open Source:

[View full width]

newsfeeds=# for $item in //items
newsfeeds-# where string($item/category)='Open Source'
newsfeeds-# return $item;
        <item>
            <title>The Metaphorical Web #10: Venality</title>
            <description>A look at the BSA's most recent campaign of FUD, and why it will
   only make the Open Source movement stronger.</description>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm#BSA"</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Open Source<
  /category>
            <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml<
  /guid>
        </item>

The query is terminated by the semicolon on the return line.

Root Node Queries in XQuery

A root node query / differs a little from that of an XML document. In most traditional XML documents, there is one known container element; with a relational database, on the other hand, the database itself is the container. Note that such a database might still allow for external resources, although they would have to be specified using absolute rather than relative URLs. For instance, suppose you want to create a parameterized query that retrieves only those items in which the category is a specific value. To do this, you need to create an XML document that holds the parameters (categoryParams.xml):

<params>
    <param name='category' value='Open Source'/>
</params>

The XQuery looks like this:

newsfeeds=# let $params := document('http://localhost/pg/categoryParams.xml') //param
newsfeeds-# for $item in //items
newsfeeds-# where string($item/category)=string($params[@name='category']/@value)
newsfeeds-# return $item;

Note that the $params value in this case is as likely as not to be a sequence, as is the <category> element of the item. Consequently, this might return the element more than once, because it is possible for both the parameter set and the categories to overlap. A better test would be to use the some operator:

newsfeeds-# where some $category in $item/category satisfies
string($category)=string($params[@name='category']/@value)

This evaluates the list of items only until it finds one case where a category has a matching parametric value; then it returns that item and moves on to the next item.

Generating XML from PostgreSQL

PostgreSQL includes a number of internal flags, indicated by slash prefixes, that can change the state of the environment as part of the operation. You can use one such flag, \root, to change the underlying root node's element. For instance, to create a root <rss> node, you pass it in on its own line:

newsfeeds=# \root = 'rss'
root is now <rss>
newsfeeds=# return 'test';
<rss>test</rss>

With this, you can create an RSS document that returns all channels and their associated items that have a category passed in through the external categoryParams.xml document:

newsfeeds=# \root = 'rss'
newsfeeds=# let $params := document('http://localhost/pg/categoryParams.xml') //param
newsfeeds-# for $channel in //channel
newsfeeds-# where string($channel//category)=string($params[@name='category'] /@value)
newsfeeds-# return
newsfeeds-# <channel>
newsfeeds-# {$channel/*[not(name(.)='item']}
newsfeeds-# {for $item in $channel/item
newsfeeds-# where string($item//category)=string($params[@name='category']/@value)
newsfeeds-# return $item
newsfeeds-$ };

For the category Open Source, this code would return:

[View full width]

<?xml version='1.0'?>
<rss version="2.0">
    <channel>
        <title>The MetaphoricalWeb</title>
        <link>http://www.metaphoricalweb.com</link>
        <description>A collection of essays and code resources for XML developers, .NET
   users, and people interested in distributed application development.</description>
        <docs>http://backend.userland.com/rss</docs>
        <managingEditor>kurt@metaphoricalweb.com</managingEditor>
        <webMaster>webmaster@metaphoricalweb.com (Kurt Cagle)</webMaster>
        <language>en-us</language>
        <copyright>Copyright 2003, Cagle Communications</copyright>
        <ttl>60</ttl>
        <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
        <lastBuildDate>Mon, 04 Feb 2003 0:00:01 GMT</lastBuildDate>
        <cloud domain="data.metaphoricalweb.com" port="80" path="/RPC2"
   registerProcedure="ourFavoriteSongs.rssPleaseNotify" protocol="xml-rpc"/>
        <generator>Cagle Communications</generator>
        <guid>http://www.metaphoricalweb.com</guid>
        <image>
            <url>http://www.metaphoricalweb.com/images/metaweb.png</url>
            <title>The MetaphoricalWeb</title>
            <link>http://www.metaphoricalweb.com</link>
        </image>
        <item>
            <title>The Metaphorical Web #10: Venality</title>
            <description>A look at the BSA's most recent campaign of FUD, and why it will
   only make the Open Source movement stronger.</description>
            <link>http://www.metaphoricalweb.com/xml/metaphor10.htm#BSA"</link>
            <author>kurt@kurtcagle.net (Kurt Cagle)</author>
            <category domain="http://www.metaphoricalweb.com/taxonomy/">BSA </category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> intellectual
   property</category>
            <category domain="http://www.metaphoricalweb.com/taxonomy/"> Open Source<
  /category>
            <pubDate>Mon, 04 Feb 2003 0:00:01 GMT</pubDate>
            <guid isPermaLink="true"> http://www.metaphoricalweb.com/xml/metaphor11.xml</guid>
        </item>
        </channel>
    </rss>

The line

newsfeeds-# {$channel/*[not(name(.)='item']}

returns a sequence consisting of those properties (elements) of the channel that are not items—in essence, everything else in the <channel> element.

Although this is pure speculation, it is likely that PostgreSQL will also include the option of passing the resulting document to an XSLT transformation, probably through a flag such as

newsfeeds-# \xslt 'http://localhost/transforms/displayRSSAsHTML.xsl'

Such a transformation would occur any time output is produced, so it's likely that passing an \xslt flag with no filename would turn the process off. The psql version would be –xslt. A very simple version of this transformation might look something like this:

[View full width]

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
    <xsl:variable name="categories" select="//category/text()"/>
    <xsl:template match="/">
        <xsl:apply-templates select="rss"/>
    </xsl:template>

    <xsl:template match="rss">
        <html>
            <head>
                <title>RSS Feed – Categories = <xsl:value-of select="string-join
  ($categories,',')"/></title>
            </head>
            <body>
                <h1>Categories:<xsl:value-of select="string-join($categories,',') "/></h1>
                <ul>
                <xsl:apply-templates select="channel"/>
                </ul>
            </body>
        </html>
    </xsl:template>

    <xsl:template match="channel">
        <li><h2><a href="{link}"><xsl:value-of select="title"/></a></h2>
            <ul>
            <xsl:apply-templates select="item"/>
            </ul>
        </li>
    </xsl:template>

    <xsl:template match="item">
        <li><a href="{link}" alt="{description}"><xsl:value-of select= "title"/></li>
    </xsl:template>
</xsl:stylesheet>

The examples here have assumed that the XQuery would be done in an interactive mode, but it is more likely that the query will be contained within a file or piped process. PostgreSQL can invoke a query script with the -f flag from outside the environment, or via the \f flag from within the environment. Thus, the previous query can be contained within a file, queryByCategory.xquery, and invoked on PostgreSQL as follows (this assumes the result will be transformed at the end by an XSLT script):

{-- queryByCategory.xquery --}
\root = 'rss'
\xslt 'http://localhost/transforms/displayRSSAsHTML.xsl'
let $params := document('http://localhost/pg/categoryParams.xml')//param
for $channel in //channel
where string($channel//category)=string($params[@name='category']/@value)
return
<channel>
{$channel/*[not(name(.)='item']}
{for $item in $channel/item
where string($item//category)=string($params[@name='category']/@value)
return $item
}
\xslt
\root

The routine would then be invoked via a command line as

%bash$psql --xquery -f queryByCategory.xquery -X newsfeeds > newsfeedResult.htm

and placed into the result newsfeedResult.htm, shown in Figure 5.3.

Figure 5.3. You can use this combination of XQuery and RSS to filter news feeds in a number of ways.

    [ Team LiB ]


In Brief

  • XQuery can be used to generate text beyond simple records, such as SQL instruction. XQuery's capability to integrate with databases opens up a number of possibilities for code generation.

  • There is no formal mechanism for passing parametric information into an XQuery, but you can use the document() function to preload an XML document that contains parametric information.

  • Scalable Vector Graphics (SVG) provides a way to convert raw data into an XML-based visual representation.

  • The Really Simple Syndication (RSS) schema is gaining prominence on the Web. The specification defines one or more channels, which in turn define one or more news items. Syndication schemas such as RSS correlate links with titles for those links and provide a thematic classification system that can be queried.


Chapter 6. Working with the XPath 2.0 Data Model

by Per Bothner

IN THIS CHAPTER

  • XML Infosets

  • Node Values and Types

  • Types and Type Systems

  • Schemas and Complex Types

  • Validation Produces Type Annotations

  • Sequence Types

  • Static Typing

  • In Brief


XML Infosets

The XQuery data model is based on the XML Information Set standard (W3C Recommendation October 24, 2001, http://www.w3.org/TR/xml-infoset). It rather abstractly defines the information content of an XML document as a document item that contains nested element items, which in turn contain namespaces, attributes, characters, and other items. This is a conceptual standard: It does not define any file formats or programming interfaces, but rather it defines the interpretation of an XML file. It is intended to be useful for defining other XML-related standards, including the XQuery/XPath data model.

XML files that are different at the character level but that have the same information set, or infoset, are for most practical purposes equivalent. For example:

<a   b='upcase'  ><![CDATA[Hello!]]></a>

and

<a b=" upcase" >Hello&#33;</a>

have the same information sets.

The Canonical XML recommendation (http://www.w3.org/TR/xml-c14n) is a related standard in that it specifies a unique (canonical) way to convert an XML infoset back into an XML document. Two XML documents that are logically equivalent (have the same infoset) translate to the same canonical XML representation. The canonical XML for the previous example is

<a b="upcase">Hello!</a>

A parsed XML file results in an infoset, but synthetic infosets can also be constructed from other sources, such as a database, or created by a program that manipulates a DOM (Document Object Model). A DOM is a popular data structure API used to encode and manipulate XML data—that is, infosets.

The XQuery language also allows you to create infoset items using element constructor expressions or predefined functions.


Node Values and Types

Node values represent the parts of a XML document, or more generally, an XML infoset. Nodes are also used to represent document fragments—standalone nodes that are not part of a document (for example, those that might be generated by an element constructor expression). (See Chapter 2, "Understanding the XPath 2.0 Specification," for an introduction to nodes.)

The XQuery 1.0 and XPath 2.0 Data Model specification (http://www.w3.org/TR/query-datamodel/) goes into detail about the different kinds of nodes. It also defines a number of functions for constructing and accessing nodes. For example, the function dm:node-kind takes a Node and returns a string value that represents the node's kind: "document", "element", "attribute", "text", "namespace", "processing-instruction", or "comment". It is important to understand that these functions are only to explain the data model: You cannot call them from an XQuery program, and the dm prefix isn't bound to any namespace. They are used to define the XQuery language, but they are not actually part of the XQuery language itself. The functions that you can use in an actual XQuery program are those specified in the Functions and Operators specification (http://www.w3.org/TR/xquery-operators/). There in the Accessors section you will find fn:node-kind among other node-related functions, defined to return the same result as dm:node-kind. However, other functions defined the Data Model specification, such as dm:children, are not available.

Node Identity and Hierarchy

Nodes in XQuery are immutable, which means you cannot change any part of a node once it has been created. This makes sense, because XQuery is a pure, side-effect-free expression language. However, nodes do have identity—two nodes that were created using different expressions are distinct nodes, even if they contain the same data. You can compare the latter using the standard function fn:deep-equal. The following examples are all true:

fn:deep-equal(<a>test</a>, </a>test</a>)
<a>test</a> isnot </a>test</a>
let $x := <a><b></b></a> return $x/b is $x/*

Because nodes have identity, you can talk about making a copy of a node because you can distinguish the original from the copy. However, atomic values do not have identity: There is no way to copy the string "xyzzy" , because there is no way to distinguish the copy from the original. That is the difference between a string atomic value and a text node, which you can copy.

The children of a node are the nodes below it in the tree hierarchy, not including attribute or namespace nodes. The expression N/node() evaluates to a sequence containing the children of N in document order. For each node X that is a child of Y, the node X has a parent property that is the node Y. You can access the parent of a node X with the expression X/parent::node(). This has some surprising consequences. Because nodes are immutable, you have to specify the children of an element or document when you create it. However, those children must have their parent property set to the new node—but you can't modify them, because they are immutable. This chicken-and-egg problem is solved by creating new copies of the children, with the parent property of the new nodes set to the new parent. Any children of the children must also be copied.

Note, however, that this copying of nodes is part of the specification, but an implementation is free to optimize away the copying if it doesn't change the result. For example, consider the following expression:

<a>Some <b/>{<c>text</c>}</a>

The specification says that <b> and <c> nodes are created, and then copied when <a> is created. But because there is no way to access the old <b> and <c> nodes, an implementation is free to just reuse the old nodes without copying them, or it can create them in place at the same time it creates <a>. This is an example of the important difference between specification and (valid) implementation. The lack of side effects in XQuery gives the implementation extra flexibility in choosing how to implement things. A possible disadvantage is that it makes it hard to estimate how much work is done for an XQuery program, unless you are very familiar with your implementation. On the other hand, you usually don't need to know.

More generally, an implementation is free to represent nodes in any way compatible with the specification. An obvious choice is to use the standard Node type specified in the W3C's DOM (http://www.w3.org/DOM). However, although DOM is a flexible and convenient API, it is quite space-inefficient. As an example of an alternative representation, the Qexo implementation (http://www.gnu.org/software/qexo/) uses a single TreeList for an entire document. The TreeList contains an internal array, and node objects are identified by indexes into that array. (The Apache Xalan XSLT processor uses a similar Document Array Model representation.) In fact, an implementation in some cases might not create actual node objects.

Consider that the ultimate result of evaluating an XQuery expression is often written out to a file as a new XML document. In that case, the XQuery processor can write out the nodes on the fly directly to the output file, without creating any nodes. More generally, the XQuery processor can "write" the output to a SAX DocumentHandler or a similar event-driven interface.

String Value and Typed Value of Nodes

Sometimes it is useful to convert a node into a string value. The function fn:string does this. The string value of a text node consists of the characters in the node. The string value of an element or document node is the concatenation of the text node descendents of the node in document order. The string value of an attribute node is the attribute value.

There is also the typed value of an element, attribute, or text node, which you can extract using the fn:data function. This is the value of a node as a sequence of atomic values, asthe result of Schema validation. If an element node has a complex type, the typed value is undefined.


Types and Type Systems

The XQuery and XPath languages are typed expression (functional) languages. This means that programs are made from expressions (which may in turn contain sub-expressions), and that evaluating an expression results in a value, which has a type. (See Chapter 2 for a discussion of the concept of types.)

The XQuery and XPath languages specify both dynamic types and static types. The static type checking is optional, both for implementors and users. An XQuery implementation need not implement the static typing feature, and implementations that do implement static typing will have the option to disable it.

We will discuss static typing later, but first let's study dynamic typing, including the kinds of values that XQuery and XPath deal with. The data model is part of dynamic typing.

The Data Model: Items and Sequences

The values an XQuery program works on are sequences of items. An item is either an atomic value (for example, an integer or a string) or a node (for example, an element or an attribute). See Chapter 2 for more information about sequences.

Atomic Values and Types

The XQuery/XPath primitive types are the same as in XML Schema, which is a standard for specifying element structures of XML data and associating types with XML data.

The following is a complete list of the built-in types defined by XML Schema. We will only list them briefly; for more information, see the W3C Recommendation (May 2, 2001) of "XML Schema Part 2: Datatypes" (http://www.w3.org/TR/xmlschema-2/). This specifies for each type its value space (the abstract values that belong to the type), its lexical space (the text representation of values using printable characters), and its facets (properties of the type itself).

XML Schema defines the following built-in types:

  • A boolean is one of the two truth values true and false.

  • A string is zero or more Unicode characters. There are various sub-types of string:

    • A normalizedString is a string that does not have any whitespace characters except for space.

    • A token is a normalizedString that has no leading or trailing spaces, and does not have two or more spaces in a row.

    • A language is a token used to specify a natural (human) language.

    • An NMTOKEN is a token consisting of one or more NameChar characters, as defined in the XML standard.

    • A Name is a token used to represent XML names, such as body or html:table.

    • An NCName is a plain Name without a colon, such as body. The types ID, IDREF, and ENTITY are sub-types of NCName used for special kinds of attribute values as specified in the XML standard.

    • The types IDS, IDREFS, ENTITIES, and NMTOKENS are used for space-separated sequences of the corresponding tokens.

  • A NOTATION is used for attributes that specify the notation (encoding) of an element. However, NOTATION is not a subtype of string.

  • An anyURI represents a Uniform Resource Identifier Reference, such as http://www.w3.org/.

  • A QName represents an XML-qualified name, which is a pair consisting of a namespace name and a local part. Note that in the value space, a namespace name is an anyURI (such as http://www.w3.org/1999/xhtml), whereas the lexical representation uses namespace prefixes (as in xhtml:body). Therefore, mapping between the two requires a context that contains the needed namespace declaration.

  • A decimal is an arbitrary-precision real number in base 10. An integer is a decimal without a fractional part. The types nonPositiveInteger, negativeInteger, nonNegativeInteger, and positiveInteger are the obvious subtypes of integer. The types long, int, short, byte, unsignedLong, unsignedInt, unsignedShort, and unsignedByte are sub-types of integer that can be encoded in binary using 64, 32, 16, or 8 bits, respectively.

  • The types float and double correspond to 32-bit and 64-bit IEEE binary floating-point real numbers. The standard lexical representation uses decimal format, with an optional exponent, such as -58.45 or 1.25e-10, even though these types are not sub-types of decimal.

  • There are a number of time-related types:

    • A date is a calendar date, such as May 31, 1999 (written 1999-05-31). A time is an instant that occurs every day, like 1:20pm (written 13:20). A dateTime is a specific instant of time, like 1:20 p.m. on May 31, 1999 (written as 1999-05-31T13:20). Any of these may have an optional time zone specified.

    • A gYear is a specific year in the Gregorian calendar, while a gMonthYear is a specific year and month. A gMonthDay is a month and day that recurs every year, a gMonth is a month that recurs every year, and a gDay is a day that recurs every month.

    • A duration is a duration of time, like 2 days and 1 hour (written as P2D1H). The XQuery/XPath committee has added two sub-types of duration, which might be added to future Schema revisions: xdt:yearMonthDuration (a duration of some number of years and months) and xdt:dayTimeDuration (a duration of some number of days, hours, minutes, and seconds).

  • The types hexBinary and base64Binary are used to encode arbitrary binary data. The value of either is zero or more octets (8-bit bytes). A hexBinary uses two hexadecimal digits for each octet, so 0FB7 encodes the 16-bit integer 4023. A base64Binary uses the Base64 MIME Content-Transfer-Encoding.

The union of all primitive types is anySimpleType. The union of all types (primitive or not, including nodes and sequences) is anyType.

These standard types names are in the http://www.w3.org/2001/XMLSchema namespace. In XQuery, the prefix xs is bound initially to this namespace, so you can write just xs:string. The prefix xsd is also commonly used, but is not predefined in XQuery.

The XQuery specifications add four types: the duration types xdt:yearMonthDuration and xdt:dayTimeDuration were mentioned previously; xdt:anyAtomicType includes all the atomic values; and xdt:untypedAtomic is a type used for untyped data, such as text that has not been validated. All are subtypes of anySimpleType, and are in a namespace pre-bound to the xdt prefix.


Schemas and Complex Types

The word schema comes from the database community, and means a description of the structure, types, and relations of a database. In the XML world, a schema is a description of the syntax and meaning (types) of a class of XML documents. A schema language is a formalism for specifying the types of documents as schemas.

The earliest XML schema language is DTD (Document Type Descriptor), which appears in the original XML specification from 1997 and goes back to the SGML roots of XML. DTD is a simple language that lets you express simple structural constraints. For example, thefollowing

<!ELEMENT tr td*>

means that a <tr> element consists of zero or more <td> elements.

DTD does not have any mechanism for specifying semantic or type information, except in a very few cases. Other schema definition languages allow you to define and specify types.

XML Schema (http://www.w3.org/XML/Schema) is a 2001 specification from W3C that can be used to specify structural constraints and associate type information with XML documents. Although there are other Schema languages in use, this is the one with the most usage and visibility, partly because it is a W3C standard. The type semantics of XQuery/XPath2 are defined in terms of XML Schema.

As an example, let's use the record of a series of dice throws. Perhaps you want to verify that the dice are fair, or you want a source of random numbers, or you want to search for mystical patterns:

<?xml version="1.0"?>
<die-tests>
  <die-test>
    <who>Nathan</who>
    <when>whenever</when>
    <throws>5 2 2 2 1 3 6 6 2 6</throws>
  </die-test>
  <die-test>
    <who>Per</who>
    <when>2002-10-09T09:07</when>
    <throws>6 2 5 2 2 3 3 3 4 1</throws>
  </die-test>
</die-tests>

The Schema for this code might look like the following:

<?xml version="1.0"?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <xsd:element name="die-tests">
    <xsd:complexType>
      <xsd:sequence>
        <xsd:element name="die-test" type="die-test-type"
          minOccurs="0" maxOccurs="unbounded"/>
      </xsd:sequence>
    </xsd:complexType>
  </xsd:element>

  <xsd:complexType name="die-test-type">
    <xsd:sequence>
      <xsd:element name="who" type="xsd:string"/>
      <xsd:element name="when" type="xsd:dateTime"/>
      <xsd:element name="throws">
        <xsd:simpleType>
          <xsd:list itemType="die6-result"/>
        </xsd:simpleType>
      </xsd:element>
    </xsd:sequence>
  </xsd:complexType>

  <xsd:simpleType name="die6-result">
    <xsd:restriction base="xsd:integer">
      <xsd:minInclusive value="1"/>
      <xsd:maxInclusive value="6"/>
    </xsd:restriction>
  </xsd:simpleType>
</xsd:schema>

This is verbose, and might be a bit intimidating, but it is relatively straightforward. It contains a top-level element declaration for the root element <die-tests>, as well as type definitions for types named die-test-type and die6-result. Recall from Chapter 2 the difference between a complex type (such as die-test-type) and a simple type (such as die6-result): A simple type can only be expressed as character data, but a complex type can include sub-elements and attribute specifications. The type of an attribute can only be simple, whereas the type of an element can be either simple (if it only contains text data) or complex. All predefined types (such as xsd:integer) are simple.

The top-level element declaration for die-tests says that any element with the die-tests tag has the structure and type specified: It is a complex type consisting of a sequence of zero or more elements that have the tag die-test, and the content of each such die-test element has the type with the name die-test-type. (You can specify that a given element tag has different types in different contexts, but we'll ignore that possibility.)

The definition of the complex type die-test-type specifies that any element declared to have that type (in this case, die-test) consists of a sequence of a <name> element, a <when> element, and a <throws> element. The latter is a space-separated list of die6-result values. The definition for the simple type die6-result says that a die-result is an integer in the range 1 through 6.


Validation Produces Type Annotations

To validate an XML document against a schema means to scan the document, verifying that the document satisfies the constraints specified in the schema. The result is a post-schema validation infoset (PSVI), which is an infoset (as defined earlier) with additional type annotations. A type annotation is the QName of a type named in a schema.

An XQuery processor may optionally implement the Schema import feature. If it does, it must be able to import definitions from external schemas, and be able to validate node trees.

Each element or attribute in XQuery has a type annotation, which is its dynamic type. If an element has not been validated or otherwise been given a type annotation, it has the default type annotation xs:anyType. The corresponding default for an attribute node is the type xdt:untypedAtomic.

Atomic (non-node) values can also have type annotations. The annotation xst:untypedAtomic indicates that the type is unknown, typically raw text from a schema-less XML file. Operations that take atomic values can cast xdt:untypedAtomic to a more specific type, such as xs:double, but if the atomic value is of the wrong kind (a string where a number is required, as in the operands of +), a runtime error might be signaled.

An XQuery application can use a validate expression:

validate ( EXPR )

This takes a sequence of elements, strips off any existing type annotations, and adds type annotations as specified by the in-context schema definitions. The latter are all the schema element declarations and type definitions that are imported by schema import declarations. (You can optionally specify a SchemaContext that can be used with context-dependent schema types.)

A schema import declaration appears in the query prolog of an XQuery program. For example:

import schema "http://www.w3.org/1999/xhtml" at "xhtml.xsd";

This tells the XQuery processor to look at the location specified (in this case by the relative URL xhtml.xsd) and add any schema components in the specified namespace (http://www.w3.org/1999/xhtml) to the set of visible schema components. These now become available for validate expressions.

Note that schema validation and type annotation are conceptually dynamic (runtime) type operations. A type annotation is a QName that is associated with a value, not associated with a static (compile-time) expression. Next we will look at static type-checking.


Sequence Types

The XQuery language provides operations to check whether a value belongs to a type, as well as mechanisms to declare that a variable or parameter has a specific type. In an XQuery program (static) types are instances of SequenceType. We won't go into detail about SequenceType, but here are some examples:

  • text()— Matches any text node.

  • element()— Matches any element node.

  • element(xhtml:td,*)— Matches any element node whose tag has the local part td and has the same namespace URI that xhtml is bound to. (It does not have to have xhtml as the actual namespace prefix.)

  • element(*, die6-result)— Matches any element of any tag that has a type annotation of die6-result.

  • element(xhtml:title)?— Matches an optional element (zero or one items) that match the tag xhtml:title, and whose type annotation matches that declared for xhtml:title in an imported schema definition.

  • node()*— Matches a sequence of zero or more nodes.

  • item()+— Matches any non-empty sequence.

  • attribute(@ID,*)— Matches any attribute node whose name is ID (in the empty namespace).

  • xs:integer— Matches any integer type or any type derived from it, such as xs:nonNegativeInteger.

These types can be used for XQuery's type-checking and conversion operators. The following is a very brief summary; see the specification or other chapters for details and examples:

  • expr instance of type— Returns true if the value of expr matches (is an instance of) type.

  • cast as type (expr)— Converts the value of expr to a given type, using certain standard conversions.

  • treat as type (expr)— Treats the expr as having static type type. At runtime, a dynamic error is signaled if the value of expr is not an instance of type.

  • typeswitch (expr) case type1 return expr1 ... default return exprd— Selects the first case whose type matches the value of the expr, and evaluates the corresponding expression.


Static Typing

An XQuery implementation can optionally implement the static typing feature. This means that the implementation is required to detect static type errors at analysis (compile) time. At the time of this writing, the specification has a number of unresolved issues, and I don't know of any implementation that implements static typing. (However, some of the precursor languages that inspired XQuery do implement static typing.) For these reasons, plus the fact that the specification of static typing is big and formal, I won't go beyond mentioning a few of the concepts.

The static type system defined in the XQuery formal semantics (http://www.w3.org/TR/query-semantics/) goes far beyond what you can express as a SequenceType. It includes most of the type specification concepts of XML Schema. The formal semantics defines extra declarations define type, define element, and define attribute. These are not in the XQuery source language (you can't write them directly), but are a formalism used in the formal semantics to express types imported from schemas. The idea is that an XQuery program is translated to core XQuery, which is simpler and more regular (but less convenient) than the actual XQuery program. Part of this translation is that schema import declarations are translated into define type, define element, and define attribute declarations. These internal declarations, as well as the whole concept of core XQuery, are purely part of the formal specification of XQuery: There is no requirement that any implementation implement the translation to core XQuery, only that it act as if it does.

Static type checking is done at the level of core XQuery at analysis (compile) time. There are a whole slew of rules that say things like "If the type of expr1 is xs:boolean, the type of expr2 is type2, and the type of expr3 is type3, then the type of if (expr1) then expr2 else expr3 is (type2|type3)." Here (type2|type3) is a type expression in the formal semantics, which you cannot write directly in the actual XQuery language.


In Brief

  • The XQuery data model defines the information content of an XML document.

  • XML files that have the same infoset are essentially equivalent.

  • XQuery programs work on sequences of items, which can be nodes or atomic values.

  • Node values represent the parts of a XML infoset and document fragments.

  • Nodes and atomic values are immutable. Nodes have identity, but atomic values do not.

  • The XQuery/XPath type system, including the primitive types, is based on XML Schema.

  • Validating an XML document against a schema means to scan the document, verifying that it satisfies the constraints specified in the schema. This adds type annotations, which are the names of types in the schema.

  • An XQuery program can check whether a value belongs to a type and declare that a variable or parameter has a specific type.

  • Static typing involves detecting static type errors at analysis (compile) time. This is an optional feature.


Chapter 7. Understanding XQuery Semantics

by Vaidyanathan Nagarajan

IN THIS CHAPTER

  • XQuery Semantics Introduction

  • Processing Model for XQuery

  • Differences Between XQuery and XQuery Core

  • XQuery Data Model

  • Schemas and Types

  • Functions

  • Notation

  • Expression of Semantics in XQuery

  • Type Level

  • Function Semantics

  • In Brief


XQuery Semantics Introduction

XQuery is a functional language (the language is built around functions). Each query is equivalent to an expression. In layman's terms, you can view XQuery as a SQL equivalent used for querying XML documents. From a traditional programming language course perspective, it is better to have a grasp of the syntax and semantics of the language. Semantics is the study of meaning in a language; knowing the semantics can help you master the nuances of any programming language.

XQuery is a strongly typed language. XML schema types can be imported that describe the documents that will be processed. Static type analysis is also supported by XQuery, through which an XQuery expression type can be inferred based on the data type of the inputs. This helps with early error detection, and is the basis for certain forms of optimization.

Currently, the XQuery semantics are still a W3C Working Draft (version 1.0). This chapter is based on the draft released as of November 15, 2002. The draft version of the XQuery semantics is available at the following URL:

http://www.w3.org/TR/query-semantics/


Processing Model for XQuery

Figure 7.1 depicts the processing model for XQuery. Any XQuery processor can implement its own algorithm for XQuery processing, but the results should be the same as if it were produced by the model described in this section.

Figure 7.1. XQuery processor phases.

The diagram in Figure 7.1 will be familiar to compiler and assembly gurus. Basic XQuery processor implementations can be represented by four phases: the first three phases are compile-time and the last one is runtime. A brief description of each of the four phases follows:

  • Parse— A syntax check is performed. If there are no errors using the defined grammar, an internal representation is built. The grammar syntax is defined by the document at http://www.w3.org/TR/xquery/.

  • Normalization— Generally in database applications, normalization makes the table structured and reduces data redundancy. The same is true when we speak of normalization in XQuery. Applying normalization to XQuery maps each query to an equivalent in the subset XQuery Core (which is defined at http://www.w3.org/TR/query-semantics/#sec_core). Literals are normalized before expressions containing literals are normalized (a bottom-up approach). In future processing models, the query prolog will be processed before normalization starts.

  • Static semantics— This step usually involves checking whether each query is type-safe. A static error condition arises if the query is not type-safe. For example, comparing a number against a String will cause a static error. The output of this step is a syntactical tree where each sub-expression is associated with its static type and with an output type for the result of the query. If a document does not contain any schema definition, a standard type is associated so as to include any well-defined XML content.

  • Dynamic semantics— This is the runtime phase of XQuery; the output result is calculated in this step. Evaluation is a bottom-up approach, starting with literals and proceeding to expressions.

The first three phases are the compile-time for XQuery, and are usually associated with syntax and type error checking. If the compilation succeeds, the resulting form is the compiled form of the query. During compile-time, only certain classes of errors can be detected, such as the int versus string comparison. Other classes of errors are detected during runtime (just as a Java "Array Out of Bounds" error is detected only in runtime and not in compile-time).

The phases defined so far deal with the internals of a query processing engine. There are other phases that these engines need to support. These other phases are summarized in Figure 7.2.

Figure 7.2. Other phases of the XQuery processor.

Inference rules serve the purpose of describing how the semantics of expressions are derived from sub-expressions (remember that it's a bottom-up approach). These inference rules are used during the last three internal XQuery Processor phases: normalization, static semantics, and dynamic semantics.


Differences Between XQuery and XQuery Core

XQuery Core is a functional subset language of XQuery. XQuery Core supports unordered sequences and recursive navigation. XQuery Core follows the four Cs principle:

  • Correctness— XQuery Core is a statically-typed language with rigid type checking. This rigidity helps to keep formal specification of static and dynamic semantics at a manageable size.

  • Composition— Composition of arbitrary operators, both semantically and syntactically, is possible.

  • Closure— Usually the input as well as the output of query expressions are fragments, meaning part of the whole XML document that is being queried. They have an associated tree structure.

  • Completeness— XQuery Core supports the equivalent of selection, projection, and operators. User-defined recursive functions are also supported.

XQuery-to-XQuery Core Mapping: An Example

Let's look at how XQuery can be mapped to XQuery Core, using my favorite Honda car example. Consider two XML documents. You will perform an XQuery join on them and show the equivalent complex expression in XQuery Core. Note that we won't go step-by-step through how this conversion takes place; such a discussion is beyond the bounds of this chapter.

First, the XML document defines car statistics with a DTD definition (note that rather than using an external DTD, the DTD is defined internally):

<?xml version="1.0" encoding=" UTF-8"?>
<!-- Inner DTD definition for XML data -->
<!DOCTYPE automobile [
<!ELEMENT automobile (car*)>
<!ELEMENT car (carMake*,model*,manufacturer*,tyreBrand*,price*)>
<!ATTLIST car year CDATA #REQUIRED>
<!ELEMENT carMake (#PCDATA)>
<!ELEMENT model (#PCDATA)>
<!ELEMENT manufacturer (#PCDATA)>
<!ELEMENT tyreBrand (#PCDATA)>
<!ELEMENT price (#PCDATA)>
]>

<!-- start of XML data -->
<automobile>
    <car year="1998">
        <carMake>Honda Civic</carMake>
        <model>Sedan4Door DX</model>
        <manufacturer>Honda</manufacturer>
        <tyreBrand>Radial ATX</tyreBrand>
        <price>$20,000</price>
    </car>
    <car year="2002">
        <carMake>Honda Accord</carMake>
        <model>Sedan 4Door Special Edition</model>
        <manufacturer>Honda</manufacturer>
        <tyreBrand>Radial ATX II</tyreBrand>
        <price>$25,000</price>
    </car>
</automobile>

The following XML document defines the reviews for each car defined in the previous XML document (again, the DTD is defined internally):

<?xml version="1.0" encoding="UTF-8"?>
<!-- DTD declaration for the XML data -->
<!DOCTYPE reviews [
<!ELEMENT reviews (car*)>
<!ELEMENT car (carMake*,review*)>
<!ELEMENT carMake (#PCDATA)>
<!ELEMENT review (#PCDATA)>
]>
<!-- XML data -->
<reviews>
    <car>
        <carMake>Honda Civic</carMake>
        <review>Affordable good car driven by Kutty!</review>
    </car>
    <car>
            <carMake>Honda Accord</carMake>
            <review>Luxury car driven by dilbert fan!</review>
    </car>
</reviews>

Evaluating the context of XPath expressions requires the entire document to be accessible to perform document operations and XPath axes traversal. XQuery path expressions support something similar to XPath's attribute, child, and descendant axes. The nodes returned in XQuery path expression are ordered so that the original document need not be referenced subsequently. XQuery's FLWR expressions generate new sequences: The mnemonic stands for for, let, where, return. The general syntax of an FLWR expression is as follows:

for variable in expression

let variable := expression

where expression

return expression

The following XQuery, which joins the two XML documents and produces the output, uses FLWR expressions. In database terminology, this code joins two tables and retrieves columns from both tables. Here we instead manipulate XML documents:

for $a in $automobile0/car, $r in $rev/car
where $a/carMake = $r/carMake
return
    <car>
        {$a/carMake, $a/manufacturer, $r/review}
    </car>

for $a in $automobile0/car, $r in $rev/car
where $a/carMake = $r/carMake
return
    <car>
        {$a/carMake, $a/manufacturer, $r/review}
    </car>

This XQuery will produce the following XML:

<car>
    <carMake>Honda Civic</carMake>
    <manufacturer>Honda</manufacturer>
    <review>Affordable good car driven by Kutty!</review>
</car>
<car>
    <carMake>Honda Accord</carMake>
    <manufacturer>Honda</manufacturer>
    <review>Luxury car driven by dilbert fan!</review>
</car>

The translated simple XQuery expression is as follows. The complex expression represents the XQuery Core expression:

for $b1 in $automobile0 return
    for $b2 in nodes ($b1) return
        typeswitch ($b2) as $b3
            case element car {anytype}
                return $b3
            default return ()

Expression Level

Both XQuery and XQuery Core are good at selecting and recombining parts of documents. However, XQuery Core's ranked information retrieval is not good.

User-defined functions can be used for full-text predicates. A flat set is a 1:1 XML representation of a relational table into a nested XML document. Group-by queries that partition flat sets can be expressed by nested queries; the XML document in those cases needs to be passed multiple times. So, early versions of XQuery Core had an explicit group-by operator.

Another challenging problem with XQuery Core is ranked information retrieval. You cannot create a generic expression for queries combining structural and related content.

Type Level

The XQuery Core type system has transferred much of the typing found in functional programming languages into regular tree expressions' underlying DTDs and schemas in XML. XQuery Core needs to overcome a couple of challenging issues: one arises from XML Schema's rich subtyping mechanism, and the other challenge comes from polymorphic parameters and recursive functions. I won't go into detail about the challenges here.


XQuery Data Model

The XQuery data model is made of the following basic components:

  • Atomic value

  • Node

  • Item

  • Sequence

Atomic Value

The atomic value is a value for one of the atomic types. An atomic type is a simple type that is not derived from a list or union.

Node

A node can be any one of the following types: document, element, attribute, namespace, comment, processing instruction, or text. As you know, XML documents are tree representations, and each root has sub-trees made of nodes and values associated with the nodes.

Examples are given here for some of the important nodes:

  • Document node:

    // Document node D0 definition
base-uri(D0) = (http://www.indiaclubhouse.com/booklist/booklist.xml)
string-value(D0) = xf:string("\n India Club House Book Sellers\n")
children(D0) = E0

  • Element node:

    // Element node E0 definition
name(E0) = xf:Qname-from-uri(http://www.indiaclubhouse.com/BookListSchema, "book")
name(E0) = xf:Qname-from-uri("http://www.indiaclubhouse.com/BookListSchema, "book")
base-uri(E0) = xf:anyURI("http://www.indiaclubhouse.com/booklist/booklist.xml")
string-value(E0) = xf:string("\n India Club House Book Sellers\n")
typed-valued(E0) = ()
parent(E0) = D0
children(E0) = (T0) //just defined a text node
attributes(E0)=A0
namespaces(E0)=N0
declaration(E0)=SC0
type(E0)=SC1
unique-ID(E0)=()

  • Attribute node:

    //Attribute Node A0 definition
name(A0) = xf:Qname("bookname")
string-value(A0) = xf:string("The Hotel Riviera")
typed-value(A0) = xf:string("Hotel Riviera")
parent(A0) = E0
declaration(A0) = ()
type(A0) = ()

  • Namespace node:

    //Namespace Node N0 definition
name(N0) = xf:Qname("xml")
string-value(N0) = xf:string(http://www.w3.org/XML/1998/namespace)

  • Text node:

    //Text Node T0 definition
string-value(T0) = xf:string("\n ")
parent(T0) = E0
// Text node T1

Sequences

A sequence is an ordered collection of simple-typed values, or an ordered collection of nodes and simple-typed values. Sequences cannot have sequences defined within them. A sequence containing one item is called a singleton sequence (for example, (a)). A sequence containing zero items is called an empty sequence (written ()).

Element and attribute nodes, as well as atomic values, are annotated with type names. This process is referred to as type annotation and occurs during validation.

Next, we will discuss two more related features: node identity and order.

Node Identity

According to the XQuery 1.0 data model, nodes have an identity. It comes from the unique location of any node within an XML document. Two nodes can be compared using either node equality (the is operator) or value equality (the = operator).

The following example shows the difference between these equalities:

Let $a := <book><author>Nathan</author></book>,
    $b := <author>Nathan</author>,
    $c := $a/author
return
{-- example of node equality --}
($a/author == $c) {-- value is true as they are both the same node}
{-- example of value equality --}
($a/author = $b) {-- value is true as they have same value}
{-- example of node equality with false value --}
($a/author is $b) {-- value is false because they do not refer to the same node}

In this example, strict equality maps to the eq concept in Lisp and value equality maps to the equal concept in Lisp.

Order

Two kinds of ordering can be identified in XQuery: sequence and document order. Sequence order refers to ordering of items within a given sequence. Document order refers to ordering among nodes in an XML document:

[View full width]

let $node1:=<book>
               <bookname name="Fuzzy Logic"/>
               <bookname name=" The Dog is Not a Toy"/>
           </book>
    $node2:= ($node1/bookname[@name='The Dog is Not a Toy'], $node1/bookname[@name='Fuzzy
   Logic'])


Schemas and Types

The XQuery data model does not establish a complete type system. As an example, it is possible to identify an AttributeNode, but the data model does not establish what kind of attribute it is or its contents. The XQuery type system is based on XML Schema, a primer for which can be found at http://www.w3.org/TR/xmlschema-0/. XML Schema is primarily composed of regular expressions, expression trees, and regular tree grammars. This section describes, in detail, the static type systems and their conceptual overview.

A type system relates values, types, and expressions. The type system of XQuery has the following ingredients:

  • Universe of possible types

  • Type notation

  • Instances and domain types

  • Subtyping

  • Rules relating expressions to types

  • Rules relating values to types

XQuery has its own built-in simple data types. Type constructors are available for constructing sequences, node types (such as attributes), and elements. Formally, a type is defined to be an infinite set of instance values that belong to the type. For example, xs:double consists of the set of all double values, and the element address {xs:string} contains all elements named address whose contents are of type single strings.

Type Annotations and Anonymous Types

A type annotation indicates the type name for each element and attribute node in an XML document. This annotation results from the validation process or is a default assignment.

For example, consider the following document fragment:

<book> The Book Lego Mindstorms is written by <author>Jin Sato</author> </book>

The following code snippet is the associated schema:

<xs:element name="book" type="BookType"/>
<xs:complexType name="BookType" mixed="true">
       <xs:sequence>
              <xs:element name="author" type="xs:String">
       </xs:sequence>
</xs:complexType>

Before validation, the two elements book and author are annonated by xs:anyType. After validation, element book is annotated with type name FactType and author is associated with type name xs:String.

If the schema type does not have an explicit name, it is called an anonymous type.

Subtyping Rules

A function is mathematically considered an operator that takes input parameters and produces output. Subtyping places an important role on function semantics. Subtyping in XQuery is related to the XML Schema rule of derivation by restriction. The following subtyping rules are defined in the XQuery semantics:

  • One type is a subtype of another if every value of the first type also belongs to the second type. For example: (xs:double)* and (xs:integer|xs:double)*.

  • Two types are equivalent if every value of one type is also a value of the other type, and vice versa. For example: (xs:integer|xs:double)* and (xs:double|xs:integer)*.

  • Passing an argument of the declared type or a subtype of the declared type can make a call to a function. This is referred to as type substitutability. Subtypes are substitutable for their supertypes, but not vice versa.

  • A derived type is always a subtype of its base type, but not the opposite. For example: (xs:double)+xs:double and (xs:integer|xs:double)*.


Functions

In this section, we will look at some of the important functions and operators, giving examples of some and listing others that are available as part of XQuery's semantics. As you know, the XQuery data model consists of simple-typed values, nodes, sequences, and so on. The functions described in this section operate on the components of the data model. Table 7.1 lists some of the XQuery functions and their equivalents in algebraic theory.

Table 7.1. Standard Node Sequence Functions in XQuery

FUNCTION

DESCRIPTION

op:intersect

Set intersection

op:union

Set union

op:except

Set difference

xf:sequence-node-equal

True if the arguments have the same set of nodes

xf:sequence-deep-equal

True if the arguments have the same values

Examples

Let's consider the following set of nodes and perform some operations on them to see the results. We'll define sets of tickerSymbols from the NASDAQ stock exchange as the nodes and apply the operators from Table 7.1 to them:

$tickerSymbols1 = ($HIG, $UBS),$tickerSymbols2 = ($HIG,$UBS),$tickerSymbols3 = ($UBS,$SUNW)

The following operator builds a sequence of nodes from the sequences parameter1 and parameter2, eliminating duplicates:

op:intersect (node*$parameter1, node*$parameter2)

The result is as follows:

node*

For this example

op:intersect($tickerSymbols1,$tickerSymbols2)

the result is

($HIG,$UBS)

Here is another example:

op:intersect($tickerSymbols2,$tickerSymbols3)

with this result:

($UBS)

To check the equality of two nodes, you use the operator xf:node-equal().

The following operator builds a sequence of nodes occurring from parameter1 and from parameter2, eliminating duplicates:

op:union (node*$parameter1, node*$parameter2)

The result is as follows:

node*

For example,

op:union($tickerSymbols1,$tickerSymbols2)

gives the following result:

($HIG,$UBS)

The next operator builds a sequence of nodes that occur in parameter1 but not in parameter2, eliminating duplicates:

op:except(node*$parameter1, node*$parameter2)

The result is

node*

Here is an example:

op:except($tickerSymbols1,$tickerSymbols2)

The result is

()

For this example,

op:except($tickerSymbols2,$tickerSymbols3)

the result is

($HIG)

Other Functions

As this chapter is a primer on XQuery semantics, here's a list of other functions that are part of the XQuery Semantics specification:

op:concatenate
op:boolean-and
op:boolean-or
op:union-value
op:except-value
op:to
op:numeric-equal
op:numeric-add
op:get-end-datetime
op:numeric-subtract
op:get-duration
op:get-start-datetime
op:numeric-multiply
op:numeric-divide
op:numeric-mod
op:boolean-equal
op:datetime-equal
op:duration-equal
op:hex-binary-equal
op:base64-binary-equal
op:numeric-greater-than
op:datetime-greater-than
op:duration-greater-than
op:numeric-less-than
op:datetime-less-than
op:duration-less-than
op:node-equal
op:node-before
op:node-after
op:numeric-unary-plus
op:numeric-unary-minus
op:operation
xf:false
xf:true
xf:length
xf:Boolean
xf:item-at
xf:get-namespace-uri
xf:get-local-name
xf:round
xf:compare
xf:not
xf:empty-sequence
xf:root
xf:index-of

Some of these functions are generic—they perform operations on arbitrary components within the data model. There are also operations defined to construct or access components of the XQuery data model; these operations are beyond the scope of this section.


Notation

In the main notation of XQuery, the important aspects are the inference and normalization rules. This section provides a high-level view of these rules.

Inference Rules

The basic building block of an inference rule is the judgment. The judgment is specified by a description containing a combination of variables (instantiated to an appropriate value) and is constructed with symbols. The judgment is made of three parts:

assumptions |- problem =>> result

The inference rules are used to express logical relationships between judgments, and define how complex judgments can be sought from basic judgments. Here's an example from the Working Draft:

$x=>0 4=>4
    $x+4=>4

If an expression containing the variable reference $x yields 0, and the literal expression 4 yields 4, you know that $x+4 yields 4.

The judgments in the first line are referred to as the premises, and the second line is referred to as the conclusion. Critically, the inference rule tells you that if all the premises are true, you may reliably infer that all the conclusions are true.

The parts of a judgment have their origin in the theory of deduction in formal semantics. The book Formal Semantics by Glynn Wiskel (MIT Press) is a good reference work for those interested in inference rules and the theory of formal semantics.

Normalization

Normalization applies a translation function to each expression in XQuery syntax; the resulting expression is in XQuery Core syntax. The concept of normalization is highly theoretical and gets into rewrite system rules. I'll just describe the basic structure for all normalization rules:

[Expression]

==

coreExpression

[Expression] represents an XQuery expression before translation, and coreExpression represents the expression in XQuery Core after the translation.

For example, consider the following XQuery:

for $a in (1,2) return
       $b in (3,4)
       return
              element pair { ($a,$b) }

It is normalized to this coreExpression:

for $a in (1,2) return
       for $b in (3,4) return
              return
                     element pair { ($a,$b) }


Expression of Semantics in XQuery

This section presents an overview of how the semantics of expressions in XQuery can be constructed. In general, an expression can be represented as shown in the following, exhausting all the possibilities:

Expr ::= SortExpr
        |OrExpr
        |AndExpr
        |FLWRExpr
        |QuantifiedExpr
        |TypeswitchExpr
        |IfExpr
        |GeneralComp
        |ValueComp
        |NodeComp
        |OrderComp
        |InstanceofExpr
        |RangeExpr
        |AdditiveExpr
        |MultiplicativeExpr
        |UnionExpr
        |IntersectExceptExpr
        |UnaryExpr
        |CastExpr
        |Constructor
        |PathExpr

Let's briefly examine the grammar for some of the expressions in the main expression on the left-hand side:

SortExpr (sort expression) ::= Expr "stable"?" sortby""("SortSpecList")"
OrExpr ::= Expr "or" Expr
AndExpr ::= Expr "and" Expr

A FLWRExpr in XQuery consists of a ForClause, a LetClause, and an optional WhereClause:

FLWRExpr ::= (ForClause|LetClause)+WhereClause?"return" Expr

A coding example for flower expressions is as follows:

<bib>
  {
    FOR $b IN document("C:\XQueryKickStart\XQueryDB\bib.xml")/bib/book
    WHERE $b/publisher = "Addison-Wesley" AND $b/@year > 1991
    RETURN
      <book year=' $b/@year '>
        { $b/title }
      </book>
  }
</bib>

This XQuery sample is taken from the example in Chapter 8, "XQuery and Databases." It clearly demonstrates a flower expression with the for clause, the where clause, and a return expression. This example returns books published by Addison-Wesley beyond the year 1991.

An example of a let clause is shown here:

let $a=5
return
       $a+10

Here are some additional grammar examples:

QuantifiedExpr ::= ("some"|"every") Variable "in" Expr "satisfies" Expr
TypeswitchExpr ::= "typeswitch""("Expr")"("as" Variable)?CaseClause+"default" "return" Expr
IfExpr ::= "if" "(" Expr")""then" Expr "else" Expr
GeneralComp ::= Expr("="|"!="|"<"|"<="|">"|">=") Expr
ValueComp ::= Expr("eq"|"ne"|"lt"|"le"|"gt"|"ge") Expr
NodeComp ::= Expr("==" | "!==") Expr
OrderComp ::= Expr("<<"|">>"|"precedes"|"follows") Expr
InstanceofExpr::= Expr "instance" "of" "only?" Datatype
RangeExpr ::= Expr "to" Expr
AdditiveExpr ::= Expr ("+"|"-") Expr
MultiplicativeExpr ::= Expr ("*"|"div"|"mod") Expr
UnionExpr ::= Expr ("union" | "|") Expr
IntersectExceptExpr ::= Expr ("intersect" | "except") Expr
UnaryExpr ::= ("-"|"+") Expr
CastExpr::=("cast""as"|"treat""as"|"asser""as")Datatype"("Expr")"
Constructor::= ElementConstructor|ComputedElementConstructor|ComputedAttributeConstructor
PathExpr::= AbsolutePathExpr|RelativePathExpr


Type Level

This section discusses XQuery types and prime types. Let's look at the grammar identified for these types.

XQuery Types

The XQuery type can be represented by the following regular expression:

Type::= TypeUnion
        |TypeBoth
        |TypeSequence
        |TypeRepeat
        |ItemType
        |TypeRef
        |TypeParenthesized
        |TypeNone
TypeUnion::= Type "|" Type
TypeBoth::= Type "&" Type
TypeRepeat::= Type TypeOccurenceIndicator
TypeSequence::= Type ","  Type
ItemType::= ElementRef | AttributeRef | Simpletype
TypeRef::= "type"  Qname
TypeParenthesized::= "("Type?")"
TypeNone::= "none"

Prime Types

Prime types play an important role in the static semantics of XQuery. Prime types in XQuery can be represented by the following grammar rule:

PrimeType ::= ItemType | (PrimeType "|"  PrimeType)


Function Semantics

Let's briefly look at the user-defined function semantics. The grammar expression for the function can be defined as follows:

[View full width]

FunctionDefn ::== "define" "function" QName "(" ParamList? ")" ("returns" Datatype)?
   EnclosedExpr
ParamList::= Param (","Param)* - the Param repeats itself.
Param::= Datatype? Variable

There are also other formal semantic functions that have been defined but are not a part of the draft document. They are as follows:

fs:document
fs:distinct-doc-order
fs:data
fs:unique-item-at.


In Brief

  • XQuery processor implementations are represented by four phases: parsing, normalization, and static semantics (all compile-time); and dynamic semantics (runtime).

  • XQuery Core is a functional subset language of XQuery that supports unordered sequences and recursive navigation.

  • FLWR expressions (for, let, where, return) generate new sequences.

  • The XQuery data model includes these components: atomic value, node, item, and sequence.

  • XQuery has its own built-in simple data types. Type constructors are available for constructing sequences, node types, and elements.

  • A function is an operator that takes input parameters and produces output. XQuery functions operate on the components of the data model.

  • Subtyping in XQuery is related to the XML Schema rule of derivation by restriction.

  • The basic building block of an inference rule is the judgment, which is specified by a description containing a combination of variables and constructed with symbols. Inference rules express logical relationships between judgments and define how complex judgments can be sought from basic judgments.

  • Normalization applies a translation function to each expression in XQuery syntax; the resulting expression is in XQuery Core syntax.


Chapter 8. XQuery and Databases

by Vaidyanathan Nagarajar

IN THIS CHAPTER

  • Relational Database Concepts

  • A Relational-to-Object Database: Concept Mapping

  • SQL Concepts

  • Relational Technology

  • Oracle 9i XQuery Support

  • IBM DB2 XQuery Support

  • GoXML DB XQuery Support

  • In Brief


Relational Database Concepts

A database is an organized collection of logically related data. Today, the relational model has become a universally accepted design. In addition, object databases and object-relational databases are the next big step in database technology. Oracle provides object-relational databases. The OO database system has its origins in OO programming languages.

XML document structure is based on a hierarchical design to store data in flat files—a tree structure with a root, subtrees, and leaves. The hierarchical model can also be found in directory services.


A Relational-to-Object Database: Concept Mapping

If we were to map the relational model to an object model, the rows would correspond to an object (an instance of a class; compare the class to a table), and the columns would correspond to the attributes of the object. This mapping has enabled the creation of object-oriented databases. Starting with Oracle 8i, it became possible to store Java objects directly in an Oracle database, thus making it an object-enabled database.

Transactions

Databases deal with transactions. A transaction is a logical unit of work. Databases support two types of transactions: a commit transaction, which marks the end of a transaction; and a rollback transaction, which marks an unsuccessful end of a transaction. Transactions have four important properties that go by the acronym ACID:

  • Atomicity— Transactions are atomic, which means all or nothing.

  • Consistency— A transaction transforms a consistent state in a database to another consistent state.

  • Isolation— Transactions in a database are isolated. Transaction updates are concealed from all the other transactions until the committing of the update transaction takes place.

  • Durability— After a transaction commit is done, even if the system crashes, the update still survives.


SQL Concepts

SQL is based on set theory and manipulates sets of data. Although we won't talk about the set theory view of SQL, we will discuss briefly the various SQL operations.

Data Manipulation Language (DML)

Data Manipulation Language (DML) provides operations or commands to work on the created tables. The four basic operations are as follows:

  • SELECT retrieves rows from tables.

  • INSERT inserts a new row into a table.

  • DELETE deletes an existing row from a table.

  • UPDATE updates existing rows in a table.

Consider a simple table called Employee that has two columns: empno and empname. You represent the table as Employee(empno,empname). The column empno uniquely identifies an employee in the Employee table. Let's see how the four basic operations can be used.

To retrieve all the employees in the Employee table the following code can be executed:

select empno, empname from Employee;

To insert rows into the Employee table, you can use the insert operation. An example is shown in the following code snippet:

insert into employee values(50,'Nathan Nagarajan);

To delete the row you added for Nathan Nagarajan, you can use the delete operation:

delete from Employee where empno=50;

To update the Nathan Nagarajan row to Vaidyanathan Nagarajan, you can use the update operation:

update Employee set empname='Vaidyanathan Nagarajan' where empno=50;

Data Definition Language (DDL)

Data Definition Language (DDL) is used to manage tables. Its set of common commands is as follows:

  • CREATE creates database objects: tables, views, triggers, indexes, and so on.

  • ALTER changes structures and permissions on tables.

  • DROP deletes database objects.


Relational Technology

Current vendors that provide relational technology include Oracle (39.4% of the market), Microsoft (11.1%), IBM (33.6%), Sybase, and Informix. Some of the key features provided by these vendors include scalability, security, sophisticated indexing, and performance tuning. Many APIs are available to query databases from programming languages. A couple of important ones are JDBC and ADO.

XML databases first appeared via a modified RDBMS, whereby XML data was stored as a CLOB data type. Databases then added shredding capability by layering on top of a standard RDBMS. After this, a number of pure XML databases appeared (usually referred to as XDBMS), where XML can be stored natively without losing information to shredding operations. As an example, Tango Server uses the Native XML database model.

Finally, major DB vendors such as Oracle and Microsoft came out with enhanced RDBMSs that extended their underlying type systems to support XML information (elements, attributes, DOM tree nodes, and so on) as built-in data types for columns. With XQuery becoming the standard for querying XML documents, all these RDBMS vendors have begun providing their own XQuery implementation engines.

Next, we'll look at two vendors' implementation support for XQuery. We begin with Oracle.


Oracle 9i XQuery Support

Oracle has been an active participant in the standardization of XQuery with the W3C's XML Query Working Group. In February 2003, Oracle released an XQuery prototype as a set of downloadable Java classes. This prototype is not a product; it can be used inside or outside the database.

Outside the database, it can be used in the middle-tier to query XML documents in the following situations:

  • All data is stored in XML.

  • Use XML elements and attributes returned as a resultset.

The prototype release version is RELEASE_0.2_030217. You can download it from the following URL:

http://otn.oracle.com/sample_code/tech/xml/xmldb/xmldb_xquerydownload.html

The Oracle XQuery implementation is packaged in a JAR file called xquery.jar. The prerequisites for running the XQuery implementation are the following:

  • Oracle XDK

  • JDK 1.3

  • JDBC driver to access the Oracle database

This new version contains a sample interactive command-line tool called XQLPlus (which is similar to SQLPlus) for connecting to the Oracle database to query it.

JXQI

JXQI is the XQuery API implementation in Java by Oracle. You can use this API to execute queries written in XQuery and fetch the resultset of the execution. The API is modeled similarly to the JDBC API in Java.

Installing JXQI

After you download JXQI, unzip it to a folder called OracleXQuery. Installation adds two JAR files to your classpath—one is from Oracle XDK, and the other is the Oracle XQuery implementation kit. Now the classpath looks something like this:

CLASSPATH=.;C:\oraclesdk\lib\xmlparserv2.jar;C:\OracleXQuery\xquery.jar;

To ensure the installation is correct, a file called exmpl1.xql is provided that contains an XQuery you can run in XQLPlus to retrieve a book list from an XML file bib.xml (supplied with the XQuery implementation). The XQuery code in the exmpl1.xql file is buggy, and we recommend using the following tested XQuery code (also available in the file exmpl.xql from the Sams site):

<bib>
  {
    FOR $b IN document("bib.xml")/bib/book
    WHERE $b/publisher = "Addison-Wesley" AND $b/@year > 1991
    RETURN
      <book year=' $b/@year' >
        { $b/title }
      </book>
  }
</bib>

You can also cut and paste the code to a file using any standard text editor, name the file exmpl.xql, and save it in the directory OracleXQuery (where you have unzipped the Oracle XQuery implementation).

Next in the command prompt cd to the OracleXQuery folder and issue the following command:

java oracle.xquery.XQLPlus exmpl.xql

You will see output similar to that shown in Figure 8.1.

Figure 8.1. Output of XQLPlus calling file directly.

The file can also be referenced using the @filename command. The output in Figure 8.2 shows the usage.

Figure 8.2. Output of XQLPlus using @filename command.

You can also type the query into XQLPlus directly. Figure 8.3 shows the code being directly executed in the XQLPlus prompt, and the result of the XQuery.

Figure 8.3. Output of XQLPlus typing the code directly in the prompt.

In addition, you can use the XQueryContext utility to test the implementation of XQuery. XQueryContext provides the context for executing XQueries. You can call it from a Java program or as follows, assuming you were in the OracleXQuery folder and using exmpl.xql:

java oracle.xquery.XQueryContext –xqfile exmpl.xql

The output of the execution will be similar to that shown in Figure 8.4.

Figure 8.4. Output of using XQueryContext class (file reference from current directory).

In this example, bib.xml is assumed to be in the same location as exmpl.xql. If you make XQueryContext reference exmpl.xql from another directory as shown here:

java oracle.xquery.XQueryContext –xqfile c:\XqueryKickStart\XQueryDB\exmpl.xql

Then bib.xml will be referenced from that folder. You can override the location of bib.xml with the baseurl parameter; see Figure 8.5.

Figure 8.5. Output of using XQueryContext referring file from another directory.

Coding Using the JXQI API

The Oracle XQuery implementation consists of the following classes:

  • PreparedXQuery— Provides the context for execution of XQueries.

  • SQLDataSource— Gets SQL data source names and iterates over the data source. This class implements the XMLDataSource class.

  • SQLIterator— Iterates over the SQL data sources. This class implements XMLDataSourceIterator.

  • XMLDataSource— An interface implemented by SQLDataSource.

  • XMLDataSourceIterator— An interface implemented by SQLIterator.

  • XQException— The exception thrown when XQuery execution throws an error.

  • XQLPlus— A utility provided with the Oracle XQuery implementation to interactively execute XQueries.

  • XQMesg— Constructs XQuery messages.

  • XQueryContext— Provides the context for executing XQueries. This class can be run as a utility or can be used in coding and executing XQueries from Java programs.

  • XQueryDataSource— This is an interface and can be used to provide classes that want to provide their own XQueryDataSource implementations.

  • XQueryResultSet— Provides the context for executing XQueries.

  • XQueryResultSet.XQueryResultType— A class enclosed by the XqueryResultSet class.

Let's look at the code example in Listing 8.1, which shows how to use JXQI.

Listing 8.1 OracleXQueryDemo.java
import oracle.xquery.XQueryContext;
import oracle.xquery.PreparedXQuery;
import oracle.xquery.XQueryResultSet;
import oracle.xml.parser.v2.XMLNode;

import java.io.Reader;
import java.io.FileReader;

/**
 * Created by IntelliJ IDEA.
 * User: Nathan Nagarajan
 * Date: Mar 17, 2003
 * Time: 10:20:10 PM
 * This programs demonstrates usage of Oracle XQuery API to read a query from a
 * file and print out the result of the XQuery. The XML file to query is found
 * as part of XQuery in the query file.
 */
public class OracleXQueryDemo {
    public static void main(String argv[]) {
        //Create an XQuery Context.
        XQueryContext ctx = new XQueryContext();
        try {

            // Read the XQuery file containing the query.
            Reader strm =
                new FileReader("C:\\XQueryKickStart\\XQueryDB\\exmpl_xquery.xql");

            // Prepare the XQuery.
            PreparedXQuery xq = ctx.prepareXQuery(strm);

            // Execute the XQuery.
            XQueryResultSet rset = xq.executeQuery(false);

            //Traverse through the result set.
            while (rset.next()) {
                // get the result nodes
                XMLNode node = rset.getNode();
                //System.out.println(" NODE " + node.getNodeName());
                //print the node.
                node.print(System.out);
            }
        } catch (Exception e) {
            e.printStackTrace();
            System.out.println(" got exception " + e.getMessage());
        }
    }
}

Just as in JDBC programming, you get a connection in JXQI that you need to get an XQuery context. After you get the XQuery context, you read the file having the query written XQuery by getting a Reader. After you read the XQuery, PreparedXQuery is called through the context. This PreparedXQuery is used in the next step to get a resultset. The nodes of the read XML stream are printed out until the resultset is empty. The output of the code is shown in Figure 8.6.

Figure 8.6. Output for OracleXQueryDemo program.

The XqueryContext keeps track of connection to the database, metadata about the database, and so on. In JDBC, for a PreparedStatement you can bind variables to the prepared statement. Similarly, in JXQI, you can bind variables to the PreparedXQuery. The sample code in Listing 8.2 demonstrates how to use bind variables in JXQI.

Listing 8.2 OracleXQueryBindDemo.java
import oracle.xquery.XQueryContext;
import oracle.xquery.PreparedXQuery;
import oracle.xquery.XQueryResultSet;
import oracle.xquery.exec.Utils;
import oracle.xml.parser.v2.XMLNode;
import oracle.xml.parser.v2.DOMParser;
import oracle.xml.parser.v2.XMLDocument;

import java.io.InputStream;

/**
 * Created by IntelliJ IDEA.
 * User: PadmaNathan
 * Date: Mar 17, 2003
 * Time: 11:10:17 PM
 * Example shows how to dynamically bind XML files to query and execute XQuery
 */
public class OracleXQueryBindDemo {
    public static void main(String argv[]) {
        //Prepare XQuery Context.
        XQueryContext ctx = new XQueryContext();

        try {
            // Prepare the XQuery String with a bind variable
            PreparedXQuery xq = ctx.prepareXQuery(
                    " for $b IN bind('1')/bib/book            " +
                    " WHERE $b/publisher = 'Addison-Wesley' " +
                    "   AND $b/@year > 1991                 " +
                    "    RETURN  <book year=' $b/@year '>   " +
                    "            { $b/title }  </book>      ");

            //use a DOM parser to read the XML stream.
            DOMParser prs = new DOMParser();
            InputStream is =
                Utils.getStream("c:\\XQueryKickStart\\XQueryDB\\bib.xml");
            prs.parse(is);

            XMLDocument doc = prs.getDocument();
            //bind the XML stream in DOM to the bind variable "1" in XQuery.
            xq.setNode((new Integer("1")).toString(), doc);

            // System.out.println(" Executing Query ");
            //Execute the result set
            XQueryResultSet rset = xq.executeQuery(false);
            int pos = 0;
            //print the nodes.
            while (rset.next()) {
                // System.out.println(" ---- Fetching Node[" + pos + "] ---- ");
                pos++;

                XMLNode node = rset.getNode();
                // System.out.println(" NODE " + node.getNodeName());
                node.print(System.out);
            }
        } catch (Exception e) {
            System.out.println(" got exception " + e.getMessage());
        }
    }
}

In this code, look at how the file name is used as a bind variable and dynamically bound before executing the query. The bound filename is the file containing the XML document. Data types that can be bound to the parameters include String, integer, float, a DOM node, and so on.

If you want to use JXQI to connect to a database like Oracle, you can substitute the following three lines of code where you would read from a XML file, because you are connecting to a database to retrieve query resultsets:

DriverManager.registerDriver(new Oracle.jdbc.driver.OracleDriver());
Connection connection =  DriverManager.getConnection("jdbc:oracle:@","scott","tiger");
 XQueryContext context = new XQueryContext(conn);

Note how the connection object is passed to the XQueryContext. The previous code used the default constructor for XQueryContext.

Using JXQI to Connect to an Oracle Database

Next we present two examples of using JXQI to connect to an Oracle database. In the first example, you'll use XQuery to connect to a database. In the second example, you'll define an XML filter to retrieve restricted results from the query.

In both of the database examples, we tested the code using Oracle9i and the Oracle thick JDBC drivers available at

http://otn.oracle.com/software/tech/java/sqlj_jdbc/htdocs/jdbc9201.html.

Note that you have to register with Oracle Technology Network to access this download. We used JDK 1.4 version of the kit to write the code. Use ojdbc14.jar if you are using JDK1.4 and include the JAR in your classpath.

Rather than building a new table, the program uses the out-of-the-box emp table shipped with the Oracle9i implementation. The following describe operation on the emp table shows its structure:

SQL> describe emp
 Name                                      Null?    Type
 ----------------------------------------- -------- ---------------------------

 EMPNO                                     NOT NULL NUMBER(4)
 ENAME                                              VARCHAR2(10)
 JOB                                                VARCHAR2(9)
 MGR                                                NUMBER(4)
 HIREDATE                                           DATE
 SAL                                                NUMBER(7,2)
 COMM                                               NUMBER(7,2)
 DEPTNO                                             NUMBER(2)

Now let's look at the code for connecting to an Oracle database. These examples are available for download from the Sams site. Listing 8.3 presents code that queries the emp table in Oracle in the user scott schema and prints the empid and salary as output.

Listing 8.3 OracleXQueryDBDemo.java
import oracle.xquery.XQueryContext;
import oracle.xquery.PreparedXQuery;
import oracle.xquery.XQueryResultSet;
import oracle.xquery.exec.Utils;
import oracle.xml.parser.v2.DOMParser;
import oracle.xml.parser.v2.XMLDocument;
import oracle.xml.parser.v2.XMLNode;

import java.io.InputStream;
import java.io.Reader;
import java.io.StringReader;
import java.sql.DriverManager;
import java.sql.Connection;

/**
 * Created by IntelliJ IDEA.
 * User: Nathan Nagarajan
 * Date: Apr 20, 2003
 * Time: 7:06:10 PM
 * This program demonstrates the usage of using XQuery calls to a database.
 * The oracle database comes with a default table called emp. This program
 * creates an XML Stream of EMP tag along with their name and salary.
 */
public class OracleXQueryDBDemo {
    public static void main(String argv[]) {
        try {
            //use oracle thick client to register the drivers.
            DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
            //get the database connection
            Connection conn = DriverManager.getConnection(
                              "jdbc:oracle:oci:@NATHAN", "scott", "tiger");
            // create a context using that connection..
            XQueryContext ctx = new XQueryContext(conn);

            // create a string from the file
            Reader strm = new StringReader(
                    "FOR $i IN sqlquery(\"select * from scott.emp\")/ROW " +
                    " RETURN <EMP empno=\"{$i/EMPNO}\">{$i/ENAME}{$i/SAL}</EMP>");

            //to get formatted result set use this strm and
            //comment the previous line.
          /*  Reader strm = new StringReader(
                                "FOR $i IN sqlquery(\"select *
                                from scott.emp\")/ROW " +
                                " RETURN <EMP empno=\"{$i/EMPNO}\
                                ">\n{$i/ENAME}\n{$i/SAL}</EMP>");

           */
            // prepare the query
            PreparedXQuery xq = ctx.prepareXQuery(strm);

            // get a resultset
            XQueryResultSet rset = xq.executeQuery(false);
            System.out.println("<EmployeeData>");
            //print the result set.
            while (rset.next()) {
                XMLNode node = rset.getNode();  // get result nodes
                //System.out.println(" NODE " + node.getNodeName());
                node.print(System.out);
            }
            System.out.println("</EmployeeData>");
        } catch (Exception e) {
            System.out.println(" got exception " + e.getMessage());
        }
    }
}

In this code, you first load the Oracle drivers using a DriverManager, and then get the connection through the Connection object by passing the url and connector strings. As you know, XQueryContext can take a Connection object for its constructor. You construct an XQuery context, pass the connection object, and execute the XQuery.

Note the following lines from the code:

"FOR $i IN sqlquery(\"select *
from scott.emp\")/ROW " +
" RETURN <EMP empno=\" {$i/EMPNO}\
">\n{$i/ENAME}\n{$i/SAL}</EMP>");

You select all records from the emp table (scott.emp refers to the schema and table name, which is of the format schema.tablename) and then create an XML stream of EMP records. The function sqlquery is used to identify the SQL query. The result of this function is a list of ROW elements. Each ROW element is an XML representation of a row of the SQL query resultset. You can join the result with other XQuery sources or format the elements. The other steps of processing the resultset are similar to other examples we have discussed. The output of the program is large, because there are 14 rows in the table. Figure 8.7 depicts the output of running the program.

Figure 8.7. Output for OracleXQueryDBDemo program.

In addition, a file called output1.xml is provided that contains the complete results from executing the program. For convenience, the output is shown here:

<EmployeeData>
<EMP empno="7369">
   <ENAME>SMITH</ENAME>
   <SAL>800</SAL>
</EMP>
<EMP empno="7499">
   <ENAME>ALLEN</ENAME>
   <SAL>1600</SAL>
</EMP>
<EMP empno="7521">
   <ENAME>WARD</ENAME>
   <SAL>1250</SAL>
</EMP>
<EMP empno="7566">
   <ENAME>JONES</ENAME>
   <SAL>2975</SAL>
</EMP>
<EMP empno="7654">
   <ENAME>MARTIN</ENAME>
   <SAL>1250</SAL>
</EMP>
<EMP empno="7698">
   <ENAME>BLAKE</ENAME>
   <SAL>2850</SAL>
</EMP>
<EMP empno="7782">
   <ENAME>CLARK</ENAME>
   <SAL>2450</SAL>
</EMP>
<EMP empno="7788">
   <ENAME>SCOTT</ENAME>
   <SAL>3000</SAL>
</EMP>
<EMP empno="7839">
   <ENAME>KING</ENAME>
   <SAL>5000</SAL>
</EMP>
<EMP empno="7844">
   <ENAME>TURNER</ENAME>
   <SAL>1500</SAL>
</EMP>
<EMP empno="7876">
   <ENAME>ADAMS</ENAME>
   <SAL>1100</SAL>
</EMP>
<EMP empno="7900">
   <ENAME>JAMES</ENAME>
   <SAL>950</SAL>
</EMP>
<EMP empno="7902">
   <ENAME>FORD</ENAME>
   <SAL>3000</SAL>
</EMP>
<EMP empno="7934">
   <ENAME>MILLER</ENAME>
   <SAL>1300</SAL>
</EMP>
</EmployeeData>

The next example shows how to apply a filter to the resultset by making an equijoin to a outer file that has just one value:

<deptno>20</deptno>

This file is available as dept.xml from the Sams site, or you can enter this code into a dept.xml file. This file acts a filter to show only records for department #20. The code in Listing 8.4 shows an example of using a filter to get a subset of the resultset.

USING ROOT TAGS

If you want to strictly adhere to the syntax and semantics of XML construction, you can include root tags for the data. You'll need to modify the program in Listing 8.4 to reference the root and child nodes, as in root/childnode.

Listing 8.4 OracleXQueryDBFilterDemo.java
import oracle.xquery.XQueryContext;
import oracle.xquery.PreparedXQuery;
import oracle.xquery.XQueryResultSet;
import oracle.xml.parser.v2.XMLNode;

import java.sql.DriverManager;
import java.sql.Connection;
import java.io.Reader;
import java.io.StringReader;

/**
 * Created by IntelliJ IDEA.
 * User: Nathan Nagarajan
 * Date: Apr 20, 2003
 * Time: 8:15:21 PM
 * This program demonstrates using XQuery calls to a database.
 * The oracle database comes with a default table called emp. This program
 * creates an XML Stream of EMP tag along with their name and salary. Also, a
 * filter is applied to filter the resultset to employees belonging to deptno
 * read from a dept.xml file.
 */
public class OracleXQueryDBFilterDemo {
    public static void main(String argv[]) {
        try {
            //use oracle thick client to register the drivers.
            DriverManager.registerDriver(new oracle.jdbc.driver.OracleDriver());
            //get the database connection
            Connection conn = DriverManager.getConnection(
                              "jdbc:oracle:oci:@NATHAN", "scott", "tiger");
            // create a context using that connection..
            XQueryContext ctx = new XQueryContext(conn);

            // create the xquery with a filter on the data read from a file.
            StringBuffer sBuffer = new StringBuffer();
            sBuffer.append("FOR $i IN sqlquery(\"select *
                           from scott.emp\")/ROW, ");
            sBuffer.append("$j IN document(\"
                           C:\\XQueryKickStart\\XQueryDB\\dept.xml\") ");
            sBuffer.append("WHERE $j/deptno = $i/DEPTNO ");
            sBuffer.append("RETURN ");
            sBuffer.append("<EMP empno=\"{$i/EMPNO}\" deptno=\"{$j/deptno}\"/>");

            //String Reader to read the stream.
            Reader strm = new StringReader(sBuffer.toString());

            // prepare the query
            PreparedXQuery xq = ctx.prepareXQuery(strm);

            // get a resultset
            XQueryResultSet rset = xq.executeQuery(false);
            System.out.println("<EmployeeData>");
            //print the result set.
            while (rset.next()) {
                XMLNode node = rset.getNode();  // get result nodes
                //System.out.println(" NODE " + node.getNodeName());
                node.print(System.out);
            }
            System.out.println("</EmployeeData>");
        } catch (Exception e) {
            System.out.println(" got exception " + e.getMessage());
        }
    }
}

In this example, note how an equijoin is made between the resultset obtained from a query and a filter read from the local file dept.xml. The following code shows the equijoin relation:

WHERE $j/deptno = $i/DEPTNO

The output of the program is shown in Figure 8.8.

Figure 8.8. Output for OracleXQueryDBFilterDemo program.

If you want to select results for different department, you can use the appropriate deptno. Note that this example has the limitation of setting only one filter in the dept.xml file. You can write XQueries to do filters on multiples conditions, like filtering on MGR, JOB, and so on in the emp table.

Using JXQI offers many advantages. The API is constructed similarly to the JDBC API. It supports PreparedStatements, so queries can be precompiled and reused many times. Allowing bind variables makes it more scalable. Oracle has enhanced the XQuery language, allowing SQL queries to be embedded within XQuery. It supports XQueryX, which is an XML representation of XQuery. Oracle also supports the use of URLs inside the document function and in file headers. In fact, one of the XqueryContext constructors provides for reading data straight from a URL.

Having looked at Oracle support for XQuery, let us move now to IBM's support for XQuery in DB2.


IBM DB2 XQuery Support

As of this writing, IBM is still working on its release of XQuery in its DB2 database (a relational storage engine). However, IBM has put up a site showing how XQuery can be used to query DB2 databases through the Xperanto demo. That demo is available at the following site:

http://xperanto.dfw.ibm.com/demo/demos/xml/xq1.jsp

The scenario of the example goes like this. Three different companies are merging, and they internally use different databases: DB2, Informix, and Excel. The challenge presented to the user is to combine data from these different data source and produce XML output.

Figure 8.9 shows the exchange of data between different servers when an XQuery is constructed for querying data across different data sources. This diagram summarizes how the XQuery architecture will be supported by DB2. The XQuery goes to the Federated Server, which translates the XQuery to SQL Query, and the output is served as XML.

Figure 8.9. XQuery architecture for DB2.

The XQuery that corresponds to obtaining the resultset is shown in the following code snippet. The XQuery combines data over the different data sources to provide the suppliers' parts:

for[$partkey=partkey $supplierKey=suppkey
$supplier=supplier$quote=quote] in part_suppliers
where $partkey=? and $supplierKey=?
return
<row>
<partkey>{$partkey}</partkey>
<supplier>{$supplier}</supplier>
<quote>{$quote}</quote>
</row>

IBM has not progressed much on implementing XQuery. No specific API is available at this time to code in Java and check out IBM's XQuery implementation. Watch the IBM site for future updates.

Next we'll look at the one of the leading native XML databases: GoXML DB, which provides XQuery support with an XML repository.


GoXML DB XQuery Support

Two types of database formats are available for storing and retrieving XML data:

  • Native XML database

  • XML-enabled relational database

The criteria on which the correct database format should be chosen can be based on the following checklist:

  • Documents versus data

  • Retrieval speed

  • Storage size

  • Static versus dynamic structures

  • XML Schema and DTD support

  • Security

  • Permanent versus temporary storage

Native XML databases are better for document-centric XML that's readable by human beings. Magazine articles are a good example of document-centric content. Native XML databases maintain comments, the CDATA section, document order, entity usage and instructions for processing. Native XML databases also let you query the contextual nature of the XML document.

Data-centric XML documents use XML for transportation. Invoices and payment orders are good examples of data-centric content.

GoXML DB is a native XML repository. It allows you to store and reuse XML documents, and lets you leverage stored data through the support of XQuery.

The benefits and return on investment (ROI) of using GoXML are as follows:

  • XML documents can be stored in their original format.

  • The hierarchical and complex structure of XML documents can be easily understood.

  • GoXML supports the latest XML technologies.

Most relational databases provide XML support by extending their existing boundaries. When the XML content becomes more document-centric, the traditional relational databases can lead to a mismatch between the relational model and the XML model (mapping between the relational and XML model will not be good). Note that the relational model is a table structure, but the XML model is more hierarchical. Some of the problems in the relational model include overhead in maintaining the hierarchical-relational mapping, an inflexible relational security model, and so on.

Most of the relational databases store the entire document along with tabular data, which adds complexity and has implications for storage size. From a security perspective, relational databases provide security at the table and view level. Thus, securing an XML document at the element level becomes difficult. Assuming the document is stored in a BLOB object, it can be made secure at the document level. If a change is made to the document at a node level, the entire document needs to be replaced; but if it's a native XML database, the replacement would take place only at the node level, where the change has to be incorporated.

Native XML databases let you set up security at the document- and element-level. Relational databases require the structure of the data to be predetermined, but a native XML database that is not dependent on any predetermined schema is easier to manage.

Relational databases are more efficient for retrieval of documents whose sizes are measured in GB, whereas native XML is best suited for data measured in MB.

GoXML was designed and built primarily for XML documents. It fits into a niche technology space rather than the mixed space of relational companies like Oracle, IBM, and Microsoft, which support XML in their relational databases.

GoXML DB Product Design

Figure 8.10 shows the various components of GoXML DB. A client API (referred to as the Administrator WorkBench) is available to interface with the server.

Figure 8.10. XQuery architecture for GoXML DB.

The company XMLGlobal, which produces GoXML DB, supports ebXML and adopts this emerging infrastructure as an important component for all its XML products.

In certain stages of the development life cycle using traditional relational databases, XML content documents can be created in a business transaction process, normally by writing proprietary XML components. Instead of developing any proprietary XML components, GoXML DB can be used; it controls the overhead associated in converting between XML and relational components. You can use the XML-based capabilities of schema validation and XQuery support in such a case. In fact, using GoXML DB turns out to be a cheaper solution than rewriting most of the existing components. This kind of structure is usually referred to as service-oriented architecture—both databases work seamlessly as a single component.

XQuery Support in GoXML

Now let's get into the details of using XQuery directly on GoXML DB. The database and the client to connect to the database are available from http://www.xmlglobal.com/index.jsp. The installation is straightforward; you must install the server first and then the client. When you register with the site, you'll receive a mail message with the registration key to use during installation.

A Content Management System for XML (CMX) is also installed as a part of the client install. CMX is a Web application that enables XML documents stored in the GoXML DB server to be exposed and accessed via HTTP protocol. Documents can be exposed through the CMX via read-only, directly via HTTP, and read-write via a Webfolder.

The client software is called the Explorer. Figure 8.11 shows the interface of the client software to the server.

Figure 8.11. Client software to connect to the GoXML DB server.

GoXML DB comes with three built-in databases: InvoiceDB, BookDB, and PersonDB. There are also three schemas—book, person, and invoice—which are shown in the schemas view.

Figure 8.12 shows how to load the sample XML provided with the software. When you select the Person template, a menu with an Add Documents option opens; you can use it to load the XML file.

Figure 8.12. Loading XML data into PersonDB.

Figure 8.13 shows the execution of the following code and the result in the XML Display window:

Root("PersonDB:Person")/Person/City
Figure 8.13. Executing XQuery in the Explorer window.

Figure 8.14 shows the execution of another XQuery in the Explorer window. The query does the following: If the City value is Paris, it's replaced with the value Vancouver. The code is as follows:

FOR $d IN Root("PersonDB:Person")
              LET $newdoc :=
              REPLACE $d/Person/City WITH <City>Vancouver</City>
              WHERE $d/Person/City = "Paris"
              RETURN $newdoc
Figure 8.14. Executing a replacement XQuery in the Explorer window.

GoXML DB also comes with an API to query XML documents. The following code snippet shows how to invoke XQuery using the API:

SystemManager.init();
            Server srv = SystemManager.getServer("MyServer");
            AuthenticationInfo ai = new AuthenticationInfo();
            xc = srv.getConnection(ai);
            xc.beginTransaction();
            XStatement xqStmt = xc.createStatement();
            String myQuery = "Root(\"PersonDB:Person\")/Person/City";
            XResultSet rs = xqStmt.execute(myQuery, XStatement.SF_DEFAULT);
            if (rs != null) {
                FlexStringBuffer fsbTemp = new FlexStringBuffer();
                rs.beforeFirst();
                int i = 1;
                while (rs.nextValue()) {
                    fsbTemp.setLength(0);
                    rs.fsbGet(fsbTemp, false);
                    if (fsbTemp.length() > 0) {
                        System.out.println(String.valueOf(i) + "." + fsbTemp);
                    }
                    i++;
                }
            }

            xc.commitTransaction();
            xc.close();

GoXML is ready for commercial use; this chapter was written using GoXML DB v3.0. Some of the product features include XQuery support, XPath support, tag compression, XML Schema and DTD support, document integrity, XML data type support, multilevel security, XML Update language, full-text indexing and search, and an XML-based Administrator and Query tool.

There are other native XML products on the market, but GoXML DB is the most popular. GoXML can be easily integrated into an existing architecture where XML is the messaging mechanism between business components; you can do this by placing GoXML between the relational database and the business components, and making the business components use the GoXML API to construct the XML data for the front-end applications to process.

SHOP TALK: USING METADATA TO GENERATE XML FROM A RESULTSET

Before XQuery support was available in databases, applications that wanted to use XML in their architecture for sending data from business components to the front end began using a combination of XML and XSLT technology. To send XML-based resultsets to the front end, middle-tier programmers had to write their own wrappers to generate XML documents from the resultsets and then stream the resulting XML document. Today, with XQuery support, it is possible to embed XQuery in database calls and retrieve the XML statements.

One useful technique for generating XML data from the resultset of a SQL query involving JDBC uses the dataset metadata and makes the column names of a table the tags for the XML stream generated from the resultset; it then streams the XML document to the front end. In the front end, you can use an XSLT parser to parse the XML stream and print the output to the browser.

This dataset metadata feature is useful for building XML documents on the fly. Using the basic SQL commands, you can grab data from the database and use the column names to build the XML document. Delivering XML documents for the front-end application also has another advantage: If the site is converted to a Web Services portal in the future, minimal conversion is necessary to convert the site into a Web Services host because the site has already been designed to use XML technology. All that needs to be done is the addition of SOAP support (Simple Object Access Protocol uses XML over HTTP for its architecture), and the site becomes ready to act as a Web Services host. Any site that uses this Web Services host site can make a call through SOAP or any proprietary technology to get the service call result as XML, and can use its own technology for display. Even if the portal supports telephony in the near future, conversion to a voice-enabled portal is easy because the portal uses XML for data exchanges. This architecture is a valuable design paradigm for sites that want to use the power of XML in document and data delivery.

The best design principle using JXQI has been the following: If you have an existing system that has data stored in Oracle and you want to generate an XML stream out of the relational data, you can construct your own XML generator layer using the JXQI API. This is an efficient solution when you compare a buy-versus-develop solution. In a buy decision, GoXML is the way to go, because it easily becomes a layer between the application and the database, and provides the layer to generate XML data from a relational database.


In Brief

  • Basic design principles used in relational databases do not apply to XML documents, which are based on a hierarchical design model.

  • Both Oracle and IBM offer demonstration Web sites showing how their XQuery prototypes work. However, Oracle's work on XQuery is further along as of this writing.

  • Oracle 9i support for XML is in the form of a native data type.

  • Currently, IBM's DB2 database translates XQuery into SQL, then to XML as output.


Chapter 9. Advanced XQuery Topics

by James Linn

IN THIS CHAPTER

  • XML and Documents

  • XML Design for Data Storage

  • XQuery Limitations

  • In Brief


XML and Documents

XML has long found important uses in document handling, as well as being a means of transporting data. The XML community has tended to divide itself into document-centric and data-centric communities. Up until this time, the data-centric group has gotten most of the attention and support. However, in the information technology community at large, the application of XML to the document-centric segment is rapidly gaining in importance.

In this context, documents mean things that can be read, such as Web pages, text fragments in document-handling applications, magazine articles, books, and brochures. It also encompasses all the items embedded in the document, such as sound files, pictures, animation, icons, links, and so on. In addition, in the case of a document designed to be printed, it will include all the layout information as well. Finally, it might well include the workflow information required by a particular organization to manage the publication process.

The difference between the two XML worlds can be seen by comparing the favorite example of the data-centric world, an XML file containing a purchase order to a Web page such as http://www.britannica.com. A Web page was chosen as the document-centric example for several reasons:

  • It fits the definition of something that translates into something that can be read. For many people, Web pages are the primary document form they see every day.

  • This Web page, like most, is a complex object, a compound of text, images, and links to other resources.

  • The page contains considerable data scattered over its surface: Data that might be useful to refer back to at some later date, and data that should be easily retrieved by a mechanism such as XQuery.

  • The expansion of mobile phone technology into Web browsing has put Web pages among the first objects that are being required to be published in a variety of formats: WML as well as HTML. Separating the data in this page (by putting it into XML) from the presentation elements is an easy and effective way to enable the efficient reformatting of documents.

The WWW is one of the largest repositories of data in the world. The Web is very difficult to search well, and impossible to search exhaustively. XML is a simple and effective way to make the Web as searchable as a conventional database.

Using XML in the tags on Web pages is just a beginning. After XML tagging, we will be able to apply semantics to all the data residing in documents. These semantics will provide the additional information necessary to make the World Wide Web readable by machines. These semantics will be applied using XML, and the information in this Semantic Web will be queried using XPath and XQuery.

The Semantic Web

The Semantic Web is a new project that will enable the stitching together of the World Wide Web into a coherent whole—a whole that can be mined, aggregated, and correlated by machines to produce information from the vast collection of data that has been dumped into the World Wide Web namespace.

This additional metadata layered onto the objects in the Internet by the Semantic Web will make business-to-business (B2B) integration much easier, because the Net will become much more transparent to machines. Therefore, much more of the ordinary business processing involved in e-commerce can be handled transparently.

Finally, the Semantic Web, coupled with Web Services and P2P networks, will enable the integration of multiple individual computers into a functioning whole.

The issues that the Semantic Web is dealing with are not isolated to that space; they are the issues that document management will have to face as XML becomes more integral to its creation and management.

The first step to making the Web machine readable is to surround all the data elements in HTML pages with XML tags that describe their meaning. Here is an example of a couple of data items in a Web page that have been tagged using XML:

<shoes>
    <shoe>
        <Manufacturer>Hush Puppies</Manufacturer>
            <size>12 D</size>
        ...
    </shoe>
</shoes>

<shoes>
    <shoe>
        <Manufacturer>BrakeShoes2U</Manufacturer>
        <size>Toyota Corolla 1994</size>
        ...
    </shoe>
</shoes>

As you can see from the isolated XML here, adding labels to the data is just the start. What does this data mean? A human will glance at the data and say that the first site has something to do with selling the kind of shoes that people wear and the second site is about selling car parts, or at least selling brake shoes for cars. We figure that out by using the semantic data we carry in our heads. We know that Hush Puppies is a brand of human shoes, and that Toyota is a car manufacturer. The Semantic Web will provide the semantic data necessary for a machine to understand the meaning of the data displayed on the Web page. Without that additional semantic data, an auto parts store might wind up with a truckload of high heel shoes, when it wanted brake shoes.

In other words, just because a page is readable by a machine doesn't mean that the machine can understand it. There has to be metadata about the labels—in other words, metadata about the metadata. The Resource Definition Framework (RDF) is such a metadata source.

Resource Definition Framework

RDF is an XML-based language that gives observers (machines) a description of the elements in your pages and how these elements relate to a larger body of data. The basic unit in RDF is the triple. It consists of a subject, a predicate, and an object. The subject is a resource identifier; the predicate is a characteristic, attribute, or relationship; and the object is another resource for literal data. Here is an example of an RDF triple for brake shoes:

<rdf:Description about=" www.BrakeShoes2U.com">
    <o:shoe>www.BrakeShoes2U.com/shoe.xml<o:shoe>
</rdf:Description>

This triple has a predicate attribute that identifies the company that owns the page. The predicate is shoe. The object tells the machine to look in the shoe.xml file on BrakeShoes2U.com for a description of the subject. The o prefix is a namespace defined earlier in the XML document that points to the location of the file that defines the elements for the BrakeShoes2U corporation. This triple is inserted in the HTML document as XML in a <meta> tag. There would be a triple for every subject (element tag) in the document that needed additional data specified for it.

This is a major leap forward for the mining of information from documents, Web or otherwise. Management of the exact wording of metadata has become decentralized. It has moved from standard-setting bodies to users. The user creates the labels and the document that describes their meaning. Here is a fragment from a shoe.xml description file that a brake manufacturer might create:

<Class ID="shoe">
    <car make model year>
    </car make model year>
</Class ID>

But the RDF information plus the information in the site-specific XML file still isn't enough. Allowing the user to do the defining will, in most cases, inject too much fuzziness into the transaction for a machine to function reliably. What is needed is another layer of metadata to add meaning to the RDF triples. An ontology is required. In this context, an ontology is a description of the relationship between objects.

ONTOLOGIES

Currently there are several projects that have been organized around ontologies.

One is Simple HTML Ontology Extensions (SHOE). The design idea behind SHOE is to produce a set of HTML extensions that enable the embedding of semantic data into Web pages. For more information about SHOE, we direct you to http://www.cs.umd.edu/projects/plus/SHOE.

Another prominent ontology project is DARPA Agent Markup Language + Ontology Inference Layer (DAML+OIL). DAML+OIL takes the concepts found in SHOE and enriches them with constructs such as data types, data ranges, data structures, and a more elaborate system of expressing semantic relationships.

These ontology frameworks have the goal of giving individual human authors the structure required to embed information in documents that will enable machines, via software, to completely understand those documents.

For more information about DAML+OIL and the DAML project, see http://www.daml.org/.

The W3C has a section that explains the requirements for a Web ontology language. See http://www.w3.org/TR/webont-req.

Ontologies

At the ontology level, we are high enough so that the objects we are describing are quite well agreed on. In addition to describing the relationships between objects, the ontology will contain rules: rules for drawing inferences about objects, and rules for constraining these objects. The following is a possible ontology that could help a machine figure out whether this Web site is talking about brake shoes, human shoes, or possibly horse shoes:

<DEF-RELATION name="shoes">
    <DEF-ARG 1="make">
    <DEF-ARG 2="model">
    <DEF-ARG 3="year">
</DEF-RELATION>

MORE ABOUT THE SEMANTIC WEB AND RDF

This chapter only touches on the possibilities presented by these technologies. For further information about the Semantic Web, start with Tim Berner-Lee's essay at http://www.w3.org/designissues/semantic.html or go to http://www.w3.org/2001/sw, which is the W3C Semantic Web Working Group homepage.

For further information about RDF, see the RDF section at http://www.w3.org/RDF.

This ontology, coupled with the previous shoe.xml description file, indicates the relationship between a car and the brake shoe that would fit it. In other words, the two files taken together tell an outside observer (which could well be a machine) the vehicle information that this company requires to fill an order for brake shoes for the car. It would be available on BrakeShoes2U's Web site for machines to peruse.

The goal of these efforts is to vastly improve machines' capability to accomplish the projects that they do best, which are searching, aggregating, and presenting the results of a task. Humans can then specify the task on a much higher, more general level. In other words, to enable machines to better extract information from the Himalayas of data residing in documents and to present that data as information to humans.

Opportunities Created by the Semantic Web

In addition to searching and organizing data, another goal for the Semantic Web is to allow for B2B transactions without human intervention. Spiders would walk the Web, using future versions of XQuery to extract the data of interest to an individual or corporation. This data would then be aggregated and correlated with the company's internal data. It would be stored in XML document form for future mining by XQuery expressions.

The constructs employed in the Semantic Web could be used to allow large numbers of standalone computers to operate together coherently.

Furthermore, the metadata hierarchy produced by the Semantic Web will allow for the transparent integration into a whole the information in the hodge-podge of stovepipe systems that constitute most enterprises and governmental entities.

SHOP TALK: DOCUMENT-CREATION TOOLS THAT USE XML AND SEMANTIC WEB

The ideas and concepts involved in the Semantic Web have grown to the point where there are toolsets available to create ontologies. See IsaViz at http://www.w3c.org/2001/11/IsaViz.

As an indication of the maturity of the Semantic Web space, Adobe has announced XML enablement of most of the applications in its document processing suite, including such stalwarts as Illustrator, Photoshop, InDesign, and GoLive.

For the XML integration of these products, Adobe uses an XML framework called Extensible Metadata Platform (XMP). The function of XMP is to provide a standard way to apply metadata in the form of labels to all the pieces and parts of a document created in any of the applications in the Adobe suite.

For more information on XMP in conjunction with Adobe's products, see http://www.adobe.com/products/xmp/main.html.

XMP is built on W3C specifications, so it is open and can be used throughout the document processing world. Adobe's hope is that all major players in the content creation space will use XMP to integrate their products with the W3C standards for RDF and later ontologies.

There are several more well established players in the XML authoring world in addition to Adobe. SoftQuad (http://www.softquad.com/top_frame.sq) and Arbortext (http://www.arbortext.com) have recently brought new software releases to the marketplace that extend XML authoring capabilities.

These new releases seek to hide the complexities of the XML labeling from the author by offering preformatted templates that encapsulate most of the tags and the work necessary to define them. The templates are also very useful in encouraging strict standardization of tags among authors.

Arbortext has incorporated data validation via XML Schemas, more flexible output production by using XSL-FOs, and dynamic dialog box production by XML User Interface Language (XUL) into its new product. These additions make it much easier to develop standard Arbortext applications that can be used by authors throughout an enterprise.

This incorporation of XML deep into the major document creation applications is proof positive that the document side of XML is of major importance.

XML and Document Publishing

The document production issues involved in the World Wide Web are a subset of the issues involved in the production of documents.

Desktop publishing has, over the last couple of decades, moved from high-end proprietary systems to the personal computer. As you can ascertain from the decision that Adobe made to integrate XML into most of its content development application, the XML language has put publishing on the verge of another big change.

It is important to remember that documents come in a variety of shapes, sizes, and formats. The same collection of information might have to be packaged in a brochure, a hardbound book, a softcover book, a condensed book, published on a Web page, or might become part of an e-learning offering, as illustrated in Figure 9.1.

Figure 9.1. Illustration of some of the elements that can be involved in a page of a publication.

This heterogeneity produces substantial challenges to the process of markup for publication. One answer to quickly and economically producing publication pages is to append computer-readable labels to all the elements that could possibly be part of a document, and then to use a desktop publishing program to coalesce these elements into the particular publication that is desired. Those labels will be in XML for the following reasons:

  • XML is readable by humans as well as computers.

  • Adding and subtracting elements is a simple, straightforward process.

  • XML integrates exceedingly well with the workflows that control the process of publication.

  • XML functions well as part of the process of persisting the document elements to a permanent store.

Making labels both machine- and human-readable is important. If humans can't read the label, some sort of program will have to be generated that will enable humans to read and modify those labels. More importantly, XML is in a human language. That means the natural integrative powers of the semantics and ontology that have been developed for human language can be applied to these labels. The problems inherent in having computers understand the meaning of a label are best handled by teaching computers the semantics and ontology of languages.

Any labeling scheme for documents must allow for the facile addition and subtraction of elements. Consider the greatly simplified document creation workflow shown in Figure 9.2.

Figure 9.2. Hypothetical workflow for the creation of a document.

The most important things to note about this workflow are that many individuals and applications participate in it, and each must be able to both read the labels attached to the individual components, and each must be able to modify them and to add (usually) or subtract (rarely) elements from the labels.

In addition, the label elements could easily participate in the workflow, as can be seen in this XML series.

The project is initiated and the pages are roughed out. The workflow section of the XML looks like this:

<document>
    <workflow>
        <currentstage>preliminary markup</current stage>
        <next stage>preliminary graphics</next stage>
    </workflow>
</document>

Preliminary markup knows that preliminary graphics gets the document next. Preliminary markup does its work and moves the document to preliminary graphics. The XML now looks like this:

<document>
    <workflow>
        <currentstage>preliminary graphics</current stage>
        <next stage>final photo</next stage>
        <next stage>final graphics</next stage>
        <stage after that>final text</stage after that>
    </workflow>
</document>

As you can see, after the preliminary graphics stage, the document process bifurcates; there are two next stages. The document goes to both the final photo and final graphics department. In addition, an entirely new element, the <stage after that> element has been added.

With document-centric XML, as the publication progresses, the XML is enriched. The elements change, grow, or shrink in a dynamic fashion. The XML object has a rich structure, with the label elements and their order an integral part of that structure.

Throughout this process, the XML and its component parts are repeatedly moved in and out of repositories. Hence, persistence of the XML is a vital part of its life cycle, and the implications of persisting XML structures as rich as the one associated with document production must be carefully considered. A solution that would be fine for a data-centric XML document might not be suitable for a complex XML document associated with document production.


XML Design for Data Storage

We have established the concept that XML contains a richness of metadata. In the next section we will examine XML design from the point of view of storing all that richness. RDBMSs, the filesystem, and native XML databases will be examined. The most suitable solution varies as to:

  • The "complexity" or "richness" of the XML. Simple XML documents contain all the metadata of interest in the data element tags. Complex or rich XML has significant data contained in things such as the order of elements.

  • How often elements are inserted into or deleted from the XML.

Basic Requirements for Data Storage

The prime requirements for a database are to:

  • Persist data in a manner that guarantees the integrity of the data.

  • Keep the data secure from attack, corruption, or destruction.

  • Enable the easy retrieval of that data.

  • Support operations that manipulate that data.

Fundamentals of RDBMS

RDBMSs have largely taken over the persistent storage of data in the business world. Their paradigm is simple and straightforward: all data elements are broken into tables of rows and columns. The column identifies the data, and the row corresponds to an individual data instance. A complex data domain such as Employee is disassembled into multiple individual tables. All rows will have a primary key that uniquely identifies that Employee instance. These primary keys will later be used to connect or relate the tables so that an Employee data domain named Jane Doe can be reassembled.

ACID

Atomic, Consistent, Isolated, and Durable (ACID) is the acronym used to describe the properties that a database transaction must exhibit.

Atomic means that all the operations are done in one unit of work; either all the operations are successfully carried out, or none of them are.

Consistent means that all the data in the database is left in a consistent state when the transaction completes.

Isolated means that the modifications that this transaction is performing are isolated from the effects of other transactions.

Durable means that the modification made during the transaction will survive the crash of the database system.

ACID is the standard that must be adhered to in commercial databases.

As another example, the data domain that is represented by a car can be broken into tables such as transmission, brake, lighting, and so on. A possible primary key for each row in each table could be something meaningful, like the combination of car make, year, and model (toyota_1977_corolla), or something completely arbitrary (abc12345).

The mathematics of the relationship between data and the table structure were worked out by Dr. E. F. Codd. He described a system to normalize the data, and formalized that system into six normal forms. When data is broken into tables using the normal form system, it's easy to guarantee the integrity of that data. Maintaining the integrity of data in a database has been an issue to be solved for every database implementation that has been developed. RDBMSs use their system of unique keys and normalization to ensure data integrity.

This approach causes performance problems. Depending on the number of tables involved, it can take considerable time and processor power to follow the map of joins and reassemble a data domain from all the tables into which the RDBMS has been broken. One of the major development areas for RDBMS vendors has been optimizing the process of querying data from all of these tables to minimize the performance hit. Query speed and efficiency is one of the few areas where RDBMS salesmen can try to differentiate their product from competing products. There are some areas where RDBMS query performance has "hit the wall." For example, to get acceptable performance for data-mining applications, the tables usually have to be denormalized to a greater or lesser degree.

RDBMS Mechanisms to Ensure Data Integrity

Data integrity in an RDBMS is handled by the theoretical underpinnings of normalization and by mechanisms to preserve the "relational integrity" of the data. All the data items that represent a particular object will normally be spread among many different tables. Each table relates to another via the primary key that is part of every data row. Relational integrity mechanisms have been developed to maintain these primary key connections. For example, primary keys are normally maintained by the database. If a key is to be added or deleted to the database, this operation has to occur in all the tables that relate to that data item. The database engine prevents editing of primary keys by users and applications. It checks the primary key for each data row to make sure that it is unique before the data is allowed to be inserted.

In addition to primary keys, data rows can be related to data in other tables by the primary key for that data. This key now becomes a foreign key in that data row. For example, the general data pertaining to the instance of a particular car would be carried in a table perhaps named CAR. It might look like this:

key | make | model | year | color | style
abc123456 | Toyota | Corolla | 1977 | white | 2 door

Each car in the inventory would have a row in this table. The BRAKE table would then look like this:

key | car_key | position | manufacturer | price
zxz0987 | abc123456 | front | Raybestos | 12.34

In this instance, the car_key column represents a foreign key in the brake table.

Other RDBMS Transaction Integrity Mechanisms

In addition to relation integrity mechanisms, data integrity during Create, Read, Update, and Delete (CRUD) operations is guaranteed by a system that first writes the data to a transactional log before attempting to store it in the database. If there is any kind of failure in the operation up to and including the database server crashing, the RDBMS can use this transaction log to keep the data in a consistent state. If there is any kind of server interruption, when it comes back online, the first thing that happens is that all the transactions that are in the log that can be completed are written to the database. The transactions that are in an incomplete state are rolled back. Thus the data in the database is kept in a consistent state through server and network outages.

RDBMSs are also designed to maintain data across more severe accidents, up to and including destruction of the server machine itself. RDBMSs enable the database manager to maintain mirror copies of the data and the transaction log and to put those copies on different drives.

All of these features enable RDBMSs to offer maximum safety for an enterprise's data. This margin of safety is one of the prime reasons that companies will pay the costs associated with an RDBMS.

Importance of Indexing

One of the major value-adds that comes from the RDBMS model is the performance enhancements that come from indexing. Data in columns upon which searches are often performed can be abstracted into a data structure called an index. Searches performed on indexed columns are orders of magnitude faster than the brute-force searches that are required when the search criteria is on a non-indexed column. Consider the following SQL statement:

SELECT * FROM employees WHERE empNumber = xyz1234

If the employee table is indexed on empNumber, the search will be very fast. If the employee table does not carry an index on empNumber, the query time will be much longer. Query speed is a prime determinate of database response, and it can separate acceptable from unacceptable performance.

RDBMSs and Metadata

The metadata for an RDBMS is separate from the data value itself, and is stored in schema. The names of the tables, the names of the columns in those tables, which column represents the primary key, which column represents the foreign key—these are a few of the metadata items that would be in the car schema. You will note that there is no requirement that anything in the schema be human-readable. Table names, column names, and primary and foreign keys must all be understandable by the RDBMS. Making them human-readable is optional.

It is important to note that, without these schema and the RDBMS that can consume them, the data in the RDBMS is inaccessible. The schema for a data set can be exported by itself, without the data, to another instance of the RDBMS and used to re-create the table structure for that data set.

Traditionally, changes to an RDBMS schema have been expensive. To add or delete a column from a table normally involves backing up the data in the table, dropping (deleting) the table, re-creating the table with the new column or minus the deleted column, and restoring the data to the table. Issues can often arise when restoring the data to a table that has a new column. If the new column prohibits nulls or has to be unique, some value will have to be inserted for each row as part of the restore operation. However, this situation is changing. Schema changes can be automated. Some RDBMSs, such as SQL Server, enable the database administrator to add a column to an existing table while the instance is still running. However, tables are still best thought of as durable entities.

Transaction Services in an RDBMS

RDBMSs have well-developed transaction management systems. They can demarcate the start and stop points of transactions, respond to fatal problems with the transaction and, by using the transaction log, reliably commit successful transactions and roll back unsuccessful ones. In addition to managing local transactions, facilities have been developed to enable the management of transactions spanning multiple RDBMSs that might be housed on multiple servers and located in multiple different parts of the planet.

Security Services in an RDBMS

RDBMSs all offer a complete set of security services. Users are required to present a valid set of credentials to gain any access to the data. Access can be further controlled with Access Control Lists (ACLs).

The concept of organizing users into groups is a powerful way to streamline security administration, and it is fully supported by all RDBMSs. It is possible to specify user access on an extremely fine level, controlling access to individual database objects such as tables, queries, views, and so on. Additionally, all or parts of the data can be encrypted.

This is just a sample of the comprehensive security services that all commercial RDBMSs offer. The services are not only comprehensive, but have been tested and debugged extensively. Perhaps more importantly, they are an integral part of the database engine. Add-on security management systems are normally less desirable because they are easier to circumvent.

Concurrency Control

A database should be built to handle access by thousands of users. It must be able to quickly perform whichever of the CRUD operations is requested of it. When more than one user is interested in the same data, problems can occur. Managing multiple access to the same data is called concurrency control.

RDBMSs control concurrency via isolating the transactions. There are three types of isolation conditions that can occur when more than one user is operating on the same piece of data.

Let's start our discussion by postulating two transactions that are running against the same row of data. Figure 9.3 shows how the rows of interest look before anything happens.

Figure 9.3. Database before any interactions.

Transaction A gets there first and begins to work with the row with the Key123 primary key. It changes the data in the second column from Hello to Southwest. The data in the table now looks like Figure 9.4.

Figure 9.4. Database after Transaction A has changed it and before the transaction is committed.

It then proceeds to work on the rest of the transaction. When a SQL select command is run against a database, it returns a snapshot of the data—a picture of the data frozen at the time it was read. At this point, Transaction B arrives and tries to get a snapshot of the data. What will happen?

The following is an exposition of the different isolation levels and how they would affect Transactions A and B:

  • Dirty reads— If Transaction B reads a row after Transaction A has updated the row but not committed the changes, a dirty read has occurred.

    If A commits the transaction, B's snapshot is still valid. However, if A rolls back the transaction, the data will revert to the previous value and the snapshot that Transaction B has will now be erroneous, or dirty.

  • Nonrepeatable reads— These are reads that are not repeatable for reasons other than a transaction rollback. If a row is deleted, for example, subsequent reads will not see that data.

  • Repeatable reads— When a repeatable read is specified by the transaction, the database guarantees that the data will be the same, no matter how many times a transaction reads it.

  • Phantom reads— Let's say that a row is inserted into a table by Transaction A after Transaction B starts. If the row is visible to Transaction B, this is called a phantom read.

You can see that, if Transaction B is allowed to read the table at this point, it will have done a dirty read. Whether it will be able to read the table before Transaction A has finished with it depends on the isolation level specified by both Transaction A and Transaction B.

The nature of the transaction should determine the isolation level that is appropriate. Top-level RDBMSs allow the transaction to set the isolation level. Enforcement of the isolation levels is done via locks. There are four kinds of locks:

  • Optimistic concurrency— This is not really a lock. When optimistic concurrency is specified, no locks are placed on the data. The transaction is hoping that the data will remain the same throughout the transactions. Unless specified differently, a JDBC driver will default to optimistic concurrency.

  • Read locks— Read locks prevent other transactions from changing the data during the time of the read. Other transactions may read the data, but not change or delete it. The transaction placing the read lock can't change the data, either.

  • Write locks— Write locks are placed when the transaction desires to change the data. Once more, other transactions may read the data, but not change it. However, the transaction placing the write lock is free to do what it will with the data. A write lock allows dirty reads not only by other transactions, but also by the transaction that placed the lock.

  • Exclusive write locks— Exclusive write locks are also used when a transaction desires to change the data. However, when an exclusive write lock is applied, other transactions can't even read the data. With some databases, the transaction that placed the exclusive write lock can't even read the data until the transaction is ended. Exclusive write locks will completely prevent dirty reads by other transactions. When the exclusive write lock also applies to the transaction placing the lock, not even it will get a dirty read.

What will happen to Transaction B depends on both the locks applied by Transaction A and itself. If Transaction A has applied an optimistic lock, Transaction B is free to apply any sort of lock it wants. However, if Transaction B applies anything higher, including a read lock, Transaction A will be prevented from changing the data until Transaction B releases its lock. But Transaction A wouldn't do that, because it plans to change the data in the Key123 row. So it will apply either a write or an exclusive write lock on the Key123 row.

The fact that Transaction A has applied a write or an exclusive write lock limits what Transaction B can do. If it wants the capability to write to the Key123 row, it will have to wait for Transaction A to finish. But let's say that it just wants to do a read. It will use either an optimistic or a read lock. If Transaction A has put a write lock on the Key123 row, Transaction B will be able to do what could potentially be a dirty read. If Transaction A has put an exclusive write lock on the row, Transaction B won't even be able to see the data.

Transaction Control

Transaction levels are what describe the safety level of a transaction. The net effect of the locking and the isolation level desired determines the transaction level applied. There are four transaction levels:

  • Read uncommitted— The transaction can read uncommitted data. Dirty and nonrepeatable reads are allowed. Phantom reads are usually possible, too.

  • Read committed— At this transaction level, uncommitted data cannot be read. This prevents dirty reads, but phantom reads are possible.

  • Repeatable reads— If a transaction has a repeatable read isolation specified, the database guarantees that no other transaction can change that data. Therefore, dirty and nonrepeatable reads are prevented. Phantom reads can still occur.

    How this is accomplished can vary among database vendors. If Transaction B specifies a repeatable read, some vendors will place a lock on the row that B is interested in. So, if Transaction B has specified that it wants a repeatable read, it would have to wait until Transaction A completes. This is because B is requesting a lock, and the database cannot honor that until Transaction A lets go of the row. Rather than place a lock on the data, some databases will just always return the same value each time it requests a read. This means that, if Transaction A rolls back its change, Transaction B will consistently get erroneous data. However, the read will be repeatable.

  • Serializable— Serializable is the most restrictive level of transaction isolation. This transaction is isolated from any changes to the data. It has exclusive read-write access to the rows. No other transaction can even read the data, much less write to it. As can be expected, setting an isolation level of serializable will have to be approved by the database administration staff.

This discussion covers the isolation levels outlined in SQL92. Different database vendors implement locking in different ways. In addition, the level of support for the different transaction and locking levels can vary from database vendor to database vendor. For example, Oracle 9i supports only Read Committed (the default) and Serializable transaction isolation levels. SQL Server 7 supports all four isolation levels.

If database access is through an Open Database Connectivity or JDBC driver, the level of isolation support provided by the ODBC or JDBC driver is another variable. Optimistic isolation can be assumed—nothing else can.

Locking Mechanisms

A final variable is the exact mechanism that the RDBMS vendor uses to apply locks to the data. With applications that have to support a high number of transactions per hour, you will have to research the locking mechanisms used by the vendor. In particular, when a transaction level of repeatable read or serializable is specified, does the database lock the entire table or tables involved (remember, an SQL statement will usually bring back data from several tables that have been joined), or does the database just lock the rows involved? Exclusive table locks are not desirable, and most database vendors have done the work necessary to support row-level locking. When in doubt, check with the database manufacturer.

This just brushes the surface of the issues that are involved in database access in a multi-user environment. Further discussion is beyond the scope of this chapter.

As we progress to contrasting XML with RDBMSs, you should be aware that much time and effort has been expended by RDBMS vendors on concurrency control, and the solutions they have developed are well-tested and reliable. It is misleading to compare the level of polish and capability that one of the top-tier industrial-strength RDBMS vendors can bring to the table with the current state of the XML language in general, and XQuery in particular.

Importance of SQL

Perhaps the thing that has been most important for the growth of RDBMS has been the development of a declarative, vendor-independent language, Structured Query Language (SQL), to carry out CRUD operations against the data store. SQL is declarative in that the user declares what she wants to have happen, and the RDBMS is free to produce that result in any way it sees fit. Here is an example of a SQL statement:

SELECT f_name, l_name, age AND ssn from personal_data WHERE empID = 12345

The user has declared that she wants to know the first and last name, age, and Social Security number for the employee whose employee ID number is 12345. The actual physical storage mechanism the database uses and the method of reassembling the data is left entirely to the RDBMS manufacturer.

You will note from the previous example that the user needs to have some knowledge of the way the data is organized (the schema) in the database. She must know the exact, properly spelled name of the table, and the exact, properly spelled names of the columns in the table. Recently RDBMSs have added native language (usually English) query capabilities. Here is an example of an English language query:

give me all of the brake shoes for 1977 Toyotas, order by year and then by price

Even though the user has asked for data as though the tables have columns such as year and price, there is no necessity for the data in the database to be organized in that way. All that is necessary is for the native language query facility to understand the translation from the actual database schema to the meaning of the words in the user's query.

Native language queries totally remove the necessity of knowing anything about the schema from the user. This is as it should be. In object-oriented terminology, the native language query facilities could be thought of as fitting into the Command Pattern, one of the reusable object-oriented recipes from the seminal book Design Patterns: Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides (Addison-Wesley). The query facility has commands that know the database schema, and it has knowledge of where the data is and what commands are appropriate to find it. With natural language queries, database operations have become both vendor- and schema-neutral. The power of this approach cannot be overestimated.

All of these systems have been well-worked out and are extremely reliable. Although issues such as maintaining relational integrity is complex, at this point, they can be considered bulletproof. The RDBMS paradigm is a very powerful approach to persisting data, and it has been the dominant data storage model for more than a decade.

XML Paradigm for Data Storage

XML documents, be they data- or document-centric, contain data and the metadata that describes that data, and hence can be considered databases. So, how do the XML and RDBMS paradigms compare?

To frame the debate, the RDBMS paradigm has well-worked out solutions for the following data issues:

  • Data integrity via normalization rules and referential integrity.

  • Indexing and other query performance enhancements.

  • A metadata set that fully describes and supports operations on the data. Metadata is external to the data, described in ways that are proprietary to the RDBMS and is difficult to modify and extend.

  • Transaction management.

  • Security systems as part of the database engine.

  • Concurrency control mechanisms.

  • Declarative language (SQL) for data query and manipulations. This language has support for Create, Read, Update and Delete operations.

  • Support for native language queries.

XML documents have more or less well-worked out solutions for the following data issues:

  • They contain data elements organized in a way that supports data integrity.

  • They carry with them a flexible, easily modified metadata set that is an integral part of the document.

  • They have a declarative language for data querying: XQuery. Currently XQuery fully supports the Create and Read elements of CRUD. Create is supported in two ways. It is possible to create an XML fragment from a larger document using the return part of a For, Where, Let, Order By, Return (FWLOR) statement. However, that is only part of what is wanted in a Create. The Create that is normally thought of creates an entirely fresh document and provides for its durable storage. The full capabilities of Create operations are provided by the new insert and remove XQuery operators. As you would expect, both operators enable you to specify where in a node list the insert should occur. As we discuss later in this chapter, the update part of CRUD is included in the XUpdate extensions to XQuery 1.0.

  • A powerful language for transforming the data in XML documents: XSLT. This is unique to XML; the RDBMS world has no counterpart.

  • A powerful method for presenting the data in XML documents to users: XSL.

Before going further, it must be pointed out that, in a way, comparing XML documents to RDBMSs is an apple/oranges comparison, or at least a peaches/nectarines comparison. On one hand, XML data stands on its own. Its primary purpose is as a vehicle to encapsulate data and its metadata in a platform-, operating system-, and application-independent manner. Persistence is not a design consideration with XML.

Also, because the data in an XML document comes packaged with its metadata, it is possible to create a series of modules that interact with the data and metadata and bring added functionality in a manner that is much more open than what is possible with an RDBMS. It might well be that you will not see a monolithic application such as an RDBMS in the XML space. What is far more likely is that there will be a plethora of modules, some or all of them open source, that accomplish a particular task.

On the other hand, with RDBMSs, the entire purpose is persisting data. The data is buried inside the application in a form that is completely platform-, operating system-, and application-specific. It is impossible to think of the data without thinking of the application that contains the data. The data is a series of bits stored somewhere in one or more files on one or more machines. The Management System part of the relational database management system provides the performance enhancements such as indexing, the transaction management, concurrency control, and security services.

This tightly integrated approach brings a lot of efficiency and power to bear onto data management, but it is a polar opposite to the open standards approach of XML. XML documents in and of themselves will never provide transaction management or concurrency control. One of the major benefits of XML is that it is human-readable, which in many cases conflicts with security. However, there are numerous W3C XML security projects such as XMLKey, XMLSignature, and XMLEncryption that are building the security infrastructure for XML.

Given that XML documents are best thought of as data collections and that persistence of the document is external to XML itself, persistence will become the responsibility of an external module. This means that XML persistence is not constrained to a particular application running on a particular operating system on a particular platform. Developers can choose the persistence solution that best suits the situation. Then, when the situation changes, they can extract their XML and move to a different persistence mechanism.

Persistence Strategies for XML

It is now time to discus how to best persist XML documents. Currently, most of XML data is stored in the filesystem as ordinary text files, or on the Internet and intranets as part of HTML pages. The XML data from these files can be easily loaded into and worked with by a variety of tools, such as XSLT, XPath, and XQuery. These resources represent the raw data from which information can be mined. Their persistence mechanism will remain as it is: Web pages and files on a filesystem.

For many applications, storage of data on the filesystem is an adequate solution.

It is possible for data storage on a filesystem to deal with the following RDBMS issues:

  • Backup— If the files are on a distributed filesystem such as Windows 2000 Server DFS, or UFS on Unix, it is possible to provide for automatic backup of an XML file store to another drive or any computer on the network. This means that the backup computer can be located in a different building or city. Therefore, XML data on the filesystem can be protected by a disaster recovery plan.

    If DFS is not available and backup of the files is a necessity, a third-party tool will have to be bought or a solution developed. If you decide to go that direction, it is important to do thorough testing to make sure that the backup system can be trusted.

    Rather than the nightly backup that requires the database to go offline that some of you might be familiar with, modern RDBMSs have honed the backup process to the point that they offer backup systems that are very close to real time. That means that the backup set is identical to or almost identical to the working data set.

    If your XML data requires that degree of congruence between the data and the backup, you will have to make sure that your files are on a filesystem with a RAID (Redundant Array of Inexpensive Disks) level 1 or greater.

  • Security— Modern filesystems from some of the OSs provide for very fine-grained control of access to folders and individual files on a filesystem. The Access Control Lists can be set up on roles and groups as well as individual identities. Security can be set so that access by a group or individual is restricted to certain times of the day and of the week. Auditing can be set up to track who accesses what and when they did it.

    In reality, the security provided by a modern filesystem will offer as comprehensive a level of protection for your data as provided by an RDBMS.

After you get past static data in Web pages and into using XML as a data vehicle as it is used by Web Services, you will be forced to conclude that persistent storage as text files in a filesystem is unsatisfactory. It is unsatisfactory for the following reasons:

  • Data integrity— There is no built-in mechanism to ensure data integrity. Text files are easily corrupted either by accident or maliciously. If the data is vital to an enterprise, persistent storage of XML data as text files in a filesystem is much too risky.

    We have already mentioned that, although certain filesystems do offer a level of backup for the files, it is not industrial-strength enough for most enterprise data needs.

  • CRUD operations— For XML documents to be useful as data, it will be necessary to be able to carry out CRUD operations against them. Having the data on a filesystem in hundreds or perhaps tens of thousands of separate files effectively precludes operations on the data as a whole, or even on a significant fraction of it. Just to begin with, exhaustive search operations will involve the opening and parsing of every file.

    Even operations on a small group of files, such as a purchase master and a series of individual order details would be much slower than a similar operation on a database because of the indexing possible on a database and the many query optimizations that all RDBMS engines provide.

Enterprises that are generating and storing data as XML documents sooner or later are going to need to wrap the data in some sort of database. Currently the choices are the following:

  • RDBMS— All the major RDBMS players offer the capability to accept and store XML. When storing XML in a conventional RDBMS, it will occur by one of the following storage mechanisms: Binary Large Object (BLOB) or its cousin the Character Large Object (CLOB), shredding, or some sort of "native XML" format. Everybody can do BLOBs/CLOBs and shred. Currently, of the "big four" RDBMSs, only Oracle and DB2 offer "native XML" storage.

  • XML database— The term XML database refers to a database management system that will give you storage of your XML documents totally in XML. Several of the players in this space offer Object Data Base Management Systems (ODBMSs) and have leveraged their expertise in the storage of objects to produce databases that specialize in the storage of XML, which can be thought of as an object.

  • Both RDBMS and an XML database— Many enterprises have gotten beyond smokestack mentalities and look at the collection of data containment applications that exist in their companies as a federated data system. Using that model, they have moved to create the means of working effectively with heterogeneous sources and stores of data.

XML Structure and its Relationship To Persistence

Let's begin our discussion of encasing XML data in a database by examining in detail the structure of a collection of data in an XML document.

For our first example, we will choose an XML document that was created by a document creation tool and whose prime function is to be presented to humans as something to be read.

As has been pointed out before, an XML document has all its nodes, elements, and attributes arranged in a definite relationship to one another. In other words, the actual structure of the document, as well as the XML tags and the data in the elements, is vital to the meaning of the document.

There are sections, such as the preface and introduction, that belong in the front of the document, and others, such as the index, appendices, and glossary, that belong in the back of the book. In addition, in most documents, the order in which elements such as chapters appear in the body is important. Hence, for a database solution to work for storing document-centric XML, it must preserve the structure of the XML as well as the tags and the attributes and elements that go with the tags.

RDBMS XML Persistence Strategies

How would an RDBMS handle our XML? It could store the XML as a CLOB (for the purposes of our discussion, a CLOB is interchangeable with a BLOB, so, for the rest of this section I will just use the term CLOB). In fact, the first approach RDBMSs took to storing XML was to store it as a CLOB. Storing the XML as a CLOB nicely preserves the structure of the XML. The row that holds the XML object will have a column for a primary key and can have other columns for data such as author, ISBN, and so on, that provide the capability to do a simple search on those of the data elements internal to the XML that the database schema designer chooses to extract as important. The problem with this approach is that the extraction of identifying elements tends to be arbitrary and inadequate. However, because this CLOB is totally opaque to the database engine, querying or searching the data requires that it be read into a DOM and having an external application apply XPath, XQuery, and XSLT on it. For many document-centric XML storage tasks, the CLOB approach is adequate, at least for the short term.

Shredding

It could "shred" the XML. Shredding refers to an automatic mechanism to break XML into what an RDBMS really understands: tables with rows and columns. The tags or attribute names become column names, and the attributes and elements become the data in the row. Primary keys are generated, foreign keys are imported from related tables, and a full relational structure is generated. Shredding does great violence to an XML document. The structure of the document is lost. Usually, attributes are treated the same as elements, so it is impossible to tell the difference when examining the schema of the tables generated by shredding a document. Storing document-centric XML by shredding is a totally inappropriate solution. The data is safely tucked away, and such things as transaction management and concurrency control can be applied by the database engine. The data can be indexed on any column or set of columns desired and queries will execute smoothly and quickly. However, shredding is not suitable for document-centric XML.

The real inadequacy of shredding becomes obvious when you attempt to get the XML back from the database. In many cases, particularly with document-centric XML, it is vital that the XML come back in the exact same structure and form as it went in. That is difficult or impossible to do with shredding.

There are switches that can be applied to the SQL SELECT statement that can give the application some level of control over the XML that is returned from the RDBMS. For example, the following Microsoft SQL Server 2000 SQL statement will return the data in the form of an XML string:

SELECT * FROM books WHERE isbn = ""  FOR XML AUTO

This statement will give you XML where the column names are the XML tags and the data in the row(s) are the element values. The FOR XML directive is quite useful; it can be used with any SELECT statement on any set of tables. The data in the tables does not have to have come from XML. FOR XML is a wonderful way to pull data from SQL Server 2000 and present it directly to the user using an XSL, perhaps with some XSLT changes.

However, if the data went in as XML and you want to get the original XML out intact—if you are planning to feed the XML into an XSL style sheet, for example—FOR XML will probably not work for you. To begin with, any attributes that went in will come out as elements. In general, it is difficult to predict how processing instructions, comments, and namespaces will be handled.

If you substitute EXPLICIT for AUTO in the previous SQL, you can gain a great deal of control over the returned XML, at the cost of having to explicitly specify the structure of the XML that you want. In other words, the application has to supply the metadata that was embodied in the structure of the XML as input instructions to the query. The lost metadata about the structure of the XML document has to be supplied by the developer. It is true that some of this lost metadata can be supplied by the clever use of a DTD; but if the structure of an XML document is important, as it is in document-centric XML, you should preserve that structure by storing the XML as a CLOB or use an XDBMS.

SQL SERVER AND XML

This discussion has just brushed the surface of the mechanics of using SQL Server 2000 to store and return XML. For more information on this topic, I direct you to the MSDN site at http://www.msdn.microsoft.com/library/default.asp. Go to the SQL Server tab.

As an example of how the structure of document-centric XML is dynamic, let's revisit our workflow XML example. At one stage, the XML looks like this:

<document>
    <workflow>
        <currentstage>preliminary markup</current stage>
        <next stage>preliminary graphics</next stage>
    </workflow>
</document>

After preliminary markup has finished with the XML, it now looks like this:

<document>
    <workflow>
        <currentstage>preliminary graphics</current stage>
        <next stage>final photo</next stage>
        <next stage>final graphics</next stage>
        <stage after that>final text></stage after that>
    </workflow>
</document>

If you are using an RDBMS and the shredding solution to storing XML, the schema of the WORKFLOW table will have changed from the first instance to the second. Somehow a new row must be added to the WORKFLOW table. Don't ask an RDBMS that is shredding the data to handle dynamic XML such as this. You will have to move to a native XML solution.

SHOP TALK: IS SHREDDING THE BE-ALL AND END-ALL?

Some will say that there is no problem in the way shredding destroys the structure of an XML Document Object Model (DOM), because arranging elements in the proper order can be accomplished by applying an XSL and some XSLT to the XML that is returned from the shredded tables, and their position in the document is irrelevant. Don't count on XSL to save you if you need your node returned in a specific order, because XPath returns a node set (or list of elements in XPath 2.0) in document order. So, even when using XSL to handle the presentation of the XML document elements, the order in which they appear is important and manipulations of the XML document can't be counted on to preserve that order.

Bottom line: Don't listen to people who airily claim that shredding works for everything. For a few corporations, shredding will work all of the time. For a few businesses, shredding is always inadequate. Locate yourself on this continuum.

One of the advantages of the way XML handles metadata, using it to wrap the data itself, is that it is simple to add or subtract attributes and nodes. Each change in an element's attributes and/or in the list of nodes in the XML means that, if you are using shredding, the database schema will have to be changed. Fortunately, this change will be carried out by the database itself, but still, this will not be a fast operation. Expect database performance to suffer when you are trying to shred rapidly-changing XML into tables.

We do not recommend shredding for document-centric XML. In spite of the fact that your database people might be totally confident that shredding will work just fine and that they can handle any inconsistencies, do not go down this road. You will most likely regret it.

Native Storage

DB2 offers an XML Extender add-on for Universal Database (UDB) 7.2. This add-on is designed for the management of DB2 interactions with XML. This XML Extender brings the following to the table:

  • Shredding— The Extender is the module that performs the shredding of the data in XML into RDBMSs. Shredding is performed under control of the developer using mapping files written in a mapping language.

  • Native XML storage— The XML Extender provides the capability to store the XML document in a "native form" in the database. It is difficult to penetrate the exact storage format of this "native XML," but it is clearly some sort of enhanced CLOB.

  • Indexing— DB2 does not offer the node-level indexing of an XDBMS, but it does allow for the splitting of elements and attributes upon which to place indexes out into what it calls side tables. These side tables are normal RDBMS tables, so searching them will be very fast. The results of the search are used to locate the desired XML document. If the attribute or element that you are searching on is not in the side table, the search engine will have to do some sort of search on the entire document store. Changes to what is indexed in the side tables probably will not be painless.

  • Updates— The XML Extender offers the capability to do updates on attribute and element data. Updates are performed by a user-defined function.

  • User-defined functions— Users can define their own functions to store the XML, extract data from it, or update the XML object. These functions can be used in SQL statements and the database engine will understand them.

  • Text searches— If another DB2 extender (the text extender) for all has been installed, it is possible to perform searches of the XML for words or phrases.

  • Location paths— DB2 offers a facility called location paths. These are XML expressions that point to elements or attributes in the XML object. The syntax used by location paths looks identical to XPath, but the term XPath is not used, and no claim is made that location paths adhere to current W3C standards. No claim is made that IBM has any plans to come into compliance with W3C standards in the future, either.

    These location paths are used by other elements in the XML Extender package, so that puts constraints on them that preclude full implementation of the W3C XPath standards.

    No mention is made of XQuery as an add-on or as something that is under development.

  • Validation— Using the Extender, DB2 can consume DTDs and use them to validate XML input into the database.

DB2 has achieved what looks like an 80/20 solution for persisting XML. It has close to 80% of the functionality that one would desire using a simple, straightforward approach:

  • The side table idea provides useful indexing while still allowing the storage of the XML in a native form.

  • Storage of the XML in CLOB form preserves the dynamic nature of XML; that is, it allows for the structure of the XML to change without incurring the overhead involved when a shredded table schema has to change.

  • DB2 enables CRUD functionality through the use of user-defined functions.

All in all useful, however much XDBMS functionality has been left out. Depending on your application, this might be a satisfactory solution. However, once given a functionality, user needs tend to expand. It is easy to envision situations where this minimalist approach will be inadequate.

Oracle first started offering XML support in 8i, v8.1.7. That came in the form of the XML Development Kit (XDK). The Oracle XDK is a set of components, libraries, and utilities that provides the following:

  • An XML parser.

  • An XSL processor and XSLT transformation engine.

  • An XML class generator.

  • Transviewer Java Beans. This set of Java Beans provides easy integration of XML into client-server applications. The set of beans includes DOM Builder, XSLT Transformer, Treeviewer, XMLSourceView, XMLTransformPanel, DBViewer, and DBAcess Beans.

  • The XQSL servlet. This useful object can produce XML from the results returned by an SQL query. In addition, it can use an XSL style sheet to transform the XML on the server for presentation to the client.

The XDK is used by 9i Release 2 as well. An XML Schema processor and an XML SQL Utility (XSU) have been added.

Oracle 9i, Release 2 has a very strong suite of XML tools. It offers storage of XML in the database in two forms. One it calls LOB, the second Object-Relational.

The LOB storage model maintains complete fidelity to the original XML, including whitespace.

The Object-Relational storage model is an extension onto the basic RDBMS that maintains DOM fidelity while breaking the data into tables. DOM fidelity includes such things as

  • Distinguishing between attributes and elements.

  • Preservation of comments, processing instructions, and namespaces.

  • Support for non-SQL data types, such as Boolean.

  • Support for W3C XML data model.

  • Support for XML constraints that are not supported in SQL, such as strings constrained to be regular expressions.

9i Release 2's native XML support is provided by the following modules:

  • XMLType— This is a data type that is used to store, query, and retrieve XML documents.

  • SYS_XMLGEN— This is a function that can be used to create XML documents from SQL queries.

  • SYS_XMLGEN— This is the database package that does the work mandated by SYS_XMLGEN to create XML from SQL.

  • SYS_XMLAGG— This is an SQL function that is used to aggregate multiple XML documents.

  • URI support— This enables the use of URIs to retrieve references that are global and/or between databases.

  • Text support— This is the functionality that provides XPath queries into XMLType and text columns (CLOBs).

In actuality, what Oracle has done is tweak the CLOB type into something that is tailored for XML document storage and then enhanced its text search engine to provide XPath searches on XMLType and CLOB columns. But native XML storage is still basically in a kind of CLOB.

In addition to XPath statements, Oracle has added XMLType functions such as Extract() and ExistsNode(), and the capability to do indexing on XMLType through the Oracle Text engine.

Oracle also offers a prototype version of an XQuery module that works with 9i Release 2, indicating that Oracle has committed itself to supporting XQuery.

Oracle's XQuery support involves a set of Java extensions called JXQI. JXQI is used to submit

XQueries to 9i Release 2.

Here is a sample code of performing an XQuery using JXQI:

[View full width]

XQueryContext xCtx = new XQueryContext();

// We want a list of purchase order nodes String query = "for $i in document("shoeOrder
  .xml)/PurchaseOrder return $1;

try {
    // Read in the query
    Reader fileString = new StringReader(query);

    // Prep XQuery and run it
    PreparedXQuery prepXq = xCtx.repareXQuery(fileString);
    XQueryResultSet xqRs = prepXq.executeQuery();

    // Do the usual thing with the result set
    while(xqRs.next) {
        XMLNode node = xqRs.getNode();
        ... // do something with node
    }
    catch(XQException xqe) {
    ... // Do the right thing
    }
    catch(Exception e) {
    ... // Do the right thing here too
    }

This example demonstrates that JXQI enables Java developers to work with XQuery using familiar JDBC-like method calls. Conversely, JXQI forces XML developers to move out of familiar XSLT and work with Java syntax.

MORE ABOUT IBM AND ORACLE SUPPORT FOR XML

For more information on DB2 UDB and the XML Extender for it, we refer you to http://www-3.ibm.com/software/data/db2/ for DB2 in general, and to http://www-3.ibm.com/software/data/db2/extenders/xmlext/index.html for information on the Extender.

Oracle's XML support is examined in detail in Chapter 8. For further information on the XML features of Oracle 9i Release 2, see the following URL: http://otn.oracle.com/tech/xml/xmldb/content.html.

In addition, JXQI is, at this time, a set of proprietary, Oracle-specific extensions. The code definitely will not port to another RDBMS. A proliferation of proprietary, nonportable XQuery extensions is not what we need. However, we predict that JXQI will appeal to a lot of existing Oracle and Java shops as they move into the world of XML.

Oracle has built extensive support for native XML into 9i Release 2. It sets the bar as to what an RDBMS can deliver in terms of native XML support. SQL Server and DB2 will be compared to 9i Release 2's functionality.

XML Databases

Storage in native XML is the solution of choice for document-centric XML. When XML is stored in native form, it is put into persistent storage intact.

XML database management systems (XDBMSs) are one way to store XML in a native form. The XML is stored in the form of a DOM and retrieved via a reference to it in the root of the database container. Those of you familiar with object databases will recognize this approach; this is the way object-oriented database management systems handled objects. This is entirely appropriate, because XML, from a persistence point of view, is very analogous to objects.

XDBMSs offer the following for XML management:

  • Node-level access to the XML.

  • All W3C standards are implemented— This means that XPath and XQuery are fully supported. Hence, XML access will be completely familiar to anyone who works with XSL, XPath, and XSLT.

  • Greatly improved performance versus storing the XML as a CLOB—XDBMSs have the capability to select and work with subsets and fragments of an original document. If the document is stored as a CLOB, the entire object will have to be read into memory every time you want to do anything with it.

    So, in the development environment, where the machine is only serving one client and the XML is short and simple, performance will be fine. When the application is deployed, hundreds of users will be working with large and complex XML, and performance will rapidly become unacceptable. If that happens to you, bring in an XML database, deploy the application on it, and do a comparison.

  • Indexing— XDBMSs support indexing on individual nodes. Indexing is the greatest performance enhancer for queries available. Developers that are used to using XPath to bring data out of an XML DOM accept the delays involved and have no idea how much faster operations of any complexity can be when run against an XML store that is indexed.

    In addition to node indexing, it is possible to index on individual words or phrases.

  • CRUD operations are supported— All XDBMSs will have a set of tools that enable CRUD operations against the XML. Updates can be of nodes or groups of nodes that satisfy an update expression.

  • Concurrency and transaction control are provided by XDBMSs— Some give you stored procedures operating on the XML from within the engine. The data can be partitioned across several machines for increased performance. XSLT transformation engines are part of the core database functionality. This promises to speed XSLT transformations considerably.

  • Flexible database schema— XDBMSs use schema. That means that the database engine will enforce rules on the data that is in the elements and attributes and on the overall structure of the XML.

    XDBMSs can consume DTDs or XML schemas provided by the developer. If no DTD or XML schema is provided, the database will generate a mapping schema of its own. When the XML is read from the persistent store, it can also return the input DTD or XML schema or provide the schema that it generated.

    Flexible database schema are the core of providing support for active data dynamic business processes.

  • Connectors to RDBMSs and mainframe systems— XDBMSs provide connector modules that allow the interfacing of XDBMSs with legacy systems, most importantly with RDBMSs. A well executed connection architecture can enable near seamless integration of data in an enterprise, no matter where it resides. XDBMSs are well aware that they will be a minor part of the data repository picture for an enterprise. Hence, XDBMS vendors have spent time and energy to develop the means to play well with others.

XDBMSs are not the only choice when it comes to the native storage of XML. DB2 and Oracle have both made claims that they have the capability to store XML in a native form inside their databases.

Persistent Storage of Data-centric XML

The preceding discussion was on document-centric XML. Document-centric XML is best thought of as active data. It is shared, updated, audited and extended often throughout its life cycle. It has a dynamic schema. The schema and data is administered by the users. In some cases, the XML lasts only as long as the business process.

Now let us turn to data-centric XML. Data-centric XML is not shared, updated, and audited. The schema is static, not dynamic. A common example of data-centric XML is a purchase order.

XML is becoming the paramount message type for the abstraction of data from objects and the transmission of that data between applications. The self-describing nature of XML automates the encapsulation of the data into the message and the extraction of the data from the message. E-commerce is an example of an area that extensively uses XML for data transmission.

The SOAP message protocol used in Web Services is XML. Enterprise integration tools such as IBM's MQIntegrator or Mercator Software's Mercator Integration Broker use XML extensively in the messages that enable legacy applications to exchange data. When developing a new application to connect to a legacy application, XML into the integration tool is the message of choice. It is easy for the integration tool to convert the data in the XML into the legacy format. The trip back from the legacy application sees the legacy message converted into XML before being sent on to the new application. Many enterprises have chosen to standardize on XML as the message format between inter-enterprise application, as well as for B2B messaging. Currently there is a lot of data moving across networks of all sorts as XML.

The storage requirements for data-centric XML are very different from the storage requirements for document-centric XML.

Shredding is perfectly acceptable for data-centric XML. A RDBMS is, in most cases, the preferred persistence repository for data-centric XML. With data-centric XML, the XML is just another message format. The data that it contains, along with the metadata specified in the tags, is all that is of interest. At this level, data is data. The main things that are desired are to prevent any sort of corruption of this data, to be able to access it quickly, and to be able to perform CRUD operations against it using a familiar language: SQL.

The transaction management, concurrency, security features, and so on of an RDBMS are naturally synergistic with data-centric XML. In addition, the capability of indexing of relational tables, along with the well-developed query optimizers, make an RDBMS a strong contender, performance-wise.

In addition, XML, in conjunction with XSL style sheets and XSLT, is a natural combination for the presentation of data in browser-based applications. To illustrate, a common browser action is to go to a database and come back with a list of items to fill an HTML Select control. An extremely powerful way to do this is to use an XSL style sheet on the XML. If the query that extracts the data from the database returns XML, the HTML Select control on the Web page can be quickly and easily created and populated by feeding the XML to an XSL style sheet, either on the server or in an XSL-aware browser.

Native XML databases have evolved their indexing and search functionality to the point where they are strong considerations for storage of data-centric XML, as well as document-centric XML. Unlike with document-centric XML, it is impossible to make any hard and fast recommendations as to which is the best solution for data-centric XML persistence and management, R or X DBMS. In most cases, your particular application requirements will dictate which is the best choice.

SHOP TALK: DOES IT REALLY MATTER IF YOUR DATABASE IS RELATIONAL OR XML-BASED?

Some people will argue that the RDBMS versus XDBMS argument is bogus. Object-oriented architectures emphasize decoupling the various parts of applications by encapsulating the desired action in an object and by presenting the action as part of an interface. The classic example is the situation where a Boy object commands a Dog object to bark. In OOD terms, the Boy object has called the bark() method on the Dog object. The Boy doesn't care how the Dog object carries out the bark(). The Dog object is free to do the barking herself, hire another dog to do the barking, play a tape recording of a dog barking, whatever. In an analogous fashion, object-oriented developers will argue that, when they command the object that is the front for the persistence functionality to save an XML document, they don't care how that is accomplished. RDBMS, XDBMS, or something else—it is immaterial, they argue. In other words, the persistence functionality has become so abstracted by the OOD paradigm that, as far as the developer is concerned, the debate between the various persistence solutions has become moot. However, on the database side, the complications of trying to force the round XML peg into the square RDBMS hole can be so costly in terms of time and money that it is impractical. Beware of sweeping generalizations in either direction.

Although the debate between RDBMS and XDBMS might be immaterial to object-oriented applications, it is still very important to the individuals charged with actually carrying out the persistence functions that have been encapsulated behind the persistence layer interface. These people need to think long and hard about exactly how the persistence store is going to carry out the data storage and data manipulations that have been mandated by the application. So, although an object-oriented developer can well skip this section, the DBA that is tasked with coming up with a storage solution that will work, scale, and perform needs to consider the points raised here.

The best model to use going forward is to move beyond the either/or of RDBMS/XDBMS and to think in terms of federated data sources. The federated data model enables each data source to do what it does best. Communication mechanisms between federated data systems have been well worked out. The Internet and Web Services have destroyed the idea of a central data store. Data has been decentralized across the enterprise, and in the case of B2B, across enterprises. The lingua franca of the modern data communication is XML, and the navigation method is up and down its parent-child relationships. The methods of exploration are XPath and XQuery.

The RDBMS community is well aware that XML presents a set of challenges that will force them to extend the envelope of their thinking in entirely new ways. For example, XML has brought back the concept of a data set. An XML document is nothing more than a data set with metadata. The data set is familiar to anyone who has worked with IMS, or who has worked with ODBMSs. Concepts such as these will have to be an integral part of any effective XML storage solution, and RDBMSs are well aware of that. But the XML paradigm is a fundamental shift, and incorporating it into the RDBMS framework will not be simple or straightforward.


XQuery Limitations

XQuery is a new specification, and it has several limitations. Some of them, such as the lack of CRUD capabilities and the need for GUI tools to make working with XPath and XQuery easier, have been addressed. Some of them, such as the limitations of XQuery as a declarative language, are just being discussed.

XQuery As a Declarative Language

We have previously discussed the constructs of the Semantic Web. To use all the systems elaborating the Semantic Web will require a declarative language. As mentioned earlier, declarative languages enable the user to declare what he would like to see happen without having to describe exactly how it is to be accomplished. Declarative languages exist on a spectrum of abstractive power. SQL is considered a declarative language, yet the creation of a SQL statement requires knowledge of the underlying data model. In the case of SQL queries that involve the joining of several tables and the extraction of particular data elements from the tables, a great degree of knowledge of the data model is necessary. To increase the declarative power of SQL, "English" or "natural language" facilities have been developed. These extensions understand the underlying data model and enable the user to declare in very simple language what information he wants. The natural language facility figures out the tables to join and the data items to ask for.

XQuery is a fairly high-level declarative language. However, XPath, as it is currently implemented, is not nearly as high-level. XPath usually requires the user to have an intimate knowledge of the contents, if not the structure, of the XML document(s) being queried.

To make XQuery more declarative, we are suggesting that XPath be extended to handle constructs such as this:

for $c in document(www)  // www tells processor to search
                         //the World Wide Web

XQuery, as it goes forward, should have the capability to consume RDF triples and use ontology layers. XQuery's facility for user-defined functions could be used to accomplish this. This extension of declarative power is a natural and logical fit to XQuery. The final object of XQuery development should be a human language front end similar to the natural query facilities offered by most RDBMSs. That will allow such queries as

"Get me all of the products named 'shoes' where shoes are 'brake shoes'

GUI XQuery Tools

Current XQuery and XPath statements are cumbersome, verbose, and not always easy to read. Tools for the GUI generation of XQuery statements are a requirement if XQuery is to move into the mainstream of enterprise development. XML Spy (from Altova Gmbh) and Stylus (from ObjectStore) are two tools that have already started moving into this space. It is expected that the open source community will produce a serviceable XQuery generation tool, too.

XQuery and CRUD

The current XQuery specification does not specify the Update part of CRUD capabilities for XQuery. When it comes to querying the World Wide Web, we could argue that the Create, Update, and Delete parts of CRUD make no sense. However, after that data is retrieved, it is best organized into XML documents. An XUpdate specification has been published that fills that gap.

XUpdate provides for the full spectrum of services. Query functionality to select the nodes to change, update, insert, and delete (both before and after the context node). For further information on XUpdate, see http://www.xmldb.org/xupdate.

After XML has been persisted as XML, there currently are two means of querying it. The first is XSQL, which has full CRUD capabilities; the second is XQuery, which is lacking the Update capability. Microsoft has a similar tool in SQLXML.

Oracle Magazine describes XSQL in detail in a recent article at http://www.oracle.com/oramag/oracle/01-jan/index.html?o11xml.html. XSQL is an important alternative to XQuery because there will be a natural inclination for application developers and the individuals in charge of managing the persistent XML stores to use the tool that is most familiar to them: XSQL, the XML extension to SQL. We predict that content providers and Web application developers, on the other hand, will gravitate to XQuery because it uses a familiar metaphor: XML. To prevent a split in the tools to work with XML that will substantially increase development and maintenance costs and greatly complicate the natural development of the XML language, XQuery will need to evolve to include full CRUD capabilities. When XQuery has reached that point, both application and content developers should be using it.

Although it is true that CRUD operations become possible if you wrap your XML documents in a database management system, for full development of the XML language, it is important that XQuery have the capability to update existing XML documents.


In Brief

  • XML is important in both document markup and as a data transmission media.

  • XML tagging will make the Web a machine-readable P2P network that will support an unprecedented level of integration.

  • The Semantic Web project is developing mechanisms to publish the additional metadata required to make machines-to-machine communication 100% accurate.

  • The integration of XML into the publishing paradigm has brought a much greater degree of automation and efficiency to the process. However, the XML produced by these applications and other document-centric applications is more complex to handle and persist than the data-centric XML from such applications as Web Services.

  • XML documents are much more than data, and they don't fit particularly well with RDBMSs. However, both the data-centric and document-centric XML worlds share a common agenda. Also, some RDBMSs are developing extensions that make them much more capable XML storage mediums.

  • XQuery, as it is presently implemented, has limitations as a declarative language that will need to be addressed.

  • The current XQuery specification doesn't provide for the Update part of CRUD operations, but the XUpdate specification will eliminate that objection.


Part III: Using XQuery

10 Using Microsoft XQuery Services

11 Generating XML and HTML Using XQuery

12 Working with Java and XQuery


Chapter 10. Using Microsoft XQuery Services

by James McGovern

IN THIS CHAPTER

  • Microsoft XQuery Services

  • Example

  • Additional Examples

  • XQuery Classes

  • In Brief


Microsoft XQuery Services

Microsoft's XQuery Services demonstration Web site at http://xqueryservices.com contains a set of classes that enables you to use XQuery expressions against a collection of XML documents that were created using the .NET framework. This chapter uses the examples that ship with the XQuery services SDK so you will have the opportunity to try it for yourself.

The class that you interact with most frequently is XQueryNavigator, which contains the result set from an XML data source (collection of XML documents) and allows for programmatic navigation. The XQueryNavigator class exposes several methods, including execute, select, InnerXml, OuterXml, and so on. The execute method is the most used and allows for a well-formed XQuery expression that is specified as part of the XQueryExpression class to execute using a specified collection (XQueryNavigatorCollection). The execute method returns an XQueryNavigator object that contains the results of the query.

A collection in XQuery simply contains an array of XML-based data sources upon which XQuery operations will be performed. The XQueryNavigatorCollection class handles the assembly of the collection.


Example

Let's use Visual Basic .NET to create a sample that uses these classes. To execute the following code, you need to download the .NET SDK from Microsoft's Web site. First, let's develop an XQuery expression that allows you to query for books written after 1991 by Addison-Wesley. For relational database aficionados, here is what the SQL looks like:

SELECT description FROM book
  WHERE publisher = 'Addison-Wesley'
  AND year > '1991'

Now, let's look at the corresponding XQuery expression:

<bib>
 {
  for $b in document("http://www.bn.com/bib.xml")/bib/book
  where $b/publisher = "Addison-Wesley" and $b/@year > 1991
  return
    <book year="{ $b/@year }">
     { $b/title }
    </book>
 }
</bib>

The Visual Basic code in Listing 10.1 executes the previous expression.

Listing 10.1 Visual Basic Sample Code
Imports System
Imports System.IO
Imports Microsoft.Xml.XQuery

Class XQueryTest

    Public Shared Sub Main()

        Dim Args() As String = System.Environment.GetCommandLineArgs
        Dim fileName As String = String.Empty
        Dim query As String = String.Empty
        Dim fileAlias As String = String.Empty
        Dim i As Integer

        For ii = 1 To Args.Length - 1
            Dim arg As String = Args(ii)
            If arg.Substring(0, 1) = "-" Then
                Select Case arg.Substring(1)
                    Case "f"
                        ii += 1
                        If ii < Args.Length Then
                            fileName = Args(ii)
                        End If

                    Case "q"
                        ii += 1
                        If ii < Args.Length Then
                            query = Args(ii)
                        End If

                    Case "a"
                        ii += 1
                        If ii < Args.Length Then
                            fileAlias = Args(ii)
                        End If

                    Case Else

                End Select
            End If
        Next ii

        Try

            ' Here is where we create a new collection
            Dim col As New XQueryNavigatorCollection()

            ' Add the file obtained from the command line to the collection
            ' It will be referenced by its alias in the XQuery statement
            col.AddNavigator(fileName, fileAlias)

            ' Read the output of the query
            Dim q As String = String.Empty

            If fileName <> String.Empty Then
                Dim sr As New StreamReader(query)
                q = sr.ReadToEnd()
                sr.Close()
            End If

            ' Compile the XQuery expression
            Dim expr As New XQueryExpression(q)

            'Execute the query and print out its result to the console
            Console.WriteLine(expr.Execute(col).ToXml())

        Catch e As XQueryException

            Console.WriteLine(e.Message)
            Console.WriteLine(e.StackTrace)

        End Try
        Return

    End Sub 'Main

End Class 'XQueryTest

The essence of this program is the creation of a new XQueryNavigatorCollection to which bib.xml is added. Compile this sample program using the Visual Basic compiler:

C:\Program Files\XQuery Demo\XQueryTest>vbc.exe XQueryTest.vb
/reference:..\Microsoft.Xml.XQuery.dll
/reference:System.xml.dll

By running this command from a DOS prompt, you should see output similar to the following:

Microsoft (R) Visual Basic .NET Compiler version 7.00.9466
for Microsoft (R) .NET Framework version 1.00.3705.209
Copyright (C) Microsoft Corporation 1987-2001. All rights reserved.

The Visual Basic compiler creates an executable named XQueryTest.exe. You can now execute the XQuery test application as follows:

XQueryTest –f bib.xml –a bib.xml –q 1.xqu

This displays the following results:

<bib>
        <book year="1994">
                <title>TCP/IP Illustrated</title>
        </book>
        <book year="1992">
                <title>Advanced Programming in the Unix environment</title>
        </book>
</bib>


Additional Examples

At the time of this writing, full support for the XQuery specification was not yet realized. You will find that it does support the majority of specifications many developers use. Let's look at several additional queries and their resulting responses.

The following examples use W3C sample queries and target documents. The samples are available from http://xqueryservices.com and are also contained in the .NET SDK.

First, here's an SGML XQuery expression:

<result>
  {
    for $s in document("sgml.xml")//section/@shorttitle
    return <stitle>{ $s }</stitle>
  }
</result>

The following is the SGML XQuery result:

<result>
        <stitle shorttitle="What is markup?" />
        <stitle shorttitle="What is SGML?" />
        <stitle shorttitle="How does SGML work?" />
</result>

Next is a namespace XQuery expression:

declare namespace dt = "http://www.w3.org/2001/XMLSchema"

<Q3>
  {
    document("auction.xml")//*[@dt:*]
  }
</Q3>

And here's the namespace XQuery result:

[View full width]

<Q3>
        <ma:Open dt:type="timeInstant" xmlns:ma="http://www.example.com/AuctionWatch"xmlns
  :xlink="http://www.w3.org/1999/xlink"xmlns:anyzone="http://www.example.com/auctioneers#anyzone"
xmlns:eachbay="http://www.example.com/auctioneers#eachbay"
xmlns:yabadoo="http://www.example.com/auctioneers#yabadoo"
xmlns:dt="http://www.w3.org/2001/XMLSchema">2000-03-21:07:41:34-05:00</ma:Open>
        <ma:Close dt:type="timeInstant" xmlns:ma="http://www.example.com/AuctionWatch"
   xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:anyzone="http://www.example.com/auctioneers#anyzone"
xmlns:eachbay="http://www.example.com/auctioneers#eachbay"
xmlns:yabadoo="http://www.example.com/auctioneers#yabadoo"
xmlns:dt="http://www.w3.org/2001/XMLSchema">2000-03-23:07:41:34-05:00</ma:Close>
        <ma:Open dt:type="timeInstant" xmlns:ma="http://www.example.com/AuctionWatch"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:anyzone="http://www.example.com/auctioneers#anyzone"
xmlns:eachbay="http://www.example.com/auctioneers#eachbay"
xmlns:yabadoo="http://www.example.com/auctioneers#yabadoo"
xmlns:dt="http://www.w3.org/2001/XMLSchema">2000-03-19:17:03:00-04:00</ma:Open>
        <ma:Close dt:type="timeInstant" xmlns:ma="http://www.example.com/AuctionWatch"
   xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:anyzone="http://www.example.com/auctioneers#anyzone"
xmlns:eachbay="http://www.example.com/auctioneers#eachbay"
xmlns:yabadoo="http://www.example.com/auctioneers#yabadoo"
xmlns:dt="http://www.w3.org/2001/XMLSchema">2000-03-29:17:03:00-04:00</ma:Close>
</Q3>

The W3C XQuery specification does not state how to clearly handle conflicts between namespaces in the input documents and namespaces declared within the queries. Namespaces defined within the input document are also supported in the output document. Namespaces that are declared within element constructors are not allowed.


XQuery Classes

Now that we have working examples, let's dive into the guts of the Microsoft XQuery classes and see some advanced features. The first thing to note before proceeding is that the .NET framework has a built-in extension point for extending XML related classes: System.XML. All XQuery-related classes are derived from one class that allows for future points of integration.

One example is the XQueryDocument class. This class can be created using an XmlReader. Likewise, the XQueryNavigator class you used in the earlier example can be created using the XQueryNavigatorCollection class. Multiple classes are used in an XQuery solution, but the five that are most widely used are the following:

  • XQueryDocument— Creates an XQueryDocument that encapsulates another XML document and uses a URI as its constructor.

  • XQueryNavigator— Embodies an XML document that has navigational features. Remember, the XQueryDocument class is solely responsible for providing an efficient method to store XML documents.

  • XQueryNavigatorCollection— Adds an XML document referred to by a URI to the XQueryNavigatorCollection. Note that this class cannot be used to load XML document fragments. If fragments exist, they must be loaded into an instance of an XmlReader to create a full XQueryDocument.

  • XQueryExpression— Encapsulates an XQuery query and either returns a result or throws an XQueryException.

  • XQueryException— If an exception occurs while processing an XQueryExpression, an XQueryException is created that contains the line number at which the error occurred.


In Brief

  • Microsoft is fully committed to supporting the W3C XQuery specifications and incorporating future implementations into the .NET framework.

  • Microsoft's XQuery demonstration Web site includes classes for working with XML documents developed with the .NET framework.

  • The .NET framework has a built-in extension point for extending XML related classes: System.XML. All XQuery-related classes are derived from one class that allows for future points of integration.

  • By building XQuery support into the .NET core, Microsoft will be one step closer to allowing systems to query metadata from heterogeneous systems in a seamless manner.


Chapter 11. Generating XML and HTML Using XQuery

by Per Bothner

IN THIS CHAPTER

  • Tools for Generating Web Pages

  • A Photo Album Application

  • Generating the Overview Page

  • Generating the Picture Pages

  • Running XQuery Programs as Servlets

  • Running XQuery Programs from CGI Scripts

  • Comparing XSLT and XQuery

  • In Brief


Tools for Generating Web Pages

There are many tools for generating Web pages. Many of these are based on templates: You write an HTML page, but you can embed within it expressions that are calculated by the Web server. Here are some popular examples:

  • JSP or JavaServer Pages (http://java.sun.com/products/jsp/) is a format in which Java expressions and commands are embedded in HTML pages. A JSP page is automatically translated into a special kind of Java class called a servlet. A Web server can be configured to execute the servlet when receiving Web requests. This is a simple example:

    <p>3*4 is: <%= 3*4%>.</p>

    Evaluating it yields this HTML fragment:

    <p>3*4 is: 12.</p>

  • PHP (http://www.php.net/) is a relatively new scripting language that is especially popular for server applications. Here is the same example:

    <p>3*4 is: <?php echo 3*4 ?>.</p>

  • ASP (Active Server Pages) from Microsoft supports embedded interpreted VBScript and JScript code; ASP.NET (http://www.asp.net) supports other compiled .NET languages. Here is the same example:

    <p>3*4 is: <%= 3*4>.</p>

  • BRL (Beautiful Report Language, http://brl.sourceforge.net/) allows expressions in the Scheme language to be embedded inside templates. BRL is similar to JSP, in that it generates Java servlets. Again, the same example:

    <p>3*4 is: [(* 3 4)].</p>

You can do the same thing with XQuery:

<p>3*4 is: {3*4}.</p>

There is one important difference between XQuery and most template systems: In the latter, you can nest expressions in some programming language inside HTML. With XQuery, you can also nest HTML inside expressions, as in this loop:

for $i in (1 to 10) return
   <p>{$i}*4 is: {$i*4}</p>

With XQuery, you can also declare functions that return HTML fragments, and you can pass those fragments through to other functions. With most template systems, you can only create output fragments as strings, and pass them around as strings.


A Photo Album Application

As an example of how to generate Web pages using XQuery, we will show how to create a photo album application. There are lots of such applications around, and although they differ in features, they all have the same basic idea: You throw a bunch of digital images (JPEG files) at the application, and it generates a bunch of Web pages. The overview page shows many smaller thumbnail images; if you click on one, you get a bigger version of that image.

There are two basic approaches to doing this:

  • A program that generates all the Web pages in advance. This is nice because you can use a browser to view the pages on your local disk using a file: URL, even if you aren't running a server. You can even burn the Web pages onto a CD-ROM and browse the pages from there. Also, you might have an ISP that enables you to put up Web pages, but running server scripts might be an extra-cost option.

  • A module or script added to a Web server that generates the Web pages when requested by a browser. This saves disk space for the HTML files, though that is trivial compared to the disk space for the JPEG images. More valuable is that the server can present the viewer with multiple styles and preferences. This is most valuable if the server resizes the JPEG images as suitable for the viewer. This approach also makes it easier to just upload a collection of new pictures without extra steps.

We will show how you can use XQuery for either approach, starting out with how to generate static Web pages in advance. The software, with usage and installation instructions, is available at http://www.gnu.org/software/qexo/qalbum/.

SHOP TALK

For larger Web applications, it is a good idea to separate out the presentation (Web page layout) from logic (calculating what to present and performing requested actions). This is especially important when different people are responsible for the different aspects of the application. For this modest application, we're more interested in simplicity, but it is still a good idea to put HTML element constructors in different functions than ones that contain non-trivial logic.

For each album, we start with an XML file that contains a list of JPEG image files, with some information about each picture. This XML file can be generated using a script that scans the JPEG files in a directory, or it can be created with a text editor. I use a combination of scripts and manual editing, as described at the preceding Web address, but that's not relevant to our goal of learning and using XQuery.

The following sample index.xml file describes six digital pictures from a car trip to the west coast of Norway in the summer of 2001. Each photograph is described by a <picture> element. Each picture may be available in three resolutions:

  • The <full-image> element gives the name (and size in pixels) of the original full-size image.

  • The <image> element gives the name and size of the image suitably scaled for viewing in a Web page by itself when browsing.

  • The <small-image> element gives the name and size of a thumbnail image.

Listing 11.1 shows a sample index.xml file (abbreviated).

Listing 11.1 An index.xml File Describing Some Pictures
<?xml version="1.0"?>
<group>
  <title>Trip to Norway</title>
  <text>July 28-30, 2001 Nathan, Per, and John took a quick round trip to the
    Norwegian fjord country.  Here are some highlights.</text>

  <picture id="Lillehammer22">
    <caption>Lillehammer: Ski-jump practice</caption>
    <date>2001-07-28</date>
    <text>The Lillehammer olympic 90m ski-jumping hill is used for Summer practice.
      You can see a jumper landing.</text>
    <full-image width="1280" height="960">Lillehammer22.jpg</full-image>
    <image width="740" height="555">Lillehammer22p.jpg</image>
    <small-image width="240" height="180">Lillehammer22t.jpg</small-image>
  </picture>

  <!-- more picture elements as desired -->

  <picture id="ViaValdres2">
    <caption>Valdres valley</caption>
    <full-image width="1280" height="960">ViaValdres2.jpg</full-image>
    <image width="740" height="555">ViaValdres2p.jpg</image>
    <small-image width="240" height="180">ViaValdres2t.jpg</small-image>
  </picture>

</group>


Generating the Overview Page

Our first task is to generate the overview page. We want at most three thumbnails per line, producing something as shown in Figure 11.1.

Figure 11.1. The overview page shows clickable thumbnails of pictures.

This is a nice example that illustrates the kind of nontrivial rearranging that would be difficult to do with a template system or with XSLT, and tedious to do with a language like Java.

Here is the "main program" to generate the overview page:

let $group := doc("index.xml")/group
  return
    make-group-page($group)

This is simple enough: We call the document function to read the index.xml file. This returns the document root, so we do /group to select the top-level group node, which we pass to the make-group-page function, which does the actual work of creating the Web page:

declare function make-group-page($group) {
<html>
  <head>
    {$group/title}
    <style type="text/css">
      a.textual {{ text-decoration: none }}
      table.row {{ padding: 10px }}
      img {{ border: 0 }}
    </style>
  </head>
  <body bgcolor="#00AAAA">
    <h2>{$group/title/node()}</h2>
{   make-rows(0, $group/picture)}
  </body>
</html>
};

The value returned by a call to make-group-page is calculated by an element constructor expression, which looks like normal XML data, except you can use curly braces ({...}) to embed XQuery expressions. For example, the <h2> HTML element contains an XQuery expression that selects the <title> child of the $group parameter, and then selects the children (usually a text node) of that <title> child. Note that double curly braces ({{...}}) in the <style> element are used for curly braces that should appear as is.

Things start to get interesting when we get to the call to make-rows. This is a recursive helper used to divide a sequence of <picture> elements into rows of at most three. It's quite straightforward if you're comfortable with recursive functions. (If you're not comfortable using recursion, it might seem mysterious—but it does work.) The $pictures parameter is a sequence of the <picture> elements we haven't dealt with yet. If the list is empty, we return nothing (the empty sequence ()). If there are three or fewer pictures, we pass them to the format-row function, which makes a single row of thumbnail pictures. If there are exactly four pictures total, we take the first two pictures and pass them the format-row function, which puts two thumbnail pictures in one row, and then we do the same with the remaining two pictures. Otherwise, there are more than three pictures, so we take the first three pictures, put them in a row using format-row, then we take the rest of the pictures (starting at picture 4), and recursively call make-rows. This processes the rest of the pictures in the same way, putting three pictures in a row each time, until we come to the end.

This is the make-rows function:

(: Process a sequence of <picture> elements, grouping them into
 : rows of at most 3, for the thumbnail page.
 : $prev:  An integer giving the number of <picture>s in
 :   this current sequence that precede those in $pictures.
 : $pictures:  Remaining <picture> elements to processes.
 : Returns formatted HTML for the input $pictures.
 :)
declare function make-rows($prev, $pictures) {
  let $count := count($pictures) return
  if ($count = 0) then ()
  else if ($count <= 3) then
    format-row($pictures)  else if ($count = 4 and $prev = 0) then
    (: A special case:  If there are 4 pictures in a row,
    : then group them as  rows of 2 rather than 3 + 1.
    (format-row(sublist($pictures, 1, 2)),
     format-row(sublist($pictures, 3, 2)))
  else
    (format-row(sublist($pictures, 1, 3)),
     make-rows($prev+3,sublist($pictures, 4)))
};

The format-row function loops over a sequence of <picture> elements and calls make-thumbnail on each one. If the <picture> has a <caption> child, it is placed underneath the thumbnail. We wrap each thumbnail plus its caption inside an <a> HTML link wrapped inside an HTML <table>, finally wrapping the entire row in another HTML <table>.

declare function format-row($row) {
  "$#10;  " (: emit a newline for readability :)
  <table width="90%" class="row"><tr>{
  for $pic in $row return
  <td>
   <table bgcolor="black" cellpadding="0" frame="border"
      border="0" rules="none">
      <tr>
        <td align="center"><a href="{$pic/@id}.html">{
          make-thumbnail($pic)}</a></td>
      </tr>
      { if ($pic/caption) then
      <tr>
        <td bgcolor="#FFFF99" align="center"><a class="textual"
          href="{$pic/@id}.html">{$pic/caption/node()}</a></td>
      </tr>
      else ()}
    </table>
  </td>
  }</tr></table>
};

Finally, make-thumbnail looks for a <small-image> child that contains the actual name of the JPEG file, and calls make-img to emit the <img> image link. If there is no <small-image>, we look for a <image> or a <full-image> element, and have the browser scale the image to a smaller size. (In that case, the browser has to download the bigger image, only to throw away most of the detail, so this is not something you want to do normally.)

declare function make-thumbnail($pic) {
  if ($pic/small-image) then
    make-img($pic/small-image, 1.0)
  else if ($pic/image) then
    make-img($pic/small-image, 0.5)
  else if ($pic/full-image) then
    make-img($pic/full-image, 0.2)
  else
  ( "(missing small-image)", string($pic), ")" )
};

declare function make-img($picture, $scale) {
  <img border="1" src="{$picture}" width="{number($picture/@width) * $scale}"
    height="{number($picture/@height) * $scale}" />
};

A Slight Refinement

The code so far ignores any children of the top-level <group> except for <picture>. However, if you look at the original index.xml example, you'll see that the <group> has a <text> child. Let's place any such <text> contents in the appropriate place on the overview page. Another sometimes useful feature is to be able to specify how the pictures are to be organized into rows, rather than depend on the default maximum of three per row. You can do that by editing the index.xml file to place a <row> element around one or more <picture> elements to specify that these should go in a row by themselves:

declare function make-group-page($group) {
<html>
    ... rest as before ..
    <h2>{$group/title/node()}</h2>
{   find-rows((), $group/*)}
  </body>
</html>
};

(: Process the children of a <group>, grouping thumbnails into rows.
 : $pictures:  Sequence of <picture>s that need to be split into rows.
 : $unseen: sequence of remaining children we have not processed yet.
 :)
declare function find-rows($pictures, $unseen) {
  if (empty($unseen)) then make-rows(0, $pictures)
  else
    let $next := item-at($unseen, 1),
        $rest := sublist($unseen, 2)
    return
      typeswitch ($next)
      case element(row,*) return
        (make-rows(0, $pictures),format-row($next/*),find-rows((), $rest))
      case element(date,*) return (: ignore <date> children here. :)
        (make-rows(0, $pictures),find-rows((), $rest))
      case element(title,*) return (: ignore <title> children here. :)
        (make-rows(0, $pictures),find-rows((), $rest))
      case element(text,*) return (: format <text> as a paragraph. :)
        (make-rows(0, $pictures),<p>{$next/node()}</p>,find-rows((), $rest))
      default return
        find-rows(($pictures,$next), $rest)
};

The initial call to find-rows sets $pictures to the empty sequence, and $unseen to the sequence of all the child elements of the top-level <group>. If $unseen is the empty sequence, we're done, and we just call make-row to wrap up the last row. Otherwise, we look at the first element of $unseen.

We use a typeswitch expression to do the appropriate thing depending on the type of that first element. A typeswitch evaluates an expression (in this case $next, the first value of the $unseen sequence). Then it searches through the case clauses, each of which specifies a type. For example, element(row,*) is a type that matches all element nodes whose tag is row, whatever their type annotation. (If you import a Scheme for the index.xml file format and validate the documents, you can get away with writing plain element(row).) The typecase selects the first case clause such that the $next value is an instance of the type, and evaluates the corresponding return expression. If there is no matching case, it evaluates the default return instead.

If the $next value is a <row> element, we first pass any previously seen $pictures to make-row so it can split those into rows, then we pass the children of the <row> element to format-row to create a single row, and then we recursively call find-rows to process the $rest of the sequence. The next two cases are to just skip any <title> and <date> elements, because they are handled elsewhere. The logic for handling a <text> element is similar to that for <row> except that we wrap the contents of the <text> in a <p> paragraph. Finally, the default case handles <picture> elements with a recursive call that moves the $next element over to the $pictures sequence.


Generating the Picture Pages

Now let's look at how we can generate a Web page for each picture, like that shown in Figure 11.2.

Figure 11.2. A Web page for viewing a single picture.

The first tricky part is dealing with the links for the previous and next picture. The other tricky part is that we want to support multiple styles. The existing code supports three styles that each image can be displayed in:

  • "Medium image" is the default style. It displays the image at around 640 pixels wide, which is fine for most screens and browsers.

  • "Large image" gives you a higher resolution, about 1024 or 1280 pixels wide. (For most of the pictures I take, 1280 pixels wide is the original camera resolution—from which you can infer that my camera is a few years old.)

  • The "Info" style shows the thumbnail image, EXIF camera information from the JPEG, and links to the original JPEG files.

Thus we need to generate three HTML pages times the number of <picture> elements. Exactly how to write all these files is somewhat implementation-dependent. There are at least three ways to do it:

  • Write a script or driver program (perhaps a batch script) that loops through the pictures, calling the XQuery implementation once for each desired output file. You need to pass the desired file as some kind of parameter that XQuery can access. The standard function inputs is one way to tell the XQuery program which file to generate. The output from XQuery is redirected to the intended HTML file.

  • Generate all the HTML output files in a single XQuery run, by putting them in a single large XML object:

    <outputs>
  <output-file filename="picture1.html">
    <html>contents of picture1.html</html>
  </output-file>
  <output-file filename="picture2.html">
    <html>contents of picture2.html</html>
  </output-file>
  ... and so on ...
</outputs>

    It is then easy to write a post-processor to split this into separate XML files.

  • Generate all the HTML output files in a single XQuery run, but use an implementation-specific function to write each HTML separately. Although nonstandard, it is the simplest and most efficient approach, so we will use it here.

The Qexo (http://www.gnu.org/software/qexo/) implementation of XQuery includes a write-to built-in function. It takes two parameters—a value (normally a document or element node) and a filename—and writes the former to the latter, using HTML-compatible XML (XHTML) syntax. It returns an empty sequence. Other XQuery implementations might have similar functions. Qexo is discussed in detail in Chapter 12.

The outermost expression of the new XQuery program writes both the index.html overview file and each of the individual picture pages, the latter in each of the three styles. Writing the picture pages is done by two nested loops: the outer one looping over <picture> elements, and the inner one looping over the three styles. Each then calls the picture function to generate each picture page, and uses write-to to write the HTML to the appropriate file. Each filename is generated by concatenating the picture's id attribute, the style string, and the .html extension.

let $group := doc("index.xml")/group,
    $group-date := $group/date,
    $group-name := $group/title,
    $pictures := $group//picture,
    $count := count($pictures)
  return (
    write-to(make-group-page($group), "index.html"),
    for $i in 1 to $count
    for $style in ("", "info", "large")
    let $picture := item-at($pictures,$i),
        $prev := if ($i > 1) then item-at($pictures, $i - 1) else (),
        $next := if ($i < $count) then item-at($pictures, $i + 1) else (),
        $date := if ($picture/date) then $picture/date else $group-date
      return
      write-to(
        picture($picture, $group, string($picture/@id),
          $prev, $next, $date, $style),
          concat(string($picture/@id), $style, ".html"))
)

The standard function item-at selects a value from a sequence using an index (starting at one). We've already looked at make-group-page. Listing 11.2 is the big picture function, which returns a single picture page as an <html> element.

Listing 11.2 The picture Function
(: Generate a page picture image with links etc.
 : $picture:  The <picture> node to use.
 : $group:  The enclosing <group>.
 : $name:  The string-value of the picture's id attribute.
 : $prev:  The previous <picture> or the empty sequence if there is none.
 : $next:  The next <picture> or the empty sequence if there is none.
 : $date:  The date the picture was taken, as a string, or the empty sequence.
 : $style:  The style and size.  Currently only "large" or "" (medium)
 :   or "info".  The "info" style shows a thumbnail, plus EXIF information,
 :   plus links to the raw JPGs.
 :)
declare function picture($picture, $group, $name, $prev, $next, $date, $style) {
<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type"/>
    <link rel="up" href="index.html" />{
    (: Note how we pre-pend whitespace to properly indent the <links>s. :)
    if (empty($prev)) then ()
    else ("
    ",<link rel="prev" href="{$prev/@id}{$style}.html" />),
    if (empty($next)) then ()
    else ("
    ",<link rel="next" href="{$next/@id}{$style}.html" />)}
    <title>{make-title($picture,$group)}</title>
    <style type="text/css">
      a {{ padding: 1 4; text-decoration: none; }}
      td {{ padding-left: 0; border-style: none }}
      span.button {{ border-width: thin; background-color: #FFFF99;
        border-style: solid }}
    </style>{( (: (Note what we have to do to add an XQuery comment here!)
     : Next we generate a JavaScript handler, to handle pressing the
     : keys 'n' and 'p' for navigation.  The indentation of the code
     : isn't "logical", but it makes the JavaScript code look ok. --} )}
    <script language="JavaScript">
      document.onkeypress = handler;
      function handler(e) {{
        var key = navigator.appName == 'Netscape' ? e.which
          : window.event.keyCode;{
        if (empty($next)) then ()
        else (: if 'n' was pressed, goto $next --}
          concat('
        if (key == 110) { location="',
          string($next/@id), $style, '.html"; return true; }'),
        if (empty($prev)) then ()
        else (: if 'p' was pressed, goto $prev :)
          concat('
        if (key == 112) { location="',
          string($prev/@id), $style, '.html"; return true; }')}
        return routeEvent(e); }}
    </script>
  </head>
  <body bgcolor="#00AAAA">
{ nav-bar($picture, $name, $prev, $next, $style)}
{ make-header($picture, $group)}
{ picture-text($picture)}
{ if (empty($date)) then () else <p>Date taken: {$date}.</p>}
{ let $full-image := $picture/full-image,
      $image := $picture/image
  return
  if ($style = "info") then (
    <table><tr>
      <td style="padding: 20 20 20 10">{make-thumbnail($picture)} </td>
      <td>{doc(concat($name,"-info.txt"))}</td>
    </tr></table>,
    <table><tr><td>Plain JPEG images:</td>
    {raw-jpg-link($picture/full-image, "Original")}
    {raw-jpg-link($picture/image,
    if ($full-image) then "Scaled" else "Original")}
    {raw-jpg-link($picture/small-image, "Thumbnail")}
    </tr></table>
  )
  else if ($style="large" and $full-image) then
    make-img($full-image, 1)
  else if ($style="large" and $image
           and number($image/@width) <= 640
           and number($image/@height) <= 640) then
    make-img($image, 2)
  else if ($full-image) then
    make-img($full-image, 0.5)
  else
    make-img($image, 1)
  }
  </body>
</html>
};

The basic structure of this should by now be familiar. The JavaScript function handler (in the <script>) goes to the next or previous page if the letter N or P on the keyboard is pressed. (The strange layout of the code is to make the generated HTML look nice.) Note how the XQuery comment before the <script> section is nested inside parentheses (so we get a syntactically valid empty sequence), which is nested inside the curly brace escapes. Note how we check the $style parameter to select which JPEG image to show, and whether we need to scale it. The "info" style generates a picture with information about the picture itself, as in Figure 11.3. This so-called EXIF information is generated by the camera, and has been extracted from the JPEG file using the jhead program (http://www.sentex.net/~mwandel/jhead/).

Figure 11.3. A page giving detailed information about a single picture.

The navigation bar (the row of buttons on the top of the page) is generated by the nav-bar function:

(: Create a 1-row navigation-bar: next, prev etc :)

declare function nav-bar($picture, $name, $prev, $next, $style) {
<table>
  <tr>
    <td><span class="button"><a href="index.html">Index</a></span></td>{
    if ($style="info") then () else ("
    ",<td><span class="button">{make-link($name, "info", "Info")}</span></td>),
    if ($style="large") then () else ("
    ",<td width="200" align="left"><span class="button">{
      make-link($name, "large", "Large image")}</span></td>),
    if ($style="") then () else ("
    ",<td width="200" align="left"><span class="button">{
      make-link($name, "", "Medium image")}</span></td>)}
    <td width="100" align="right">{
      if ($prev) then
        <span class="button">{make-link($prev/@id, $style, " < Previous ")
      }</span> else ()}</td>
    <td width="100" align="left">{
      if ($next) then
        <span class="button">{make-link($next/@id, $style, " Next > ")}</span>
      else ()}</td>
  </tr>
</table>
};

This is standard use of HTML tables to arrange the clickable buttons in a row. The <span class="button" > acts with the top-level <style>. Notice the use of conditionals to generate only the buttons that make sense.

Finally, the last five small functions are as follows:

declare function picture-text($picture) {
  for $text in $picture/text return <p>{$text/node()}</p>
};

declare function make-title($picture, $group) {
  concat(string($group/title), " - ",
         if (empty($picture/caption)) then string($picture/@id)
         else string($picture/caption))
};

declare function make-header($picture, $group) {
  <h2>{if ($picture/caption) then $picture/caption/node()
                else $group/title)/node()}</h2>
};

declare function raw-jpg-link($image, $description) {
  if (empty($image)) then () else
  <td><span class="button"><a href="{$image}">{$description} ({
    string($image/@width)}x{string($image/@height)})</a></span></td>
};

declare function make-link($picture-name, $style, $text) {
  <a href="{$picture-name}{$style}.html">{$text}</a>
};


Running XQuery Programs as Servlets

Servlets are a popular framework for having a Web server process a Web (HTTP) request. This approach is both efficient and powerful, and it is part of the Java Enterprise framework. JSP (JavaServer Pages, mentioned earlier) is built on top of servlets. A JSP page is an HTML template that can contain embedded Java expressions and commands. A JSP-capable Web server automatically translates (if needed) a JSP page into a Java servlet class. Such a server has various ways you can configure it to select which servlet is executed for a given request URL.

The photo album application would be very difficult to write using JSP. However, it is quite easy to do it using XQuery. There is at this time no standard for how Web servers can cause XQuery programs to be executed in response to a Web request, so we will look at how the Qexo implementation supports servlets. Most of the XQuery program is the same as before. What is different is the top-level "main" expression:

let $path := request-servlet-path(),
    $last-slash := last-index-of($path, "/"),
    $file := substring($path, $last-slash+1),
    $xml-file := concat("file", servlet-context-realpath(),
                        substring($path, 1, $last-slash), "index.xml"),
    (: or just plain: $ml-file := "index.xml" :)
    $group := doc($xml-file)/group
return
  if ($file = "index.html") then
    make-group-page($group)
  else
    let $group-date := $group/date,
        $group-name := $group/title,
        $pictures := $group//picture,
        $count := count($pictures),
        $style := if (last-index-of($file, "info.html") > 0) then "info"
            else if (last-index-of($file, "large.html") > 0) then "large"
            else "",
        $base-length := string-length($file) - string-length($style) - 5,
        $base := substring($file, 1, $base-length),
        $i := (for $j in 1 to $count return
            if ($base = string(item-at($pictures,$j)/@id)) then $j else ())
      return
            if (empty($i)) then
                    error-response(404, $file)
        else
          let $picture := item-at($pictures,$i),
              $id := string($picture/@id),
              $prev :=
                if ($i = 0) then ()
                else item-at($pictures, $i - 1),
              $next :=
                if ($i >= $count) then ()
                else item-at($pictures, $i + 1),
              $date :=
                if ($picture/date) then $picture/date
                else $group-date
            return
              picture($picture, $group, string($picture/@id),
                      $prev, $next, $date, $style)

To get the request URL relative to the servlet context, we use Qexo's request-servlet-path function. We extract the directory name and the filename using some string operations. (The last-index-of function is a Qexo function that returns the last position in the first argument of a substring matching the second argument, or 0 if the first argument does not contain the second argument.) Next we need to read the correct index.xml file. One way to find it is to use the Qexo function servlet-context-realpath, which gives us the root directory of the current (pics) Web application; to that we append the directory that we extracted from the request, and the constant "index.xml". (We prepend "file:" to make it an absolute URL.) A simpler solution is commented-out: just pass "index.xml" to the doc function. This works because a relative URL is resolved against the current base-uri(), which is the URL of the original request. However, the resulting URL is an http: URL, since that's what the base-uri() was, which is less efficient than opening a file directly.

If the requested $file was "index.html" , we pass the read $group to make-group-page as before, and we're done. Otherwise, things are a little complicated.

We need to find the correct <picture> element in the read document by searching for one that matches the incoming request. There are various ways of doing this, but we need to keep a few concerns in mind:

  • We need to find not just the current <picture>, but also the previous and next pictures, so we create links to them. Thus we need not just the requested <picture> element, but also its index in the sequence $pictures. We do that by using the for-loop that sets the value of $i.

  • We want to emit some suitable error response if the request is for a <picture> that doesn't exist. We do that by testing empty($i), and calling the Qexo error-response function. This function takes an HTTP error code and a message, and creates an appropriate error value that gets sent back to the browser.

  • It is more efficient if the HTML page generated by the picture function can be written directly to the servlet output buffer, as opposed to creating a temporary document value, and later writing that out. To make it easier for an XQuery compiler to avoid unnecessary copying, we call the picture function so its output directly becomes the output of the top-level expression, without first storing it in a temporary variable.

The call to doc in the picture function must be similarly modified:

declare function picture($picture, $group, $name, $prev, $next, $date, $style) {
  ...
  if ($style = "info") then (
    make-thumbnail($picture),
   (let $path := request-servlet-path(),
         $last-slash := last-index-of($path, "/"),
         $dir := substring($path, 1, $last-slash) return
    doc(concat("file:",servlet-context-realpath(),$dir,$name,"-info.txt"))),
  ...
};

Otherwise, the servlet version is same as the batch version.

Installing the Servlet Under the Tomcat Server

A number of Web servers can run servlets. The servlet specification describes how a Web application consists of some number of servlets, static pages, and a web.xml "deployment descriptor." We will show how you can create an application running under the Tomcat server from the photo album program. Tomcat (http://jakarta.apache.org/tomcat/) is an open-source Web server written in Java, released by the Apache Foundation's Jakarta project (http://jakarta.apache.org/). It is the reference platform for servlets. The instructions here assume you have version 4.1.x of Tomcat installed, and that the value of $CATALINA_HOME where Tomcat is installed is /opt/Tomcat. We also assume a Posix-compatible environment, including an sh-style shell such as GNU bash. (If you use Microsoft Windows, you can get these as part of the Cygwin [http://www.cygwin.com/] environment.) If you have a different environment, or need information beyond these very basic instructions, check out the Tomcat documentation.

We assume you're using the Qexo implementation of XQuery. The Qexo implementation classes are included in the kawa-N.M.jar file containing the Kawa compiled classes, where N.M is the Kawa version number. You will need to install this JAR file somewhere where Tomcat can find it. For example, you can install it in $CATALINA_HOME/shared/lib/kawa-N.M.jar.

Create the actual pics Web application. You need to create the directories $CATALINA_HOME/webapps/pics, $CATALINA_HOME/webapps/pics/WEB-INF, and $CATALINA_HOME/webapps/pics/WEB-INF/classes. Under Unix, Linux, or MS Windows, do the following:

mkdir $CATALINA_HOME/webapps/pics
mkdir $CATALINA_HOME/webapps/pics/WEB-INF
mkdir $CATALINA_HOME/webapps/pics/WEB-INF/classes

You next need to compile the pictures XQuery program to a servlet:

kawa --xquery --servlet -C pictures.xql

This creates a pictures.class file, which you need to install in the $CATALINA_HOME/webapps/pics/WEB-INF/classes directory, so Tomcat can find it:

cp pictures.class $CATALINA_HOME/webapps/pics/WEB-INF/classes/pictures.class

You next need to write the web.xml deployment descriptor. This is an optional file that describes the Web application, and goes into the $CATALINA_HOME/webapps/pics/WEB-INF directory. The following is sufficient for the pics application:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE web-app
  PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
  "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
  <display-name>Web Photo Album</display-name>

  <servlet>
    <servlet-name>pictures</servlet-name>
    <servlet-class>pictures</servlet-class>
  </servlet>

  <servlet-mapping>
    <servlet-name>pictures</servlet-name>
    <url-pattern>*.html</url-pattern>
  </servlet-mapping>
</web-app>

The <display-name> element is just a human-readable description of the Web application. Each <servlet> element names a servlet, and gives the fully qualified name of the Java class used to implement it. (In this example, they're the same.) The <servlet-mapping> element specifies which URLs (within the pics Web application) cause the named servlet to be called; in this case, all requests that end in the .html extension.

Finally, we need to create some content! For example, we can take the *.jpg files and the index.xml file from the earlier NorwayHighlights and copy them into the Web application's directory $CATALINA_HOME/webapps/pics.

If Tomcat isn't already running, start it now:

$CATALINA_HOME/bin/startup.sh start

After you have started up Tomcat, it listens by default to port 8080. To view the NorwayHighlights album, point your browser at http://localhost:8080/pics/NorwayHighlights/index.html. When Tomcat receives this request, it will invoke the pictures servlet, and the call request-servlet-path() will return /NorwayHighlights/index.html. The $file variable will get set to index.html, so make-group-page is called. If you click a link, the browser sends back a new request, which the servlet handles. Any .jpg image files that might be requested via an <img> or <a> tag, such as http://localhost:8080/pics/NorwayHighlights/Skjolden1.jpg, will not be handled by the servlet. Instead, Tomcat handles those using its default rules, and looks for correspondingly named files relative to the Web application directory, such as $CATALINA_HOME/webapps/pics.

Comparison with JavaServer Pages

We can summarize the advantages of XQuery versus JSP and similar template systems as follows:

  • Better integration than JSP's HTML+Java. You can use the same programming language for both the presentation and the logic.

  • XQuery is a higher-level language compared to Java, with powerful features for manipulating XML.

  • XQuery is a true expression language; values can be structured XML that you can pass to functions. You don't need to clutter your program with print invocations.

  • Qexo can run XQuery programs in different modes, even expressions typed at a command prompt. This helps development and testing.

  • Errors (such as stack traces) refer to the line numbers in the XQuery source, as opposed to line numbers in generated Java code.

  • XQuery is statically typed, which helps in catching errors early.


Running XQuery Programs from CGI Scripts

CGI (Common Gateway Interface) scripts are programs or scripts that are executed by a Web server in response to a request. The request parameters are passed to the script as environment variables. The script writes the result parameters followed by the data (usually an HTML page) to standard output, and the server takes that output and sends it back as an HTTP response.

You can have a CGI script call an XQuery processor, and use XQuery to generate the result that way. Using CGI scripts will not give you high performance, especially if you have to start up an XQuery processor on each request. On the plus side, it does not require any server modifications or configuration, and it might be fast enough for some applications. It might be most useful for testing simple XQuery applications.

Another problem running XQuery from CGI scripts is that there is no standard for accessing request parameters from XQuery. The Qexo implementation provides a set of functions that is a subset of the servlet functionality. Qexo provides a wrapper script that runs a servlet, and using the CGI environment variable to provide a subset of the servlet functionality. For example, you can write a little XQuery program like this:

response-content-type("text/html"),
<html>
<p>The request URL was: request-url()</p>
<p>{let $query := request-query-string() return
    if ($query)
    then ("The query string was: ",$query)
    else "There was no query string."}</p>
</html>

If this is in the file hello.xql, you compile it to a servlet like this:

kawa --servlet --xquery -C hello.xql

Copy the resulting .class file(s) to your Web server's CGI directory. On Red Hat GNU/Linux using the Apache server, you can do the following (as root):

cp hello*.class /var/www/cgi-bin/

Next find the cgi-servlet program that Kawa builds and installs. If you installed Kawa in the default place, it will be in /usr/local/bin/cgi-servlet. (You'll have this if you installed Kawa from source, but not if you're just using the Kawa JAR file.) Copy this program into the same CGI directory, renaming it to hello:

cp /usr/local/bin/cgi-servlet /var/www/cgi-bin/hello

Make sure the files have the correct permissions:

chmod a+r /var/www/cgi-bin/hello*.class /var/www/cgi-bin/hello
chmod a+x /var/www/cgi-bin/hello

Now you should be able to run the Kawa program, using the URL http://localhost/cgi-bin/hello. It might take a few seconds to get the reply, mainly because of the startup time of the Java VM. That is why servlets are preferred. The CGI interface can still be useful for testing, or when you can't run servlets.


Comparing XSLT and XQuery

XSLT (XSL Transformations, where XSL stands for Extensible Stylesheet Language) is a popular and powerful language for transforming an input XML document into an output document. The latter can be XML, HTML, or plain text. XSLT became a W3C recommendation (standard) in 1999; it is being revised in conjunction with the XQuery standardization process. This is because XSLT uses XPath for expressions and patterns, and XPath is also a subset of XQuery.

The most visible difference between XSLT and XQuery is that an XSLT program (a stylesheet) is actually also an XML document. This can sometimes be useful, but it has the big disadvantage that XSLT stylesheets, in spite of their simplicity, can be both verbose and hard to read.

The more significant difference between XSLT and XQuery is in the execution model, specifically the flow of control. Except for XQuery's unusual data types, it is otherwise a relatively normal programming language, with explicit control flow. (Strictly speaking, XQuery does not have a defined control flow, as an implementation is free to evaluate expressions in whatever order is consistent with the standard. Still, you can understand an XQuery program in terms of a conventional execution model.) In contrast, executing an XSLT stylesheet is controlled by a template processor, which matches a node against a set of templates, selects the template whose pattern most closely matches the input node, and then executes that template. This process might then be repeated by the <xsl:apply-templates> instruction, which recursively calls the template processor on the child nodes of the current node.

Using pattern matching to drive the execution in this way is very powerful and usually convenient. It works best when doing relatively simple conversions that can be expressed using patterns. You can do more complex programming using XSLT, but it can quickly become very awkward and verbose.

Transforming Docbook to HTML Using XSLT

As a sample application, we will look at converting Docbook to HTML, using both XSLT and XQuery. Docbook (http://www.docbook.org) is a popular SGML/XML format for writing technical documentation. Groups that use it for manuals and help files include the Linux Documentation Project and the Gnome project.

Listing 11.3 is a simple (and incomplete) XSLT stylesheet for translating Docbook to HTML.

Listing 11.3 An XSLT Stylesheet for Transforming Docbook to HTML
<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

<xsl:template match="sect1/title">
<h2 class="title"><xsl:apply-templates/></h2>
</xsl:template>

<xsl:template match="sect2/title">
<h3 class="title"><xsl:apply-templates/></h3>
</xsl:template>

<xsl:template match="artheader/title">
<h1><xsl:apply-templates/></h1>
</xsl:template>

<xsl:template match="artheader/subtitle">
<h2><xsl:apply-templates/></h2>
</xsl:template>

<xsl:template match="artheader/authorgroup">
<h3><i><xsl:apply-templates/></i></h3>
</xsl:template>

<xsl:template match="biblioentry/abbrev">[<xsl:apply-templates/>]</xsl:template>
<xsl:template match="biblioentry/title"><cite><xsl:apply-templates/></cite>
</xsl:template>
<xsl:template match="biblioentry/authorgroup"><xsl:apply-templates/>.
</xsl:template>

<xsl:template match="article">
<html>
<head>
<xsl:if test="artheader/title">
<title><xsl:value-of select="artheader/title"/></title>
</xsl:if>
</head>
<body>
<xsl:apply-templates/>
</body>
</html>
</xsl:template>

<xsl:template match="title">
<title><xsl:apply-templates/></title>
</xsl:template>

<xsl:template match="abstract">
<div type='abstract'><h3>Abstract</h3><xsl:apply-templates/></div>
</xsl:template>

<xsl:template match="element">
<p><xsl:apply-templates/></p>
</xsl:template>

<xsl:template match="para">
<p><xsl:apply-templates/></p>
</xsl:template>

<xsl:template match="screenshot">
<img src='{mediaobject/imageobject/imagedata[@format="PNG"]/@fileref}' />
</xsl:template>

<xsl:template match="caption">
<p><b><xsl:apply-templates/></b></p>
</xsl:template>

<xsl:template match="emphasis"><em><xsl:apply-templates/></em></xsl:template>

<xsl:template match="citation">[<xsl:apply-templates/>]</xsl:template>

<xsl:template match="quote">"<xsl:apply-templates/>"</xsl:template>

<xsl:template match="classname"><code><xsl:apply-templates/></code>
</xsl:template>

<xsl:template match="function"><code><xsl:apply-templates/></code>
</xsl:template>

<xsl:template match="itemizedlist">
<ul><xsl:apply-templates/></ul>
</xsl:template>

<xsl:template match="listitem">
<li><xsl:apply-templates/></li>
</xsl:template>

<xsl:template match="sect1">
<div class="sect1">
<xsl:apply-templates/>
</div>
</xsl:template>

<xsl:template match="sect2">
<div class="sect2">
<xsl:apply-templates/>
</div>
</xsl:template>

<xsl:template match="programlisting">
<pre><xsl:apply-templates/></pre>
</xsl:template>

<xsl:template match="bibliography">
<div class="bibliography"><h2>Bibliography</h2><xsl:apply-templates/></div>
</xsl:template>

<xsl:template match="biblioentry">
<p><xsl:apply-templates/></p>
</xsl:template>

</xsl:stylesheet>

Transforming Docbook to HTML Using XQuery

In contrast, Listing 11.4 is an XQuery program for doing the same Docbook-to-HTML translation. Because XQuery doesn't have XSLT's template-matching driver, we have to write explicit control flow using recursive function calls. The typeswitch expression is useful for selecting between different node types, but unfortunately, XQuery type expressions are not as expressive as XPath patterns, so more complex patterns might need explicit logic.

Note especially how context patterns like artheader/title are handled. The idea is that in place of the XSLT rules that use the artheader context, we use a convert-artheader function.

An alternative method for handling "context" is to pass it as an explicit parameter like the $level parameter of convert-title.

Listing 11.4 An XQuery Program to Transform Docbook to HTML
declare function convert-children ($x) {
  for $y in $x/node() return convert-node($y)
};

declare function convert-title($x, $level) {
  if ($level <= 1) then <h1 class="title">{$x/node()}</h1>
  else if ($level = 2) then <h2 class="title">{$x/node()}</h2>
  else if ($level = 3) then <h3 class="title">{$x/node()}</h3>
  else if ($level = 4) then <h4 class="title">{$x/node()}</h4>
  else if ($level = 5) then <h5 class="title">{$x/node()}</h5>
  else <h6 class="title">{$x/node()}</h6>
};

declare function convert-div-children ($x, $level) {
  for $y in $x/node() return
    typeswitch ($y)
      case element(title,*) return
        convert-title($y, $level)
      default return convert-node($y)
};

declare function convert-artheader ($x) {
  for $y in $x/node() return
    typeswitch ($y)
      case element(title,*) return
        <h1>{convert-children($y)}</h1>
      case element(subtitle,*) return
        <h2>{convert-children($y)}</h2>
      case element(authorgroup,*) return
        <h3><i>{convert-children($y)}</i></h3>
      default return ( )
};

declare function convert-biblioentry ($x) {
  for $y in $x/node() return
    typeswitch ($y)
      case element(abbrev,*) return
        ('[',convert-children($y),']')
      case element(title,*) return
        (<cite>{convert-children($y)}</cite>,'.')
      case element(authorgroup,*) return
        (convert-node($y),'.')
      default return convert-node($y)
};

declare function convert-node ($x) {
  typeswitch ($x)
    case element(article,*) return
      <html>{convert-children($x)}</html>
    case element(artheader,*) return
      convert-artheader($x)
    case element(title,*)return
      <title>{convert-children($x)}</title>
    case element(authorgroup,*) return
      convert-children($x)
    case element(author,*) return convert-children($x)
    case element(abstract,*) return
     <div type='abstract'><h3>Abstract</h3>{convert-children($x)}</div>
    case element(para,*) return <p>{convert-children($x)}</p>
    case element(screenshot,*) return
      <img src="{string($x/mediaobject/imageobject
      /imagedata[@format="PNG"]/@fileref)}" />
    case element(caption,*) return
      <p><b>{convert-children($x)}</b></p>
    case element(emphasis,*) return <em>{convert-children($x)}</em>
    case element(citation,*) return ('[',convert-children($x),']')
    case element(quote,*) return ('"',convert-children($x),'"')
    case element(classname,*) return <code>{convert-children($x)}</code>
    case element(function,*) return <code>{convert-children($x)}</code>
    case element(itemizedlist,*) return <ul>{convert-children($x)}</ul>
    case element(listitem,*) return <li>{convert-children($x)}</li>
    case element(sect1,*) return
    <div class="sect1">{convert-div-children($x, 2)}</div>
    case element(sect2,*) return
    <div class="sect2">{convert-div-children($x, 3)}</div>
    case element(programlisting,*) return <pre>{convert-children($x)}</pre>
    case element(informalfigure,*) return convert-children($x)
    case element(bibliography,*) return
      <div class="bibliography" ><h2>Bibliography</h2>
      {convert-children($x)}</div>
    case element(biblioentry,*) return <p>{convert-biblioentry($x)}</p>
    case element(firstname,*) return convert-children($x)
    case element(surname,*) return convert-children($x)
    default return ($x)
};

let $doc := input()
let $artheader := $doc/article/artheader
let $title := $artheader/title
return
<html>
<head>
{if ($title) then <title>{convert-children($title)}</title> else ()}
</head>
<body>
{convert-children($doc)}
</body>
</html>

These two stylesheets are of comparable size and complexity. The XSLT is slightly shorter, but the difference is minor, and this is the kind of application that is XSLT's strength. So, my advice is: If you have a task that matches XSLT's strength, by all means use XSLT. However, if you have a task that is a mix of XSLT-style transformation combined with some control logic, consider using XQuery, even for the part of the task that is XSLT-like. The advantage of XSLT over XQuery in applications best suited for XSLT is relatively minor, whereas the pain of writing more complex logic in XSLT instead of XQuery is considerable! The photo album is an application I first wrote in XSLT, but I was able to easily make significant improvements when I rewrote it in XQuery.


In Brief

  • It is convenient to generate XML and HTML Web pages with XQuery.

  • You can use XQuery servlets for applications inside Web servers.

  • XQuery can be used as an alternative to XSLT.


Chapter 12. Working with Java and XQuery

by Per Bothner and Vaidyanathan Nagarajan

IN THIS CHAPTER

  • Qexo: An XQuery-to-Java Compiler

  • Running Qexo

  • XQEngine

  • In Brief


Qexo: An XQuery-to-Java Compiler

Both Qexo and XQEngine, discussed later, are implemented using Java, but you don't have to know Java to use them. Both are open source, which means you are free to download and use them without charge, study and modify the source code, and redistribute them without charge. (However, check the respective licenses, especially if you're thinking of using open-source code as part of a commercial application.)

Qexo is written in Java, and it compiles XQuery expressions and programs to Java bytecodes (.class files). Qexo is available from http://www.gnu.org/software/qexo/.

SHOP TALK: QEXO AND KAWA

Qexo is based on and is a part of the Kawa framework (http://www.gnu.org/software/kawa/). The easiest way is get Qexo is to download the latest version of the Kawa JAR file. (The following examples use kawa-1.7.jar from the 1.7 release of Kawa.)

Kawa depends on a Java feature called a ClassLoader, which can take a bytecode representation of a Java program (the same format as a .class file, but stored in an array in memory) and convert it into a runnable class in an existing Java executable (or virtual machine).

Compiling the bytecodes and then using a ClassLoader gives Qexo both fast interactive responsiveness and fast execution of repetitive code. You can also save compiled code in a .class file so it's available for future use, and it can even be compiled to machine code using a Java-to-machine-code compiler such as GCJ.


Running Qexo

In the following sections, we show examples of different ways you might use Qexo, looking at both standalone use (for those who know and care nothing about Java) and how to use Qexo with Java.

Running the Qexo Application

In this section, we write qexo to indicate the command you use to start Qexo. You can run Qexo a number of ways. If you have downloaded Kawa as a JAR file (for example, kawa-1.7.jar), you can start Qexo using either this command

java -cp kawa-1.7.jar kawa.repl --xquery

or

java -jar kawa-1.7.jar --xquery

In our discussion, we'll assume you've defined qexo as an alias for one of the above commands. On a Unix or GNU/Linux system, you can make an alias like this:

$ alias qexo='java -jar kawa-1.7.jar --xquery'

Then enter the following:

$ qexo

where $ stands for the prompt in your command-line processor (shell or console), and we use boldface for commands you type.

Alternatively, you can place the kawa-1.7.jar file in your class path, and use the following command instead of qexo:

$ java kawa.repl --xquery

Interactive Use

If you start Qexo without specifying any file parameters, it will enter an interactive loop. Here are some examples, with user input shown in bold:

$ qexo
(: 1 :) for $i in 1 to 3 return 10*$i
10 20 30

The command-line prompt includes the current input line number, and has the form of an XQuery comment, to make it easier to cut and paste. Following the prompt, you can type some complete XQuery expression (in the example, for $i in 1 to 3 return 10*$i) and press Enter. The Qexo processor evaluates the expression and writes out the result—in this case, a sequence of three integers.

How does Qexo know when an expression is complete? When should it evaluate what it has, as opposed to prompting for more input? The rule is that if current input line forms a complete valid expression, Qexo evaluates it. If Qexo has identified a syntax error, it prints out a message and discards the input. Otherwise, it prints a prompt and waits for more input.

Let's continue, this time with a multiline expression:

(: 2 :) (3+
(: 3(:) 10)
13
(: 4 :) if (3<2)
(: 5i:) then "it's true"
(: 6i:) else "it's false"
it's false

Notice how the prompt changes to ( or an i to indicate that you're inside an incomplete parenthetical or if expression, respectively.

Next, here are some examples of syntax errors:

(: 7 :) (for $x := 10 return $x
<stdin>:7:9: missing 'in' in 'for' clause
(: 8 :) %+1
<stdin>:8:1: invalid character '%'
(: 9 :) = 5
<stdin>:9:1: missing expression

Qexo prints out the file name of the error (in this case, the standard console input), followed by the line and column numbers. For the last error, it couldn't be more specific than missing expression.

Here is an example of an element constructor expression. Notice how the prompt changes to an XML comment:

(: 2 :) <a>
<!--3--><b>{for $i in 1 to 3 return 10*$i}</b>
<!--4--></a>
<a>
  <b>10 20 30</b>
</a>

You can also define XQuery functions interactively:

(: 5 :) declare function repeat ($count, $values) {
(: 6{:)   for $i in 1 to $count return $values
(: 7{:) }
(: 8 :) "[", repeat(4, (1,2)), "]"
[1 2 1 2 1 2 1 2]
(: 9 :)

Running XQuery Programs

The XQuery specification defines a program as a collection of declarations followed by a top-level expression. The normal way of running a program is to put it in a file and evaluate it. You can use the -f command-line flag to specify the name of a file containing a program:

$ qexo -f pictures.xql

You can also specify a (short) XQuery program on the command line following a -e flag:

$ qexo -e '<img src="file.png"></img>'
<img src="file.png" />

The output is by default printed using the XHTML style, which is XML in a style that most HTML browsers can handle. You can override the output format using an --output-format option. For example, you can specify HTML format:

$ qexo --output-format html -e '<img src="file.png"></img>'
<img src="file.png">

You can even specify a format for Scheme programmers:

$ qexo --output-format scheme -e '<img src="file.png"></img>'
(img src: file.png )

Compiling an XQuery Program to an Application

If you run Qexo with the -C flag followed by one or more filenames, those files will be compiled, producing one or more .class files. The --main option specifies that Qexo should generate a main method, creating an application that can be run using the java command.

Assume pictures.xql is the name of a file containing an XQuery program:

$ qexo --main -C pictures.xql

This creates a file pictures.class. (The command may in rare cases create some other classes as well. These have the form pictures*.class.) You can run this as follows:

$ java –cp .:kawa-1.7.jar pictures

This should be the same as, but faster than, running:

$ qexo -f pictures.xql

Running XQuery Servlets in a Web Server

A servlet is a Java class that can be loaded into a Web server to process and answer HTTP requests. It is an efficient way to provide server-side computation, because the servlet can be loaded and allocated once, and then process thousands of requests. An XQuery program can be compiled by Qexo into a servlet. See Chapter 11, "Generating XML and HTML Using XQuery," and http://www.gnu.org/software/qexo/servlet.html for more information and examples of servlets using Qexo.

Calling Java Methods from XQuery

A Qexo extension allows you to call an arbitrary Java method in an XQuery expression, using XQuery function call notation. The following example uses Drew Noakes' EXIF extraction library (http://www.drewnoakes.com/code/exif/) for extracting EXIF metadata (time-stamps, focal-length, and so on) commonly produced by digital cameras. The code assumes that exifExtractor.jar is in your classpath. Start your XQuery program with these namespace declarations that are used as aliases for Java classes:

declare namespace exif-extractor =
  "class:com.drew.imaging.exif.ExifExtractor";
declare namespace exif-loader = "class:com.drew.imaging.exif.ExifLoader";
declare namespace ImageInfo = "class:com.drew.imaging.exif.ImageInfo";
declare namespace File = "class:java.io.File";

Remember that a namespace defines a prefix alias for a URL literal, which can be any string, used as a unique name. Qexo uses the convention that a URL string starting with class: refers to a Java class. Specifically, it acts as if all Java methods are prebound to a QName whose local name is the method name, and whose namespace URI is class: followed by the fully qualified Java class name. For example, if the Qexo processor sees a call to a function exif-loader:getImageInfo, with the namespaces as defined previously, it will translate that into a call to a method named getImageInfo in the class com.drew.imaging.exif.ExifLoader. (That is, assuming you haven't explicitly defined a function by that name!) If the method is overloaded, Qexo uses the argument types to select a method. The method name new is used specially for creating a new objects, being equivalent to a Java new expression:

declare function get-image-info ($filename as xs:string)
{
<pre>{
  let $info := exif-loader:getImageInfo(File:new($filename))
  for $i in iterator-items(ImageInfo:getTagIterator($info)) return
 ( "
", ImageInfo:getTagName($i),": ", ImageInfo:getDescription($info, $i))
}</pre>
};

The function takes a single parameter, $filename, which is the name of a JPEG image file as a string. It uses that to create a new file, which is used to create an ImageInfo object. The getTagIterator method creates a java.util.Iterator instance, which you can use to get all the EXIF tags in the image. The Qexo function iterator-items takes an Iterator and turns it into an XQuery sequence consisting of the values returned by the Iterator. The for loops over this sequence, and you format each tag item into a readable output line. (For more information, visit http://www.gnu.org/software/qexo/Extensions.html.)

Calling XQuery Expressions from Java

XQuery is often used as part of a larger Java application, and here we explain how you can use Qexo to evaluate an XQuery expression in a Java program.

The following Java statement creates an XQuery evaluation context and assigns it to the variable named xq:

XQuery xq = new XQuery();

You can then use the eval method to evaluate an XQuery expression, returning a Java Object:

Object result = xq.eval(expression);

The following application reads the strings on the command line, evaluates them as XQuery expressions, and prints the result:

import gnu.xquery.lang.XQuery;
public class RunXQuery
{
  public static void main (String[] args) throws Throwable
  {
    XQuery xq = new XQuery();
    for (int i = 0;  i < args.length;  i++)
      {
        String exp = args[i];
        Object result = xq.eval(exp);
        System.out.print(exp);
        System.out.print(" => ");
        System.out.println(result);
      }
  }
}

(Here and in the following examples I use boldface to emphasize the interesting bits, as opposed to boilerplate or repeated bits.)

You can use these commands to compile and run this application, assuming that kawa-1.7.jar is in your class path:

$ javac -g RunXQuery.java

$ java RunXQuery '3+4' 'for $i in 1 to 5 return $i+10' '<a>{3+4}</a>'
3+4 => 7
for $i in 1 to 5 return $i+10 => 11, 12, 13, 14, 15
<a>{3+4}</a> => <a>7</a>

The println method calls the generic toString method, which is fine for quick-and-dirty output (such as for debugging), but isn't recommended for printing real data. One reason is that it requires allocating a temporary string, which then has to be copied into the PrintStream's output buffer—a wasteful process for large data structures. Another reason is that none of the output shows up until it has all been converted, which can also hurt performance. (If the toString method gets into a loop, which is quite possible for cyclic data structures, you sit there waiting with no idea what is going on.) The toString method also doesn't provide any control over the output format, such as whether you want characters like < escaped as &lt;, or whether you want HTML-style or XML-style output, for example. Formatting to a specific line width is also difficult.

In Qexo, you can instead send the output to a special Consumer. It's like a Writer (or a SAX2 ContentHandler), but it works with abstract data rather than characters. The gnu.xml.XMLPrinter class implements Consumer and extends PrintWriter, so you can use it as either of those two. It writes out the received data in XML format, although there are options to produce HTML and other styles. Here is a revised version of RunXQuery that uses an XMLPrinter:

import gnu.xquery.lang.XQuery;
import gnu.xml.XMLPrinter;
public class RunXQuery
{
  public static void main (String[] args) throws Throwable
  {
    XQuery xq = new XQuery();
    XMLPrinter pp = new XMLPrinter(System.out);
    for (int i = 0;  i < args.length;  i++)
      {
        String exp = args[i];
        System.out.print(exp);
        System.out.print(" => ");
        Object x = xq.eval(exp);
        pp.writeObject(x);
        pp.println();
        pp.flush();
      }
  }
}
$ java RunXQuery 'for $i in 1 to 5 return $i+10'
for $i in 1 to 5 return $i+10 => 11 12 13 14 15

Note the flush call to make sure that the output from the XMLPrinter is sent to the System.out before you write anything on the latter directly. This produces mostly the same output as before, except that sequence items are separated by spaces instead of comma-spaces. (Also, XML quoting is handled correctly.)

This still isn't the best way to evaluate and print. It is more efficient to have the evaluator print directly to the output, rather than to create an intermediate data structure. To do that, you can pass the XMLPrinter directly to the eval call:

import gnu.xquery.lang.XQuery;
import gnu.xml.XMLPrinter;
public class RunXQuery
{
  public static void main (String[] args) throws Throwable
  {
    XQuery xq = new XQuery();
    XMLPrinter pp = new XMLPrinter(System.out);
    for (int i = 0;  i < args.length;  i++)
      {
        String exp = args[i];
        System.out.print(exp);
        System.out.print(" => ");
        xq.eval(exp, pp);
        pp.println();
        pp.flush();
      }
  }
}

This produces the same output as before. Whether it is more efficient will depend on the expression you evaluate (and how clever Qexo is). But for XQuery programs that generate large XML data sets, it can make a significant difference, and in general it's a good idea to pass the Consumer directly to the evaluator.

If the XQuery program is in a file, rather than a String, you can use an eval method that takes a Reader:

xq.eval (new FileReader("file.xql"), new XMLPrinter(System.out));

You can also call Qexo functions that have been compiled to .class files, directly using Java method invocation. How to do so is a bit complicated and likely to change, so check the online version of this chapter at http://www.gnu.org/software/qexo/Running.html for future updates.

Using Qexo with SAX2

The Simple API for XML (SAX; http://www.saxproject.org) is a set of classes for copying XML data (infosets) using method calls, without necessarily doing any physical copying. It is a popular API because it is an efficient way to process large data sets. The Consumer interface used by Qexo is similar to the SAX2 ContentHandler interface. If you have a class that implements ContentHandler, you can use a ContentConsumer filter to convert it to a Consumer. The following code snippet shows how you can pass the result of evaluating an XQuery expression to a ContentHandler:

import org.xml.sax.ContentHandler;
  ...
  ContentHandler ch = ...;
  xq.eval(exp, new ContentConsumer(ch));

The Consumer Interface

The Consumer interface (like the SAX2 ContentHandler) is very useful and efficient for any kind of processing of XML data that can be done in a single pass. A Consumer is a passive output sink. It doesn't do anything on its own. Instead, it is used as the output of a producer, which is the application that does the actual work and sends the results to the Consumer. The separation between a producer (which generates results) and a Consumer (which uses the results) permits great flexibility in plugging together modules. Note that a Consumer can pass the data along to another Consumer, acting as the latter's producer. This allows you to chain together a pipeline of Consumer filters.

Listing 12.1 shows a Java program that counts the number of different kinds of elements produced by evaluating XQuery expressions. It is a class that extends the basic gnu.lists.FilterConsumer, which provides dummy implementations of the Consumer methods.

Listing 12.1 CountElements.java
import gnu.xquery.lang.XQuery;
import java.util.*;
import gnu.lists.*;
import java.io.PrintStream;

public class CountElements extends FilterConsumer
{
  CountElements()
  {
    super(VoidConsumer.getInstance());
  }

  List elementNames = new ArrayList();
  int numAttributes = 0;
  int numInts = 0;
  int numObjects = 0;

  public void beginGroup(String typeName, Object type)
  {
    elementNames.add(typeName);
    super.beginGroup(typeName, type);
  }

  public void beginAttribute(String attrName, Object attrType)
  {
    numAttributes++;
    super.beginAttribute(attrName, attrType);
  }

  public void writeInt(int v)
  {
    numInts++;
    super.writeInt(v);
  }

  public void writeObject(Object v)
  {
    numObjects++;
    super.writeObject(v);
  }

  void dump (PrintStream out)
  {
    Collections.sort(elementNames);
    int total = 0;
    ListIterator it = elementNames.listIterator();
    String previous = null;
    int count = 0;
    for (;;)
      {
        boolean done = ! it.hasNext();
        String cur = done ? "" : (String) it.next();
        if (previous != null && ! previous.equals(cur))
          {
            out.println("<" + previous + "> - " + count + " times");
            count = 0;
          }
        if (done)
          break;
        previous = cur;
        count++;
        total++;
      }
    out.println("TOTAL: " + total);
    if (numAttributes > 0)
      out.println("Attributes: " + numAttributes);
    if (numInts > 0)
      out.println("ints: " + numInts);
    if (numObjects > 0)
      out.println("Objects: " + numObjects);
  }

  public static void main(String[] args) throws Throwable
  {
    XQuery xq = new XQuery();
    CountElements counter = new CountElements();
    for (int i = 0;  i < args.length;  i++)
      {
        String exp = args[i];
        xq.eval(exp, counter);
      }
    counter.dump(System.out);
  }
}

The producer (in this case, the XQuery.eval method called by the main method) calls the beginGroup method when it wants to write out an XML element. The beginGroup implementation in this class just adds the element's string name (the typeName) to the List called elementNames. It then calls super.beginGroup to do the default processing of beginGroup, which calls beginGroup in the next Consumer in the filter. In this case, that is a VoidConsumer, which ignores everything it receives, so the super.beginGroup isn't really needed; we include it to illustrate the general idea.

The code also counts attributes using the beginAttribute method as well as calls to writeInt and writeObject. These are used for non-XML typed values, which SAX doesn't handle.

At the end, the dump method is called. It sorts the list of elements and writes out the number of times each has been seen, along with some other statistics. Here is a sample run:

$ javac -g CountElements.java
$ java CountElements '<a><b/>{10 to 20}<b/>{1+1}<b/></a>'
<a> - 1 times
<b> - 3 times
TOTAL: 4
ints: 11
Objects: 1

Note how the sequence 10 to 20 produces 11 calls to writeInt, whereas the expression 1+1 produces a single call to writeObject. Whether an XQuery integer produces a calls to writeInt or writeObject may change in future Qexo versions.

The TreeList DOM Class

When Qexo needs to store a document in a data structure, it uses an instance of the class gnu.lists.TreeList. The name of the class isn't Document because it's actually a lot more general than what is needed for plain XML documents. It can handle typed values, and it is also used to represent sequences containing multiple items.

The TreeList class is used to implement a Document Object Model (DOM), but it does not implement the standard org.w3c.dom.Node or org.w3c.dom.Document interface. The reason is that the W3C DOM APIs use a separate Node object for each conceptual node (element, attribute, and so on) in a document. This is very inefficient, as it wastes a lot of space and makes a lot of work for the garbage collector. Instead, TreeList uses a much more compact array-based representation, using one char array and one Object array for the entire document. A node is just an index into the former array, which makes it efficient to traverse a document.

The following example shows how you can modify the CountElements application so that the command-line arguments are the URLs of XML files (instead of XQuery expressions). Replace the main method with the following, leaving the rest of the CountElements class as before:

public static void main(String[] args) throws Throwable
{
  CountElements counter = new CountElements();
  for (int i = 0;  i < args.length;  i++)
    {
      String url = args[i];
      TreeList doc = gnu.kawa.xml.Document.parse(url);
      doc.consume(counter);
    }
  counter.dump(System.out);
}

Each URL is opened and parsed as an XML file, to create a TreeList object. You can now do a lot of things with this TreeList; in this example, all we do is invoke its consume method, which writes out all its data to a Consumer (in this case, a CountElements object).


XQEngine

XQEngine is an open-source, full-text search engine for querying XML documents. You can embed the XQEngine API package in Java to enable the querying of XML documents. The size of the package is around 190K. The current version is based on a single-thread model, which is useful for sites that have moderate traffic. XQEngine lets you query multiple XML documents using the standard XQuery query combinations, and is thus quite similar to SQL. So for example, a query like //title simultaneously searches across every document in your XML repository for the <title> element.

To run XQEngine on your machine, download the zip file from http://www.fatdog.com/download_056.html. Also, include in your Java classpath the following JAR files:

  • swing-1.1\swing.jar

  • sax\sax2.jar

  • jaxp-1.1\jaxp.jar

  • jaxp-1.1\crimson.jar

XQEngine comes with a tester application that uses the Java Swing API. You can test the example asterixbooks.xml file we present in this section against the tester application.

Features of XQEngine

As we mentioned, XQEngine is open-source software and has the restrictions of a GPL license. It currently can handle the querying of 32,767 XML documents. Future implementations of this engine will not have this limit. Multiple documents up to that limit are handled simultaneously. No schemas or DTDs are required. If the programmer exceeds this limit, undesirable exceptions will be thrown. With the limit restriction removed in future versions, query results for large documents will be faster than they are currently.

XQEngine has an indexing capability, so any XML documents that are added as part of the searchable repository for the application using XQEngine need to index those documents through XQEngine before they can be used.

XQEngine is a full-text search engine and extends the XQuery/Xpath specifications. It provides a syntactic shorthand for searching on keywords. For example, assume an XML document contains the list of Asterix Comics currently available on the market. To search for the keyword Legionary in the title of the book, you can query as specified here:

//BOOK[TITLE &='legionary']

This query returns <BOOK> elements that include a <TITLE> containing the word legionary.

XQEngine also lets you search based on Boolean operators and logical operators. For example, if you restrict the search in the previous example to the words the legionary, you'll use the following query:

//BOOK[TITLE &='the legionary']

The search query is greatly narrowed, and the query will return <BOOK> elements that include a <TITLE> containing both the words the and legionary. Note that both the words need not be next to each other in a line that contains them.

In earlier versions of the XQEngine, query-parsing code was written using antlr, but now it is done using the JavaCC compiler. Support for XQL has been reduced a bit, along with the proprietary extensions that were available in the earlier versions of XQEngine. XQL is a query language using XML as the data model. Expressions in XQL are easily parsable, easy to type, and can be used in a variety of software environments. It is implemented in Web browsers, XML document repositories, XML middle-ware and command-line utilities.

Three types of Listeners were available in the earlier version of XQEngine:

  • Standard Listener

  • Summary Listener

  • CSV Listener

The current version has only the Standard Listener available. The other listeners will be added on demand. This means that if the Summary Listener or CSV listener is needed in a future release, it will be added based on the usage.

Phrasing support for text searches has been removed in the current version. For example, looking for the keyword Jakarta-Tomcat will find the exact version in previous implementations. So, the - and . operators no longer have special meaning in XQEngine.

XQEngine is SAX2 parser-compliant. This means you must use an XML parser that is SAX2-capable. SAX2 is a push-model parser: When SAX2 parses an XML document, it reads the XML document and passes a series of events to the event handlers. Some of the events that are generated include those occurring in the content of an XML document, those that occur in DTDs, and those that occur as errors.

XQEngine supports all the features of the XQuery specification. The primary additions include FLWR expressions and element and attribute constructors. Constructors let you specify the output, as well as the search that is being done. Using constructors allows for a flexible procedural approach for XSLT transformations.

Future versions of XQEngine will have a faster retrieved resultset, will remove the limit on the number of documents that can be searched, and will have a small memory footprint.

XQEngine APIs

The XQEngine API uses the JavaBeans concept of set and get methods for most method calls. Most of the methods within classes in this package are public void (they do not return a value). Exceptions that are thrown internally by different class methods belong to the package com.fatdog.xmlEngine.exceptions.

The following are the important classes that are useful when you're writing code using the XQEngine API:

  • class.com.fatdog.xmlEngine.XQEngine

  • interface com.fatdog.xmlEngine.query.ResultListener

  • interface com.fatdog.xmlEngine.query.ExplicitResultListener

For the XQEngine package, the methods can be further classified according to operation, as illustrated in Figure 12.1. The XQEngine class diagram appears in Figure 12.2.

Figure 12.1. Classification of methods in the XQEngine class based on operations.

Figure 12.2. Class diagram for XQEngine.

The class ResultListener supports the following method:

results ()

The class ExplicitResultListener supports the following methods:

results()
getExplicitDocument ()

Table 12.1 lists the usage of various methods in the XQEngine class.

Table 12.1. Methods in the XQEngine Class

METHOD NAME

USAGE

org.xml.sax.XMLReader getXMLReader ()

Returns a SAX2 parser instance.

setXMLReader (org.xml.sax.XMLReader xmlReader)

Installs a SAX2 XMLReader object for parsing an XML document.

setStopList (String filePath)

Pointer to a text file containing one word per line, indicating those that need not be indexed.

setMinIndexableWordLength (int len)

Default value is 1. Sets the length of words that will not be indexed.

setDoIndexNumbers (boolean indexNum)

True indicates that every word should be indexed. False indicates that words beginning with digits are ignored.

setDoFullText (Boolean doIndexElementFullText)

True indicates that attribute and element text context is indexed. False indicates that indexing takes place only for attribute text.

setResultListener (ResultListener l)

Registers a listener object to get the results from an XQuery search.

setDocument (String filePath)

Useful for indexing a document or documents; would be called in a for loop. Remember, the limit is 32,767 XML documents.

int setExplicitDocument (String xmlContents)

Used for indexing literal documents.

setEmitDocID (boolean emitDocID)

If True, emits document IDs for element and attribute nodes. The default value is False.

setQuery (String xQueryString)

Sends the XQuery query and result to the listener registered through the setResultListener method.

setIndexFileBaseName(String basename)

Sets the index file base name. Usually, two files starting with the base name are created with the extensions dict and xref. For example, if the basename is file1, the file1.dict and file1.xref files are created.

saveIndex()

Saves the state of the current index. The two files with extensions dict and xref are created at this method call using the base name set from the previous method.

restoreIndex()

Restores the saved current index.

printSessionStats()

Prints the current session's statistical summary of elements, attributes, text content, namespace, prefixes, and index memory usage.

printSessionStats (long dur,PrintWriter output)

Prints a statistical summary of the elements, attributes, text content, namespace, prefixes, and index memory usage, providing the duration of an index in milliseconds and the output where the summary is to be directed.

String getDocumentName(int docId)

Uses the document ID to get the XML document filename.

String getElementName(int docId, int elemIx)

Uses a document ID and the number of elements to retrieve the element name.

int getNumDocuments()

Gets the number of documents.

int getNumElements(int docId)

Gets the number of elements in a document, given the document ID.

int getNumWords(int docId, int elemIx)

Gets the number of words in element elemIx, given the document ID.

String showDocTree(int docId)

Given a document ID, returns a simple hierarchical tree representation of the elements for the document matching the passed document ID.

static String fileToURLString(File file)

Converts a file-based path to an InputSource argument for the SAXParser.

static mouseClickWindow(String windowtitle, int x, int y, int width, int height)

Keeps any output window after the program is exited.

Let's look at the other methods available in the two interfaces that are a part of the XQEngine API. The results(String results) method in the interface ResultListener returns the results of an XQuery query. In the ExplicitResultListener interface, the method results(String xqueryResults) returns results via the callback method. The getExplicitDocument(int docId) method takes the document ID and returns the external literal document as a String. The results method in both the ResultListener and ExplicitResult Listener interfaces returns a serialized resultset.

Having looked at the API, next let's look at some sample code that uses the XQEngine APIs.

Using the XQEngine API

The first step involved in writing a Java application using XQEngine is getting an instance of the XQEngine before you start indexing. The following code snippet shows how to get the instance of XQEngine:

XQEngine xqEngine = new XQEngine();

In this snippet, the default settings are invoked, so everything will be indexed. The following code prevents indexing of certain words, ignores numbers, and discards words of a particular length:

xqEngine.setStopList("\asterixavoid.txt");
xqEngine.setDoIndexNumbers(false);
xqEngine.setMinIndexableWordLength(5);

The first line gives the URL of the file where words that need not be indexed are provided. The second line says not to index numbers, and the third line sets indexing for words that are more than five characters in length.

Before you proceed, you must instantiate a SAX parser. You can use the JAXP parser provided by Sun Microsystems. The following code snippet constructs the instance of the SAX parser. Note that the code creates a SAX2 parser, because XQEngine currently supports it:

SAXParserFactory factory = SAXParserFactory.newInstance();
try{
       SAXParser saxParser = factory.newSAXParser();
       XMLReader xmlReader = saxParser.getXMLReader();
       xqEngine.setXMLReader(xmlReader);
}catch(Exception e){
       System.err.println("Could not set a XML Reader for the xqEngine: + e.getMessage());
}

Look at how this code gets the SAXParser instance. First, you use the SAXParserFactory to get a factory; then you use the factory to get a new SAXParser. Next, you get the XML reader for that parser and set it to the XQEngine. The SAXParserFactory uses the Factory Pattern to generate the different parser instance. The Factory Method pattern defines an interface for creating objects and retains control of which class needs to be instantiated.

The next step is to index the XML document against which the queries will be done:

xqEngine.setDocument("C:\\javaXQuery\\asterixbooks.xml");

This code assumes that an XML document called asterixbooks lists the Asterix books currently in the library. The relative path name or absolute path name can be used in the setDocument method.

This example does not use explicit documents. To index explicit documents, you need to use the method setExplicitDocument().

The next step is to set the XQuery query. Note that the query should follow the semantics and syntax defined in the W3C draft proposal for XQuery. The following code snippet sets the query for querying the asterixbooks file:

xqEngine.set("//BOOK/TITLE");

This line sets the query to return a result of all <TITLE> attributes for the element <BOOK>.

The resultset looks like this:

<xqe:result query="//BOOK/TITLE" xmlns:xql=http://www.fatdog.com/XQEngine.html>
<TITLE>Asterix in Britain</TITLE>
<TITLE>Asterix the Gladiator</TITLE>
<TITLE>Asterix at the Olympic Games</TITLE>
<TITLE>Asterix and the Big Fight</TITLE>
<TITLE>Asterix and the Roman Agent</TITLE>
<TITLE>Asterix in Switzerland</TITLE>
<TITLE>Asterix the Gaul</TITLE>
<TITLE>Asterix and Cleopatra</TITLE>
<TITLE>The Mansions of the Gods</TITLE>
<TITLE>Asterix and the Goths</TITLE>
<TITLE>Asterix the Legionary</TITLE>
<TITLE>Asterix in Spain</TITLE>
</xqe:result>

XQEngine uses a delegation event model like the Java AWT model. You need to register a listener with the XQEngine instance to retrieve the result. The following code snippet shows how you can register a listener for XQEngine:

engine.setResultListener(this);

Note that, as we discussed, XQEngine currently supports the Standard Listener. The Summary and CSV Listeners, which were part of the earlier versions of the API, will be available in the future.

You can also use XQuery queries to return attributes—for example, if the <TITLE> tag includes an attribute year. Consider this line from the asterixbooks.xml document:

<TITLE year=1975>Asterix and the Great Crossing</TITLE>.

If you ran a query like //@year on the XML document, the following resultset would be returned:

<?xml version="1.0"?>
<xqe:result query=//@year xmlns:xqe=http://www.fatdog.com.XQEngine.html>
<xqe:attribute name="year" value="1975"/>
</xqe:result>

Note how in the resultset the attribute name and its value are returned. If you wanted to get the element containing the attribute and not the attributes, a query like //*[@year] would produce the resultset shown here:

<?xml version="1.0"?>
<xqe:result query=//@year xmlns:xqe=http://www.fatdog.com.XQEngine.html>
<TITLE year=1975>Asterix the Legionary</TITLE>
</xqe:result>

Note that in this resultset, the whole element is returned.

XQEngine is a full text-based XML search engine. So, a standard query in both XQuery and Xpath looks like the following:

//firstPublished[ .= "Published in French"]

This returns a resultset that exactly matches the element <firstPublished>. Note that each word in the query string is mapped to each word in the retrieved resultset. Again, as we discussed earlier, you can use a combination of logical and conditional operators in your querying of XML documents.

SHOP TALK: USES FOR XQENGINE

The best use of XQEngine is creating zero-cost tools for the data mining of XML documents—personal productivity tools that run on a single machine. Because of its single-thread model, it can be a part of a CD-based application or used for providing XML querying on a server that has moderate traffic. Its most useful when you want to return resultsets as XML; you don't need to write a proprietary API to do that, but rather you can use an existing XQuery Engine API.

XQEngine has a simple API and is open source. There is no need to have any special XML storage databases; all the XML documents can be stored locally in the hard drive. Writing XQueries to process the stored XML documents is straightforward and elegant.


In Brief

  • Qexo compiles XQuery expressions or programs to bytecodes, and uses a ClassLoader for quicker interactive responses and fast execution of repetitive code.

  • You can use Qexo interactively, to process standalone XQuery programs, or to invoke it from Java.

  • Qexo can be run with Java Servlets to provide efficient server-side computation.

  • XQEngine utilizes XQuery as the front-end language and allows interrogation of collections of XML documents using Boolean combinations of keywords, just like any HTML engine, such as Google or Alta Vista.

  • XQEngine is used primarily for writing personal productivity tools that run on a single machine. Because of its single-thread model, it can be a part of a CD-based application, or used for providing XML querying on a server that has a moderate traffic.

  • XQEngine is useful for returning resultsets as XML using an existing XQuery Engine API.


Part IV: Appendixes

A XML Fundamentals

B Other Sources of Information


Appendix A. XML Fundamentals

by Vaidyanathan Nagarajan

IN THIS APPENDIX

  • Processing Models for XML Data

  • XML Documents

  • XML Documents and DTDs

  • Well-Formedness and Validity of XML Documents

  • XML Syntax and Semantics

  • Entity References

  • Rules for Writing XML

  • CDATA Section

This appendix covers XML fundamentals from a layman's perspective. It is written as a quick and easy reference for both programmers and managers: Programmers who are learning to code in XML can use this appendix as a base reference to get started with XML, and managers who want to know about XML can refer to this appendix to understand the basic XML technology.

Both HTML and XML use tags as part of the language semantics. The basic building blocks of an XML document are elements, attributes, and values. An element is the most basic unit of an XML document. The element can include other elements and text. All information in XML is coded between tags. Usually an element is written between an opening tag (<) and a closing tag (>). XML documents need to have both opening and closing tags to present information, but HTML is usually easy-going with tags. The term XML application refers to data described within an application-specific custom markup language tags.

Attributes are contained within an element's opening tag and use quotation marks with delimited values that describe the purpose and the content of the element. The association of a value with an attribute is considered to be metadata in XML terms. It's called metadata because it describes information about the data captured in the XML document. This code snippet shows the usage of tags, elements, and attributes:

<book name="Asterix and the Cauldron" year="1966"></book>

The example also introduces an important concept: empty elements in an XML document. Any element without a value between the opening and closing tags is referred to as an empty element.

Software programs can be written just to extract sufficient information needed from a XML stream. XML parsers are currently supported in all browsers, and viewing XML documents is quite easy.


Processing Models for XML Data

There are two models for processing XML data: DOM and SAX. The DOM model is costlier than SAX in terms of memory usage. DOM reads the whole XML tree into memory, and the more XML streams, the more memory is used to store the XML tree. SAX, on the other hand, is event-driven: Data is read when it's needed. Currently, SAX2 is the industry standard. More useful information on the SAX model can be found at http://www.saxproject.org.

Web designers can manage information effectively using XML as the standard technology. Lately, it's common for programmers to build XML streams directly from databases through their own proprietary API packages and using XSLT (another offshoot of XML technology), which functions like a database language and provides functions to manipulate XML streams to extract information. Nowadays, applications are designed using XSLT, XML, and business objects (business-processing code written in a programming language that provides capabilities to interact with databases to pull out information). There are also a large number of standard methods for producing XML, including Java APIs, SQL extensions, and so on. The diagram in Figure A.1 depicts this Web application architecture.

Figure A.1. Web application architecture using XSLT and XML.

This architecture works as follows:

  1. Business objects interact with the database to pull information and store in data beans.

  2. Dynamic pages call the data beans and create an embedded XML document.

  3. The template and XML document are parsed through an XSLT processor to present the information to the client.


XML Documents

We will use the XML SPY IDE for developing XML code and illustration purposes. XML SPY is currently at version 5.0 and is available for download at http://www.xmlspy.com. Three editions of this software are available: the Enterprise Edition is useful for building XML applications that interact with databases; the Professional Edition is useful for building enterprise-level XML applications; and the Developer Edition is useful for academics and people learning to write XML.

As an example, let's take the simple data information provided here and map it to XML:

COLUMN NAME | COLUMN VALUE
Name | Nathan Nagarajan
Address | 200 Great Pond Drive
City | Windsor
State | CT
Zip | 06095

You can convert this data information into an XML document, as shown here:

<?xml version="1.0" encoding="UTF-8"?>
<!--  Address book as an XML document -->
<addressbook>
    <person>
        <name>Nathan Nagarajan</name>
        <address>200 Great Pond Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06095</zip>
    </person>
</addressbook>

The data information is converted into a self-describing XML document. Note the logical way in which the order of the information is provided. Information is presented between data-describing tags—you have constructed an address book holding personal information. Opening tags are usually written as <tagName>, and closing tags as </tagName>.

Sometimes XML documents embed data type declaration (DTD) documents, which describe the order, sequence, and occurrence of tags (for example, required versus nonrequired) within the XML document. Nowadays, DTDs are vanishing from XML. However, we will describe them to give you a clear understanding of the various data type definitions available in XML.


XML Documents and DTDs

The following XML document is an extension of the previous example document with a reference to an external DTD:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Address book as an XML document -->
<!DOCTYPE addressbook SYSTEM "c:\addressbook.dtd">
<addressbook>
    <person>
        <name>Nathan Nagarajan</name>
        <address>200 Hop Meadow Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06040</zip>
    </person>
</addressbook>

The output of the document is shown in browser view in Figure A.2.

Figure A.2. Browser view of the XML document.

The code for the addressbook.dtd DTD file referenced in the XML document is as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT address (#PCDATA)>
<!ELEMENT addressbook (person)>
<!ELEMENT city (#PCDATA)>
<!ELEMENT name (#PCDATA)>
<!ELEMENT person (name, address, city, state, zip)>
<!ELEMENT state (#PCDATA)>
<!ELEMENT zip (#PCDATA)>

A couple of things are worth mentioning here. DTDs are being replaced currently by XML Schema (XSD, which is beyond the scope of this appendix). In addition, the example includes a DOCTYPE declaration—this directive is important, particularly in the HTML/XHTML world with respect to DTDs, because they are still the method by which specific versions of HTML are defined.


Well-Formedness and Validity of XML Documents

The concepts of well-formedness and validity are important in the world of XML. XML documents have to be both well-formed and valid.

Well-formed means each opening tag has a corresponding closing tag. For example, the <person> opening tag has a corresponding </person> closing tag. Valid means that the order, occurrence, and sequence of tags within a XML stream match the DTD.


XML Syntax and Semantics

We begin this section by describing the sample XML document in detail and presenting the syntax and semantics in writing XML applications.

XML Declaration

The first line in the XML document is called an XML declaration. This line is mandatory for generating XML documents:

<?xml version="1.0" encoding="UTF-8"?>

This code snippet tells the XML parser the version of XML being used to generate the XML application and also the encoding schema being used. The attribute version="1.0" provides the version of the XML specification. The attribute encoding="UTF-8" provides the character encoding for the XML document. (A few of the character-encoding schemas available to an XML document are UTF-8, UTF-16, EUC-JP, and ISO-10646-UCS2.)

There is an additional attribute we have not included in the example: standalone="no". This attribute can have the value yes or no. If the value is yes, there is no reference to any external DTD and the parser can simply parse the application. The value of no for the attribute implies that there is a reference to an external DTD, and the parser has to use the DTD to parse the XML document.

DTD Declaration

If there is a DTD file describing the tags used in the XML document, the following code snippet should follow the XML declaration. It's called the DTD declaration:

<!DOCTYPE addressbook SYSTEM "c:\addressbook.dtd">

Note that DTDs can be declared either external or internal. For example, if the DTD was served off a Web server, the DTD declaration would be as shown here:

<!DOCTYPE addressbook SYSTEM "http://www.xmlspy.com/XMLChapter/addressbook.dtd">.

The DTD declaration in our example is an external declaration. If our DTD was internal, the XML document would be as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!--  Address book as an XML document -->
<!DOCTYPE addressbook [
    <!ELEMENT address (#PCDATA)>
    <!ELEMENT addressbook (person)>
    <!ELEMENT city (#PCDATA)>
    <!ELEMENT name (#PCDATA)>
    <!ELEMENT person (name, address, city, state, zip)>
    <!ELEMENT state (#PCDATA)>
    <!ELEMENT zip (#PCDATA)>
]>
<addressbook>
    <person>
        <name>Nathan Nagarajan</name>
        <address>200 Great Pond Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06095</zip>
    </person>
</addressbook>

Root Element

Each XML document contains a tag called the root element, between which all the information is described. In our example, the root element is defined by <addressbook></addressbook> tags. Each Person object is defined between <person></person> tags.

Tag Attributes

It is possible to define attributes for a tag. So, for example, if you write the address book in a different way using attributes, the generated XML document looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE addressbook [
    <!ELEMENT address (#PCDATA)>
    <!ELEMENT addressbook (person)>
    <!ELEMENT city (#PCDATA)>
    <!ELEMENT name (#PCDATA)>
    <!ELEMENT person (name, address, city, state, zip)>
    <!ATTLIST person nickname CDATA #REQUIRED>
    <!ELEMENT state (#PCDATA)>
    <!ELEMENT zip (#PCDATA)>
]>
<addressbook>
    <person nickname="Nathan">
        <name>Nathan Nagarajan</name>
        <address>200 Great Pond Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06095</zip>
    </person>
</addressbook>

This example defines an attribute called nickname for the Person tag and incorporates it into the DTD. Going forward, you will see that we follow an internal DTD type declaration style to shrink the space required to present code.

It's good programming practice to always keep the DTD and XML document separate. Never complicate your XML documents by putting in attributes for tags.

Comments

Comments in an XML document are represented as in the following code snippet:

<!--  Address book as an XML document -->

Empty Elements

The empty element in an XML document can be written in two ways. In the sample XML document we have been using, assume that the address was empty for a person's name. You can represent empty tags in either of these two ways:

<address> </address>

or

<address/ >

The second way is more efficient. The first way, the XML parser has to parse three tags: <address>, empty string, </address>. In the second representation, it has to parse only one tag: <address/>.

CDATA

We will talk about the CDATA tag briefly here; the final section of the appendix also discusses CDATA. If the elements in an XML document are to be represented as text, a CDATA tag is useful. The syntax for a CDATA tag is as shown here:

<![CDATA[ ]]>


Entity References

Entity references are a voluminous topic. We'll examine a few important concepts from a DTD perspective.

With an ENTITY tag, you create an entity reference and the data the entity reference should expand to when referred to either in the DTD or the XML document. Entities can be broadly classified into two types: Parameter Entity declarations and General Entity declarations. Each is further subdivided into external or internal. If a General Entity is used, data is loaded directly into the XML document. If a Parameter Entity is used, it is a part of the DTD definition. The classification of internal or external within the General Entity is based on whether the entity reference is defined in a DTD or an external file.

Let's look at a simple example. In our XML document, add a description tag to each Person object. The XML code is shown here:

<?xml version="1.0" encoding="UTF-8"?>
<!--  Address book as an XML document -->
<!DOCTYPE addressbook [
    <!ELEMENT address (#PCDATA)>
    <!ELEMENT addressbook (person)>
    <!ELEMENT city (#PCDATA)>
    <!ELEMENT name (#PCDATA)>
    <!ELEMENT description (#PCDATA)>
    <!ELEMENT person (name, address, city, state, zip, description)>
    <!ATTLIST person nickname CDATA #REQUIRED>
    <!ELEMENT state (#PCDATA)>
    <!ELEMENT zip (#PCDATA)>
    <!ENTITY author "Nathan Nagarajan's">
]>
<addressbook>
    <person nickname="Nathan">
        <name>Nathan Nagarajan</name>
        <address>200 Great Pond Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06095</zip>
        <description>This is &author; work address.</description>
    </person>
</addressbook>

We have added a new tag called description and made it part of the Person element. The data type of the new tag is data. We also defined the following line in the DTD definition:

<!ENTITY author "Nathan Nagarajan's">

The ENTITY tag has an entity reference called author and associates the value "Nathan Nagarajan's" with the author.

The following code snippet shows how to reference the ENTITY in the main XML datastream:

<description>This is &author; work address.</description>

We reference the creator ENTITY defined in the DTD using an ampersand (&) at the beginning and a semicolon at the end.

If you save the program into an XML file and load the file in a browser, the output will be similar to Figure A.3. (The output shown here and in the remaining figures in the appendix is in the browser built into the XMLSPY IDE.)

Figure A.3. Browser view of the revised XML file.

In the diagram, note how the author entity was replaced with the words Nathan Nagarajan's.

The next example defines an ENTITY tag with reference to an external file and shows the output of the XML document. Assuming the description is defined in an external file named description.txt, the code is shown here:

<?xml version="1.0" encoding="UTF-8"?>
<!--  Address book as an XML document -->
<!DOCTYPE addressbook [
    <!ELEMENT address (#PCDATA)>
    <!ELEMENT addressbook (person)>
    <!ELEMENT city (#PCDATA)>
    <!ELEMENT name (#PCDATA)>
    <!ELEMENT description (#PCDATA)>
    <!ELEMENT person (name, address, city, state, zip, description)>
    <!ATTLIST person nickname CDATA #REQUIRED>
    <!ELEMENT state (#PCDATA)>
    <!ELEMENT zip (#PCDATA)>
    <!ENTITY author SYSTEM "description.txt">
]>
<addressbook>
    <person nickname="Nathan">
        <name>Nathan Nagarajan</name>
        <address>200 Great Pond Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06095</zip>
        <description>&author;</description>
    </person>
</addressbook>

The ENTITY description has been changed to reference an external file. The attribute SYSTEM within the ENTITY tag indicates to the XML parser that the value of the ENTITY reference is defined in another document—an external file called description.txt.

Create a file named description.txt using Notepad and paste the following line into it:

This is Nathan Nagarajan's work address.

Save the file in the same directory where you have stored the XML file. If you store your description file in a different place, fully qualify the path to the file in the ENTITY tag. Now when you run the file in a browser (either Internet Explorer or Mozilla), the output should be identical to that shown in Figure A.3—the text from the description.txt file is placed in the XML stream between the <description> and </description> tags.


Rules for Writing XML

In this section, we present some rules for writing XML. If an XML document satisfies this set of rules, it is considered well-formed. Being well-formed implies that the document can be displayed by the browser.

Root Element Is Required

There cannot be more than one root element in an XML document. Only comments and processing instructions can precede a root element in an XML document.

Closing Tags Are Necessary

In an XML document, there should be a closing tag for every opening tag. Note that the closing tags start with a /. The slash distinguishes an opening tag from a closing tag.

Proper Nesting of Elements

Elements should be properly nested in an XML document. For example, in the following code snippet, the element <author></author> is inside <book></book>, so the author tag should be closed before the book tag:

<book>
<author>Goscinny</author>
</book>

The following code is wrong because the author tag is closed after book, but author is defined inside book:

<book>
<author>Goscinny
</book>
</author>

Case Matters

XML is case sensitive. In XML, there is a difference between the book and BOOK tags—these tags are separate and unrelated.

Whitespace

Extra whitespaces can be added around elements in an XML document. Refer to any of the examples in this appendix; they have whitespace around the elements. Extra whitespaces are ignored by the parser in the browsers.

Quotations Are Needed for Attribute Values

Attribute values should be in quotation marks. Single or double quotation marks are OK.

Entity References Should Be Declared

Any entity reference except the following five should be declared in the DTD before being referenced:

&amp;

&lt;

&gt;

&quot;

&apos;

These five special symbol entities are built into XML. You're only allowed to use these five entities. If other entities are to be defined, they must be defined in the DTD before being used in the XML document.

The general syntax for referencing entities is as shown here (ampersand followed by the reference name followed by a semicolon):

&entity-reference-name;

Table A.1 lists the predefined General Entities and their values, along with their ASCII character equivalents.

Table A.1. Mapping of Predefined General Entities

ENTITY

DISPLAY

ASCII CODE

&amp;

&

&#38; #38;

&lt;

<

&#38;#60;

&gt;

>

&#62;

&apos;

'

&#39;

&quot;

"

&#34;

The following sample XML code references all the predefined General Entities shown in Table A.1:

[View full width]

<?xml version="1.0" encoding="UTF-8"?>
<!-- This example generates all the pre-defined Entity symbols -->
<exampleroot>
    <title>Authors of Professional Cartooning Book and annual salary are listed below</title>
    <author>
        <name>Nathan</name>
        <salarydescription>Nathan&apos;s annual salary is &gt;60,000 &amp; &lt;65,000 <
  /salarydescription>
        <description>Nathan says, &quot;No Cartoonist can equal my salary. &quot;<
  /description>
    </author>
    <author>
        <name>Nagarajan</name>
        <salarydescription>Nagarajan&apos;s annual salary is &gt;200,000 &amp; &lt;250,000
   </salarydescription>
        <description>Nagarajan says, &quot;I am paid to keep Cartoons in BAY. &quot;<
  /description>
    </author>
</exampleroot>

In this code, note the usage of the predefined entity symbols. An apostrophe is used in the <title/> tag. Less than, greater than, and ampersand are used in the <salarydescription/> tag. Quotes are used in the <description/> tag. If you save the code and run it in a browser, the output will be similar to that shown in Figure A.4.

Figure A.4. Browser view using predefined entity symbols.

DTD Elements

Next, we will take a sample code snippet of a DTD and talk about important elements within the DTD from our example's viewpoint. Consider the following code snippet:

<!DOCTYPE addressbook[
<!ELEMENT address (#PCDATA)>
<!ELEMENT addressbook (person)>
<!ELEMENT city (#PCDATA)>
<!ELEMENT name (#PCDATA)>
<!ELEMENT person (name, address, city, state, zip, description?)*>
<!ATTLIST person
    nickname CDATA #REQUIRED
>
<!ELEMENT state (#PCDATA)>
<!ELEMENT zip (#PCDATA)>
<!ELEMENT description (#PCDATA)>
]>

The ELEMENT tag name is associated to a data type called #PCDATA, which means the <name/> tag can contain text data. In the <person/> tag, the children tags (name, address, city, state, zip, description?) are separated by commas. The comma indicates the sequence in which the children tags appear.

The asterisk (*) at the end of the <person/> tag indicates that the tag can have zero, one, or more appearances within the XML document:

<!ELEMENT person (name, address, city, state, zip, description?)*>

If it were like this:

<!ELEMENT person (name, address, city, state, zip, description?)+>

the plus (+) at the end would indicate that the tag should appear at least once (or more).

Parentheses ( ) indicate the start and end of a grouping, as in (name, address, city, state, zip, description?). The question mark (?) associated with description indicates that the <description/> tag is optional.

The following code snippet indicates that the nickname attribute for <person/> is required:

<!ATTLIST person nickname CDATA #REQUIRED>

If the nickname attribute is not present, the XML parser will throw an error.

A Notation tag is usually declared in the DTD. This tag includes information about the type of content. Here's an example of such a usage:

<!NOTATION JPG SYSTEM "image/jpeg">


CDATA Section

As we said earlier, if the elements in an XML document are to be represented as static text, a CDATA tag is useful. The following code snippet shows the use of CDATA:

<?xml version="1.0" encoding="UTF-8"?>
<!--  Address book as an XML document -->
<!DOCTYPE addressbook SYSTEM "c:\addressbook_cdata.dtd">
<addressbook>
    <person>
        <name>Nathan Nagarajan</name>
        <address>200 Great Pond Drive</address>
        <city>Windsor</city>
        <state>CT</state>
        <zip>06095</zip>
    </person><![CDATA[
        <person>
            <name>Nathan Nagarajan</name>
            <address>200 Great Pond Drive</address>
            <city>Windsor</city>
            <state>CT</state>
            <zip>06095</zip>
        </person>
]]></addressbook>

The example defines CDATA and defines the entire person data within the CDATA block. The output in Figure A.5 shows how CDATA is rendered by a browser that supports XML—the CDATA block has the text embedded within it.

Figure A.5. Browser view of CDATA.

SHOP TALK: XML: THE WEB SERVICES STANDARD

The best way to design future Web applications will be to use XML as the data-passing mechanism. Currently, the best practice I have seen is to use a combination of XML and XSLT to generate HTML pages. Following this best practice enables you to change the look and feel of your Web site whenever needed without incurring overhead; in addition, maintenance costs are low and changes are easy to implement.

Currently, most Web applications are designed as multitier systems, separating presentation from business processing. A widely adopted framework in the world of Java applications has been the use of Apache's Struts framework, which provides a Model-View-Controller pattern for Web application development. Another presentation framework from Apache is Cocoon; it combines the power of XSLT and XML for the Presentation layer in a Web application. Adopting a framework increases the life expectancy of your Web application.

Web Services is the future of Web applications. To join the bandwagon of supporting Web Services, you need to design all your applications to generate XML content. The current de facto standard for Web Services is XML. Designing your current Web site with a vision of the future to support Web Services lends itself to having XML as the data mechanism for Web Services support.

Microsoft's strategy for the future is to use XML as a data communicator for all data passing between different components in Windows. In designing a portal with a standard security framework, I have found that XML is the way to go; the solution has worked well in several implementations of the framework across different partner sites.


Appendix B. Other Sources of Information

IN THIS APPENDIX

  • Specifications

  • Books

  • List Servers

  • Implementations

The first and primary source of information on XQuery is the W3C Web site at http://www.w3c.org. The following publications are related to topics covered within this book and might be of interest to you.


Specifications

  • Enterprise JavaBeans Specification, Sun Microsystems, Inc.— http://java.sun.com/j2ee/docs.html

  • Java API for XML Processing (JAXP) Specification— http://java.sun.com/xml/jaxp/index.html

  • Java API for XML Registries (JAXR) Specification— http://java.sun.com/xml/jaxr/index.html

  • Java Community Process— http://www.jcp.org

  • Microsoft Developer Network— http://msdn.microsoft.com

  • Universal Description, Discovery, and Integration (UDDI) Specification— http://www.uddi.org

  • XML Query Use Cases— http://www.w3.org/TR/xmlquery-use-cases/

  • XSL Transformations (XSLT) 2.0— http://www.w3.org/TR/xslt20/

  • XQuery 1.0 Requirements— http://www.w3.org/TR/xmlquery-req


Books

• Daconta, Michael, Leo Obrst, and Kevin Smith. The Semantic Web: A Guide to the Future of XML, Web Services, and Knowledge Management. Indianapolis: John Wiley and Sons, 2003.

• Kassem, Nicholas and the Enterprise Team. Designing Enterprise Applications with the Java 2 Platform, Enterprise Edition. Boston: Addison-Wesley, 2000.

• Kay, Michael. XSLT Programmer's Reference, 2nd ed. Birmingham, UK: Wrox Press, 2003.

• Mangano, Sal. XSLT Cookbook. Cambridge, Mass.: O'Reilly, 2002.

• Marchal, Benoît. Applied XML Solutions. Indianapolis: Sams, 2000.

• McGovern, James, Sameer Tyagi, Michael Stevens, and Sunil Mathew. Java Web Services Architecture. San Francisco: Morgan Kaufmann, 2003.

• Ray, Erik T. Learning XML. Cambridge, Mass.: O'Reilly, 2001.

• Richter, Jeffrey. Applied Microsoft .NET Framework Programming. Redmond, Wa.: Microsoft Press, 2002.

• Wahlin, Dan. XML for ASP.NET Developers. Indianapolis: Sams, 2001.

• Wilson, Mark and Tracey Wilson. XML Programming with VB and ASP. Greenwich, Conn.: Manning, 1999.


List Servers

  • XML applications— http://groups.yahoo.com/group/xml-applications-tools

  • XQuery— http://groups.yahoo.com/group/xquery


Implementations

  • Derby, from GAEL— http://www.gael.fr/derby/

  • Galax, from Lucent— http://db.bell-labs.com/galax/

  • IPSI-XQ, from IPSI— http://ipsi.fhg.de/oasys/projects/ipsi-xq/index_e.html

  • Liquid Data for WebLogic 1.0, from BEA— http://edocs.bea.com/liquiddata/docs10/prodover/concepts.html

  • Quip XQuery prototype, from Software AG— http://www.softwareag.com/developer/quip/default.htm

  • Saxon, from SourceForge— http://saxon.sourceforge.net/

  • SQL/XML-IMDB, from QuiLogic— http://www.quilogic.cc/xml.htm

  • Virtuoso Universal Server, from OpenLink Software— http://demo.openlinksw.com

  • X-Hive XQuery demo— http://www.x-hive.com/xquery

  • XML DB, from Oracle— http://otn.oracle.com/tech/xml/xmldb

  • XQBE, from Politecnico di Milano— http://dbgroup.elet.polimi.it/xquery/xqbedownload.html

  • XQE, from Cerisent— http://cerisent.com/cerisent-xqe.html

  • XQuantum, from Cognetic Systems— http://www.cogneticsystems.com/xquery/xquery.html

  • XQuench XML Query parser and engine, from SourceForge— http://sourceforge.net/projects/xquench

  • XQEngine full-text XML search engine, from FatDog Software— http://www.fatdog.com

  • XQuery database demo, from X-Hive— http://www.x-hive.com/xquery

  • XQuery language demo, from Microsoft— http://xqueryservices.com

  • XStreamDB, from Bluestream Database Software Corp.— http://www.bluestream.com/dr/?page=Home/Products/XStreamDB/


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

***underline to operator

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

abstraction
access
concurrency control 2nd 3rd 4th 5th 6th 7th 8th
accessibility
functions
ACID (Atomic, Consistent, Isolated, and Durable) 2nd
ACID (atomicity, consistency, isolation, and durability)
ACLs
RDBMSs
addressing
nodes 2nd
XPath 1.0 2nd 3rd
XPath 2.0 2nd
Adobe
XMP
aggreagte functions 2nd
algorithms
processing models 2nd 3rd
analysis
static typing 2nd 3rd
annotation
type 2nd 3rd
annotations
type 2nd
anonymous types 2nd
APIs
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
XQEngine 2nd 3rd 4th 5th 6th 7th
applications
CGI scripts
executing as 2nd 3rd
compiling 2nd
execuitng Qexo 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
photo albums
creating 2nd 3rd 4th 5th
RSS 2nd 3rd 4th 5th 6th
buildinng syndication engines 2nd 3rd 4th 5th
generating from PostgreSQL 2nd 3rd 4th 5th 6th
querying feeds 2nd 3rd
root node queries 2nd
servlets
executing as 2nd 3rd 4th 5th 6th 7th
XML
portable data 2nd
XUL
applying
root tags
Semantic Web project 2nd
XPath 2nd
Arbortext
architecture
CORBA
GoXML DB
IBM DB2
arguments
aggregate functions 2nd
ascending keyword
assignments 2nd
association
schemas
atomic value
atomic values
types 2nd 3rd 4th
Atomic, Consistent, Isolated, and Durable. [See ACID]
atomicity, consistency, isolation, and durability (ACID)
Attribute node. [See also nodes]
Attribute nodes
attributes
tags
XML
axes
XPath

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

B2B (Business-to-Business)
books on XQuery
Boolean values
expressions
optimizing 2nd 3rd 4th
borrowing library modules 2nd 3rd 4th
building
code libraries 2nd
custom switch structures 2nd
syndication engines 2nd 3rd 4th 5th
Business-to-Business (B2B)
bytecodes
compiling

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

calling
imported variables 2nd
Java expressions 2nd 3rd 4th 5th
Java methods
schemas 2nd 3rd 4th
CDATA 2nd 3rd
CData-Sections
CGI (Common Gateway Interface)
scripts
executing applications from 2nd 3rd
characters
logical expressions 2nd 3rd 4th
ordering
child elements
XML
children nodes
order by command 2nd 3rd 4th 5th
children. [See also nodes]
classes 2nd
Microsoft XQuery Services 2nd 3rd 4th 5th 6th
classes 2nd
TreeList DOM 2nd
clauses
return
where 2nd 3rd 4th
closure (XQuery Core)
code
compiling 2nd 3rd
consolidating
JXQI 2nd 3rd 4th 5th
libraries 2nd
semantics
data models 2nd 3rd 4th 5th 6th 7th
expressions 2nd 3rd
expressions (XQuery Core)
functions 2nd 3rd 4th 5th
inference rules 2nd
mapping XQuery Core 2nd 3rd
normalization 2nd
notation
prime types 2nd
processing models 2nd 3rd
subtyping rules 2nd
type (XQuery Core)
types
XQuery Core
XPath 2nd
codes
Visual Basic
collation
Command Pattern
commands
DDL
declare namespace
DML 2nd
import schema 2nd 3rd 4th
order by 2nd 3rd 4th 5th
structures
building custom switches 2nd
for-return statements 2nd 3rd
if-then-else statements 2nd 3rd 4th
XPath 2nd
Comment nodes
comments
commit transactions. [See also transactions]
Common Object Broker Architecture (CORBA)
comparisons
JSP
XSLT 2nd 3rd 4th 5th 6th 7th 8th
compilers
Qexo 2nd
executing 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
compiling 2nd 3rd
static typing 2nd 3rd
completeness (XQuery Core)
complex types 2nd 3rd 4th
composition (XQuery Core)
concatenate() function
conclusions
concurrency control 2nd 3rd 4th 5th 6th 7th 8th
conditional logic. [See also logical expressions]
conditions
expressions
optimizing 2nd 3rd 4th
where keyword 2nd 3rd 4th
configuration
code libraries 2nd
photo album applications 2nd 3rd 4th 5th
configuring
paths
connections
GoXML DB servers
JXQI 2nd 3rd 4th 5th 6th 7th
consolidation
reused code
Consumer interface 2nd 3rd 4th
context 2nd 3rd
conversion
tokenize() function 2nd 3rd
CORBA (Common Object Broker Architecture)
coreExpression
correctness (XQuery Core)
CountElements.java
Create, Read, Update, and Delete (CRUD)
CRUD
XQuery 2nd
CRUD (Create, Read, Update, and Delete)
custom switch structures, building 2nd
customization
tables 2nd 3rd 4th 5th 6th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

DAML+OIL (DARPA Agent Markup Language + Ontology Inference Layer)
DARPA Agent Markup Language + Ontology Inference Layer (DAML+OIL) 2nd
Data Definition Language (DDL)
data elements (XML)
Data Manipulation Language (DML) 2nd
data models
atomic value
nodes
identity 2nd
order
sequences
data storage
XML 2nd
data-centric XML
databases 2nd 3rd
paradigms for 2nd 3rd 4th
persistence strategies for 2nd 3rd 4th
RDMS persistence strategies for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
requirements for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
data types
XPath 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
XML
Xpath
XML
XPath
XML
Xpath
XML
XPath
XML
Xpath
XML
data-centric XML, persistent storage of 2nd 3rd 4th
databases 2nd
ACID 2nd
directories 2nd
GoXML DB support 2nd 3rd 4th 5th 6th 7th
IBM DB2 support 2nd
JXQI
connecting 2nd 3rd 4th 5th 6th 7th
nodes 2nd
addressing 2nd
Oracle 9i support 2nd
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
relational technology 2nd
SQL. [See also SQL]
UDB 2nd 3rd 4th 5th 6th 7th 8th
XDBMSs
XML 2nd 3rd
data-centric XML
requirements for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
XPath
applying 2nd
dates
XPath 2nd 3rd 4th 5th 6th
Xpath
XML 2nd 3rd
DDL (Data Definition Language)
declarations
DTDs 2nd
XML 2nd
declare namespace command
defining
conditions
where keyword 2nd 3rd 4th
external functions 2nd 3rd
functions 2nd 3rd 4th 5th 6th 7th
schemas
variables
descending keyword
design
data storage 2nd
data-centric-XML
databases 2nd 3rd
paradigms for 2nd 3rd 4th
persistence strategies for 2nd 3rd 4th
RDMS persistence strategies for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
requirements for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
SVG 2nd 3rd 4th 5th
tables
customizing 2nd 3rd 4th 5th 6th
detection
static typing 2nd 3rd
development
expressions 2nd 3rd 4th 5th 6th
XDK
directories
nodes. [See also nodes]
XPath 2nd
dirty reads
DML (Data Manipulation Language) 2nd
DocBook
HTML
transformaing to 2nd 3rd 4th 5th 6th 7th
Document node. [See also nodes]
Document nodes
Document Type Definition)
document() function 2nd
documents
Oracle 9i support 2nd
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
order
ordering 2nd 3rd 4th 5th
XML 2nd 3rd 4th
building custom switches 2nd
command structures 2nd
DTDs 2nd 3rd
external data sources 2nd 3rd
for-return 2nd 3rd
if-then-else statements 2nd 3rd 4th
infosets 2nd
node values 2nd
publishing 2nd 3rd 4th 5th 6th 7th
rules of 2nd 3rd 4th
Semantic Web project 2nd 3rd 4th 5th 6th
validity of
well-formedness
XML. [See also XML]
XQEngine 2nd 3rd 4th
APIs 2nd 3rd 4th 5th 6th 7th
DOM
XML processing models 2nd
DTD (Document Type Definition)
DTD. [See also schemas]
DTDs 2nd 3rd
declarations 2nd
entity references 2nd 3rd 4th
dynamic semantics
dynamic type

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

EAI (Enterprise Application Integration)
EDI (Electronic Data Interchange)
editing
command structures 2nd
building custom switches 2nd
for-return statements 2nd 3rd
if-then-else statements 2nd 3rd 4th
overview pages 2nd 3rd
Electronic Data Interchange (EDI)
element constructor expression
Element node. [See also nodes]
Element nodes
elements
child
Xml
data (XML)
empty 2nd
extensions
processing with 2nd
internal/external
nesting
parameter
root 2nd
XML
vaidating 2nd
empty elements 2nd
engines
syndication
building 2nd 3rd 4th 5th
Enterprise Application Integration (EAI)
enterprise applications
integrating
entity references 2nd 3rd 4th
enumeration
to operators 2nd 3rd 4th
equality
nodes
errors
static typing 2nd 3rd
every keyword
evolution of XML 2nd
exclusive write locks
executing
Qexo 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
replacement XQuery
execution
CGI scripts
applications as 2nd 3rd
queries
servlets
applications as 2nd 3rd 4th 5th 6th 7th
experssions
coreExpression
expressions
compiling 2nd
executing Qexo 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
developing 2nd 3rd 4th 5th 6th
element constructor
functions
defining 2nd 3rd 4th 5th 6th 7th
namespaces 2nd
Java
calling 2nd 3rd 4th 5th
logical 2nd 3rd 4th
optimizing 2nd 3rd 4th
order by command 2nd 3rd 4th 5th
regular 2nd 3rd 4th
flags
matches() function 2nd
replace() function 2nd
tokenize() function 2nd
results
returning 2nd 3rd 4th 5th 6th 7th 8th
semantics 2nd 3rd
SQL 2nd
to
XPath 2nd 3rd 4th 5th 6th 7th 8th
XQuery Core
extenal functions
invoking 2nd 3rd
Extensible Markup Language. [See XML]
Extensible Metadata Platform. [See XMP]
Extensible Stylesheet Language. [See XSL]
extension functions
extensions
SHOE
extenssions
processing with 2nd
external data sources
XPath 2nd 3rd
external elements
external libraries
externals DTDs 2nd 3rd

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

feeds, querying RSS 2nd 3rd
files
XML
infosets 2nd
nodes values 2nd
XML. [See also XML]
flags
for keyword
iterations 2nd 3rd
for loops
to operators
enumerating 2nd 3rd 4th
For, Where, Let, Order By, Return (FWLOR) statements
for-return statements 2nd 3rd
format-row function
formatting
code libraries 2nd
HTML 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
inheritance
paths
photo album applications 2nd 3rd 4th 5th
pipelines 2nd 3rd
SVG 2nd 3rd 4th 5th
Web pages 2nd
creating photo album applications 2nd 3rd 4th 5th
editing overview pages 2nd 3rd
image pages 2nd 3rd 4th 5th 6th 7th 8th
overview pages 2nd 3rd 4th 5th 6th
forms
normalization
fragmentation
JavaScript
fragments
documents
frameworks
RDF. [See also RDF]
XMP 2nd
fucntions
accessibility
full-text search engines
XQEngine 2nd 3rd 4th
APIs 2nd 3rd 4th 5th 6th 7th
functions 2nd 3rd 4th
aggreagte 2nd
code
libraries 2nd
concatenate()
defining 2nd 3rd 4th 5th 6th 7th
document ()
document()
external
invoking 2nd 3rd
format-row
graph()
index-of()
insert()
libraries
make-rows
matches() 2nd
namespaces 2nd
picture
position()
remove()
replace() 2nd
semantics
sequences 2nd 3rd 4th 5th 6th 7th
modifying set functions 2nd 3rd
subsequence
subtyping rules 2nd
tokenize() 2nd 3rd 4th 5th
XML
portable data 2nd
XPath 2nd 3rd
FWLOR (For, Where, Let, Order By, Return) statements

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

generating
XML
from resultsets
generating
XML from PostgreSQL 2nd 3rd 4th 5th 6th
generating Web pages 2nd
creating photo album applications 2nd 3rd 4th 5th
editing overview pages 2nd 3rd
image pages 2nd 3rd 4th 5th 6th 7th 8th
overview pages 2nd 3rd 4th 5th 6th
Goldfarb, Dr. Charles
GoXML DB support 2nd 3rd 4th 5th 6th 7th
graph() function
graphics
indexes
photo album applications
creating 2nd 3rd 4th 5th
SVG 2nd 3rd 4th 5th
SVG. [See also SVG]
GUI
XQuery tools

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

hierarchies
databases. [See databases]
Semantic Web project
XML
nodes 2nd 3rd
HTML
DocBook
transforming to 2nd 3rd 4th 5th 6th 7th
formatting 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
SHOE
Web pages
editing overview pages 2nd 3rd
generating 2nd 3rd 4th 5th 6th 7th
image pages 2nd 3rd 4th 5th 6th 7th 8th
overview pages 2nd 3rd 4th 5th 6th
HTML Hyper Text Markup Language
Hyper Text Markup Language. [See HTML]
hyperlinks
XLink 2nd

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

IBM DB2 support 2nd
identities
nodes
XML 2nd 3rd
identity
nodes 2nd
if keyword 2nd 3rd 4th
if-then-else statements 2nd 3rd 4th
images
indexes
photo album applications
creating 2nd 3rd 4th 5th
SVG 2nd 3rd 4th 5th
Web pages
generating 2nd 3rd 4th 5th 6th 7th 8th
implementation
code
compiling 2nd 3rd
extensions
pragmas 2nd
static typing 2nd 3rd
importing
modules
borrowing library 2nd 3rd 4th
calling variables 2nd
schemas 2nd 3rd 4th
index-of() function
indexes
images
indexing
DB2
RDBMSs
XDBMSs
inference rules 2nd
infosets
XML 2nd
inheritance
creating
insert() function
installation
JXQI 2nd 3rd
servlets
Tomcat servers 2nd 3rd
instances
integration
enterprise applications
interactions
databases
interactive loops
Qexo
interfaces
CGI
executing applications from 2nd 3rd
Consumer 2nd 3rd 4th
XML
XQuery tools
XUL
internal elements
invocation
RMI
invoking
external functions 2nd 3rd
librarieslmain modules
isolation levels. [See also access]
items 2nd
aggregate functions 2nd
expressions
optimizing 2nd 3rd 4th
for keyword
iterations 2nd 3rd
sequences 2nd
enumerating to operators 2nd 3rd 4th
functions 2nd 3rd 4th 5th 6th 7th
modifying set functions 2nd 3rd
node-sets 2nd
iteration
for loops
enumerating to operators 2nd 3rd 4th
iterations
for keyword 2nd 3rd

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

Java
APIs
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
expressions
calling 2nd 3rd 4th 5th
methods
calling
Qexo 2nd
executing 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
RMI
Java Messaging Service (JMS)
JavaScript
fragmentation
JMS (Java Messaging Service)
JSP
servlets
comparing
judgements
inference rules 2nd
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

keywords 2nd 3rd 4th
ascending
descending
every
for
iterations 2nd 3rd
if 2nd 3rd 4th
order by 2nd 3rd 4th 5th
return 2nd 3rd 4th 5th 6th 7th 8th
where 2nd 3rd 4th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

lanaguages
assignments 2nd
languages
functions
HTML. [See also HTML]
JavaScript
fragmentation
RDF. [See also RDF]
regular expressions
semantics
dat models 2nd 3rd 4th 5th 6th 7th
expressions 2nd 3rd
expressions (XQuery Core)
functions 2nd 3rd 4th
inference rules 2nd
mapping XQuery Core 2nd 3rd
normalization 2nd
notation
prime types 2nd
processing models 2nd 3rd
subtyping rules 2nd
type (XQuery Core)
types
XQuery Core
SGML. [See also SGML]
SQL
expressions 2nd
SQL. [See also SQL]
structure of 2nd 3rd 4th
Xlink. [See also XLink]
XML. [See also XML]
XPath
XPATH. [See also XPath]
XPointer. [See also XPointer]
XSL. [See also XSL]
XSLT
XUL. [See also XUL]
levels
expressions
XQuery Core
transaction
type
XQuery Core
lexical space
libraries
code 2nd
main modules
invoking
managing 2nd 3rd 4th
library modules
borrowing 2nd 3rd 4th
optimizing 2nd 3rd 4th 5th
variables
calling 2nd
limitations
of XQuery 2nd
list servers for XQuery
lists
logical expressions 2nd 3rd 4th
sequences 2nd
enumerating to operators 2nd 3rd 4th
functions 2nd 3rd 4th 5th 6th 7th
modifying set functions 2nd 3rd
node-sets 2nd
loading
XML
LOB storage model
location paths
locking mechanisms 2nd
logical expressions 2nd 3rd 4th
loops
for
enumerating to operators 2nd 3rd 4th
interactive
Qexo

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

main modules
libraries
invoking
make-rows function
management
concurrency control 2nd 3rd 4th 5th 6th 7th 8th
libraries 2nd 3rd 4th
XDBMSs
mapping
relational-to-object databases
transactions 2nd
XQuery Core 2nd 3rd
matches() function 2nd
messages
XML
metadata
RDBMSs 2nd
resultsets
generating XML from
XMP 2nd
metadata hierarchies
Semantic Web project
methods
Java
calling
RMI
Microsoft XQuery Services 2nd 3rd 4th 5th 6th 7th
classes 2nd
models
data
atomic value
identity (nodes) 2nd
nodes
order
sequences
processing 2nd 3rd 4th 5th
modifying
command structures 2nd
building custom switches 2nd
for-return statements 2nd 3rd
if-then-else statements 2nd 3rd 4th
set functions 2nd 3rd
modules
importing
borrowing library 2nd 3rd 4th
calling variables 2nd
schemas 2nd 3rd 4th
library
optimizing 2nd 3rd 4th 5th
multiple access
concurrency control 2nd 3rd 4th 5th 6th 7th 8th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

names
QName
Namespace node. [See also nodes]
Namespace nodes
namespaces 2nd 3rd
library modules
borrowing 2nd 3rd 4th
pragmas 2nd
native storage
XML 2nd 3rd 4th 5th 6th 7th 8th
navigation
XPath 2nd
nesting elements
node-sets
sequences 2nd
enumerating to operators 2nd 3rd 4th
functions 2nd 3rd 4th 5th 6th 7th
modifying set functions 2nd 3rd
nodes
data models
identity 2nd
equality
results
returning 2nd 3rd 4th 5th 6th 7th 8th
XML
document fragements
identities 2nd 3rd
string/typed values
values 2nd
XPATH
XPath 2nd 3rd 4th
addressing 2nd
XPath 1.0 2nd 3rd
XPath 2.0 2nd
nonrepeatable reads
normalization 2nd 3rd
normalization of data
notation

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

object databases 2nd
object-relational databases 2nd
Object-Relational storage model
objects
CORBA
ontologies 2nd 3rd
RDF 2nd
relational-to-object databases
transactions 2nd
ontologies 2nd 3rd
creating
openness
operations
DDL
DML 2nd
operators
to
enumerating 2nd 3rd 4th
optimistic concurrency
optimization
expressions 2nd 3rd 4th
extensions
extenstions
processing with 2nd
library modules 2nd 3rd 4th 5th
pragmas
processing 2nd
tables 2nd 3rd 4th 5th 6th
Oracle 9i support 2nd
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
OracleXQueryBindDemo.java
OracleXQueryDBDemo program
OracleXQueryDBDemo.java
OracleXQueryDBFilterDemo program
OracleXQueryDBFilterDemo.java
OracleXQueryDemo program
order
data models
order by command 2nd 3rd 4th 5th
organization
of data. [See also Semantic Web project]
output
results
returning 2nd 3rd 4th 5th 6th 7th 8th
overview pages
generating 2nd 3rd 4th 5th 6th
editing 2nd 3rd

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

paradigms for XML data storage 2nd 3rd 4th
parameterized XQuery 2nd 3rd 4th
parent nodes
order by command 2nd 3rd 4th 5th
parsers
XML
validating
parsing
order by command 2nd 3rd 4th 5th
paths
creating
location
performance
expressions
optimizing 2nd 3rd 4th
persistence strategies for XML data storage 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th
persistent storage
data-centric-XML
phantom reads
phases
processing models 2nd
photo album applications, creating 2nd 3rd 4th 5th
picture function
pictures.
images [See graphics]
pipelines
creating 2nd 3rd
portable data
XML 2nd
position() function
post-schema validation infoset (PSVI)
PostgreSQL, generating XML from 2nd 3rd 4th 5th 6th
pragmas
predicate clauses 2nd 3rd 4th
predicates
RDF 2nd
premises
prgamas
processing with 2nd
procedures
library modules
optimizing 2nd 3rd 4th 5th
processing
extensions 2nd
pragmas 2nd
Processing Instruction nodes
processing models 2nd 3rd
XML 2nd
processing-instructions
programming
HTML. [See also HTML]
RDF. [See also RDF]
semantics
data models 2nd 3rd 4th 5th 6th 7th
expressions 2nd 3rd
expressions (XQuery Core)
functions 2nd 3rd 4th 5th
inference rules 2nd
mapping XQuery Core 2nd 3rd
normalization 2nd
notation
prime types 2nd
processing models 2nd 3rd
subtyping rules 2nd
type (XQuery Core)
types
XQuery Core
SGML. [See also SGML]
SQL. [See also SQL]
Xlink. [See also XLink]
XML. [See also XML]
XPATH. [See also XPath]
XPointer. [See also XPointer]
XSL. [See also XSL]
XUL. [See also XUL]
programs. [See applications]
properties
databases
ACID 2nd
protocols
SOAP 2nd
XML
XLink 2nd
XPath 2nd 3rd 4th 5th 6th 7th 8th 9th
XPointer
XSL 2nd
XSLT 2nd 3rd
PSVI (post-schema validation infoset)
publishing
XML documents 2nd 3rd 4th 5th 6th 7th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

Qexo 2nd
executing 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th
QName
queries
executing
GoXML DB support 2nd 3rd 4th 5th 6th 7th
IBM DB2 support 2nd
Oracle 9i support 2nd
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
ordering
parameterized XQuery 2nd 3rd 4th
RSS feeds 2nd 3rd
SQL. [See also SQL]
XPath
XQEngine 2nd 3rd 4th
APIs 2nd 3rd 4th 5th 6th 7th
XSLT
quotations
XML attributes

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

RAID (Redundant Array of Inexpensive Disks)
RDBMSs 2nd 3rd
indexing
locking mechanisms 2nd
metadata 2nd
paradigms for XML data storage 2nd 3rd 4th
security
transactions
RDF (Resource Definition Framework) 2nd 3rd
RDMS
XML persistence strategies 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
read locks
realtionships
to persistence (XML)
Redundant Array of Inexpensive Disks (RAID)
references
entity 2nd 3rd 4th
XML fundementals 2nd
regular expressions
XPath 2nd 3rd 4th
flags
matches() function 2nd
replace() function 2nd
tokenize() function 2nd
relational databases 2nd
relational-to-object
transactions 2nd
relational databases. [See also databases]
relational technology 2nd
relational-to-object databases
transactions 2nd
Remote Method Invocation (RMI)
remote procedure calls (RPC)
remove() function
repeatable reads
replace() function 2nd
requirements
for XML data storage 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
Resource Definition Framework. [See RDF]
resources
books
list servers
specifications 2nd
XML fundamentals 2nd
results
returning 2nd 3rd 4th 5th 6th 7th 8th
resultsets, generating XML from
return clause
return keyword 2nd 3rd 4th 5th 6th 7th 8th
reused code
consolidating
revisions
overview pages 2nd 3rd
RMI (Remote Method Invocation)
rollback transactions. [See also transactions]
root elements 2nd
root node queries 2nd
root tags
applying
rows
format-rows
make-rows function
RPC (remote procedure calls)
RSS (Really Simple Syndication) 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
building syndication engines 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
generating from PostgreSQL 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
querying feeds 2nd 3rd 4th 5th 6th
root node queries 2nd 3rd 4th
rules
inference 2nd
of XML 2nd 3rd 4th
DTDs/schemas
ontologies 2nd 3rd
subtyping 2nd
writing XML 2nd 3rd 4th 5th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

SAX
XML processing models 2nd
SAX2
Qexo
Scalable Vector Graphics (SVG)
Scalable Vector Graphics. [See SVG]2nd [See SVG]
schemas 2nd 3rd 4th 5th 6th 7th
associating
importing 2nd 3rd 4th
XML
XMLXSU
scope
pragmas
scripts
CGI
executing applications as 2nd 3rd
search engines
XQEngine 2nd 3rd 4th
APIs 2nd 3rd 4th 5th 6th 7th
searches
where clauses 2nd 3rd 4th
searching
Sematic Web project. [See also Semantic Web project]
XPath
security
lockign mechanisms 2nd
RDBMSs
Semantic Web project 2nd 3rd 4th 5th 6th
semantics
data models 2nd 3rd 4th 5th 6th 7th
subtyping rules 2nd
expressions 2nd 3rd
functions 2nd 3rd 4th 5th
normalization 2nd
notation
inference rules 2nd
processing models 2nd 3rd
types
prime 2nd
XML
CDATA
comments
DTD declarations 2nd
empty elements 2nd
root elements
tag attributes
XQuery Core
expressions
mapping 2nd 3rd
type
sequences 2nd
aggregate functions 2nd
data models
expressions
optimizing 2nd 3rd 4th
for keyword
iterations 2nd 3rd
functions 2nd 3rd 4th 5th 6th 7th
modifying 2nd 3rd
node-sets 2nd
order
to operator
enumerating 2nd 3rd 4th
tokenize() function 2nd 3rd
types 2nd
sequences of items 2nd
serlets
Web servers
executing
servers
GoXML DB
Tomcat
installing servlets under 2nd 3rd
Web
executing servlets
services
Microsoft XQuery Services 2nd 3rd 4th 5th 6th 7th
classes 2nd
RSS 2nd 3rd 4th 5th 6th
buildinng syndication engines 2nd 3rd 4th 5th
generating from PostgreSQL 2nd 3rd 4th 5th 6th
querying feeds 2nd 3rd
root node queries 2nd
Web. [See Web Services]
servlets
applications
executing as 2nd 3rd 4th 5th 6th 7th
Tomcat servers
installing under 2nd 3rd
set functions
modifying 2nd 3rd
SGML (Standard Generalized Markup Language)
SHOE (Simple HTML Ontology Extensions)
shredding 2nd 3rd 4th
side effect
Simple HTML Ontology Extensions (SHOE)
Simple Object Access Protocol (SOAP)
Simple Operating Application Protocol (SOAP)
singleton sequences
SOAP
Web pages
creating 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
SOAP (Simple Object Access Protocol)
SOAP (Simple Operating Application Protocol)
SoftQuad
sorting. [See also order by command]
specifications 2nd
SQL
DDL
DML 2nd
expressions 2nd
SQL (Structured Query Language) 2nd 3rd
standalone nodes
document fragments (XML)
Standard Generalized Markup Language. [See SGML]
statements
for-return 2nd 3rd
FWLOR
if-then-else 2nd 3rd 4th
XPath 2nd 3rd 4th 5th 6th 7th 8th
statementslresults
returning 2nd 3rd 4th 5th 6th 7th 8th
static semantics
static typing 2nd 3rd
storage
native 2nd 3rd 4th 5th 6th 7th 8th
XML 2nd
data-centric-XML
databases 2nd 3rd
paradigms for 2nd 3rd 4th
persistence strategies for 2nd 3rd 4th
RDMS persistence strategies for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
requirements for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
stored procedures
library modules
optimizing 2nd 3rd 4th 5th
strategies
for XML data storage
stratigies
for XML data storage 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
strings
nodes
XML
structure of XQuery 2nd 3rd 4th
Structured Query Language. [See SQL]
structures
XML
relationship to persistence
subjects
RDF 2nd
subsequence() function
subtyping rules 2nd
support
GoXML DB 2nd 3rd 4th 5th 6th 7th
IBM DB2 2nd
Oracle 9i 2nd
Oracle 9I
JXQI 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th
SVG (Scalable Vector Graphics) 2nd 3rd 4th 5th 6th
symbols
regular ecpressions
syndication
RSS 2nd 3rd 4th 5th 6th
building syndication engines 2nd 3rd 4th 5th
generating from PostgreSQL 2nd 3rd 4th 5th 6th
querying feeds 2nd 3rd
root node queries 2nd
syntax 2nd 3rd 4th
XML
CDATA
comments
DTD declarations 2nd
empty elements 2nd
root elements
tag attributes
XPath 2nd
systems
types
atomic values 2nd 3rd 4th
XDBMSs

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

tables
creating 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
customizing 2nd 3rd 4th 5th 6th
DDL
DML 2nd
parameterized XQuery 2nd 3rd 4th
tags
attributes
CDATA 2nd 3rd
root
applying
XML
XML. [See also XML]
templates
Web pages
editing overview pages 2nd 3rd
generating 2nd 3rd 4th 5th 6th 7th
image pages 2nd 3rd 4th 5th 6th 7th 8th
overview pages 2nd 3rd 4th 5th 6th
testing
nodes
XPath 2nd
Text node. [See also nodes]
Text nodes
text. [See also documents]
thumbnails
photo applications
creating 2nd 3rd 4th 5th
to operators
enumerating 2nd 3rd 4th
tokenize() function 2nd 3rd 4th 5th
Tomcat servers
servlets
installing under 2nd 3rd
tools
ontologies
Web pages
editing overview pages 2nd 3rd
generating 2nd 3rd 4th 5th 6th 7th
image pages 2nd 3rd 4th 5th 6th 7th 8th
overview pages 2nd 3rd 4th 5th 6th
XML
XMP
XQuery
XSU
transactions
concurrency control 2nd 3rd 4th 5th 6th 7th 8th
databases 2nd
ACID 2nd
isolation levels
levels
RDBMSs
transformations
DocBook to HTML 2nd 3rd 4th 5th 6th 7th
transportability
TreeList DOM class 2nd
trees
order by command 2nd 3rd 4th 5th
XPath
troubleshooting
static typing 2nd 3rd
type
annotation
annotations 2nd
anonymous 2nd
dynamic
XQuery Core
typed values
nodes (XML)
types 2nd 3rd
annotation 2nd
atomic values 2nd 3rd 4th
complex 2nd 3rd 4th
data
XML 2nd 3rd
XPath 2nd 3rd 4th 5th 6th
of nodes 2nd
prime 2nd
semantics
sequences 2nd
static typing 2nd 3rd
XML 2nd

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

UDB (Universal Database) 2nd 3rd 4th 5th 6th 7th 8th
Uniform Resource Identifiers. [See URIs]
Universal Database. [See UDB]
URIs (Uniform Resource Identifiers). [See also hyperlinks]

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

validation
XML 2nd
type annotations 2nd
validity of XML documents
value space
values
atomic 2nd 3rd 4th 5th
Boolean
optimizing expressions 2nd 3rd 4th
nodes
XML 2nd
types
sequences 2nd
static typing 2nd 3rd
variables
defining
importing
calling 2nd
schemas 2nd 3rd 4th
vendors
relational technology 2nd
viewing
XML 2nd
Visual Basic
code

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

W3C (World Wide Web Consortium)
Web
Semantic Web project 2nd 3rd 4th 5th 6th
Web pages
creating 2nd 3rd 4th 5th 6th 7th 8th 9th 10th
generating 2nd
creating photo album applications 2nd 3rd 4th 5th
editing overview pages 2nd 3rd
image pages 2nd 3rd 4th 5th 6th 7th 8th
overview pages 2nd 3rd 4th 5th 6th
parameterized XQuery 2nd 3rd 4th
RSS 2nd 3rd 4th 5th 6th
building syndication engines 2nd 3rd 4th 5th
generating from PostgreSQL 2nd 3rd 4th 5th 6th
querying feeds 2nd 3rd
root node qeuries 2nd
SVG 2nd 3rd 4th 5th
tables
customizing 2nd 3rd 4th 5th 6th
XML
Web pages. [See also documents]
Web servers
servlets
executing
Web Services
XML
Web sites
ontologies 2nd 3rd
well-formed XML documents.
XML [See also documents]
well-formedness of XML documents
where clause 2nd 3rd 4th
where keyword 2nd 3rd 4th
wildcards
XPath
World Wide Web Consortium (W3C)
write locks
writing
XML 2nd 3rd 4th 5th

[ Team LiB ]


[ Team LiB ]

[SYMBOL] [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [N] [O] [P] [Q] [R] [S] [T] [U] [V] [W] [X]

XDBMSs (XML database management systems)
XDK (XML Development Kit)
Xlink
Xlink (XML Linking Language) 2nd
XML 2nd
data elements
data storage 2nd
data-centric-XML
databases 2nd 3rd
paradigms for 2nd 3rd 4th
persistence strategies for 2nd 3rd 4th
RDMS persistence strategies for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th
requirements for 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th
data types 2nd 3rd
databases. [See databases]
documents 2nd 3rd 4th
building custom switches 2nd
command structures 2nd
DTDs 2nd 3rd
external data sources 2nd 3rd
for-return statements 2nd 3rd
if-then-else statements 2nd 3rd 4th
publishing 2nd 3rd 4th 5th 6th 7th
rules of 2nd 3rd 4th
Semantic Web project 2nd 3rd 4th 5th 6th
validity of
well-formedness
evolution of 2nd
fundamentals 2nd
infosets 2nd
loading
nodes
identities 2nd 3rd
string/typed values
values 2nd
ontologies 2nd 3rd
order by command 2nd 3rd 4th 5th
portable data 2nd
processing models 2nd
protocols
XLink 2nd
XPath 2nd 3rd 4th 5th 6th 7th 8th 9th
XPointer
XSL 2nd
XSLT 2nd 3rd
RDF 2nd
structures
relationships to persistence
syntax
CDATA
comments
DTD declarations 2nd
empty elements 2nd
root elements
tag attributes
types 2nd
validating 2nd
type annotations 2nd
XMl
viewing 2nd
XML
writing 2nd 3rd 4th 5th
XQEngine 2nd 3rd 4th
APIs 2nd 3rd 4th 5th 6th 7th
XML (Extensible Markup Language)
XML database management systems. [See XDBMSs]
XML Development Kit (XDK)
XML Linking Language. [See XLink]
XML Pointer Language. [See XPointer]
XML SQL Utility (XSU)
XML User Interface Language (XUL)
XMP (Extensible Metadata Platform) 2nd
XPath 2nd 3rd
applying 2nd
as query languages
command structures 2nd
building custom switches 2nd
for-return statements 2nd 3rd
if-then-else statements 2nd 3rd 4th
data types
Xpath
data types
XPath
data types 2nd 3rd 4th
XML 2nd 3rd
directories 2nd
external data sources 2nd 3rd
nodes 2nd
addressing 2nd
regular expressions 2nd 3rd 4th
flags
matches() function 2nd
replace() function 2nd
tokenize() function 2nd
tokenize() function 2nd 3rd
XPath (XML Path Language) 2nd 3rd 4th 5th 6th 7th 8th 9th
XPath 1.0
addresing 2nd 3rd
XPath 2.0
addresing 2nd
XPointer (XML Pointer Language)
XQEngine 2nd 3rd 4th
APIs 2nd 3rd 4th 5th 6th 7th
XQuery Core
expressions
mapping 2nd 3rd
type
XQuery processor
XQueryDocument
XQueryException
XQueryExpression
XQueryNavigator
XQueryNavigatorCollection
XSL (Extensible Stytlesheet Language) 2nd
XSLT 2nd 3rd
as query languages
DocBook to HTML 2nd 3rd 4th 5th 6th 7th
XSLT (XSL Transformations)
comparing 2nd 3rd 4th 5th 6th 7th 8th
XSU (XML SQL Utility)
XUL (XML User Interface Language)

[ Team LiB ]


Team LiB

Brought to You by

Like the book? Buy it!