XQuery

Version 2.0¹

Introduction

XQuery builds on the XPath language, extending it to build so-called FLWOR (“flower”) expressions, which are similar in function to the SQL select statement.

Example 1: Building on XPath

• XPath Notes
• One more reason for learning Unix paths.

This query gives us the pet elements (from pets.xml), but the output is not well-formed XML:

/pets/pet

To correct this, we can add a root element. We then need { ... } to enclose the XPath expression which must be evaluated:

<petshop>
{
/pets/pet
}
</petshop>

Using XQilla

XQilla (xqilla) is a command-line tool which can run XQuery expressions.

Simple usage: to run an XQuery query from file query.xq with an input XML document source.xml:

$ xqilla -i source.xml query.xq

The preceding command will produce unformatted output which is hard to read. To make the output prettier and add an xml declaration, filter it through xmllint as follows:

$ xqilla -i source.xml query.xq | xmllint --format -

The final - tells xmllint to read from stdin. You can, of course, also pipe the output through less or redirect it to an output file.

But with xmllint, the result of the query must be valid XML, not just an XML fragment (with no single root element).

If we want our output to go to a file, we can use standard Unix file redirection (with or without xmllint):

$ xqilla -i source.xml query.xq | xmllint --format - > query.out
$ xqilla -i source.xml query.xq > query.out

These scripts can be helpful:

• README
• xq
• xqp

Basic FLWOR Expressions: Querying Pets XML

FLWOR stands for for, let, where, order by, and return; expressions are built out of clauses beginning with these words. The return clauses are required, and with it, you must have either a let or a for clause (you can have both); the rest are optional.

Both let and for bind variables. That is, they both create a variable and associate it with a value. The difference between let and for is that with let, the variable has a single value; with for, it is associated with a sequence of values (remember tuples are the basic data type in XPath) and takes on each value in the sequence in turn.

Example 2: Using the for and return clauses

The for clause creates variables using XPath expressions; it tells where the data comes from, and thus functions like the from clause in an SQL select statement.

The return clause specifies what is to be output. It does not “end the procedure” as it would in Python or Java; it simply yields or produces the next output value. That is, it returns a value, without returning control.

Simple example: for-ret1.xq, using pets.xml as the source document

xquery version "1.0";

for $pet in /pets/pet
return $pet

• The version declaration is optional and defaults to XQuery version 1.0. So if you need features of a later version, you need to declare it.
• /pets/pet is an XPath expression
• This example does not produce well-formed XML output; just a series of <pet> elements. We need a root element to contain them.
• Same as /pets/pet …

Example 3: Inserting tags and data

<petshop>
{
for $pet in /pets/pet
return $pet
}
</petshop>

• for-ret2.xq
• Inserts tags for a root element to make the output document well-formed XML.
• We need { ... } for XQuery within the tags.
• Same as enclosed /pets/pet from above, but now with a variable, other things will become easier

<petshop>
{
for $pet in /pets/pet
return <animal>
         <name>{data($pet/@name)}</name>
     <weight>{data($pet/@weight)}</weight>
     <sex>{data($pet/@sex)}</sex>
     <price>{data($pet/@price)}</price>
     <blurb>{data($pet/description)}</blurb>
       </animal>
}
</petshop>

• for-ret3.xq
• Uses { ... } again
• data(node) returns the content of node
• @ means “attribute”

Example 4: Using the let clause

The let clause introduces (local) variables in the query.

<pets>
{
for $pet in /pets/pet
let $price := $pet/@price,
    $tax := $price * 0.07,
    $with_tax := $price + $tax
return <pet>
         <name>{data($pet/@name)}</name>
         <price>{$price}</price>
     <tax>{$tax}</tax>
     <with_tax>{$with_tax}</with_tax>
       </pet>
}
</pets>

• let.xq
• let $variable := expression
• Introduces three variables
• The variables are then used in the return clause

Example 5: Using the order by clause

The order by clause, like in SQL, specifies how to sort the output.

<names>
{
for $pet in /pets/pet
order by $pet/@name
return <name>{data($pet/@name)}</name>
}
</names>

• orderby1.xq
• Simple sorting on one field, in the default order

<pets>
{
for $pet in /pets/pet
order by number($pet/@price) descending
return <pet>
         <price>{data($pet/@price)}</price>
         <breed>{data($pet/@breed)}</breed>
         <name>{data($pet/@name)}</name>
       </pet>
}
</pets>

• orderby2.xq
• Use number or xs:double to convert numeric text to a number for sorting, where 10.00 > 9.99; otherwise “9.99” > “10.00”.

We can also sort on two (or more) fields, with explicit ascending or descending order:

<pets>
{
for $pet in /pets/pet
order by $pet/@breed ascending, xs:double($pet/@price) descending
return <pet>
         <breed>{data($pet/@breed)}</breed>
         <price>{data($pet/@price)}</price>
         <name>{data($pet/@name)}</name>
       </pet>
}
</pets>

• orderby3.xq

Example 6: Using the where clause

The where clause, like in SQL, specifies conditions for selecting elements for output:

<pets>
{
for $pet in /pets/pet
where $pet/@price < 100
return <pet>
         <name>{data($pet/@name)}</name>
         <price>{data($pet/@price)}</price>
       </pet>
}
</pets>

• where1.xq
• We can also use and, or, not, != (or ne), etc.

But you don’t always need to use where for this, because you can filter with XPath expressions using predicates as well:

<pets>
{
for $pet in /pets/pet[@price < 100]
return <pet>
         <name>{data($pet/@name)}</name>
         <price>{data($pet/@price)}</price>
       </pet>
}
</pets>

• where2.xq

Example 7: All the FLWOR parts

The next example uses all the FLWOR parts. Remember, FLWOR stands for For, Let, Where, Order by, Return.

<pets>
{
for $pet in /pets/pet
let $price := $pet/@price,
    $tax := $price * 0.07,
    $with_tax := $price + $tax
where $pet/@breed != "garter snake"
(: use number or xs:double to convert string to number :)
order by number($with_tax)
return <pet>
         <name>{data($pet/@name)}</name>
         <price>{$price}</price>
     <tax>{$tax}</tax>
     <with_tax>{$with_tax}</with_tax>
       </pet>
}
</pets>

• flwor.xq

Example 8: Mixing it up

FLWOR expressions can use either let or for or both, in any order, and as many of them as you like.

let $x := 10
for $y in (5, 15, 25)
let $z := $x + $y
return <values>{$x, $y, $z}</values>

A return clause is needed in every FLWOR expression, and needs either let or for. But even without a return clause, you can have a valid XQuery expressions, since all XPath expressions are XQuery expressions; only in that case it is not a FLWOR expression.

Deeper FLWOR Expressions: Querying Stores XML

We need to look at something with deeper structure (i.e., more levels), so we’ll use stores.xml for the remaining examples. For some of these queries, we will use nested FLWOR expressions, similar to nested for loops in other programming languages.

Example 9

List all products, ordered by product name, and enclose in <products> ... </products> tags so that we have a well-formed XML output.

<products>
{
   for $p in //product
   order by $p/name
   return $p
}
</products>

Example 10

Similarly, list products by price, but in descending order. Be sure to sort numerically so that 9 < 10.

<products>
{
   for $p in //product
   order by number($p/price) descending
   return $p
}
</products>

(Either number or xs:double can be used to convert price from string to a numeric type.)

Example 11

Find the number of products, the sum of prices, and the average price.

<stats>
  <n>{count(//product)}</n>
  <sum-price>{sum(//product/price)}</sum-price>
  <avg-price>{avg(//product/price)}</avg-price>
</stats>

Example 12

Find items with price  ≥ $50.

Note: Comments in XQuery may be written like this: (: This is a comment. :)

<high-price>
  for $p in //product
  where $p/price ge 50
  return $p
</high-price>

(: oops -- have to use { } to evaluate the inner part,
and / should be div :)

<high-price>
{
  for $p in //product
  where $p/price ge 50
  return $p
}
</high-price>

(: oops need to convert $p/price to number, using either number
or xs:double :)

<high-price>
{
  for $p in //product
  where number($p/price) ge 50
  return $p
}
</high-price>

Example 13

Restructure the store elements, so that store names (attributes) become a store-name child element.

<stores>
{
  for $s in /stores/store
  return
  <store id="$s/@id">
    <store-name>$s/@name</store-name>
  </store>
}
</stores>

(: oops, need to use { ... } to evaluate some parts of this :)

<stores>
{
  for $s in /stores/store
  return
  <store id="{$s/@id}">
    <store-name>{$s/@name}</store-name>
  </store>
}
</stores>

(: But the stores appear to be empty!  Let's put the departments
back in! :)

<stores>
{
  for $s in /stores/store
  return
  <store id="{$s/@id}">
    <store-name>{$s/@name}</store-name>
    {
    for $d in $s/department
    return $d
    }
  </store>
}
</stores>

Example 14

Restructure the product elements, so that code and price become attributes, and the name element becomes a product-name element.

For now we will omit a root element.

for $p in //product
return
<product code="{$p/code}" price="{$p/price}">
  <product-name>{$p/name}</product-name>
</product>

(: Good start, but what about the details?
It's either software, book, clothing, or other-description
(no DTD tells us this, just inspection).
:)

for $p in //product
return
<product code="{$p/code}" price="{$p/price}">
  <product-name>{$p/name}</product-name>
  {  for $x in $p/software   return $x  }
</product>

(: That's a start.  Now for the rest. :)

for $p in //product
return
<product code="{$p/code}" price="{$p/price}">
  <product-name>{$p/name}</product-name>
  {  for $x in $p/software            return $x  }
  {  for $x in $p/book                return $x  }
  {  for $x in $p/clothing            return $x  }
  {  for $x in $p/other-description   return $x  }
</product>

(: Can we use the set union operator | to combine these into one sequence? :)

for $p in //product
return
<product code="{$p/code}" price="{$p/price}">
  <product-name>{$p/name}</product-name>
  {
  for $x in $p/software | $p/book | $p/clothing | $p/other-description
  return $x
  }
</product>

(: Yes! :)

Example 15

Combining aspects of Examples 13 and 14, restructure the entire document, using nested for/return loops.

Best to develop something like this incrementally.

Top levels, restructured, from Example 13:

<stores>
{
  for $s in /stores/store
  return
  <store id="{$s/@id}">
    <store-name>{$s/@name}</store-name>
  </store>
}
</stores>

Add department elements, without content:

<stores>
{
  for $s in /stores/store
  return
  <store id="{$s/@id}">
    <store-name>{$s/@name}</store-name>
    {
    for $d in $s/department
    return
    <department name="{$d/@name}">
    </department>
    }
  </store>
}
</stores>

Add restructured products, from Example 14, under department. But note we change for $p in //product to for $p in $d/product so we get only the products for the current department.

<stores>
{
  for $s in /stores/store
  return
  <store id="{$s/@id}">
    <store-name>{$s/@name}</store-name>
    {
    for $d in $s/department
    return
    <department name="{$d/@name}">
    {
      for $p in $d/product
      return
      <product code="{$p/code}" price="{$p/price}">
        <product-name>{$p/name}</product-name>
        {
        for $x in $p/software | $p/book | $p/clothing | $p/other-description
        return $x
        }
      </product>
    }
    </department>
    }
  </store>
}
</stores>

More Goodies …

There is much more to XQuery (and XPath). But just to mention a few, there are many string manipulation functions, such as concat for concatenating strings and format-number. There’s an if then else expression. You can define functions, including recursive and higher-order functions. There is way more than I have time to tell you about. So if you’re interested pursuing any of these, consult the references listed below.

References

Tutorials and Documentation

• Fawcett, Quin, and Ayers, Beginning XML, 5th edition. Chapter 7, “Extracting Data from XML” (for XPath); Chapter 9, “XQuery.”
• W3C Specifications for XQuery and related languages, including
  • XQuery 3.0
  • XPath 2.0
  • Xquery and XPath Functions and Operators
• W3Schools XQuery Tutorial
• Dr. Michael Kay’s Tutorials
  • Learn XQuery in 10 Minutes
  • Blooming FLWOR - An Introduction to the XQuery FLWOR Expression
• XQuery Update Tutorial: how to modify a document (insert, modify, delete nodes) with XQuery
• Wikibooks, XQuery

Software

• XQilla provides a command line tool and C/C++ library for XQuery and XPath. Installed on merlin; Fedora packages xqilla, xqilla-devel; command: xqilla. (open source)

• Saxon provides Java-based XQuery, XPath, and XSLT tools. There are an open source “home edition” and closed-source commercial editions. If you have it, you can use it for XQuery like this:

  $ saxon-xquery -q:QUERYFILE -s:INPUTFILE

  Either QUERYFILE or INPUTFILE may be given as “-” to read from standard input.