XQuery

= XQuery Examples Collection = Welcome to the XQuery Examples Collection Wikibook! XQuery is a World Wide Web Consortium recommendation for selecting data from documents and databases.

2019 refurbishment
There is a Github project to track issues with this book and collaborate amongst eXist db folk at least to bring this resource up-to-date.

Main tasks in the current refurbishment are to:


 * get example code executable again
 * remove complex case studies which use obsolete sources
 * add XQuery 3.0 examples.

Recent Changes

Search
You may search the book here:

About this Project
This is a collaborative project and we encourage everyone who is using XQuery to contribute their XQuery examples. All example programs must conform to the creative-commons-2.5 share-alike with attribution license agreement.

Execution of examples use an /eXist demo server/.
 * 1) Instructors: please sign our /Guest Registry/ if you are using this book for learning or teaching XQuery
 * 2) Contributors: please see our /Naming Conventions/ to ensure your examples are consistent with the textbook
 * 3) Learners: If you are looking for an example of a specific XQuery language construct, technique or problem but can't find an example, please add a suggestion to the /Examples Wanted/ section.

Introduction

 * 1) /Background/ - A brief history and motivation for the XQuery standard.
 * 2) /Benefits/ - Why use XQuery?
 * 3) /Installing and Testing/ - How to install an XQuery server on your system.
 * 4) /Naming Conventions/ - Naming standards used throughout this book.

Using XQuery
Common tasks in using XQuery with XML
 * 1) /Executing a script/ - Executing an XQuery script
 * 2) /Loading data/ - A tour of several different ways to load data into a native XML database

= Example Scripts =

Beginning Examples
Examples that do not assume knowledge of functions and modules.
 * 1) /HelloWorld/ - A simple test to see if XQuery is installed correctly.
 * 2) /FLWOR Expression/ - A basic example of how XQuery FLWOR statements work.
 * 3) /Sequences/ - Working with sequences is central to XQuery.
 * 4) /XPath examples/ - Sample XPath samples for people new to XML and XPath
 * 5) /Regular Expressions/ - Regular expressions make it easy to parse text.
 * 6) /Searching multiple collections/ - How to search multiple collections in a database.
 * 7) /Displaying Lists/ - How to take a list of values in an XML structure and return a comma separated list.
 * 8) /Extracting data from XHTML files/ - How to use the doc function to get data from XHTML pages.
 * 9) /Displaying data in HTML Tables/ - How to display XML data in an HTML table.
 * 10) /Limiting Result Sets/ - How to limit the number of records returned in an XQuery.
 * 11) /Filtering Words/ - How to test to see if a word is on a list.
 * 12) /Saving and Updating Data/ - How to have a single XQuery that saves new records or updates existing records.
 * 13) /Quantified Expressions/ - Testing all the items in a sequence.
 * 14) /Dates and Time/ - Sample expressions that work with date and time values

Intermediate Examples
Assumes knowledge of functions and modules.
 * 1) /Using XQuery Functions/ - How to read XQuery function documents and user XQuery functions
 * 2) /Creating XQuery Functions/ - How to create your own local XQuery functions
 * 3) /Returning the Longest String/ - A function to find the longest string from a list of strings
 * 4) /Net Working Days/ - How to calculate the number of working days between two dates
 * 5) /Tag Cloud/ - Counting and viewing the number of keywords
 * 6) /String Analysis// - Regular expression string analysis
 * 7) /Manipulating URIs/ - How to get and manage URIs
 * 8) /Parsing Query Strings/ - Parsing query strings using alternate delimiters.
 * 9) /Splitting Files/ - Splitting a large XML files into many smaller files.
 * 10) /Filling Portlets/ - How to fill regions of a web page with XQuery
 * 11) /Filtering Nodes/ - How to use the identity transform to filter out nodes and add nodes
 * 12) /Limiting Child Trees/ - You have a tree of information and you want to "prune" only at a specific level
 * 13) /Higher Order Functions/ - Passing functions as arguments to functions
 * 14) /Timing Fibonacci algorithms/ - A couple of Fibonacci algorithms and timing display
 * 15) /Using Intermediate Documents/ - Analysis of a MusicXML file
 * 16) /Formatting Numbers/ - using picture formats to format numbers
 * 17) /Uploading Files/ - how to upload files using HTML forms
 * 18) /TEI Concordance/ - How to build a TEI-based concordance
 * 19) /Queries on Tables/ - How to extract data from tabular data
 * 20) /Namespace Constructors/ - how to dynamically add namespaces when constructing documents

Search

 * 1) /Introduction to XML Search/ - An overview of XML search terminology
 * 2) /Basic Search/ - A simple search page
 * 3) /Searching, Paging and Sorting/ - Searching and Viewing search results
 * 4) /Keyword Search/ - full text search with Google-style results
 * 5) /Employee Search/ - an Ajax example
 * 6) /Lucene Search/ - using eXist's Lucene-based fulltext search
 * 7) /Advanced Search/ - creating complex searches using multiple search fields
 * 8) /Open Search/ - creating an OpenSearch file to describe your search page
 * 9) /Auto-generation of Index Config Files/ - scripts to automatically generate the index configuration file

Transformations
Using XQuery to transform between XML and other representations


 * 1) /Parsing CSV/ - Using FLWOR on sequences to parse CSV text.

Interaction

 * 1) /Getting URL Parameters/ - How to get parameters from the URL.
 * 2) /Getting POST Data/ - How to get XML data posted to an XQuery.
 * 3) /Checking for Required Parameters/ - How to check for a required parameter using if/then/else.
 * 4) /Adder/ - Retaining sate in interactions :Creating a web service that adds two numbers.
 * 5) /Chaining Web Forms/ - Passing data from one web page to another using URL parameters, sessions or cookies
 * 6) /Simple XForms Examples/
 * 7) /Incremental Search of the Chemical Elements/ - AJAX
 * 8) /Sending E-mail/ - How to send an e-mail message from within an XQuery

Visualization

 * 1) /Graph Visualization with Graphviz/
 * 2) /Google Chart Sparkline/ - Creating a Tufte Sparkline  using the Google Charts API
 * 3) /Google Chart Bullet Bar/ - Creating a bullet bar using the Google Charts API

Creating Custom Views
These examples use reflection on the structure of an XML document using name to implement generic functions for XML transformations.
 * 1) /HTML Table View/ - A generic HTML table representation
 * 2) /Tree View/ - A generic HTML tree representation
 * 3) /Grouping Items/ - how to group items in a report

Transforming Complex XML Documents
XQuery has many features that allow you to transform XML and create full document-style transformation libraries. Unlike traditional "database" documents, complex XML documents have "complex content" that includes in-line elements in unpredictable order. This section provides a foundation for these transformations based on using the XQuery typeswitch functions. Typeswitch function transformations replace XSLT transforms but can also access indexes for very fast transforms of large collections.
 * 1) /Transformation Styles/ overview of the three styles of transformation
 * 2) /Typeswitch Transformations/ Using the typeswitch function for document-style transforms.
 * 3) /Transformation idioms/ Handling transformation tasks
 * 4) /Generating Skeleton Typeswitch Transformation Modules/ Using XQuery to generate a skeleton module for typeswitch-based document transformation
 * 5) /Web XML Viewer/ Using the typeswitch function to transform an XML document to HTML

Paginated Reports
Unlike HTML pages, paginated reports use the concept of text flows between pages. These examples show you how to convert raw XML into high-quality PDF files suitable for printing. The examples use a markup standard called XSL-FO for "Formatted Objects"


 * 1) /Installing the XSL-FO module/ - update your 1.4 configuration to get the current software from the Apache web site
 * 2) /Generating PDF from XSL-FO files/ - Converting XML-FO to PDF files
 * 3) /XSL-FO Tables/ - Generating XSL-FO tables from XML files
 * 4) /Converting HTML tables to XSL-FO tables/ - use a XQuery typeswitch transform to convert HTML tables to XSL-FO tables
 * 5) /XSL-FO Images/ - Embedding images in generated (PDF) files

Content Publishing

 * 1) /Publishing Overview/ - How to transfer a document from an internal intranet server to a public web site
 * 2) /Publishing to Subversion/ - How to transfer a document from an internal intranet to a public SVN server using SSL and digest authentication

XML Document Comparison and Merging

 * 1) /Compare two XML files/ - using the eXist compare function to test to see if two XML files are exactly the same
 * 2) /XML Differences/ - displaying the difference between two XML files
 * 3) /Compare with XQuery/ - Using XQuery to Compare Lists
 * 4) /Time Comparison with XQuery/ - Using XQuery to Compare Dated Items
 * 5) /Synchronizing Remote Collections/ - Using lastModified time stamps to see what items have changed
 * 6) /Finding Duplicate Documents/ - Using a hash function to find duplicate documents

Time Based Queries

 * 1) /Time Based Queries/ - using dates and times to limit search results
 * 2) /Timing a Query/ - profiling how long a query takes

TEI documents
Text Encoding Initiative.


 * 1) /TEI Concordance/ - How to build a TEI-based concordance
 * 2) /TEI Document Timeline/ - Using Simile Timeline to visualize a TEI document

DocBook Documents

 * 1) /DocBook to HTML/
 * 2) /DocBook to PDF/
 * 3) /DocBook to ePub/
 * 4) /DocBook to Microsoft Word/

OpenOffice

 * 1) /OpenOffice to HTML/

Office Open XML

 * 1) /Office Open XML/

XML Schemas

 * 1) /XML Schema to Instance/
 * 2) /XML Schema to XForms/
 * 3) /XML Schema to SVG/

Processing Special Characters

 * 1) /Special Characters/ - dealing with newlines and other special characters.

= XQuery and other languages =

MusicXML

 * 1) /Using Intermediate Documents/ Analysing MusicXML documents
 * 2) /MusicXML to Arduino/

Python

 * 1) /XQuery and Python/

SQL

 * 1) /XQuery SQL Module/ - Calling SQL from within your XQuery
 * 2) /XQuery from SQL/ - Using XQuery to access a classic Relational database - Employee/Department/Salary

RDF/OWL

 * 1) /List OWL Classes/ - A simple XQuery script that will display all the OWL classes in an OWL file

Excel

 * 1) /Excel and XML/

JavaScript

 * 1) /Navigating Collections/ - basic AJAX
 * 2) /Employee Search/ - basic AJAX
 * 3) /DOJO data/ - basic JSON

SQL

 * 1) /XML to SQL/
 * 2) Microsoft-Access

XHTML + Voice

 * 1) /Simple RSS reader/
 * 2) /XHTML + Voice/ Twitter Radio for Opera

XSLT

 * 1) /XQuery and XSLT/ Executing an XSLT transform from within XQuery

= Data Mashups =

Authentication

 * 1) /Basic Authentication/ - Logging in to a remote web server using HTTP Basic Authentication
 * 2) /Digest Authentication/ - Logging in to a remote web server using HTTP Digest Authentication
 * 3) /OAuth/ - A standard for protecting a set of user-owed data within a web service

Wikipedia interaction

 * 1) /Wikipedia Page scraping/
 * 2) /Wikipedia Lookup/
 * 3) /Wikipedia Events RSS/
 * 4) /Wiki weapons page/

Wikibook applications

 * 1) /Wikibook index page/
 * 2) /Wikibook list of code links/

Freebase

 * 1) /Freebase/ - use XQuery to get data via JSON from Freebase

Google Docs

 * 1) /Google Docs/ - use XQuery to get data from Google Docs

Visualization

 * 1) /Graph Visualization/
 * 2) /Graphing from RDF/
 * 3) /Dataflow diagrams/
 * 4) /Sequence Diagrams/
 * 5) /Example Sequencer/ - Step-by-step tutorial

Google Charts
Although the Google Charts functions only work when you are connected to the Internet, these examples show that XQuery is an ideal tool for converting XML data into charts.
 * 1) /Google Charts/ Using XML and XQuery to generate Google Charts using REST
 * 2) /Google Chart Sparkline/ - A demonstration of how to create a chart using the Google Charts API
 * 3) /Google Chart Bullet Bar/ - A demonstration of how to a dashboard bullet bar using the Google Charts API
 * 4) /Histogram of File Sizes/ - An XQuery report that generates a histogram of file sizes

There are also sample XForms that can be used to create front-ends in the XForms Tutorial and Cookbook

Digital Dashboards
Digital dashboards are single screens that compress a great deal of information into a single web page. This section will leverage many of the Google Charts examples from the prior section.
 * 1) Dashboard Architecture - How to design dashboards that have fast response times

Page Scraping
Page scraping is the process of extracting well-formed XML data from any HTML web page. When creating mashup applications this is also known as the harvesting process.
 * 1) /Overview of Page Scraping Techniques/
 * 2) /Page scraping and Yahoo Weather/
 * 3) /UK shipping forecast/
 * 4) /BBC Weather Forecast/
 * 5) /Page scraping and Mashup/
 * 6) /Simple RSS reader/
 * 7) /Multiple page scraping and Voting behaviour/
 * 8) /Link gathering/
 * 9) /REST interface definition/
 * 10) /Caching and indexes/

Mapping

 * 1) /Google Geocoding/
 * 2) /String Analysis Mapping Car Registrations
 * 3) /Flickr GoogleEarth/
 * 4) /Nationalgrid and Google Maps/
 * 5) /SMS tracker/

Timelines

 * 1) /Creating a Timeline/ - Creating a simple timeline view of events
 * 2) /Timelines of Resource/ - Using creation and modification dates to create timelines
 * 3) /TEI Document Timeline/ - Creating a timeline of all dates within a single TEI document

The Semantic Web

 * 1) /DBpedia with SPARQL - Football teams/
 * 2) /DBpedia with SPARQL and Simile Timeline - Album Chronology/ Creating a timeline of album releases using data from Wikipedia
 * 3) /DBpedia with SPARQL - Stadium locations/
 * 4) /The Emp-Dept case study/
 * 5) /XML to RDF/
 * 6) /SPARQL Tutorial/
 * 7) /SPARQL interface/
 * 8) /Graphing Triples/
 * 9) /SPARQLing Country Calling Codes/
 * 10) /Southampton Pubs/
 * 11) /Alphabet Poster/
 * 12) /Simile Exhibit/ Browser visualizations using the Simile JavaScript libraries
 * 13) /Latent Semantic Indexing/ Finding the semantic distance between documents

= Development Tools =
 * 1) /Sitemap for Content Management System/ XQuery functions can easily perform many common web site content management functions
 * 2) /Uptime monitor// use XQuery to monitor a remote web service
 * 3) /XQuery IDE/ - XQuery Integrated development environment
 * 4) /Image Library/ - using an XQuery to preview your images
 * 5) /XML Schema to Instance/ - XQuery function to generate a sample XML instance from an XML Schema file (.xsd)
 * 6) /Lorum Ipsum text/ - generating sample text for inserting into test page layouts
 * 7) /XQuery and XML Schema/ - Generating an XML instance document
 * 8) /Generating XQDocs/ - Automating the generation of XQuery documentation for Modules and Functions
 * 9) XqUSEme - Firefox extension to allow XQueries including against the loaded document (even against originally non-XML (poorly formed) HTML).
 * 10) /Call Graphs/ - Generating an call graph as an SVG file from module introspection
 * 11) /System Properties/ - getting a list of all standard and java system properties and their values
 * 12) /Environment Variables/ - getting a list of all the external environment variables and their values
 * 13) /Generating xqDoc-based XQuery Documentation/ - generating module documentation using the xqDoc commenting standard

Validation

 * 1) /Validating a document/ - Validate a document with an XML Schema
 * 2) /Validation using a Catalog/ - Using a Catalog file to validate documents
 * 3) /Validating a hierarchy/ -
 * 4) /Validation with Schematron/ - Apply specific rules to check a document

Path Analysis

 * 1) /All Paths/ - A report of all paths in a document or collection
 * 2) /All Leaf Paths/ - A report of all leaf paths in a document or collection

Security

 * 1) /Login and Logout/ - How to log users in and log them out
 * 2) /URL Driven Authorization/ How to use URL rewriting to check for valid users
 * 3) /Digital Signatures/ - How to use a custom module to use the XML Digital Signature standards
 * 4) /Changing Permissions on Collections and Resources/ - how to change permissions on collections and resources

Unit Testing Tools

 * 1) /XUnit Testing/ - what is XUnit testing and how to set them up
 * 2) /XUnit Annotations/ - how to add annotations to your XQuery functions to drive your XUnit tests

= Case Studies =


 * 1) /Fizzbuzz/
 * 2) /Project Euler/
 * 3) /Topological Sort/
 * 4) /Slideshow/
 * 5) /Sudoku/
 * 6) /Pachube feed/
 * 7) /World Temperature records/ - conversion of text data formats to XML, indexing and data presentation
 * 8) /UWE StudentsOnline/

= eXist db specific Functions and Configuration =

Installing modules

 * 1) /Installing the XSL-FO module/

Setting HTTP Headers

 * 1) /Setting HTTP Headers/

dbutil

 * 1) /Database Utilities/ - a set of database utility functions for database administration

compression
Function Reference
 * 1) /Get zipped XML file/
 * 2) /Unzipping an Office Open XML docx file/ - Uncompressing and storing a docx file

eXgit
Module for interacting with any git revision control system
 * 1) /Installing eXgit/

EXPath File Transfer XQuery Extension Module
This module does both FTP, SFTP (using the SSH protocol) using a standardized and consistent format.
 * 1) /File Transfer Client/

ftp client
This module allows you to interact with a remote FTP server on a remote system. It includes functions for listing, getting and putting files.
 * 1) /FTP Client/

httpclient
Function Reference
 * 1) /Digest Authentication/
 * 2) /UK shipping forecast/

JSON

 * 1) /Convert XML to JSON/

Lucene
Function Reference Help
 * 1) /Lucene Search/

mail
Function Reference
 * 1) /Sending E-mail/
 * 2) /Basic Feedback Form/

math
Function Reference
 * 1) /Using the Math Module/

memcached

 * 1) /Using the Memcached Module/

process

 * 1) /Execute External Process/ - execute a shell command from within XQuery

request
Function Reference
 * 1) /Getting URL Parameters//
 * 2) /Getting POST Data//
 * 3) /Checking for Required Parameters/
 * 4) /Manipulating URIs/
 * 5) /Parsing Query Strings/
 * 6) /Adder/ simple client-server interaction

scheduler
Function Reference Help
 * 1) /XQuery Batch Jobs/

sequences
Function Reference
 * 1) /Sequences Module/ - three additional functions (map, fold and filter)

session
Function Reference
 * 1) /Basic Session Management/ - the basics of session management including getting a setting session variables

subversion
Function Reference
 * 1) /Subversion/ - how to update a subversion repository from within XQuery using the subversion client

transform
Function Reference
 * 1) /String Analysis/

util
Function Reference
 * 1) /Registered Modules/ : util:registered-modules
 * 2) /Registered Functions/ : util:registered-functions
 * 3) /Dynamic Module Loading/ : util:import-module, util:eval
 * 4) /Higher Order Functions/ : util:function, util:call
 * 5) /Timing Fibonacci algorithms/ : util:function, util:call, util:system-time
 * 6) /XMP data/ : util:binary-doc, util:binary-to-string, util:parse
 * 7) /Basic Authentication/ : util:string-to-binary, httpclient:get

validation
Function Reference Help
 * 1) /Validating a document/

xmldb
Function Reference
 * 1) /Saving and Updating Data/
 * 2) /Splitting Files/

xqdoc
Function Reference
 * 1) /Generating xqDoc-based XQuery Documentation/

xslfo
XSL-FO (Formatted Objects) is a way of converting XML into PDF. Function Reference
 * 1) /Installing the XSL-FO module/ - setting up your XSL-Module within eXist
 * 2) /Generating PDF from XSL-FO files/ - generating PDF from a FO file
 * 3) /XSL-FO Tables/ - adding tables to your PDF
 * 4) /XSL-FO Images/ - adding images to your PDF
 * 5) /XSL-FO SVG/ - adding SVG images to your PDF

Triggers

 * 1) /Using Triggers to Log Events/ - how to set up a trigger to log store, update and remove events on a collection
 * 2) /Using Triggers to assign identifiers/ - how to use triggers to assign identifiers to new documents or new nodes
 * 3) /Sending E-mail/ Email is one way to notify when a trigger has fired

XQuery Updates

 * 1) /Inserting and Updating Attributes/
 * 2) /Updates and Namespaces/ - How updates can change serialization

URL Rewriting

 * 1) /URL Rewriting Basics/ How to make your URLs look nice

Apache Ant Tasks

 * 1) /Reindex a Collection/

Replication

 * 1) /eXist Replication /

General guidance

 * /eXist Crib sheet/
 * eXist-db maintenance - delete versions of binary files

= Appendixes =

Systems that Support XQuery
Using native and hybrid XML databases that implement XQuery
 * 1)  BaseX  - Native open source XML Database with visual frontend
 * 2) /DataDirect XQuery/ - Java XQuery engine supporting relational, EDI, flat files and XML input/output
 * 3)  eXist - Open source native XML database
 * 4)  DB2 pureXML - DB2 9.1 includes the pureXML feature
 * 5)  MarkLogic Server - MarkLogic Server commercial XML Content Server
 * 6) /Microsoft SQL Server 2005/ -
 * 7) /NetKernel/ - NetKernel
 * 8) /Oracle Berkeley DB XML/ - Open source embedded storage management
 * 9)  Oracle XML DB - Oracle Server 11g includes the XML DB (XDB) feature
 * 10)  Sedna  - Open source native XML Database
 * 11) /Stylus Studio/ - XQuery mapping/editing/debugging, ships with Saxon (and SA) and DataDirect XQuery
 * 12) EMC xDB - EMC Documentum xDB commercial native XML database
 * 13) /XQilla/ - Open source XQuery library and command line utility
 * 14) Zorba - Open source XQuery engine C++ implementation with C, Java, Php, Python, Ruby library bindings and command line utility
 * 15) Qizx - Open source and pro XQuery engine Java implementation

Debugging XQuery

 * 1) /Gotchas/ - some pitfalls
 * 2) /Ah-has// - some ah-ha moments

= Other sources =

Function Libraries

 * 1) FunctX XQuery Function Library by Priscilla Walmsley

Discussion Groups

 * 1) XQuery General

= Indexes =
 * /Index of Application Areas/ - edited

Key to symbols:

indicates an XQuery/Best practice

XQuery