XQuery/Synchronizing Remote Collections

Motivation
You want to update items on collections that are new or newer than another collection.

Method
Many database store creation dates and last-modified dates along with resources. These dates can be used to see if a local collection is out of sync with a remote collection. An XQuery script can be written that will only list the new files or files that are newer then the creation date on your local collection.

For the eXist database, here are the two functions that are used to access the timestamps.

xmldb:last-modified($collection, $resource) xmldb:created($collection, $resource) Where: $collection is the path to the collection (xs:string) $resource is the name of the resource (xs:string)

For example:

let $my-file-last-modified := xmldb:last-modified('/db/test', 'myfile.xml')

Will return the date and time that the file myfile.xml in the collection /db/text was last modified. The format of the timestamp is the XML Schema dateTime format:

"2009-06-04T07:50:04.828-05:00"

For example, this indicates the time is 7:50 am on June the 4th, 2009 for Central Standard Time which is 5 hours behind Coordinated Universal Time (UTC).

Sample Recursive Collection Last Modified Function
You can combine the xmldb:last-modified function with another function xmldb:get-child-collections($collection) that returns all of the child collections of the current collection. By calling itself using tail recursion you can find all the last modified dates within a collection and all its subcollections.

Here is a sample XQuery function that returns a list of all the last-modified date-times of the resources in a collection and all of the subcollections under it.

Note that two attributes are added to each resource. One is the resource id which must be unique in each collection an the other is the date the resource was last modified.

Sample Driver
You can call this function by simply passing the collection root you wish to start at.

This returns the following file:

Driving Syncs with Apache Ant
You can now use these transforms to create batch files that will transfer only the files that have changed or are new.

Many databases provide Apache Ant tasks that have functions that extract and store operations.

Here is a sample Apache ant target that does an extract on a local file and stores it on a remote file.

Note that the following properties must be set in this Ant file.