XQuery/Freebase

= Motivation = You want to access a web service that takes a JSON document as a query parameter using REST that returns a JSON document as a result. This example uses Freebase, a popular open database that was purchase by Google.

= Basic Example = We will create a Freebase query that will return the names of the first three albums with the artist ID: '/en/bob_dylan'.

We will use the EXPath http client library and send a HTTP GET request to the freebase server using a URI to encode our query.

Sample Results
The result returned is a string in JSON format. I have added spaces to show the structure of the result

Freebase API Key
Here is a link to a blog that describes the process of obtaining a Freebase API key

http://anchetawern.github.io/blog/2013/02/11/getting-started-with-freebase-api/

= Using Cursors to Get Additional Data = The default freebase query limits the number of results returned. Freebase provides database cursors to facilitate retrieval of the entire set of query results. You ask for a cursor (see the example API call below for the form of the initial request) to be returned with your query results and this acts as a link to the next set of query results.

https://developers.google.com/freebase/v1/mql-overview#querying-with-cursor-paging-results

The next set of results is obtained by supplying the value of the cursor returned from the previous invocation. Along with that next set you get another cursor that points to the set after that. When the final set of results are retrieved the cursor is set to the string 'false'.

The example on the Freebase query overview webpage has sample Python code that invokes libraries to take care of all the cursor handling for you.

https://developers.google.com/freebase/v1/mql-overview#looping-through-cursor-results

However the same thing can easily be achieved from XQuery with a little bit of tail recursion. We will use as an example the following MQL query that returns all films with their netflix_id's.

A few brief comments about MQL. You ask for something by giving the field name and a value null. Null gets replaced by the actual value. However if the field can have multiple values MQL will return an array and cause your null query to error. This may happen even when you are expecting a singular value so you can avoid this problem by using the symbol for an empty array instead of null as in the query above.

You can paste the query above into http://www.freebase.com/query to see the results (we will take care of the cursor in the code example). Now to the code, which assumes XQuery 3.0

Freebase returns JSON so we use the above package to convert it to XML. From eXist you can install the package by just clicking it on the eXist Package Manager which you can get to from the eXist Dashboard.

We declare a variable for our query.

Since we are going to be doing tail recursion we need to put the API call in a function that will will make the API call and store the results in the db.

declare function local:freebaseCall($cursor as xs:string,$i as xs:integer)

This function will take two parameters. The first is the cursor and the second an integer that provisions an auto-incremented unique file name.

To kick it all off pass the null string as the initial cursor value and initialise your counter

Alternatively you can prime the function call with a cursor to restart retrieval from where you left off.

= Acknowledgements = Original example provided by Ihe Onwuka and Michael Westbay on eXist list in March of 2014

= References =
 * eXist package for JSON Parser and Serializer for XQuery http://exist-db.org/exist/apps/public-repo/packages/xqjson.html
 * Joe Wicentowski JSON serializer on Github taken from John Snelson's tools https://github.com/joewiz/xqjson
 * Chris Wallace JSON tools on Github https://gist.github.com/KitWallace/3646277
 * JSON Serializer: http://exist-db.org/exist/apps/wiki/blogs/eXist/JSONSerializer