Using the new architecture to build a new page

The new architecture uses three modules:

• synchroniser - puts all the data in a BaseX repository, synchronised on a regular schedule
• api - API layer to pull the data from the repository
• site - site pages

Overview of the steps needed

• Step 1: Analyse the page you're building
• Step 2: API: Build the XQuery elements needed to retrieve the data from BaseX
• Step 3: API: Building the API to pull back all the data
• Step 4: Site: Create API classes turning the JSON into objects to be used in the HTML pages
• Step 5: Site: Modify the page structures to include the data from the API
• Step 6: Site: Update the Middleman controller config.rb

Steps to build a new page

Step 1: Analyse the page you're building

Have a look at the page (both the page.html.erb file and the actual page in DevTracker) and note down all the data elements needed.

Remember to look at the data needed by the partials - much of this data may come from the project.

Step 2: API: Build the XQuery elements needed to retrieve the data from BaseX

Build and test the XQuery code in BaseX client.

Add the XQ file to the folder api/app/conf/xq - this holds all the XQ queries to extract the data in XML from BaseX.

A good example is (DN:insert documents XQ).

        (: A query to get all the documents for a projects :)

        let $db      := db:open('iati')
        let $allProjectDocuments := for $project in $db//iati-activity[@hierarchy=1]

        let $documents := for $document in $project//document-link
         let $categories := for $theCategory in $document//category
             return <category>{ data($theCategory) }</category>
                return ( <document>
                         <title>{data($document//title)}</title>
                         <url>{data($document/@url)}</url>
                         <format>{data($document/@format)}</format>
                         <categories>
                             {$categories}
                         </categories>
                         </document>   )

                 return (
                         <project>
                            <iatiId>{data($project//iati-identifier)}</iatiId>
                            <title>{data($project//title)}</title>
                            <participating-orgs>{data($project//participating-org)}</participating-orgs>
                            <recipient></recipient>
                            <documents>{$documents}</documents>
                          </project> )

        return json:serialize(
            <json arrays="json documents categories participating-org" objects="project document" strings="iatiId title recipient url format category" >
              {$allProjectDocuments}
            </json> )

Top tip: retrieve everything at once, don't query each project in turn.

Step 3: API: Building the API to pull back all the data

• api/app/controllers/Aggregator.scala - add a new method to reference the BaseX call

• Build a route in api/conf/routes

GET /api/projects/documents controllers.Aggregator.allDocumentsForAllProjects

• api/app/controllers/PagesData.scala - defines the data structures to be returned (in JSON) from an API call

• api/app/controllers/PagesAPI.scala - API method calls defined

Top tip: BaseX server is not on until you turn on the synchroniser. You'll see this if you get a [connectException: Connection refused] with a socket.connect() error.

cd research/pipeline/synchroniser
./sbt run

Step 4: Site: Create API classes turning the JSON into objects to be used in the HTML pages

lib/api/projects_documents_api_client.rb

lib/api/project_documents_api_client.rb Create objects referencing the JSON data

(remember needs underscores/all lower case).

Creates objects referencing the JSON objects. Makes it much easier to wire up the pages. This

• calls the API
• creates objects

For example for ProjectDocuments then the objects are:

• class ProjectDocumentsApiClient
• class Documents

Step 5: Site: Modify the page structures to include the data from the API

<pagename>.html.erb pages call the objects referenced in the API calls in Step 4.

Step 6: Site: Update the Middleman controller config.rb

config.rb is now cut down to be much lighter than with the previous architecture.