Pagination

Pagination is a method for dividing the result of a query into several pages of results.

Pagination of results allows :

• Better performance: When processing large amounts of data, retrieving everything at once can be very heavy on the server and can result in longer response times. By limiting the number of results returned at a time, we can reduce server effort and thus get faster responses.

• Better user experience: if your API requests are one-off, it will be more convenient for you to work with smaller amounts of data. Paginating allows you to work with a more manageable subset of data before moving on to the next “batch” of data.

GraphQL pagination at iAdvize

iAdvize GraphQL API limits results to a maximum of 100 per page.

To traverse through a paginated data set, you need to use cursors. A cursor represents a specific position in the data set. You can get the first and last cursor on a page, and also indications on previous page or next page existence by asking the pageInfo property:

• startCursor: cursor of the first element of the page. Could be used for backward pagination.

• endCursor: cursor of to the last element of the page. Could be used for forward pagination.

• hasPreviousPage: boolean which indicates if there are elements to retrieve in a previous page.

• hasNextPage: boolean which indicates if there are elements to retrieve in a next page.

Forward Pagination

Forward pagination is accomplished using two arguments:

• first: number of elements to retrieves.

• after: the cursor to retrieve elements after. Typically, you will pass the endCursor of the current page as after.

Backward Pagination

Backward pagination is accomplished using two arguments:

• last: number of elements to retrieves.

• before: the cursor to retrieve elements before. Typically, you will pass the startCursor of the current page as before.

Forward Pagination example

Here is an example of a query to retrieve the id, email, first name and the date time of the last connection of agents in project 1.

 query MyQuery {
  users(projectIds: [1], after: null, first: 2) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email,
          firstName
          lastLoggedAt
        }
      }
    }
  }
}

In this query, users is the resource we use to obtain users information. We then use three arguments: after, projectIds and first:

• projectIds: identifiers of projects for which we want to retrieve the list of advisors and their last connection date. In our example, only the project 1.

• after: In our example, it is null, which means we want to start from the very beginning. We will see below how to go to the result of the 2nd page.

• first: Here we put 2, so we will obtain the desired information for the first 2 agents.

Here is the result obtained after running this query (1st page of results):

{
  "data": {
    "users": {
      "pageInfo": {
        "endCursor": "YXJyYXljb25uZWN0aW9uOjE=",
        "hasNextPage": true
      },
      "edges": [
        {
          "node": {
            "id": 1,
            "email": "[e-mail]",
            "firstName": "Émilie",
            "lastLoggedAt": "2023-07-18T16:16:17Z"
          }
        },
        {
          "node": {
            "id": 2,
            "email": "[e-mail]",
            "firstName": "Davy",
            "lastLoggedAt": "2023-07-31T14:32:01Z"
          }
        }
      ]
    }
  }
}

In the pageInfo object, we obtain the information necessary to retrieve the following page:

"pageInfo": {
  "endCursor": "YXJyYXljb25uZWN0aW9uOjE=",
  "hasNextPage": true
}

endCursor : YXJyYXljb25uZWN0aW9uOjE= gives us the cursor of the end of this results page. This sequence of characters is intended to mark a stop: “I got there”. You can use the value of this cursor by specifying it in the "after" filter in a new request to retrieve the page that follows this cursor. Note that cursors are only meant to be re-transmitted back when paginating, without any modification.

hasNextPage : true tells us that there is at least one other result page. If you get hasNextPage : false this means that there is no other page of results after the one you are currently viewing.

How to move to results on subsequent pages?

To move on to the results on page #2 of our initial query, simply assign the value of the end cursor to the argument after:

query MyQuery {
  users(after: "YXJyYXljb25uZWN0aW9uOjE=", projectIds: [1], first: 2) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}

To access to all of the results, you must repeat this same exercise of changing the content of after by the value of the last endCursor you get in your previous query, as long as property hasNextPage is true.

You will know you have reached the last page of results when hasNextPage is false.

Automate the retrieval of all query results

If you want to set up a data export system on a regular basis, you will not be able to apply the manual method described previously.

To automate pagination with GraphQL, you can use a loop that performs successive queries until there are no more pages to retrieve. Each query uses the endCursor property from the previous page as the value for the argument after, which tells GraphQL where to start fetching data for the next page.

From our example query, here is an example of a Python/Javascript script:

NodeJS

const axios = require('axios');

// API URL
const url = 'https://api.iadvize.com/graphql';

// Headers for the request
const headers = {
    "Authorization": "Bearer YOUR_AUTHORIZATION_TOKEN",
    "Content-Type": "application/json"
}

// Initial query
const query = `
query MyQuery($after: String) {
  users(after: $after, projectIds: [1], first: 100) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}
`;
let variables = {
    "after": null
};

// Fetch all pages
let allResults = [];
const getData = async () => {
    try {
        const response = await axios.post(url, { query, variables }, { headers });

        // Extract data from the response
        const users = response.data.data.users;
        if (!users) {
            throw new Error("No users in response.");
        }
        allResults.push(...users.edges);

        // Check if there is a next page
        if (!users.pageInfo.hasNextPage) {
            return allResults;
        }

        // Update the cursor for the next query
        variables.after = users.pageInfo.endCursor;
        return await getData();
    } catch (error) {
        console.error(`An error occurred: ${error}`);
        return allResults;
    }
}

// Fetch all data
getData().then(allResults => {
    // Display all results 
    allResults.forEach(result => {
        console.log(result);
    });
});

This script uses a recursive function getData to retrieve all pages. This function queries the first 100 users, then checks to see if a next page exists. If so, she uses the endCursor from the current page to make a new query for the next page, and adds the results to the list allResults.

This process is repeated until there are no more pages to retrieve. In case of error, it is displayed in the console. Note that you will need axios package to run this script.

Python

import requests
import json

# API URL
url = "https://api.iadvize.com/graphql"

# Header for the request
headers = {
    "Authorization": "Bearer YOUR_AUTHORIZATION_TOKEN",
    "Content-Type": "application/json"
}

# Initial query
query = """
query MyQuery($after: String) {
  users(after: $after, projectIds: 9999, first: 100) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}
"""
variables = {
    "after": null
}

# Fetch all pages
all_results = []
while True:
    try:
        # Send the request
        response = requests.post(
            url, 
            headers=headers, 
            json={'query': query, 'variables': variables}
        )

        # Check the request succeeded
        response.raise_for_status()

        # Extract data from the response 
        data = response.json()
        users = data.get('data', {}).get('users')
        if not users:
            raise Exception("No users in response.")

        all_results.extend(users['edges'])

        # Check if there is a next page
        if not users['pageInfo']['hasNextPage']:
            break

        # Update the cursor for the next query
        variables['after'] = users['pageInfo']['endCursor']
    except requests.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
        break
    except Exception as err:
        print(f"An error occurred: {err}")
        break

# Display all results
for result in all_results:
    print(result)

This script runs a query for the first 100 users, then checks if a following page exists. If so, it uses the endCursor from the current page to make a new query for the next page, and adds the results to the list all_results. This process is repeated until there are no more pages to retrieve.

This script also includes error handling (response.raise_for_status()) to ensure that it correctly handles any errors that may arise during query execution.

Depending on the BI tool you use (PowerBI, Tableau, Talend, etc.), you will need to adapt this script.

Lean more about pagination

If you want to learn more about pagination in GraphQL, here are two additional resources you might find helpful:

1. Pagination in GraphQL: a detailed explanation of pagination in GraphQL from the creators of GraphQL.

2. Shopify Developer Guide: Pagination with GraphQL: a how-to guide from Shopify explaining how they implemented pagination with GraphQL.