Pagination

Pagination is a method for dividing the result of a query into several pages of results.

🤔 What's pagination?

Pagination of results allows :

• Better performance: When processing large amounts of data, retrieving everything at once can be very heavy on the server and can result in longer response times. By limiting the number of results returned at a time, we can reduce server effort and thus get faster responses.

• Better user experience: if your API requests are one-off, it will be more convenient for you to work with smaller amounts of data. Paginating allows you to work with a more manageable subset of data before moving on to the next “batch” of data.

🤓 GraphQL pagination at iAdvize

iAdvize GraphQL API limits results to a maximum of 100 per page.

Here is an example of a query to retrieve the date/time of the last connection (“lastLoggedAt”) of “agents” type profiles (“on Professional”) in project 9999 (“projectIds: 9999”) associated to their id (“id”), email (“email”) and first name (“firstName”):

 query MyQuery {
  users(after: "", projectIds: 9999, first: 100) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}

In this query, users is the resource we use to obtain user profile information. We then use three parameters: after, projectIds and first:

• after: which is commonly called “cursor”. In our example, it is empty ("", but could also be null), which means we start from the first element (1st page result). We will see below how to go to the result of the 2nd page.

• projectIds: This is the identifier of the project for which we want to retrieve the list of advisors and their last connection date. In our example, it is the project 9999 (also called SID 9999).

• first: This is the number of items we want to retrieve. Here we put 2, so we will obtain the desired information for the first 2 agents. Note that if you want to retrieve as much data as possible in a single query, you can change this number to the maximum 100.

The section pageInfo allows you to retrieve information used for pagination:

• endCursor: this is the "end cursor" corresponding to the last element retrieved by your query. This field will be populated automatically when you launch your query. This end cursor will allow you to move to the next page.

• hasNextPage: This is a boolean (true or false) which indicates if there are elements to retrieve in a future page. This field is automatically populated when you run your query.

Here is the result obtained running this query (1st page of results):

{
  "data": {
    "users": {
      "pageInfo": {
        "endCursor": "YXJyYXljb25uZWN0aW9uOjk=",
        "hasNextPage": true
      },
      "edges": [
        {
          "node": {
            "id": 263636,
            "email": "[e-mail]",
            "firstName": "Émilie",
            "lastLoggedAt": "2023-07-18T16:16:17Z"
          }
        },
        {
          "node": {
            "id": 263640,
            "email": "[e-mail]",
            "firstName": "Davy",
            "lastLoggedAt": "2023-07-31T14:32:01Z"
          }
        }
      ]
    }
  }
}

In the pageInfo object, we obtain the information necessary to retrieve the following page:

"pageInfo": {
  "endCursor": "YXJyYXljb25uZWN0aW9uOjk=",
  "hasNextPage": true
}

endCursor : YXJyYXljb25uZWN0aW9uOjk= gives us the cursor of the end of this results page. This sequence of characters is intended to mark a stop: “I got there”. You can use the value of this cursor by specifying it in the "after" filter in a new request to retrieve the page that follows this cursor. Note that cursors are only meant to be re-transmitted back when paginating, without any modification.

hasNextPage : true tells us that there is at least one other result page. If you get hasNextPage : false this means that there is no other page of results after the one you are currently viewing.

⏭️ How to move to results on subsequent pages?

To move on to the results on page #2 of our initial query, simply assign the value of the end cursor to the field after:

query MyQuery {
  users(after: "YXJyYXljb25uZWN0aW9uOjk=", projectIds: 9999, first: 100) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}

To access all of the results, you must repeat this same exercise of changing the content of after by the value of the last endCursor you get in your previous query, as long as hasNextPage is true.

You will know you have reached the last page of results when hasNextPage is false.

This pagination system is now applicable to most iAdvize GraphQL API resources. However, some resources have specificities. So, for searchClosedConversations, you can’t specify first and for satisfactionSurveyResponses first is named cursor but the function remains similar. If you have any doubts or problems, do not hesitate to contact your usual contact at iAdvize.

🎡 Automate the retrieval of all query results

If you want to set up a data export system on a regular basis, you will not be able to apply the manual method described previously.

To automate pagination with GraphQL, you can use a loop that performs successive queries until there are no more pages to retrieve. Each query uses the endCursor from the previous page as the value for the parameter after, which tells GraphQL where to start fetching data for the next page.

From our example query, here is an example of a Python/Javascript script:

JavaScript

const axios = require('axios');

// API URL
const url = 'https://api.iadvize.com/graphql';

// Headers for the request
const headers = {
    "Authorization": "Bearer YOUR_AUTHORIZATION_TOKEN",
    "Content-Type": "application/json"
}

// Initial query
const query = `
query MyQuery($after: String) {
  users(after: $after, projectIds: 9999, first: 100) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}
`;
let variables = {
    "after": ""
};

// Fetch all pages
let allResults = [];
const getData = async () => {
    try {
        const response = await axios.post(url, { query, variables }, { headers });

        // Extract data from the response
        const users = response.data.data.users;
        if (!users) {
            throw new Error("No users in response.");
        }
        allResults.push(...users.edges);

        // Check if there is a next page
        if (!users.pageInfo.hasNextPage) {
            return allResults;
        }

        // Update the cursor for the next query
        variables.after = users.pageInfo.endCursor;
        return await getData();
    } catch (error) {
        console.error(`An error occurred: ${error}`);
        return allResults;
    }
}

// Fetch all data
getData().then(allResults => {
    // Display all results 
    allResults.forEach(result => {
        console.log(result);
    });
});

This script uses a recursive function getData to retrieve all pages. This function queries the first 100 users, then checks to see if a next page exists. If so, she uses the endCursor from the current page to make a new query for the next page, and adds the results to the list allResults.

This process is repeated until there are no more pages to retrieve. In case of error, it is displayed in the console. Note that you will need node.js and axios packages to run this script.

Python

import requests
import json

# API URL
url = "https://api.iadvize.com/graphql"

# Header for the request
headers = {
    "Authorization": "Bearer YOUR_AUTHORIZATION_TOKEN",
    "Content-Type": "application/json"
}

# Initial query
query = """
query MyQuery($after: String) {
  users(after: $after, projectIds: 9999, first: 100) {
    pageInfo {
      endCursor
      hasNextPage
    }
    edges {
      node {
        ... on Professional {
          id
          email
          firstName
          lastLoggedAt
        }
      }
    }
  }
}
"""
variables = {
    "after": ""
}

# Fetch all pages
all_results = []
while True:
    try:
        # Send the request
        response = requests.post(
            url, 
            headers=headers, 
            json={'query': query, 'variables': variables}
        )

        # Check the request succeeded
        response.raise_for_status()

        # Extract data from the response 
        data = response.json()
        users = data.get('data', {}).get('users')
        if not users:
            raise Exception("No users in response.")

        all_results.extend(users['edges'])

        # Check if there is a next page
        if not users['pageInfo']['hasNextPage']:
            break

        # Update the cursor for the next query
        variables['after'] = users['pageInfo']['endCursor']
    except requests.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
        break
    except Exception as err:
        print(f"An error occurred: {err}")
        break

# Display all results
for result in all_results:
    print(result)

This script runs a query for the first 100 users, then checks if a following page exists. If so, it uses the endCursor from the current page to make a new query for the next page, and adds the results to the list all_results. This process is repeated until there are no more pages to retrieve.

This script also includes error handling (response.raise_for_status()) to ensure that it correctly handles any errors that may arise during query execution.

Depending on the BI tool you use (PowerBI, Tableau, Talend, etc.), you will need to adapt this script.

🧑‍🎓 Lean more about pagination

If you want to learn more about pagination in GraphQL, here are two additional resources you might find helpful:

1. Pagination in GraphQL: a detailed explanation of pagination in GraphQL from the creators of GraphQL.

2. Shopify Developer Guide: Pagination with GraphQL: a how-to guide from Shopify explaining how they implemented pagination with GraphQL.