Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi-page queries, cursor, offset etc. #9

Open
chanansh opened this issue Mar 1, 2022 · 10 comments
Open

multi-page queries, cursor, offset etc. #9

chanansh opened this issue Mar 1, 2022 · 10 comments

Comments

@chanansh
Copy link

chanansh commented Mar 1, 2022

Hi,
new here. How should multipage queries work?
should I get the cursor and then feed it back to the next query? when do I know I got to the end?

How are offset and cursor related?
Please provide documentation of best practice to collect multi-page results.
@ABusyHippie
Copy link

^I second this

@ABusyHippie
Copy link

@chanansh I found this documentation on calling API's with pagination and offset more generally that helped by allowing me to offset by the current page limit I'm getting of 500, but since I'm working with a data set of 10,000 it's still impractical to do this long term and would like to see a solution.

However in the meantime this may help you too: https://stackoverflow.com/questions/18314687/how-to-implement-cursors-for-pagination-in-an-api

@chanansh
Copy link
Author

chanansh commented Mar 4, 2022

I just called the api with the cursor=previous_result['cursor] and if cursor is an empty string the stop. However, this method does not allow threading

@ABusyHippie
Copy link

I just called with &offset=500 to get the 2nd page of results and so on. It's very manual but works for now

@chanansh
Copy link
Author

chanansh commented Mar 4, 2022

I just called with &offset=500 to get the 2nd page of results and so on. It's very manual but works for now

Can you please verify that offset =page size is equivalent to using the next cursor as the cursor?

@ABusyHippie
Copy link

Idk that for sure, I wasn't able to get cursor to work and settled for offset solution for time being. Interested to hear if anyone from the dev team has a better solution or documentation though

@AssFaceNFT
Copy link

How I paginated responses into a single array in javascript:

  1. Run an initial fetch to extract the data.total and data.page_size

image

The linkforMoreFrens function I called at the end is an asynchronous for loop that works out how many offsets it needs to run!
  1. The linkforMoreFrens Math.ceil is where the magic happens. The Math.ceil rounds up the number of pages and then multiplies this by 500, to ensure that every offset is accounted for!

image

Extra for retrying failed attempts via a recursive function: it takes away 500 from the offset to go back to the previous query! Where n is the number of retries.
image

  1. In image 2 you can see that the 'listofResults'.json returned by my fetch_retry function is passed to outputData, another function I used to concatenate every result into a single array! I defined storageArray as an empty global array.
    Then in output data, each result is applied to that array! Worked for me!

image

Hope this helps!

@chanansh
Copy link
Author

chanansh commented Mar 22, 2022 via email

@chanansh
Copy link
Author

chanansh commented Mar 22, 2022 via email

@ivan-liljeqvist
Copy link
Contributor

ivan-liljeqvist commented Mar 30, 2022 •
edited

Hey guys - please ask in https://forum.moralis.io and devs will check!

I know they are working on scaling the API to make the requests faster

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@chanansh @ivan-liljeqvist @ABusyHippie @AssFaceNFT