Scraping Data from Pagination (or multiple pages)

Most websites make use of the feature of pagination to restrict the amount of information and data on a page. However, it can become an obstacle for you.

To extract multiple pages of data, you will also need to add pagination functionality.

This Pagination functionality allows scrapers to easily scrape data from the page without special modifications.

To add pagination functionality, create scraper with any website URL which has pagination.

(Take this as Example URL: https://www.flipkart.com/mobiles/mi~brand/pr?sid=tyy,4io&otracker=nmenu_sub_Electronics_0_Mi).

Load that website and select all the data fields you want to extract.

Now you can click “Add column”, then locate & click on Pagination Link of the website.

To apply pagination, go to column settings => click on "Capture Next Page link" to enable it which will have a checkmark next to it when enabled.

If you observe after applying this column setting, you will see how all the text of pagination will convert into pagination links.

By applying this, PWS algorithm will automatically scrape listing from all the pages at runtime.

After adding pagination functionality, you can go ahead, click “Save”.

Then enable button →  is this listing page and click “Save & Run” to get our list from all the pages.