Manage Scraper Inputs with API

In this guide, we will use ProWebScraper’s API to manage input URL lists.

We will:

1. Upload Input URL list for Scraper
2. Fetch a list of Input URLs for scraper
3. Delete Input URL list for scraper

1. Upload Input URL Lists for scraper

When you want to scrape data from a list of URLs, it can be verified either by uploading a list from the dashboard, or you can use this API to upload a list to scrape data from it.

Once the URL list is uploaded for scraper, you can run scraper by API.

Using the Add API, you can directly upload inputs to scraper without having to log into ProWebScraper Dashboard.

Note : 

1. URLs should be from the same website [same to scraper configured URL] to extract data from it.
2. For better results please upload valid URLs.

API endpoint

GET https://api.prowebscraper.com/v1/explicit_url/{token}/list

Example Request Response

To upload Input URL list, a request should be made as follows :

curl --location --request POST 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/add' \

--header 'Authorization: {API KEY}' \

--header 'Content-Type: application/json' \

--data-raw '{ "urls": ["https://www.netpages.co.za/Latest-Updates","https://www.netpages.co.za/Pretoria/Transportation+with+Taxi","https://www.netpages.co.za/Pretoria/Music+Class","https://www.netpages.co.za/Pretoria/Painting+Class","https://www.netpages.co.za/Pretoria/Farm+Equipment+Suppliers"]}'

Which should respond something like :

{"statusCode":200,"message":{"stats":{"total_urls":5,"valid_urls":5,"saved_urls":5}}}

2. Fetch a list of Input URLs for scraper

Once the URLs are uploaded for scraper, you may need to fetch a list of Input URLs.

This List API will retrieve a list of Input URLs for the scraper.

API endpoint:

GET https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/list

Example Request Response

To fetch the Input URL list, a request should be made as follows :

curl --location --request GET 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/list' \

--header 'Authorization: {API KEY}

Which should respond something like :

{"stats":{"total":3,"limit":1000,"offset":0}}

{"pages":["https://www.netpages.co.za/Latest-Updates","https://www.netpages.co.za/Pretoria/Painting+Class","https://www.netpages.co.za/Pretoria/Farm+Equipment+Suppliers"]}

3. Delete Input URLs for scraper

If you are uploading Input URLs via API, you might want to delete the URLs from which scraped data successfully. 

Using this Delete API, you can delete your Input URLs directly from your application without having to log into ProWebScraper Dashboard.

3.1 Delete Specific Input URLs

If you want to delete a specific list of URLs, this URL will help you.

API endpoint

DELETE https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/delete

Example Request Response

To delete specific URL list,  a request should be made as follows :

curl --location --request DELETE 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/delete' \

--header 'Authorization: {API KEY}' \

--header 'Content-Type: application/json' \

--data-raw '{"urls": ["https://www.netpages.co.za/Pretoria/Transportation+with+Taxi","hps://www.netpages.co.za/Pretoria/Music+Class","https://www.netpages.co.za/Pretoria/Painting+Class","https://www.netpages.co.za/Pretoria/Farm+Equipment+Suppliers"]}'

Which should respond something like :

{"statusCode":200,"message":{"stats":{"total_urls":4,"valid_urls":4,"deleted_urls":4}}}

3.2 Delete All Input URLs

If you want to delete all the input URLs [except default input URL], this deleteALL API will help you.

API Endpoint

DELETE https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/deleteAll

Example Request Response

To delete a specific URL list,  a request should be made as follows :

curl --location --request DELETE 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/deleteAll' \

--header 'Authorization: {API_KEY}'

Which should respond something like :

{"statusCode":200,"message":{"stats":{"deleted_urls":6}}}