In this guide, we will use ProWebScraper’s API to manage input URL lists.
We will:
- Upload Input URL list for Scraper
- Fetch a list of Input URLs for scraper
- Delete Input URL list for scraper
1. Upload Input URL Lists for scraper
When you want to scrape data from a list of URLs, it can be verified either by uploading a list from the dashboard, or you can use this API to upload a list to scrape data from it.
Once the URL list is uploaded for scraper, you can run scraper by API.
Using the Add API, you can directly upload inputs to scraper without having to log into ProWebScraper Dashboard.
Note :
- URLs should be from the same website [same to scraper configured URL] to extract data from it.
- For better results please upload valid URLs.
API endpoint
GET https://api.prowebscraper.com/v1/explicit_url/{token}/list
Example Request Response
To upload Input URL list, a request should be made as follows :
curl --location --request POST 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/add' \
--header 'Authorization: {API KEY}' \
--header 'Content-Type: application/json' \
--data-raw '{ "urls": ["https://www.netpages.co.za/Latest-Updates","https://www.netpages.co.za/Pretoria/Transportation+with+Taxi","https://www.netpages.co.za/Pretoria/Music+Class","https://www.netpages.co.za/Pretoria/Painting+Class","https://www.netpages.co.za/Pretoria/Farm+Equipment+Suppliers"]}'
Which should respond something like :
{"statusCode":200,"message":{"stats":{"total_urls":5,"valid_urls":5,"saved_urls":5}}}
2. Fetch a list of Input URLs for scraper
Once the URLs are uploaded for scraper, you may need to fetch a list of Input URLs.
This List API will retrieve a list of Input URLs for the scraper.
API endpoint:
GET https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/list
Example Request Response
To fetch the Input URL list, a request should be made as follows :
curl --location --request GET 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/list' \
--header 'Authorization: {API KEY}
Which should respond something like :
{"stats":{"total":3,"limit":1000,"offset":0}}
{"pages":["https://www.netpages.co.za/Latest-Updates","https://www.netpages.co.za/Pretoria/Painting+Class","https://www.netpages.co.za/Pretoria/Farm+Equipment+Suppliers"]}
3. Delete Input URLs for scraper
If you are uploading Input URLs via API, you might want to delete the URLs from which scraped data successfully.
Using this Delete API, you can delete your Input URLs directly from your application without having to log into ProWebScraper Dashboard.
3.1 Delete Specific Input URLs
If you want to delete a specific list of URLs, this URL will help you.
API endpoint
DELETE https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/delete
Example Request Response
To delete specific URL list, a request should be made as follows :
curl --location --request DELETE 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/delete' \
--header 'Authorization: {API KEY}' \
--header 'Content-Type: application/json' \
--data-raw '{"urls": ["https://www.netpages.co.za/Pretoria/Transportation+with+Taxi","hps://www.netpages.co.za/Pretoria/Music+Class","https://www.netpages.co.za/Pretoria/Painting+Class","https://www.netpages.co.za/Pretoria/Farm+Equipment+Suppliers"]}'
Which should respond something like :
{"statusCode":200,"message":{"stats":{"total_urls":4,"valid_urls":4,"deleted_urls":4}}}
3.2 Delete All Input URLs
If you want to delete all the input URLs [except default input URL], this deleteALL API will help you.
API Endpoint
DELETE https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/deleteAll
Example Request Response
To delete a specific URL list, a request should be made as follows :
curl --location --request DELETE 'https://api.prowebscraper.com/v1/explicit_url/{TOKEN}/deleteAll' \
--header 'Authorization: {API_KEY}'
Which should respond something like :
{"statusCode":200,"message":{"stats":{"deleted_urls":6}}}
Comments
0 comments
Please sign in to leave a comment.