This scraper is based on the bs4 scripts. We have also included the data frame to make a csv output with all the logging info with a separate file.
Steps: -
- Install all the dependencies from requirements.txt
- Ensure that you must have installed the right version of the libraries.
Note: -
- Not throttle the server by making multiple requests at a time.
- Obey robots.txt