README.md

AZLyrics scraper

Box folder URL | Static repo website | Kaggle dataset

🎵 AZLyrics scraper for getting all the song lyrics and publishing to Box.

Python requirements

This project is using Python3. All these requirements have been specified in the requirements.lock file.

1. Requests: used for retrieving the HTML content of a website.
2. BeautifulSoup: used for scraping an HTML content.
3. Tor: used for making requests anonymous using other IPs.
4. Stem: used for authentificating every request with a different IP.
5. Fake User-Agent: used for using random User-Agent's for every request.
6. Unidecode: used for cleaning strings from weird characters.
7. Box SDK: used for uploading/downloading files to/from Box Cloud Storage.

Recommendations

Usage of virtualenv is recommended for package library / runtime isolation.

Usage

To run this script, please execute the following from the root directory:

1. Setup virutal environment

2. Install dependencies

pip3 install -r requirements.lock

3. Move JWT configuration file from Box API

4. Install Tor browser

5. Configure Tor IP renewal editting /etc/tor/torrc file

   ControlPort 9051
   CookieAuthentication 1
   

6. Restart Tor browser

sudo service tor restart

7. Run the script

python3 -m src

JWT configuration

In order to use Box Cloud Storage API in a secure way, this project is configured for using their service with the JWT authentication. After following the tutorial, we will obtain a configuration file which will have to be located under data folder with the name of jwt_config.json as the __init__.py configuration file says:

# Box integration
BOX_CONFIG_FILE_PATH = 'data/jwt_config.json'

Authors

• Albert Suàrez

License

MIT © AZLyrics scraper