Skip to content

Tool for scraping text from websites

License

Notifications You must be signed in to change notification settings

artie-inc/web_corpus

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 

Repository files navigation

webScraper.py

Usage

There are three main flags you need feed the script:

python3 webScraper.py --seed --verbose --outfile

--seed is a requirement, it is the seed URL from which the program scrapes

--verbose will print out to the terminal which URLs were accepted or rejected

--outfile is the file to which the scraped text is saved

About

Tool for scraping text from websites

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%