Merge pull request #2 from AlbertSuarez/issue-1

WorldOMeter data source

README.md

@@ -13,7 +13,7 @@
 
 ## Data source
 
-Real time data is being retrieved from the [Novel Coronavirus (COVID-19) Cases repository](https://github.com/CSSEGISandData/COVID-19), by [JHU CSSE](https://systems.jhu.edu/research/public-health/ncov/).
+Real time data is being retrieved from the [Worldometer website](https://www.worldometers.info/coronavirus/).
 
 ## Python requirements
 
@@ -23,6 +23,9 @@ This project is using Python3.7. All these requirements have been specified in t
 2. [Dictdiffer](https://dictdiffer.readthedocs.io/en/latest/): used for checking data differences easier.
 3. [APScheduler](https://apscheduler.readthedocs.io/en/stable/): used for scheduling jobs in a certain time.
 4. [Twitter](https://python-twitter.readthedocs.io/en/latest/): used for posting tweets.
+5. [BeautifulSoup](https://pypi.org/project/beautifulsoup4/): used for scraping an HTML content.
+6. [Tor](https://2019.www.torproject.org/docs/debian.html.en): used for making requests anonymous using other IPs.
+7. [Fake User-Agent](https://pypi.org/project/fake-useragent/): used for using random User-Agent's for every request.
 
 ## Recommendations
 
@@ -49,13 +52,7 @@ To run this script, please execute the following from the root directory:
   TWITTER_ACCESS_TOKEN_SECRET={TWITTER_ACCESS_TOKEN_SECRET}
   ```
 
-4. Run the script as a Python module
-
-  ```bash
-  python3 -m src
-  ```
-
-  or as a Docker container
+4. Run the script as a Docker container
 
   ```bash
   docker-compose up -d --build

docker-compose.yml

@@ -1,5 +1,6 @@
-version: '3.5'
+version: '3'
 services:
+
   covid19-bot:
     image: covid19-bot
     container_name: covid19-bot
@@ -11,3 +12,12 @@ services:
     restart: always
     volumes:
       - './data:/srv/covid19-bot/data'
+    links:
+      - covid19-bot-tor
+
+  covid19-bot-tor:
+    image: dperson/torproxy
+    container_name: covid19-bot-tor
+    ports:
+      - 8118:8118
+      - 9050:9050

requirements.lock

@@ -2,14 +2,18 @@ requests==2.23.0
 APScheduler==3.6.3
 python-twitter==3.5
 dictdiffer==0.8.1
+beautifulsoup4==4.8.2
+fake-useragent==0.1.11
 ## The following requirements were added by pip freeze:
 certifi==2019.11.28
 chardet==3.0.4
 future==0.18.2
 idna==2.9
 oauthlib==3.1.0
+PySocks==1.7.1
 pytz==2019.3
 requests-oauthlib==1.3.0
 six==1.14.0
+soupsieve==2.0
 tzlocal==2.0.0
 urllib3==1.25.8

requirements.txt

@@ -1,4 +1,6 @@
-requests
+requests[socks]
 APScheduler
 python-twitter
-dictdiffer
+dictdiffer
+beautifulsoup4
+fake_useragent

src/__main__.py

@@ -12,10 +12,9 @@
             scheduler.scheduler.start()
         else:
             log.info('Running jobs manually since DEBUG is enabled.')
-            time.sleep(5)
-            scheduler.run()
             while True:
-                time.sleep(10000)
+                time.sleep(30)
+                scheduler.run()
 
     except Exception as e:
         log.error(f'Unexpected error {e} in Covid-19')