This service uses Twitter’s streaming API to stream tweets filtering some subjects: formula 1, f1, formula1
For every tweet look up for pilot names, and team names and save the outputs into CSV files:
The outputs are:
A CSV file with raw data received.
A CSV with drivers names, aggregated by mentions.
A CSV with team names, aggregated by mentions and drivers name.
A CSV with tweets aggregated by country.
This repository already contains some data streamed within a 60-minutes time frame from Twitter.
Before you start getting your hands dirty it's important that you learn the concepts behind the code structure.
Our architecture is based on Clean Architecture, if you want to get into the details about this architecture
you can take a look at Robert C. Martin's blog post.
As mentioned, our architecture is based on Clean Architecture so there are some minor differences, this is how the files should be organized:
src/
commom/
extractor/
handlers/
loader/
logger/
transformer/
tests/
unit/
These instructions will get you a copy of the service up and running on your local machine for development and testing purposes.
Make sure you have all the following prerequisites:
This service uses Python version 3.8. To download and install the version got to the python page here.
This project uses third parthies python libs. Make sure you have it all installed in your enviroment.
pip install -r requirements.txt
Before run the commands you will need to store your Bearer Token inside src/commom/config.py file on get_config() method.
To execute the services, use IPython
After install the requirements, on root folder type ipython and run the following commands
In [1]: from src.handlers.stream_data import main
In [2]: main()
In [1]: from src.handlers.aggregate_data import main
In [2]: main()