Glue scripts for converting AWS Service Logs for use in Athena

Build and deploy a serverless data pipeline on AWS with no effort.

Extract, transform, and load data for analytic processing using AWS Glue

AWS streaming pipeline for real-time analysis

This is a data pipeline built with the purpose of serving a business team.

Terraform configuration that creates several AWS services, uploads data in S3 and starts the Glue Crawler and Glue Job.

Pipeline ETL na AWS

This project creates a serverless data pipeline to extract data from the Colombo Stock Market ASI Index API using AWS Lambda, Kinesis Firehose, and S3. An AWS Glue workflow processes and transforms the data, storing it in an Apache Iceberg table via Athena and Glue ETL jobs.

IMDB Movie Data ETL Pipeline using S3, Glue, Redshift, EventBridge, SNS

ETL pipeline using Apache Airflow

Data Streaming and Batch processing using AWS Services

O projeto foi elaborado com o objetivo de estabelecer uma arquitetura na AWS, originada a partir de uma migração de um banco de dados existente em um ambiente local (on-premise).

This project aims to analyze the popularity of YouTube content across different regions by leveraging datasets sourced from Kaggle. It employs a systematic approach to data preprocessing, cleaning, and analysis using various AWS (Amazon Web Services) services including S3, Lambda, Glue, and others, to build an automated ETL pipeline.

AWS Glue & Airflow automate weekly churn analysis pipeline (S3, Redshift) for telco (1M+ customers) with actionable Tableau dashboard (user segmentation, churn reasons, geographical distribution).

First experimentation into the world of AWS Serverless Data Engineering