Glue scripts for converting AWS Service Logs for use in Athena

The Sensitive Data Protection on AWS solution allows enterprise customers to create data catalogs, discover, protect, and visualize sensitive data across multiple AWS accounts. The solution eliminates the need for manual tagging to track sensitive data such as Personal Identifiable Information (PII) and classified information.

Build and deploy a serverless data pipeline on AWS with no effort.

AWS ETL Pipleine

Extract, transform, and load data for analytic processing using AWS Glue

AWS streaming pipeline for real-time analysis

This is a data pipeline built with the purpose of serving a business team.

Terraform configuration that creates several AWS services, uploads data in S3 and starts the Glue Crawler and Glue Job.

Terraform module which creates Glue Job resources on AWS.

This project outlines the final project requirements for DAV6100 - Information Architectures, focusing on group assignments, scoring criteria, topic selection, core requirements, and project components such as design, development, visualization, and executive presentation.

Terraform module to create and manage a AWS Glue job

Pipeline ETL na AWS

Data Engineering project using data streaming produced by python applications, ETL process and availability for ad-hoc SQL queries in the AWS cloud

This project creates a serverless data pipeline to extract data from the Colombo Stock Market ASI Index API using AWS Lambda, Kinesis Firehose, and S3. An AWS Glue workflow processes and transforms the data, storing it in an Apache Iceberg table via Athena and Glue ETL jobs.

IMDB Movie Data ETL Pipeline using S3, Glue, Redshift, EventBridge, SNS

Data Streaming and Batch processing using AWS Services

ETL using application streaming and creating a Data Lake

ETL pipeline using Apache Airflow

Used AWS Glue to perform ETL operations and load resultant data to AWS Redshift. In the second phase used AWS CloudWatch rules and LAMBDA to automatically run GLUE Jobs