Project #1
Project: FDP – Finance Data Platform
Domain: EDMp-Group Functions
Role: Senior Development Manager
Work Location: Malaysia
Duration: December 2021 – Current
Technologies used: Hadoop, Hive, Spark, Flask, Apache Airflow, Spark SQL, API, Apache Superset
Monitoring Tools: Grafana
Description: Finance Data Platform (FDP) is a journey to deliver data in a consistent and standardized manner across the Finance Landscape, this project is also aimed to reduce the cost of multiple use cases within Finance scope that currently source and transform data in silos. The objective is to create a platform that will source once and use across multiple consumers.
Responsibilities:
✔ Developed Metadata Driven Framework using Python,Spark,Spark SQL
✔ Designed UI using Flask for product catalogue View and build API using routes
✔ Initial Dag with Airflow designed and build for job orchestration for batch processing
✔ Worked on Integration of Airflow statistics with Grafana Dashboard
✔ Managing the Team and Code Reviews will be part of the role
✔ Monitoring the Hadoop echo system with Ambari and Yarn applications.
✔ SQL Generation in JSON format from Excel input has been developed, based on the input details provided by BA SQL will be generated
✔. Optimized/Rewritten the existing spark applications framework which gave 15x faster than the existing application.