
Project #1
Project: FDP – Finance Data Platform
Domain: EDMp-Group Functions
Role: Senior Development Manager
Work Location: Malaysia
Duration: December 2021 – Current
Technologies used: Hadoop, Hive, Spark, Flask, Apache Airflow, Spark SQL, API, Apache Superset
Monitoring Tools: Grafana
Description: Finance Data Platform (FDP) is a journey to deliver data in a consistent and standardized manner across the Finance Landscape, this project is also aimed to reduce the cost of multiple use cases within Finance scope that currently source and transform data in silos. The objective is to create a platform that will source once and use across multiple consumers.
Responsibilities:
✔ Developed Metadata Driven Framework using Python,Spark,Spark SQL
✔ Designed UI using Flask for product catalogue View and build API using routes
✔ Initial Dag with Airflow designed and build for job orchestration for batch processing
✔ Worked on Integration of Airflow statistics with Grafana Dashboard
✔ Managing the Team and Code Reviews will be part of the role
✔ Monitoring the Hadoop echo system with Ambari and Yarn applications.
✔ SQL Generation in JSON format from Excel input has been developed, based on the input details provided by BA SQL will be generated
✔. Optimized/Rewritten the existing spark applications framework which gave 15x faster than the existing application.
Apache Frameworks: Hadoop, Spark, Hive, Zookeeper, Airflow,Superset
J2EE Technologies: Spring Boot, Rest API, EJB30, Servlets, JSP, Web Services [SOAP]
Big Data Technologies: Hadoop 30, Hive 2, Spark 22, Spark SQL, Py-Spark 22Application/Web Server WebSphere 85, Tomcat 90,JBOSS
Databases ORACLE9i, 11g and DB2Web Interface: Flask/React JS, Applet, JSP, XML, HTML, CSSMonitoring tools: Graphite, Grafana, PrometheusLanguages: JAVA (J2SE, J2EE) , PythonBuild tools: Maven, GradleDevops: Ansible, Azure Dev-opsTools Query Tool, Aqua Studio, TOAD, JBUILDERFramework: Hibernate 43, DroolsRepository: Azure Repo, SVN, GITOther tools: Jenkins, git, Tableau, MicroStrategy