Orkhan Dadashov: Graduate Teaching Assistant for Object-Oriented Design in San Mateo, CA

May 2023

Analyze Iceberg usage service - Daily petabytes of data were ingested into thousands of Iceberg tables, and queried by Apache Spark, Trino and Flink. This project has helped the AIML organization to measure and prioritize (data/metadata) optimizations on thousands Iceberg tables. I have implemented this service from scratch and deployed into production - the iceberg usage data is collected from every Spark job when they accessediceberg tables. Iceberg usage data was ingested into Iceberg tables on cloud and visualized via Apache Superset. ? Global orphan file removal service - Iceberg tables are stored on Amazon S3 object storage. This project has helped to apply a strict privacy policy to cleanup and remove files beyond a point, particularly removing orphan files for which pointers from Iceberg tables were already removed. I have implemented this project using technologies such as Apache Spark, HiveMetastore, Iceberg tables, Amazon S3 and Airflow (for timely scheduling Spark jobs).