Soorya Ravichandran

Data Engineering → Exposure Management

• Leading infrastructure scaling strategy for 10x load increase across HBase, Solr, Hadoop and Azure services to support 180k enterprise tenants, designing critical bottleneck pipelines for Threat Intelligence.
• Architected Global Insights to Lake initiative end-to-end, defining scalable ingestion architecture and implementing core components including label-derived applicability model and metadata dataset.
• Implemented guardrails in Microsoft’s Discovery services to prevent high-blast radius blocking events, validating safety across 81 million IPs and reducing operational risk.

Data Lake Engineering

• Lead for the firm’s enterprise Data Lake platform, overseeing core services (ingest, sandbox and simulator) to support reporting and analytics across ~160K unique datasets ingesting ~450K daily data pipelines.
• Spearheaded the data & operational catalog, and built the ingest framework using HBase, Flink, HDFS, AWS S3 and Hadoop YARN.
• Developed a production-like simulation test framework that preemptively caught bugs, reducing post-deployment defects by 75% and enabling first-time-right releases.

Core Data Engineering

• Lead Architect for Lake to Lakehouse migration, designing and implementing ingest migration architecture to enhance performance and reduce costs.
• Designed and implemented workflows to decommission 25% of Lake refiners & datastores, migrating unused data to cold storage and reducing quarterly storage costs by ~$500K.
• Engineered parallel ingest pipelines for seamless migration, improved scalability, and enhanced failover reliability.
• Collaborated with Snowflake Engineers to develop the Snowpark Migration Accelerator, automating refiner categorization and streamlining migration to Lakehouse.

• Architected and developed key components of Data Lake, including AWS S3 Ingest service, Data Partitioner, Snowflake Ingest service, Avro-Parquet Scanner & Converter utility, ensuring high reliability and on-time delivery.
• Built a Simulator tool that enabled early bug detection, fostered developer mindfulness, and improved performance tuning, resulting in an 80% increase in customer satisfaction.

• Contributed to the core backend metadata services for data cataloging, ingestion, and refinement in an early-stage enterprise Data Lake; migrated ingest framework from MapReduce to Flink, reducing latency by 25% and memory usage by 33%.
• Built a robust testing framework, standardized for 100% releases, ensuring stable deployments and minimizing post-release firefighting.

• Led the Big Data Club as the primary instructor, designing hands-on projects, course content, and learning pathways in collaboration with the Center for CS Education & Outreach at The University of Texas at Dallas (UTD) to enhance student engagement, coding skills, and Big Data mastery, earning the CS Outreach Excellence Award for impactful contributions.

• Developed a high-precision Emergency Alert Solution with 100% predictability, warning Australian communities of fire and flood hazards while delivering mission-critical ETL data to Telstra’s Network Decision Support Database.

• Developed an online shopping website that markets seasonal clothes based on customer geolocation, rapidly prototyping new features to validate feasibility and user engagement.