Optimizing Spark in the Cloud - with Jean-Yves Stephan
Listen now
Description
This episode features Jean-Yves Stephan Co-Founder & CEO @ Data Mechanics (recently Acq. by Spot by NetApp), during our discussion we talk about optimizing Spark to run in the cloud at a low cost. Top 3 Value Bombs: Running Spark CAN be expensive but there are ways to reduce your current operating costs by 50-75% by smart automations (i.e.  tune for node type, memory and CPU). Spot instances can lower your costs by utilizing unused instances. Creating serverless architectures and using containers will allow for more flexibility with deployment models and scalability. 
More Episodes
In this episode we speak with Justin Borgman, Chairman & CEO at Starburst, which is based on open source Trino (formerly PrestoSQL) and was recently valued at $3.35 billion after securing their series D funding.  In this episode we discuss convergence of DW’s / DL's, why data lakes fail and...
Published 03/15/22
In this episode we speak with Paul Singman Developer Advocate at Treeverse / LakeFS. LakeFS is an open source project  that allows you to transform your object storage into a Git-like repository.  Top 3 takeaways LakeFS enables use cases like debugging to quickly view historical versions of your...
Published 03/01/22