Omkar Joshi offers an overview on how performance challenges were addressed at Uber while rolling out its newly built flagship ingestion system, Marmaray (open-sourced) for data ingestion from various sources like Kafka, MySQL, Cassandra, and Hadoop. This system is rolled out in production and has been running for over a year now, with more ingestion systems onboarded on top of it. Omkar and team heavily used jvm-profiler during their analysis to give them valuable insights. This new system is built using the Spark framework for data ingestion. It’s designed to ingest billions of Kafka messages per topic from thousands of topics every 30 minutes. The amount of data handled by the pipeline is of the order hundreds of TBs. At this scale, every byte and millisecond saved counts. Omkar detail how to tackle such problems and insights into the optimizations already done in production.

Some key highlights are:

- how to understand your bottlenecks in Spark applications, to cache or not to cache your Spark DAG to avoid…

This talk explains how Spark 3.0 can improve the performance of SQL applications. Spark 3.0 provides many performance features such as dynamic partitioning and enhanced pushdown. Each of them can improve the performance of a different type of SQL application. Since the number of features is large, it is not easy for application developers to understand these features at a glance. This talk gives a brief explanation of these features with an example program and explains how it works and how we can improve the performance.

Here are takeaways of this talk:
1. What optimization features for SQL Spark 3.0 support.
2. Which programs Spark 3.0 can accelerate.

About:
Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unifie...
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks…

Yes I admit, most Chinese people have this funny sounding accent. lol and now you can learn it.  I will not accept anymore answers when this video reaches over 5000 views. To qualify you have to be one of my subscribers and GOOD LUCK! :)


Twitter:  
http://twitter.com/popking161

Facebook:
http://www.facebook.com/#!/pages/Popking161-Magician-Joe/184251214939311

Understand why native speakers of Chinese languages have a Chinese accent when speaking English with the help of Jackie Chan.
Speak clearly and confidently with my online course: https://www.ImproveYourAccent.co.uk/Course
Instagram: https://www.instagram.com/ImproveYourAccent/
Twitter: https://twitter.com/ImproveAccent
Facebook: https://www.facebook.com/ImproveYourAccent/
Free pronunciation learning resources: https://www.ImproveYourAccent.co.uk/Learn

Sounds from FreeSound.org
Slide Guitar 1 by Redafs, Bamboo Swing, B8 by InspectorJ, Cartoon Bounce by Alexir, Wind swish swoosh 2 by daalvinz
Background Music: Inevitable from YouTube Audio Library

Специализированные процессоры ASIC для конкретных областей — один из способов «перезапустить» закон Мура и преодолеть ограничения универсальных CPU общего назна...