![]() ![]() ![]() While deploying our first applications, we found that using the below set of tools were useful when debugging Flink: Having the right profiling tools on hand are key to getting insights into how to solve a performance problem. Find the Right Profiling Toolsįirst things first. We’ll start off by covering recommended tooling, then focus on performance and resiliency aspects. Some data models require storing immense state (for example, 13 TB for the sales data, as we shared in our Storing State Forever: Why It Can Be Good For Your Analytics talk) and we’ve spent a lot of time on performance tuning, learning many lessons along the way.īelow we’ll walk you through key lessons for optimizing large stateful Apache Flink applications. Keeping large stateful applications resilient is difficult. ![]() Making sure our Flink applications stay performant and resilient is one of our top priorities. We also use RocksDB state backend and write our checkpoints and savepoints to Google Cloud Storage (GCS). Our clusters are configured to use High Availability mode to avoid the Job Manager being the single point of failure. Our Flink applications are deployed in a Kubernetes environment leveraging Google Kubernetes Engine. By: Yaroslav Tkachenko, Kevin Lam, and Rafael AguiarĪt Shopify, we’ve adopted Apache Flink as a standard stateful streaming engine that powers a variety of use cases like our BFCM Live Map.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |