Scaling I/O Bound Microservices
- Haggai Philip Zagury (hagzag)
- Development , Developer experience ( dev ex) , Presentations , Webinar , Kubernetes , Youtube , Production readiness
- March 4, 2020
Table of Contents
In this meetup, we will continue our #2ndhalf journey to the next^2. You can see the talks from the 1st meetup of this series here - http://bit.ly/second-half-p1 Nowadays, scaling and auto-scaling have become relatively easy tasks. Everyone knows how to set up auto-scaling environments - Auto-Scaling groups, Swarm, Kubernetes, etc.
But when we try to scale I/O Bound workloads: Message queues (Kafka, Rabbit, NATS) Distributed databases (Hadoop, Cassandra) Storage subsystems (CEPH, GlusterFS, HDFS), the traditional auto-scaling mechanisms are just not enough.
Heavy calculations must be performed to determine the I/O bottlenecks. Rebalancing the data after a scaling event can take up to hours depending on your data & could, resulting in data loss if not properly designed.
We will deep dive into this type of workload and walk you through code samples you can apply in your own environment.
