Docker

We recently Dockerized the main part of our event processing pipeline using the 1.12-rc4 release. It’s been awesome using Docker in Swarm mode, and we’ve been really impressed with the ease of setup and use of Swarm mode. Our event pipeline processes around 220 million application errors per day, approximately 150k per minute, and performs a massive variety of processing tasks — it’s a critical piece of our infrastructure.

A few weeks ago we were alerted that Docker Swarm was using over 8GB of RAM. Our investigation led us to discover an unexpected factor that determines its memory usage. After a bit of graphing and math, we were able to locate the code behind this unexpected behavior.