November 15th 2016

Bringing App Servers Back Online

A heartbeat failure in the primary storage array controller triggered a failover to the secondary controller. The failover took approximately 1 minute to complete. During this 1 minute, PHP threads stacked up leading to an eventual crash of the PHP-FPM process.

We are currently brining all app servers back online and placing them back into the load balancer pool.