Embrace automatic incident remediation with PagerDuty and GorillaStack

PagerDuty is the first external service we are supporting via AWS EventBridge.

PagerDuty is our eyes and ears online, alerting us to incidents as they occur on more than 370 different online services.

GorillaStack is our arms and legs, letting us quickly and automatically act to change our digital infrastructure on AWS and Azure.

With GorillaStack’s new integration for PagerDuty, these two great tools have come together — creating the first Automatic Incident Remediation (AIR) system that you can start using in minutes.

Let’s say that a particular piece of your infrastructure is prone to many problems, each of which will trigger an event on PagerDuty.

Let’s also say that in response to this recurring problem, you typically do one of five things:

1. take a snapshot of the volume attached to the EC2 instance in case everything dies
2. reboot the EC2 instance
3. enlarge the Autoscaling Group that contains the EC2 instance
4. execute a custom shellscript on the EC2 instance
5. send a custom message to someone who needs to know about the situation

Now it’s easy to create five rules in GorillaStack that do this:

1. Observe the incident on PagerDuty => ask for manual approval from you => execute response no# 1
2. Observe the incident on PagerDuty => ask for manual approval from you => execute response no# 2
3. Observe the incident on PagerDuty => ask for manual approval from you => execute response no# 3
4. Observe the incident on PagerDuty => ask for manual approval from you => execute response no# 4
5. Observe the incident on PagerDuty => ask for manual approval from you => execute response no# 5

Then when the incident occurs, all five rules will be triggered, and you can easily choose which of these will go on to execute their pre-canned responses. Click Yes to the approval requests on rules no# 3 and 5 to execute their pre-canned responses. Just ignore the others. By automating your responses in this way, you can resolve each incident not just more quickly, but also in a flexible yet predictable way.

Want to know more? Take a moment to read the documentation:

• PagerDuty Event trigger
• Notify PagerDuty Event action
• PagerDuty integration

Then please give it a test drive and let us know what you think.