PagerDuty



Introduction

PagerDuty is one of many integrations supported by Rootly. With PagerDuty, you can manage and automate on-call contact info, teams, schedules, alerting, and escalation policies. It can serve as a central point for ensuring the right people are notified when incidents happen.

There are a few key features in PagerDuty that work well with Rootly:

  • Services are a core feature in PagerDuty that represent an application or component against which you create alerts.
  • Self-service contact management allows users to set up profiles in PagerDuty with their contact info where they can be reached when an incident occurs.
  • On call schedule management allows those creating on call schedules to schedule team members in an easy to use interface. Those same team members can also be notified when they have an upcoming on-call shift, and they can also create overrides when someone else is covering some or all of their shift.
  • Team organization gathers together staff in a logical grouping to make schedule and service management easier.
  • Alert escalations are a feature that will automatically escalate unacknowledged alerts, or allow the person being paged to escalate to another engineer should they need assistance.



Rootly and PagerDuty

Rootly serves as a central point for incident coordination, communication, and data collection when things break. It integrates with a number of external services to enhance and automate incident response workflows.

When you are using Rootly with PagerDuty, you can:

  • Import PagerDuty services into Rootly Services to provide consistency between the two when using the PagerDuty to Rootly integration.
  • Create a Rootly alert when creating an incident in PagerDuty to kick off the incident management process.
  • Create a PagerDuty incident when creating an incident in Rootly, which will ensure those listed in the on-call schedule in PagerDuty are notified via their preferred contact methods.
  • Resolve a PagerDuty incident right from Rootly.
  • Page directly from Slack, if the Slack and Rootly Integration is enabled.

In the following section, we'll show you how to configure the integration between PagerDuty and Rootly.



Integration

Configuring the integration between PagerDuty and Rootly is simple and painless. You will need to be an administrator in Rootly to do so.

Log into PagerDuty with the account you wish to integrate with Rootly.

While logged into Rootly, click the drop down in the upper right corner next to your username. Then click Integrations.

Document image



On the Integrations screen, scroll down until you find PagerDuty and click Setup.

Document image



You will see a screen similar to the one below, where you will find the PagerDuty account where you logged in. The one listed below is a test account we use. Yours will be named differently. Click the blue button with the account you wish to integrate.

Document image



On the next screen, click Authorize to allow Rootly to have access to your PagerDuty account.

Document image



You will then see the PagerDuty integration configuration page, where you can choose services to add.

Note that there are two fields:

  • One for Default PagerDuty Services to show during incident creation in Rootly.
  • The other for PagerDuty services where Rootly will listen to create an incident in Rootly.

There is a drop down that will list all available services to choose. Be sure to choose all applicable services and which direction you wish them to integrate. In this case, we just have a single demo service we are going to select for both directions.

When finished, be sure to click Save Settings.

Document image



That's all there is to it! Your integration is complete.



Importing Services

It is also possible to import services from PagerDuty into Rootly so that these services can be managed from within Rootly for full incident enhancement and management. This is done from the Configuration → Services page in the top navigation bar. Doing so imports the services you select from PagerDuty into the Rootly service catalog.

In the navigation bar at the top of the Rootly screen, click Configuration, then Services.

Document image



Next, click the Import from PagerDuty button.

Document image



Check the boxes next to the services you want to import, then click Import Services.

Document image



You should then see the services you selected appear on the Services page.

Document image



From the Services page, you can configure your newly imported services to use additional Rootly features. To configure the new service, click the Edit link for that service on the right side of the page.

When you do, you will see something like the following:

Document image



From this page, you will have the ability to enhance the service with additional Rootly features such as Functionalities, Playbooks, Team Owner, User Owner, Slack Channels, additional email addresses to send notifications, and so on.

To learn more about these features, see the Rootly documentation.

Incident Management

Incident management is at the core of how Rootly and PagerDuty work together. Once you've configured your integration, you will want to test the incident creation and management features in Rootly.

To create an incident, do the following:

Towards the upper-right side of any screen in Rootly, locate the Create Incident button and click it.

Document image



You will then be presented with the New Incident Details dialog, where you can then begin to add details. At a minimum, you will need to put in a title for the incident. If you don't know the rest of the information, you can always create the incident and come back to edit it after the fact to add more details.

Once you are done filling in the details, click Create Incident.

Document image



You will be taken to a screen with all of the current details of the incident.

Document image



As you can see from the incident timeline, once the incident was created, it was also created for PagerDuty. If you login to PagerDuty, and click on Incidents in the upper left navigation bar, you will see the incident created there as well.

Document image

To resolve an incident in Rootly, click the Mark as Mitigated button on the incident details screen.

Document image



You will be presented with a dialog to explain any actions taken to mitigate the incident. Once you've filled it out, click the Mark as Mitigated button once more.

Document image



Once all of the followup for the incident has been done, you can resolve it through Rootly by clicking the Mark as Resolved button.



Document image



You will be presented with a dialog where you can enter any additional details about the incident resolution. Once completed, click the Mark as Resolved button once more.

Document image



If you return to the PagerDuty Incidents screen, and click on Resolved, you will see the incident has now been resolved in PagerDuty as well.

Document image



When resolving incidents, note the following:

  • Marking an incident as resolved in Rootly will mark this incident as resolved in PagerDuty for all linked services associated with this incident.
  • Marking an incident as resolved in PagerDuty will not mark this incident as resolved in Rootly.

Auto Assign On-calls to Incident Roles



If you'd like to automatically assign on-calls of a given PagerDuty Schedule to a Incident Role (e.g. Commander), you have two options:

Assign through Incident Role

You can configure an incident role to have a PagerDuty Schedule associated with it. This will assign however is currently on-call for that Schedule to the Incident Role when an incident is created. Here's how to set this up:

  • Go to Incident Roles in the Configuration tab
Document image
  • Add/Edit an Incident Role and select the PagerDuty Schedule to link.
Document image

Now, the next time an Incident is created, the active on-call for the linked PagerDuty Schedule will be assigned automatically to the role!

Please note that:

  • The created Incident will only associate active on-calls to the roles. In other words, if the PD Schedule is not linked to a PD Escalation Policy, for example, the on-call is considered inactive and will not be linked to the incident.
  • The email used for the on-call in PagerDuty must match the user's email in Rootly.



Assign though Teams

Another option for auto-assigning roles is to set up a Team with members, each linked to a PagerDuty Schedule. By bringing that Team into a new or ongoing incident, the current on-calls will be assigned to their Roles accordingly, overriding any existing assignments. Here's how to set that up:

  • Go to Teams in the Configuration tab
Document image
  • Add or select a Team
Document image
  • Add or select a Membership
Document image
  • Select a PD Schedule in the User or On-call field and an Incident role to associate it
Document image

Now, the next time you add a Team to an Incident (new or ongoing), the users on-call will be auto-assigned to their roles!

Please note that:

  • This feature overrides the Incident Role auto-assign. In other words, if you have an Incident Role that has a PD Schedule associated with it and a Team that has on-call members is added to an Incident, the Team members will override the Incident Role on-call.
  • The last team added to an Incident will always override the Roles of the previously added Teams.
  • Use the table below to know when Rootly will invite users to your Incident channel

Configuration

Expected outcome

On-call without role associated

Will invite to incident channel

Rootly user without role associated

Will not invite to incident channel

On-call with role associated

Will invite to incident channel & assign role

Rootly user with role associated

Will invite to incident channel & assign role



Paging Automatically via Genius Workflows

Set up automated workflows to page various on-call schedules and escalation policies based on the condition of an incident (e.g SEV1 or greater pages Infrastructure).

Creating Alerts in PagerDuty

All PD alerts can flow into Rootly for incident management if desired. By now, you should have your integration between Rootly and PagerDuty configured, as well as the services on which you wish to alert.

To create an alert from PagerDuty, do the following:

Log into the PagerDuty account where you first created the new integration.

Click the New Incident button, which should be available from most pages within PagerDuty.

Document image



In the Create New Incident dialog, select the Impact Service, which should be one of the services you integrated earlier with Rootly. Add a descriptive Title, and fill in any other fields as needed.

Document image



Click Create Incident.

You will then be taken to the page for that incident.

Document image



Now, login to Rootly if you aren't already, and navigate to Configuration, then Alerts.

Document image



There, you will see the list of current alerts, where you can create an incident for that alert if desired. To do so, click the Create Incident button on the right side, and follow the standard incident workflow outlined previously.

Document image



When you create an incident in Rootly from an alert in PagerDuty, management of that incident can then be done via Rootly. Resolving the incident in Rootly will also resolve it in PagerDuty.



Using Slack with PagerDuty and Rootly

Rootly has the capability to integrate with Slack as well as PagerDuty. You can have alerts fired into a Slack channel, as well as in Rootly and PagerDuty at the same time.

We'll assume for the sake of brevity that you've already configured your Slack integration. If not, see the documentation for the Rootly and Slack integration to configure this first. We've provided our demo Slack configuration in the image below:

Document image



When following the incident creation workflow in Rootly that we mentioned earlier, you can see that Rootly has already created a Slack channel and posted the incident information.

Document image



To interact with the incident, click the Join Channel button in Slack. You can then run any number of commands depending on the desired outcome.

Supported commands that can be used anywhere in Slack:

  • /rootly create | new <title> - This will create a new incident with a dedicated channel.
  • /rootly list - This will display a list of up to ten incidents.
  • /rootly help - Shows available commands for interacting with the Rootly Slack bot.

Supported commands that can only be used from an incident's Slack channel:

  • /rootly add event <note> - This will create a new event and add it to the timeline event.
  • /rootly add action item - This will create a new action item and add it to your incident.
  • /rootly add team - This will attach a team to an incident.
  • /rootly add service - This will attach a service to an incident.
  • /rootly add functionality - This will attach a functionality to an incident.
  • /rootly mitigate - This will mark an incident as mitigated.
  • /rootly resolve - This will mark an incident as resolved.
  • /rootly publish - This will publish an incident and make it visible to your public status pages.

For example, to close out the incident we previously created via the web UI, in the Slack channel for the incident, simply type:

  • /rootly resolve

You will be presented with a dialog in Slack to resolve the incident.

Document image



It is also possible to create an incident and page people directly from Slack. As an example, in Slack, type:

  • /rootly create "Help! This is a test incident"

You will be presented with a different dialog to create the incident. When doing so, be sure to choose a service that you have already integrated for Rootly and PagerDuty.

Document image



After a moment, a new channel will be created in Slack for the new incident, and you will also see it appear in PagerDuty:

Document image





When resolving incidents, be sure you resolve them through either the Rootly web UI, or via the Slack channel for that incident. Resolving them in PagerDuty will not resolve them in Rootly.

The best way to become familiar with using Slack with Rootly and PagerDuty is to try creating incidents, updating them, adding teams, and resolving them until you understand how each service interacts with the other.



FAQ

Q: What is the difference between configuring services from the Integration page, and importing them from PagerDuty on the Services page?

A: Configuring services on the Integration page allows Rootly to read and/or write to services in PagerDuty for alerting purposes. Importing services from PagerDuty on the Services page adds them to the Rootly service catalog, where additional options can be configured to enhance the incident or provide extra capabilities such as Slack integration, status page, etc.



Q: PagerDuty has some incident management capabilities, how is this different?

A: When it comes to incident management capabilities, there is some overlap between Rootly and PagerDuty. However, Rootly provides a more complete and centralized incident management platform designed to work with the alerting capabilities of PagerDuty, or other similar alerting services such as OpsGenie or VictorOps.



Q: Should I keep paying for PagerDuty or does Rootly replace it?

A: PagerDuty provides excellent on call scheduling management, and can act as an enhancement to the incident management capabilities in Rootly. They work best together, but it isn't strictly necessary to use PagerDuty if you would prefer a different on-call management solution.

Q: I don't use PagerDuty, is that okay?

A: You don't need to use PagerDuty if you don't want it. We do recommend using some kind of on-call management solution such as VictorOps or OpsGenie, if PagerDuty doesn't meet your requirements for those functions.



Q: I'm getting a Failed to authenticate. Reason: Invalid Credentials message when I try to authenticate my PagerDuty account. What is happening?

A: Chances are you are trying to authenticate with a Trial PagerDuty account that has expired. Check your PD account to make sure it is still active. If that's not the case, contact us at support@rootly.io or use the lower right chat widget to get connected with an engineer.



Q: Where can I get more help if I need it with Rootly or other integrations?

A: If you need help or more information about this integration, please contact support@rootly.io or use the lower right chat widget to get connected with an engineer.