Alerts in LangSmith

Self-hosted Version Requirement

Access to alerts requires Helm chart version 0.10.3 or later.

Overview

Effective observability in LLM applications requires proactive detection of failures, performance degradations, and regressions. LangSmith’s alerts feature helps identify critical issues such as:

• API rate limit violations from model providers
• Latency increases for your application
• Application changes that affect feedback scores reflecting end-user experience

Alerts in LangSmith are project-scoped, requiring separate configuration for each monitored project.

Configuring an alert

Step 1: Navigate to create alert

First navigate to the Tracing project that you would like to configure alerts for. Click the Alerts icon on the top right hand corner of the page to view existing alerts for that project and set up a new alert.

Step 2: Select metric type

LangSmith offers threshold-based alerting on three core metrics:

Metric Type	Description	Use Case
Errored Runs	Track runs with an error status	Monitors for failures in an application.
Feedback Score	Measures the average feedback score	Track feedback from end users or online evaluation results to alert on regressions.
Latency	Measures average run execution time	Tracks the latency of your application to alert on spikes and performance bottlenecks.

Additionally, for Errored Runs and Run Latency, you can define filters to narrow down the runs that trigger alerts. For example, you might create an error alert filter for all llm runs tagged with support_agent that encounter a RateLimitExceeded error.

Step 2: Define alert conditions

Alert conditions consist of several components:

• Aggregation Method: Average, Percentage, or Count
• Comparison Operator: >=, <=, or exceeds threshold
• Threshold Value: Numerical value triggering the alert
• Aggregation Window: Time period for metric calculation (currently choose between 5 or 15 minutes)
• Feedback Key (Feedback Score alerts only): Specific feedback metric to monitor

Example: The configuration shown above would generate an alert when more than 5% of runs within the past 5 minutes result in errors.

You can preview alert behavior over a historical time window to understand how many datapoints—and which ones—would have triggered an alert at a chosen threshold (indicated in red). For example, setting an average latency threshold of 60 seconds for a project lets you visualize potential alerts, as shown in the image below.

Step 3: Configure notification channel

PagerDuty

Configure PagerDuty as a notification channel using PagerDuty’s Events API v2. This integration allows critical LLM application issues to trigger PagerDuty incidents, enabling rapid response through your established incident management workflow.

Prerequisites

• An active PagerDuty account with administrator access
• Appropriate service-level permissions in PagerDuty

If on a custom deployment of LangSmith, make sure there are no firewall settings blocking egress traffic from LangSmith services.

1. Create a Service in PagerDuty

1. Log in to your PagerDuty account
2. Navigate to Services → Service Directory
3. Click + New Service
4. Complete the following fields:
   • Name: Provide a descriptive name (e.g., “LangSmith Monitoring”)
   • Description: Add details about the monitored application
   • Escalation Policy: Select the appropriate team escalation policy
   • Integration Type: Select “Events API V2”
5. Click Add Service to create the service

2. Obtain integration key

After creating the service, retrieve the Integration Key:

1. From the Service Directory, locate and click on your newly created service

2. Select the Integrations tab

3. Find the “Events API V2” integration

4. Copy the Integration Key (a 32-character alphanumeric string)

   

3. Configure LangSmith alert with PagerDuty

To receive the same alert again within an hour of it being triggered, you must resolve the active incident created by the alert in PagerDuty.

1. In the notification section of your alert set-up in LangSmith, select PagerDuty
2. Click the key icon to save the Integration Key as a Workspace secret or select an existing Workspace secret. As a best practice, we recommend saving the Integration Key as a Workspace Secret rather than adding it directly. This will allow you to re-use the same key across alerts for a workspace.
3. Configure additional notification options:
   • Severity: Maps to PagerDuty incident priority
4. Send a test alert by clicking Send Test Alert
5. Verify the incident is triggered by PagerDuty and contains relevant LangSmith alert information

Troubleshooting

If incidents aren’t being created in PagerDuty:

• Verify the Integration Key is entered correctly in LangSmith
• Ensure the PagerDuty service is active and not in maintenance mode
• Check that your PagerDuty account has Events API v2 enabled
• If an alert trigger appears to be missing in PagerDuty, check whether the expected trigger occurred within one hour of a previous trigger from the same alert rule, and whether the incident created by the previous alert is still open.
• Review network connectivity if your LangSmith instance is behind a firewall

Additional resources

• PagerDuty Events API v2 Documentation
• PagerDuty Integration Guide

Webhook

Webhooks enable integration with custom services and third-party platforms by sending HTTP POST requests when alert conditions are triggered. Use webhooks to forward alert data to ticketing systems, chat applications, or custom monitoring solutions.

Prerequisites

• An endpoint that can receive HTTP POST requests
• Appropriate authentication credentials for your receiving service (if required)

1. Prepare your receiving endpoint

Before configuring the webhook in LangSmith, ensure your receiving endpoint:

• Accepts HTTP POST requests
• Can process JSON payloads
• Is accessible from external services
• Has appropriate authentication mechanisms (if required)

If on a custom deployment of LangSmith, make sure there are no firewall settings blocking egress traffic from LangSmith services.

2. Configure webhook parameters

In the notification section of your alert, complete the webhook configuration with the following parameters:

Required fields

• URL: The complete URL of your receiving endpoint
  • Example: https://api.example.com/incident-webhook

Optional fields

• Headers: JSON key-value pairs sent with the webhook request

  • Common headers include:
    • Authorization: For authentication tokens
    • Content-Type: Usually set to application/json (default)
    • X-Source: To identify the source as LangSmith
  • If no headers, use {}

• Request Body Template: Customize the JSON payload sent to your endpoint

  • Default: LangSmith sends the payload defined and the following additional key-value pairs appended to the payload:
    • project_name: Name of the triggered alert
    • alert_rule_id: A UUID to identify the LangSmith alert. This can be used as a de-duplication key in the webhook service.
    • alert_rule_name: The name of the alert rule.
    • alert_rule_type: The type of alert (as of 04/01/2025 all alerts are of type threshold).
    • alert_rule_attribute: The attribute associated with the alert rule - error_count, feedback_score or latency.
    • triggered_metric_value: The value of the metric at the time the threshold was triggered.
    • triggered_threshold: The threshold that triggered the alert.
    • timestamp: The timestamp that triggered the alert.

3. Test the webhook

Click Send Test Alert to send the webhook notification to ensure the notification works as intended.

Troubleshooting

If webhook notifications aren’t being delivered:

• Verify the webhook URL is correct and accessible
• Ensure any authentication headers are properly formatted
• Check that your receiving endpoint accepts POST requests
• Examine your endpoint’s logs for received but rejected requests
• Verify your custom payload template is valid JSON format

Security considerations

• Use HTTPS for your webhook endpoints
• Implement authentication for your webhook endpoint
• Consider adding a shared secret in your headers to verify webhook sources
• Validate incoming webhook requests before processing them

Send alerts to Slack

Configure Slack notifications via webhook

Here is an example for configuring LangSmith alerts to send notifications to Slack channels using the chat.postMessage API.

Prerequisites

• Access to a Slack workspace
• A LangSmith project to set up alerts
• Permissions to create Slack applications

Step 1: Create a Slack app

1. Visit the Slack API Applications page
2. Click Create New App
3. Select From scratch
4. Provide an App Name (e.g., “LangSmith Alerts”)
5. Select the workspace where you want to install the app
6. Click Create App

Step 2: Configure bot permissions

1. In the left sidebar of your Slack app configuration, click OAuth & Permissions
2. Scroll down to Bot Token Scopes under Scopes and click Add an OAuth Scope
3. Add the following scopes:
   • chat:write (Send messages as the app)
   • chat:write.public (Send messages to channels the app isn’t in)
   • channels:read (View basic channel information)

Step 3: Install the app to your workspace

1. Scroll up to the top of the OAuth & Permissions page
2. Click Install to Workspace
3. Review the permissions and click Allow
4. Copy the Bot User OAuth Token that appears (begins with xoxb-)

Step 4: Add the bot to a Slack channel

Add the bot to the specific channel you want to receive alerts in. You can add a bot to a Slack channel by mentioning it in the message field (e.g., @botname).

You also need the channel ID to configure the webhook alert in LangSmith. You can find the channel ID by opening channel details > About.

Step 5: Configure the webhook alert in LangSmith

1. In LangSmith, navigate to your project
2. Select Alerts → Create Alert
3. Define your alert metrics and conditions
4. In the notification section, select Webhook
5. Configure the webhook with the following settings:

Webhook URL

https://slack.com/api/chat.postMessage

Headers

{
  "Content-Type": "application/json",
  "Authorization": "Bearer xoxb-your-token-here"
}

Request Body Template

{
  "channel": "{channel_id}",
  "text": "{alert_name} triggered for {project_name}",
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "🚨{alert_name} has been triggered"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "Please check the following link for more information:"
      }
    },
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "<{project-url}|View in LangSmith>"
      }
    }
  ]
}

6. Click Save to activate the webhook configuration

Step 6: Test the integration

1. In the LangSmith alert configuration, click Test Alert
2. Check your specified Slack channel for the test notification
3. Verify that the message contains the expected alert information

(Optional) Step 7: Link to the alert preview in the request body

After creating an alert, you can optionally link to its preview in the webhook’s request body.

To configure this:

1. Save your alert
2. Find your saved alert in the alerts table and click it
3. Copy the displayed URL
4. Click “Edit Alert”
5. Replace the existing project URL with the copied alert preview URL

Additional resources

• Slack chat.postMessage API Documentation
• Slack Block Kit Builder

Best practices

• Adjust sensitivity based on application criticality
• Start with broader thresholds and refine based on observed patterns
• Ensure alert routing reaches appropriate on-call personnel

---

Edit this page on GitHub or file an issue.

Connect these docs to Claude, VSCode, and more via MCP for real-time answers.