Skip to main content

Incident Playbook

Response procedures for common issues.

Connection Issues​

WebSocket Disconnect​

Symptoms: WebSocket connection lost, no messages received

Actions:

  1. Implement auto-reconnect with exponential backoff
  2. Poll REST endpoints to catch up on missed updates
  3. Resubscribe to all channels after reconnect

Prevention: Monitor connection status and implement robust reconnection logic

API Timeout​

Symptoms: REST requests timeout or return 5xx errors

Actions:

  1. Retry with exponential backoff
  2. Check health endpoint: GET /health
  3. Reduce request rate if system is overloaded

Prevention: Implement request throttling and circuit breakers

Order Issues​

High Rejection Rate​

Symptoms: Many orders rejected

Investigation:

  1. Check rejection reasons via GET /orders?wallet=...
  2. Review margin: GET /portfolio?wallet=...
  3. Check tier: GET /user-tier?wallet=...
  4. Verify instruments not expired: GET /instruments

Actions:

  • If margin issues: Reduce position size or add collateral
  • If tier issues: Upgrade to tier2 or cover sells
  • If expired: Use different instruments

Missing Fills​

Symptoms: Orders filled but no fill notifications

Investigation:

  1. Check GET /fills?wallet=... for fills
  2. Verify WebSocket fills channel subscription
  3. Check WebSocket connection status

Actions:

  • Resubscribe to fills channel
  • Poll REST endpoint for missed fills
  • Reconcile fills with order status

Stale Order Status​

Symptoms: Order status in REST doesn't match WS

Investigation:

  1. Check WebSocket connection status
  2. Verify order cache is up to date
  3. Compare REST and WS order status

Actions:

  • Poll REST after WS reconnect to catch up
  • Use REST as source of truth for reconciliation

MMP Issues​

MMP Triggers Too Frequently​

Symptoms: Many orders canceled by MMP

Investigation:

  1. Check MMP config: GET /mmp-config?wallet=...&currency=...
  2. Review fill patterns and cumulative metrics
  3. Check if limits are too low

Actions:

  • Increase MMP limits (qty, delta, vega)
  • Increase interval_ms to allow more fills in window
  • Reduce quoting frequency

MMP Doesn't Trigger​

Symptoms: Fills exceed limits but MMP doesn't trigger

Investigation:

  1. Verify MMP is enabled: GET /mmp-config?wallet=...&currency=...
  2. Check mmp_enabled=true on orders
  3. Verify currency matches order underlying

Actions:

  • Enable MMP on orders: mmp_enabled=true
  • Configure MMP for correct currency
  • Adjust limits if needed

Margin Issues​

Insufficient Margin​

Symptoms: Orders rejected with "Insufficient margin"

Investigation:

  1. Check portfolio: GET /portfolio?wallet=...
  2. Review margin calculation
  3. Check which scenario is failing

Actions:

  • Reduce position size
  • Add collateral (when deposit flow implemented)
  • Close positions to free margin
  • Hedge exposure to reduce worst-case loss

Missing Spot Price​

Symptoms: Orders rejected with "No spot price available"

Investigation:

  1. Check Hyperliquid spot price feed connectivity
  2. Verify underlying symbol is correct
  3. Check if spot price feed is operational

Actions:

  • Wait for spot price feed to recover
  • Use different underlying if available
  • Contact support if issue persists

System Issues​

High Latency​

Symptoms: Slow API responses or WebSocket message delays

Investigation:

  1. Check system load
  2. Monitor response times
  3. Check network connectivity

Actions:

  • Reduce request rate
  • Implement request throttling
  • Contact support if issue persists

Rate Limiting​

Symptoms: Requests rejected or throttled

Current: Rate limiting is not enforced; self-throttle recommended

Actions:

  • Implement client-side throttling
  • Reduce request rate
  • Use bulk endpoints when possible

Emergency Procedures​

Kill Switch​

Immediate actions:

  1. Cancel all orders: DELETE /bulk_order or DELETE /bulk_order_cloid
  2. Disconnect WebSocket
  3. Stop quoting system

Recovery:

  1. Verify all orders canceled: GET /orders?wallet=...
  2. Review portfolio: GET /portfolio?wallet=...
  3. Investigate root cause
  4. Resume quoting after issue resolved

Data Reconciliation​

After incident:

  1. Poll REST endpoints for current state
  2. Reconcile orders: GET /orders?wallet=...
  3. Reconcile fills: GET /fills?wallet=...
  4. Reconcile portfolio: GET /portfolio?wallet=...
  5. Resume WebSocket subscriptions

Escalation​

If issue persists:

  1. Review known issues and staging notices provided with access
  2. Check Runbooks for detailed procedures
  3. Contact support with:
    • Wallet address
    • Error messages
    • Timestamps
    • Steps to reproduce