Agent Reliability Kit View Dockerfile PR
Independent project · Not affiliated with Ramp

Know if it's your agent — or the API.

When ramp --agent or MCP calls fail, you need receipts: latency, success, redacted logs, and a clear verdict — not another guess in the loop.

5‑min walkthrough See sample report
plan → report
four-step harness
0 creds
offline contract tests
ALL_PASS
verified on ramp-cli
cli-offline-contracts run d9762b027228
Plan
Run
Doctor
Report
$ ramp-kit plan scenarios/cli-offline-contracts.yaml
pass config-list 430ms exit 0
pass auth-status-offline 312ms exit 0
pass env-show 298ms exit 0
ALL_PASS — contracts OK
Start here

What is this, in plain English?

You're building an AI agent that calls Ramp (CLI or MCP). When something breaks, this kit tells you where — your code, OAuth, API latency, or permissions.

Agents + finance = you need proof

Ramp ships ramp --agent (JSON for scripts) and MCP (natural language). Their own engineering blog admits MCP can be slow and unreliable on large datasets.

  • Your agent might parse JSON wrong (schema_version drift)
  • OAuth might be expired or missing scopes
  • Ramp API might timeout on big pulls
  • Roles might block admin-only commands

Without receipts, every failure feels like "the model hallucinated." This kit fixes that.

Plan — see what will run before you run it

Scenarios are YAML files listing CLI commands and expected outcomes. ramp-kit plan prints the step list — no API calls, no credentials.

  • cli-offline-contracts — works logged out (config, env, auth status)
  • cli-sandbox-identity — needs ramp auth login
  • permission-boundary-demo — teaches role limits

Run — capture a receipt per tool call

Each step executes ramp --agent … in an isolated temp config (so tests never touch your real tokens). Every step writes a receipt:

  • latency_ms — how long the call took
  • exit_code + success
  • redacted stdout/stderr — secrets stripped
  • trace_id — when Ramp returns x-trace-id

Doctor — classify the failure

One command answers the question every agent builder asks:

"Is it my code or Ramp?"

  • AUTH_BLOCKER → log in again, check scopes
  • LATENCY_BLOCKER → API timeout (known MCP pain)
  • AGENT_CODE_ISSUE → your harness mishandled JSON
  • ALL_PASS → contracts satisfied

Report — shareable HTML artifact

ramp-kit all runs the full pipeline and writes report.html you can open, screenshot, or attach to a PR.

  • Same discipline as a Tinker Workbench run folder
  • Audit-safe: no prod credentials in the repo
  • Reproducible: same scenario → comparable receipts
Open real sample report → 

          

        

      

    


    
    

      

        How it works
        
Built for Ramp's public agent surfaces

        

          

            
⌘

            
ramp-cli --agent

            
Scriptable JSON envelope with schema_version: 1.0. The kit contract-tests every step.

          

          

            
◈

            
Ramp MCP

            
SQL/ETL over Developer API. Latency is the known bottleneck — receipts capture it.

          

          

            
◇

            
Trust boundaries

            
Actions respect the logged-in user's role. Demo scenarios show permission limits.

          

        


        

          
Doctor verdicts explained

          

            

              ALL_PASS
              
Every step met contract expectations. Your harness and CLI envelope agree.

            

            

              AUTH_BLOCKER
              
OAuth, scopes, or permissions. Re-login — not an agent logic bug.

            

            

              LATENCY_BLOCKER
              
Timeouts on large datasets. Ramp's blog calls this out for MCP.

            

            

              AGENT_CODE_ISSUE
              
JSON envelope or schema_version drift in your code.

            

          

        

      

    


    
    

      

        

          
Try it in 60 seconds (no Ramp login)

          
Clone the project, run offline contracts against ramp-cli source. Zero credentials.

          

            
            
cd ramp/kit
uv sync
uv run ramp-kit all scenarios/cli-offline-contracts.yaml \
  --ramp-cli-src ../upstream/ramp-cli \
  --runs-dir ../runs
open ../runs/*/report.html

          

          

            With sandbox OAuth: ramp auth login then run
            cli-sandbox-identity.yaml
          

        

      

    


    
    

      

        Live artifact
        
Sample report — ALL_PASS

        
Real output from the offline scenario. Five steps, zero failures.

        

          

            cli-offline-contracts · run d9762b027228
            Open full page →