README.md

metadata

title: Support Triage Env Environment Server
emoji: 🏒
colorFrom: green
colorTo: pink
sdk: docker
pinned: false
app_port: 8000
base_path: /web
tags:
  - openenv

Support Triage Env Environment

An OpenEnv execution environment that simulates a realistic customer-support workflow: the agent triages an incoming email, optionally requests missing information, and drafts a policy-compliant reply.

This is built for agent evaluation (not a toy/game): it models concrete operator actions (intent classification, clarification questions, escalation decisions) and provides shaped rewards and deterministic graders across 3 tasks (easy → hard).

Quick Start

Use the environment via the standard OpenEnv reset() / step() API. Each task episode advances through stages using task_stage:

from support_triage_env import SupportTriageAction, SupportTriageEnv

with SupportTriageEnv(base_url="http://localhost:8000") as env:
    # Pick a task; Task 1 is the easiest.
    result = env.reset(seed=1, task_id="task_1")
    print("stage:", result.observation.task_stage)

    # Stage 1: classify intent
    result = env.step(
        SupportTriageAction(
            task_id="task_1",
            task_stage="classify",
            intent_category="billing_refund",
            finalize=False,
        )
    )

    # Stage 2: finalize a policy-compliant reply
    result = env.step(
        SupportTriageAction(
            task_id="task_1",
            task_stage="finalize",
            reply_template_id="tpl_refund_ack_next",
            draft_reply="Thanks for reaching out. We received your request. Next step: please confirm details.",
            finalize=True,
        )
    )

    print("done:", result.done, "reward:", result.reward)

That's it! The SupportTriageEnv.from_docker_image() method handles:

• Starting the Docker container
• Waiting for the server to be ready
• Connecting to the environment
• Container cleanup when you call close()

Building the Docker Image

Before using the environment, you need to build the Docker image:

# From project root
docker build -t support_triage_env-env:latest -f server/Dockerfile .

Deploying to Hugging Face Spaces

You can easily deploy your OpenEnv environment to Hugging Face Spaces using the openenv push command:

# From the environment directory (where openenv.yaml is located)
openenv push

# Or specify options
openenv push --namespace my-org --private

The openenv push command will:

1. Validate that the directory is an OpenEnv environment (checks for openenv.yaml)
2. Prepare a custom build for Hugging Face Docker space (enables web interface)
3. Upload to Hugging Face (ensuring you're logged in)

Prerequisites

• Authenticate with Hugging Face: The command will prompt for login if not already authenticated

Options

• --directory, -d: Directory containing the OpenEnv environment (defaults to current directory)
• --repo-id, -r: Repository ID in format 'username/repo-name' (defaults to 'username/env-name' from openenv.yaml)
• --base-image, -b: Base Docker image to use (overrides Dockerfile FROM)
• --private: Deploy the space as private (default: public)

Examples

# Push to your personal namespace (defaults to username/env-name from openenv.yaml)
openenv push

# Push to a specific repository
openenv push --repo-id my-org/my-env

# Push with a custom base image
openenv push --base-image ghcr.io/meta-pytorch/openenv-base:latest

# Push as a private space
openenv push --private

# Combine options
openenv push --repo-id my-org/my-env --base-image custom-base:latest --private

After deployment, your space will be available at: https://huggingface.co/spaces/<repo-id>

The deployed space includes:

• Web Interface at /web - Interactive UI for exploring the environment
• API Documentation at /docs - Full OpenAPI/Swagger interface
• Health Check at /health - Container health monitoring
• WebSocket at /ws - Persistent session endpoint for low-latency interactions

Environment Details

Action Space (SupportTriageAction / SupportAction)

The environment uses a typed, stage-based action schema:

• task_id (str): one of task_1, task_2, task_3
• task_stage (str): classify, request_info, or finalize
• intent_category (str|null): predicted intent category (used during classify)
• reply_template_id (str|null): selected policy template (used during finalize)
• draft_reply (str): drafted reply text (scored with deterministic keyword/rubric checks)
• missing_info_questions (list[str]): questions asked for missing details (Task 2 request_info)
• escalate_to_human (bool): escalation decision (Task 3 finalize)
• finalize (bool): must be true to end the episode on finalize

Observation Space (SupportTriageObservation)

• task_id (str)
• task_stage (str): the next expected stage
• email_subject (str)
• email_body (str)
• thread_history (list[str]): includes customer-provided details for multi-turn tasks
• current_requirements (list[str]): which deterministic requirements are still unsatisfied
• episode_seed (int|null): seed used to generate the simulated email/thread
• done (bool), reward (float), and metadata (dict) are standard OpenEnv fields

Reward Shaping (partial signals)

Rewards are shaped at each stage and delivered in the reward field of step() observations:

• classify gives partial reward for correct intent classification.
• request_info (Task 2) rewards asking for the correct missing fields.
• finalize returns the final shaped reward (0.0–1.0) based on a deterministic rubric:
  • correctness of template selection
  • presence of required reply sections (acknowledgement + next step keywords)
  • correctness of Task 2 value references / Task 3 escalation + safety compliance

Episodes have a max step budget per task; exceeding it ends the episode with a penalty.

Tasks and Grading

Grading is deterministic and returns a score in [0.0, 1.0]:

• task_1 (easy): intent routing + one-shot policy reply
  • checks: intent category, reply template id, and whether reply contains acknowledgement + next-step keywords
• task_2 (medium): missing information collection over 2 turns
  • checks: intent category, correct missing-info questions, correct reply template id
  • also checks that the final reply references the simulated order_id and account email provided in thread_history
• task_3 (hard): high-risk case + escalation decision
  • uses a deterministic high-risk flag derived from episode_seed
  • checks: intent category, correct template id, correct escalate_to_human decision
  • also checks reply safety/compliance keywords (escalation language for high-risk vs self-serve security steps otherwise)

Evaluation Endpoints

• GET /tasks: lists tasks and the supported action/observation schema contract
• POST /baseline: runs the baseline agent across all 3 tasks and returns scores
• POST /grader: deterministically scores a provided episode transcript

Baseline Scores

Example run from the /baseline endpoint with the default seed 123:

• task_1 (easy): 0.50
• task_2 (medium): 1.00
• task_3 (hard): 1.00
• overall: 0.83

If OPENAI_API_KEY is set, /baseline will use the OpenAI-backed agent; otherwise it uses the deterministic heuristic policy to keep the environment runnable for development.

Advanced Usage

Connecting to an Existing Server

If you already have a Support Triage Env environment server running, you can connect directly:

from support_triage_env import SupportTriageEnv

# Connect to existing server
support_triage_envenv = SupportTriageEnv(base_url="<ENV_HTTP_URL_HERE>")

# Use as normal
result = support_triage_envenv.reset(seed=1, task_id="task_1")
result = support_triage_envenv.step(
    SupportTriageAction(
        task_id="task_1",
        task_stage="classify",
        intent_category="billing_refund",
        finalize=False,
    )
)
result = support_triage_envenv.step(
    SupportTriageAction(
        task_id="task_1",
        task_stage="finalize",
        reply_template_id="tpl_refund_ack_next",
        draft_reply="Thanks for reaching out. We received your request. Next step: please confirm details.",
        finalize=True,
    )
)

Note: When connecting to an existing server, support_triage_envenv.close() will NOT stop the server.

Using the Context Manager

The client supports context manager usage for automatic connection management:

from support_triage_env import SupportTriageAction, SupportTriageEnv

# Connect with context manager (auto-connects and closes)
with SupportTriageEnv(base_url="http://localhost:8000") as env:
    result = env.reset(seed=1, task_id="task_1")
    print("stage:", result.observation.task_stage)
    result = env.step(
        SupportTriageAction(
            task_id="task_1",
            task_stage="classify",
            intent_category="billing_refund",
            finalize=False,
        )
    )
    result = env.step(
        SupportTriageAction(
            task_id="task_1",
            task_stage="finalize",
            reply_template_id="tpl_refund_ack_next",
            draft_reply="Thanks for reaching out. We received your request. Next step: please confirm details.",
            finalize=True,
        )
    )
    print("done:", result.done, "reward:", result.reward)

The client uses WebSocket connections for:

• Lower latency: No HTTP connection overhead per request
• Persistent session: Server maintains your environment state
• Efficient for episodes: Better for many sequential steps

Concurrent WebSocket Sessions

The server supports multiple concurrent WebSocket connections. To enable this, modify server/app.py to use factory mode:

# In server/app.py - use factory mode for concurrent sessions
app = create_app(
    SupportTriageEnvironment,  # Pass class, not instance
    SupportTriageAction,
    SupportTriageObservation,
    max_concurrent_envs=4,  # Allow 4 concurrent sessions
)

Then multiple clients can connect simultaneously:

from support_triage_env import SupportTriageAction, SupportTriageEnv
from concurrent.futures import ThreadPoolExecutor

def run_episode(client_id: int):
    with SupportTriageEnv(base_url="http://localhost:8000") as env:
        result = env.reset(seed=1, task_id="task_1")
        result = env.step(
            SupportTriageAction(
                task_id="task_1",
                task_stage="classify",
                intent_category="billing_refund",
                finalize=False,
            )
        )
        result = env.step(
            SupportTriageAction(
                task_id="task_1",
                task_stage="finalize",
                reply_template_id="tpl_refund_ack_next",
                draft_reply="Thanks. We received your request. Next step: please confirm details.",
                finalize=True,
            )
        )
        return client_id, float(result.reward)

# Run 4 episodes concurrently
with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(run_episode, range(4)))

Development & Testing

Direct Environment Testing

Test the environment logic directly without starting the HTTP server:

# From the server directory
python3 server/support_triage_env_environment.py

This verifies that:

• Environment resets correctly
• Step executes actions properly
• State tracking works
• Rewards are calculated correctly

Running Locally

Run the server locally for development:

uvicorn server.app:app --reload

Project Structure

support_triage_env/
├── .dockerignore         # Docker build exclusions
├── __init__.py            # Module exports
├── README.md              # This file
├── openenv.yaml           # OpenEnv manifest
├── pyproject.toml         # Project metadata and dependencies
├── uv.lock                # Locked dependencies (generated)
├── client.py              # SupportTriageEnv client
├── models.py              # Action and Observation models
└── server/
    ├── __init__.py        # Server module exports
    ├── support_triage_env_environment.py  # Core environment logic
    ├── app.py             # FastAPI application (HTTP + WebSocket endpoints)
    └── Dockerfile         # Container image definition