# Troubleshooting Runbook — `@coda/sandbox`

This runbook covers build failures, test failures, runtime errors, and operational issues for the `@coda/sandbox` package.

---

## Build Failures

### `isolated-vm` fails to build (C++ compilation error)

**Symptom:** `pnpm install` fails with compiler errors mentioning `concept`, `requires`, or other C++20 syntax. The error appears in the `isolated-vm` gyp build step.

**Cause:** isolated-vm v5 uses C++20 `concept` syntax that is not supported by older compilers, including Apple Clang versions below 15 (shipped with Xcode < 15).

**Fix:**

1. Verify that `isolated-vm` is at v6 or later in `sandbox/package.json` (`^6.1.2` or higher). isolated-vm v6 was rewritten to support the broader compiler range required for Node 24.
2. Run `pnpm install` again after correcting the version.
3. On macOS, ensure Xcode Command Line Tools are up to date: `xcode-select --install`

**Verify the addon loads correctly:**

```bash
# ESM check (matches project's module type)
node --input-type=module -e "import ivm from 'isolated-vm'; console.log('ok')"
```

### TypeScript compile errors after upgrading isolated-vm

**Symptom:** `Property 'Isolate' does not exist on type ...` or similar type errors referencing the `isolated-vm` module.

**Cause:** isolated-vm v5 and v6 have different TypeScript type shapes. In v5, types were provided by a separate `@types/isolated-vm` package. In v6, types are bundled inside the package itself. Having both installed causes conflicts.

**Fix:**

1. Remove any `@types/isolated-vm` entry from `devDependencies` in `sandbox/package.json`.
2. Ensure `isolated-vm` is at v6+.
3. Run `pnpm install` and `pnpm typecheck` to verify.

---

## Test Failures

### `SyntaxError: Cannot use import statement in a CommonJS module`

**Cause:** Jest is being run without the `--experimental-vm-modules` Node.js flag. Without this flag, Jest's module system treats `.ts` files as CommonJS and fails on the first `import` statement.

**Fix:** Always invoke Jest via:

```bash
node --experimental-vm-modules node_modules/jest/bin/jest.js
```

Do not use `npx jest`, `./node_modules/.bin/jest`, or plain `jest` — these do not pass the required Node.js flag.

The `pnpm test:unit` script handles this automatically.

### `Error: Jest encountered an unexpected token`

**Cause:** `extensionsToTreatAsEsm` is not set, or `useESM: true` is missing from the ts-jest transform configuration. Jest is attempting to parse TypeScript ESM as CommonJS.

**Fix:** Verify `sandbox/jest.config.json` contains:

```json
{
  "extensionsToTreatAsEsm": [".ts"],
  "transform": {
    "^.+\\.ts$": ["ts-jest", { "useESM": true }]
  }
}
```

Both settings are required. Either one alone is insufficient.

### `Cannot find module './foo.js'` in tests

**Cause:** TypeScript source files use `.ts` extensions but ESM convention requires `.js` extensions in import statements. During testing, ts-jest resolves `.ts` files but the `.js` extension does not match. The `moduleNameMapper` must strip the `.js` extension so ts-jest can find the source.

**Fix:** Verify `sandbox/jest.config.json` includes:

```json
{
  "moduleNameMapper": {
    "^(\\.{1,2}/.*)\\.js$": "$1"
  }
}
```

### Isolate tests hang (never complete)

**Cause:** A test creates an isolate or engine but does not dispose it. The Node.js process stays alive waiting for the native addon to release its handles.

**Fix:** Ensure every integration test has teardown:

```typescript
afterAll(async () => {
  await engine.shutdown();
  // or: await isolateManager.drain();
});
```

Check for missing `finally` blocks in test helpers that create isolates.

### `Error: IsolateManager at capacity`

**Cause:** A test creates many isolates in rapid succession without releasing or disposing them. The manager hits `maxConcurrentIsolates` and rejects new requests.

**Fix:** Call `release(handle)` in test teardown (or in a `finally` block). Ensure no test leaks a handle. If tests run in parallel, reduce `maxConcurrentIsolates` per test or run them serially.

---

## Runtime Errors

### `SandboxTimeoutError` on code that should complete quickly

**Check 1:** Is the user code using `await dataProxy.request()`? The timeout clock runs from the start of execution and includes all wait time for host data responses. If the host handler is slow, the execution times out even if the LLM code itself is fast. Increase `timeoutMs` or optimize the host data handler.

**Check 2:** Is the warm pool empty? When all pre-warmed isolates are in active use, new isolate creation adds ~50ms overhead that counts against the timeout. Verify `warmPoolSize > 0` in engine config.

### `SandboxOOMError` on small scripts

**Check 1:** Is the code loading DuckDB-WASM? DuckDB uses significant heap during initialization — often 100–300 MB. Increase `memoryMb` to 512 for executions that use DuckDB.

**Check 2:** Is the result large? Even if `maxResultBytes` will ultimately reject the result, the full object must be allocated in the isolate heap and serialized to JSON before the size check runs. A large result allocation can trigger OOM before the size limit is enforced.

### `SandboxValidationError: Code size exceeds limit`

**Cause:** The LLM generated code larger than `maxCodeBytes` (default 512 KB).

**Fix:** Increase `maxCodeBytes` in `isolateLimits` configuration, or add a system prompt instruction telling the LLM to write shorter, more concise code.

### `SandboxValidationError: Result size ... exceeds limit`

**Cause:** The execution result serialized to more than `maxResultBytes` (default 5 MB).

**Fix:** Prompt the LLM to return summarized, paginated, or aggregated data rather than raw rows. Alternatively, increase `maxResultBytes` in `bridgeLimits` if the larger result is genuinely needed.

### `SandboxAbortedError` unexpectedly

**Cause:** `session.abort()` was called before the execution completed. Common causes:

- A race condition in the calling code where a timeout on the caller's side fires before the sandbox timeout does, triggering an abort.
- An HTTP request handler that aborts the session when the client disconnects, but the client disconnected prematurely (e.g., load balancer timeout).

**Check:** Trace where `abort()` is called in the agent layer. Verify that caller-side timeouts are longer than `timeoutMs` in the sandbox config.

### `dataProxy` not defined in user code

**Cause:** `ContextBuilder.build()` failed silently, or the bootstrap script injected into the isolate context has a syntax error introduced by a recent change.

**Fix:**

1. Check engine startup logs for context build errors.
2. Verify `src/execution/isolate/context-builder.ts` compiles without errors: `pnpm typecheck`.
3. Run the context builder unit tests to confirm the bootstrap script is injected correctly: `node --experimental-vm-modules node_modules/jest/bin/jest.js src/execution/isolate/context-builder.test.ts`

---

## Operational Issues

### High memory usage

**Check `activeCount`** on IsolateManager. Each active isolate can consume up to `memoryMb` MB of heap. With `maxConcurrentIsolates: 10` and `memoryMb: 256`, peak usage can reach ~2.5 GB.

**Check warm pool:** `warmPoolSize × memoryMb` is the baseline memory consumed at idle, even with no active executions.

**Fix options:**

- Reduce `warmPoolSize` if idle memory is the concern (trade: slower first execution after idle period)
- Reduce `maxConcurrentIsolates` if peak memory is the concern (trade: more request queuing under load)
- Reduce `memoryMb` if executions don't genuinely need 256 MB (trade: more `SandboxOOMError` risk for data-heavy executions)

### Slow execution times

**Check `wallMs` vs `cpuMs`** in the `ExecutionStats` on the result event:

- High `wallMs` with low `cpuMs` means time is spent waiting on data requests, not in computation. Optimize the host data handler or reduce the number of `dataProxy.request()` calls.
- High `wallMs` matching high `cpuMs` means the computation itself is expensive. Consider whether the LLM code can be simplified.

**Check WASM initialization:** The first execution that uses a WASM module (e.g., DuckDB) initializes the module inside the isolate, which takes 100–500ms. Subsequent executions from the warm pool may or may not have pre-initialized WASM depending on configuration. If DuckDB init time dominates, it will appear as high `wallMs` on the first execution after pool creation.

### Pool always empty (stats show high create time on every execution)

**Cause:** `warmPoolSize` is 0 or not large enough for the concurrency level. The warm pool is consumed immediately and subsequent requests create fresh isolates.

**Fix:** Increase `warmPoolSize` to match expected peak concurrency. Pool isolates are used in order — if all are consumed before executions complete, cold starts occur for the overflow.

---

## Debugging Tips

- **Verbose error logging:** All `SandboxError` subclasses include `.stack`. Log the full stack trace, not just `.message`, to locate which internal component threw.

- **Console output from user code:** `console.log()` inside user code is captured and emitted as a status event with a `[log]` prefix. Check the session's status event stream to see what the user code printed.

- **Structured debug output:** User code can call `reportStatus('debug', message)` to emit a structured status event. This is more reliable than `console.log` for structured data.

- **Execution stats:** Every successful result includes `ExecutionStats` with `wallMs`, `cpuMs`, and `heapMb`. These tell you exactly where time and memory went during execution. Log these for every production execution to build a baseline.

- **Isolate manager state:** Log `activeCount` and `warmCount` from IsolateManager on each execution to identify pool exhaustion or memory pressure before they become errors.