← Back to all posts

Controlling API Floods in a Large-Scale State Restoration

January 20, 2025

Our OrgChart visualization is the heart of our application—enabling managers to explore complex reporting hierarchies at a glance. Until recently, expanding around 1 000 roles in a single view pushed the limits of our D3-based SVG renderer. After a deep-dive performance overhaul, users can now effortlessly open 100 000 roles on screen. Expansion remains user-driven—click a node one by one, or expand two or three levels at once.

But there was a catch: when a user hits Refresh, our client-side “state restoration” logic dutifully replays the entire expansion tree in one go, firing off thousands of API calls in parallel. The result? A sudden stampede of fetch requests, backend rate-limit errors, timeouts, and a frozen UI.

The Challenge: Uncontrolled Fetch Storms

  1. State Restoration at Scale On page load, we reconstruct the last-known expansion state—potentially hundreds of sub-trees deep. Each expanded node triggers its own HTTP request to fetch child data.

  2. Burst Concurrency A heavy user’s previous session might have spanned 5+ levels across multiple branches. In that scenario, a refresh instantly kicks off thousands of promises.

  3. Backend Overload Our REST API sees sudden spikes:

    • HTTP 429 (“Too Many Requests”) as rate-limiters trip.
    • HTTP 500 when downstream services overload.

  4. Degraded UX

    • Endless spinners as the browser juggles thousands of in-flight promises.
    • Partial rendering and error dialogs wherever calls fail.
    • User confidence in the OrgChart takes a hit.

We needed a way to throttle those refresh-time requests—batching them into manageable chunks—so the UI could restore state without drowning our API.

The Solution: runWithConcurrencyLimit

We built a small utility, runWithConcurrencyLimit, that:

  • Takes an array of “tasks” (functions returning Promise<T>).
  • Ensures no more than N tasks run in parallel.
  • Queues the rest and kicks off the next task as soon as one finishes.
  • Collects all results (or errors) in the original order.

Here’s the implementation:

type Task<T> = () => Promise<T>;

export async function runWithConcurrencyLimit<T>(
  tasks: Task<T>[],
  concurrency = 6
): Promise<T[]> {
  const responses: (T | Error)[] = new Array(tasks.length);
  const executing = new Set<Promise<any>>();

  for (let i = 0; i < tasks.length; i++) {
    const promise = tasks[i]()
      .then(response => void (responses[i] = response))
      .catch(error => void (responses[i] = error))
      .finally(() => executing.delete(promise));

    executing.add(promise);

    // Pause launching new tasks if we've reached the limit
    if (executing.size >= concurrency) {
      await Promise.race(executing);
    }
  }

  // Wait for any remaining tasks
  await Promise.all(executing);
  return responses as T[];
}

How It Works

  1. Initialize a responses array to store each task’s outcome by index.

  2. Maintain an executing set of in-flight promises.

  3. Loop through all tasks:

    • Invoke the task, attach .then/.catch to capture results, and .finally to remove it from executing.
    • Add the resulting promise to executing.

  4. Throttle: whenever executing.size ≥ concurrency, await Promise.race(executing) waits for the fastest promise to settle before proceeding.

  5. Flush: after enqueuing all tasks, await Promise.all(executing) ensures every task completes.

  6. Return the ordered responses array.

This pattern keeps at most concurrency requests in flight, spacing out the load on our API while still eventually running every task.

Future Improvements

  • Async Iterators for Input Accept an AsyncIterable<Task<T>> so tasks (e.g., from a stream or cursor) can feed in dynamically.

  • Async Generators for Output Return an AsyncGenerator<T> that yields results as they arrive—perfect for progressive UI rendering.

  • Backoff & Retry Strategies Integrate exponential backoff (with jitter) and retry caps for transient failures (such as 429s).

  • Cancellation Support Pass an AbortSignal into each task to let users cancel long-running or stale state restorations.

  • Dynamic Concurrency Monitor API responses (e.g., 429s) or measure client network throughput and adjust concurrency on the fly.

  • Telemetry & Metrics Emit events on task start, completion, and errors. Track latency distributions to inform tuning and detect regressions.

Conclusion

By batching our OrgChart’s state-restoration fetches into controlled concurrent groups, we tame the refresh-time storm of API calls, dramatically reduce error rates, and deliver a smoother, faster user experience—even at 100 000 expanded roles. The runWithConcurrencyLimit utility is lightweight, reusable, and ready to evolve with iterators, backoff strategies, cancellation, and dynamic tuning as your application grows. Happy coding!