I want to have a pool of workers completing multiple jobs with rate-limiting / concurrency, but because there are a lot of jobs, and each job description can take up a big chunk of memory, I would like to only create the job "on-the-fly" as soon as a worker slot is ready. Is that easy to do with p-queue ?

For instance a "solution" like that Promise.all(myHugeListOfJobs) would not work because it would require creating all the tasks at once and fill up the memory.

For now I use async-sema with a semaphore, and a util like this:

import { Sema } from "async-sema";

export const runTasksWithConcurrency = async (
  taskGenerator: AsyncGenerator<() => Promise<any>, void, unknown>,
  concurrency: number
): Promise<PromiseSettledResult<any>[]> => {
  const semaphore = new Sema(concurrency);
  const tasks: Promise<any>[] = [];

  for await (const task of taskGenerator) {
    await semaphore.acquire();
    tasks.push(
      (async () => {
        try {
          return await task();
        } finally {
          semaphore.release();
        }
      })()
    );
  }

  return Promise.allSettled(tasks);
};

that I use like this

  async function* taskGenerator() {
    for (const data of myHugeListOfItemsGenerator) {
      yield async () => {
          await doSomeWork();
      };
    }
  }

  const runResults = await runTasksWithConcurrency(taskGenerator(), 50);