Cost estimate for running this benchmark?

[Unturned3]

Hi authors, thank you for this amazing piece of work!

Do you have a rough cost estimate (e.g. API costs) for running all the evals on this benchmark?

[HUST-AI-HYZ]

Hi,

Thanks for you concern on our work. Currently we do not have a detailed number of how much API costs for running all the evals. Sorry for the inconvenience brought by this.

But I think the mainly cost is used on Long Context Agents and the Book Summary task, since they need to memorize the context for every time with a single question. While for other agents and tasks, it would be more API efficient because you only need to construct the memory for once for answering multiple questions. What is more, As most computational time is used on memory construction, the total computational time would be smaller.

I think adding the API cost statistics is a good advice, and I would do this in the next version of our paper.

Best,
Yuanzhe