Borgstore S3 performance

[tve]

Has anyone done some performance measurements / observations when using borg 2.0.0 b20 with an S3 backend? (Not necessarily AWS)

I'm starting to try Backblaze and one issue I've hit is that it has a quota for "class B" operations, which include HeadObject. Apparently, when performing a backup borg performs a lot of HeadObject operations.

According to the docs: https://www.backblaze.com/cloud-storage/transaction-pricing

Transactions Class B
Cost: The first 2,500 of these calls are free each day, then $0.004 per 10,000

Are the HeadObject calls expected?

In terms of performance, I'm wondering how much concurrency around S3 operations there is...

[ThomasWaldmann]

See #8431 and #8572.

borg2 currently stores each chunk separately. That's very simple to deal with (especially for "compacting"), but also does a lot of api calls and can be slow.

Implementing packs are one of the bigger changes still needing to be done before borg2 release.

There is no concurrency. But packs need to be done first and then we'll see whether we need concurrency.

[tve]

When I last used AWS S3 the single stream upload bandwidth wasn't anything to write home about. Maybe that has changed. Or maybe using multi-part uploads make it better and only require very localized concurrency, but at least my (now somewhat dated) experience says that concurrency is needed.

[tve]

[I didn't see TW's response above before writing this.]

I've been running a backup of a ~8GB directory structure for a while and after ~200 minutes borg managed to push around 5GB to backblaze. This is over a residential fiber with 400-500MB/s fiber and backblaze is ~18ms away. I need to back up several hundred GB so this isn't going to work. I assume there just isn't enough/any concurrency.

[Please take this as beta feedback, I know v2 isn't ready for prime time yet. ... but I'm curious what the plans are to deal with the tiny objects / lack of concurrency problem.]

[ThomasWaldmann]

#8572

[tve]

Yup, note that the most recent comment is mine... ;-)

May I ask where working on this problem set is in your calendar?

[ThomasWaldmann]

I have already started on this a while ago. But didn't continue working on it recently.

Will need a lot of work to finish, so can't say a date.

[tve]

Just as points of comparison:

• borgv2 file backend to local SSD: 1 minute
• borgv2 sftp backend to local rpi4 with USB3 SSD: 17 minutes
• borgv2 file backend onto ZeroFS to Hetzner object store (S3): 16 minutes