feat: add raw_rows option for high-performance worksheet generation

[mbaertschi]

Bypass the Cell protocol and XmlStream when raw_rows: true is passed to Worksheet.new/4, generating SpreadsheetML XML directly as iodata. Rows are processed in parallel batches via Task.async_stream, yielding ~4x speedup for large exports.

[sax]

Thanks for the contribution! I have a few thoughts:

• This is a lot of special case function heads to add to Worksheet. I don't think I should put a major refactor to a renderer behaviour on you... but maybe they could be pulled out into another module so that I could do so more easily.
• :raw_rows as another struct key... we have :opts that could handle this, and be more extensible for future options. Maybe just :raw?
• Guarding on is_list seems fragile. If there's a way to use :opts, it would help against future regressions. I can clean this one up later, though.

Do you have benchmarks? I have personally tried and failed to improve the performance of xml_stream, so it would be nice to have a benchee example to try again with.

[mbaertschi]

Hei @sax

Thanks for the feedback. Unfortunately I am under heavy pressure right now (for the project for which I created this PR :-)) I wont have time to work on this for like at least two weeks. In our case, this feature reduced the export time by the amount of available processes. For my local MacBook Pro from 40s to 8s, on k8s cluster from like 80s to 25s.

The main benefit is parallel processing where number of concurrent processes is System.schedulers_online()

[mbaertschi]

@sax any plans for this?

[sax]

I'm finally back to updating our open source, so I'm looking at this this week. I want to try a few things pulled from your changes first, to see if I can get the main path as fast as your version while preserving the functionality.

[sax]

Hey @mbaertschi, before I pull in your raw module, can you try out the changes I've pushed to main (and just released in v0.7.3)? I'm taking the core of what you did, but continuing to use xml_stream and allowing unknown data structures to fall through to the Cell protocol.

I also have this PR open in xml_stream: activesphere/xml_stream#5, which ekes out more throughput by combining two stream operations into one. I'm using the changes to binary pattern matching in that PR as overrides in exceed, so the main benefit of the PR is from the stream changes (if/when it merges, I'll remove some code from exceed).

With everything combined, I'm seeing up to a 4.5x improvement in throughput for simple data types, with some slowdown for large binaries.

In the benchmarks (exceed v0.7.3, with the xml_stream PR) on my M2, I see integers going from ~32k rows/sec to ~150k rows/sec. With all 10-20 character binaries, I see it go from ~19k/sec to ~55k/sec.

When I change it to use Task.async_stream, I'm seeing ~10% - 15% worse throughput locally (worse the higher the batch size). It would be interesting to see if it's different for you with real workloads. I could release a version where concurrent processing could be opted into with either through opts or via compile-time configuration.

[mbaertschi]

Hei @sax so here are the numbers:

My PR
xlsx export: total=11213ms (raw query)
xlsx export: total=11413ms (raw query)

Exceed 0.7.3
xlsx export: total=12880ms (raw query)
xlsx export: total=12784ms (raw query)

Exceed 0.7.3 with xml-stream PR
xlsx export: total=10047ms (raw query)
xlsx export: total=10435ms (raw query)

For me the v0.7.3 solution is already good enough, I'll close my PR. Thanks for your work!