Skip to content

fix: reduce memory overhead when loading templates - #1246

Open
Aias00 wants to merge 2 commits into
apache:mainfrom
Aias00:perf/template-stream-buffering-185
Open

Aias00 wants to merge 2 commits into
apache:mainfrom
Aias00:perf/template-stream-buffering-185

Conversation

@Aias00

@Aias00 Aias00 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Purpose of the pull request

Related: #185

Reduce the extra heap required by withTemplate(...).build() without changing template fill behavior.
Previously Fesod copied an entire template into a byte array and retained it until writer finish,
alongside POI's parsed workbook. InputStream templates additionally incurred a growing
ByteArrayOutputStream buffer and its final array copy.

This addresses the redundant template-copy overhead, not fully streaming template filling.
POI's parsed template workbook still resides in memory, including with inMemory(false).

What's changed?

  • Read file and stream templates through BufferedInputStream instead of materializing their bytes.
  • Close the prepared stream during workbook initialization, including parser and callback failures.
    Preserve autoCloseStream(false) for caller-owned streams using the existing commons-io dependency.
  • Keep the closed template stream non-null because existing fill and append paths also use it as
    a template-present flag.
  • Clean up an initialized workbook and owned output stream if construction fails. This preserves
    output cleanup when template I/O failures move from eager copying to POI initialization;
    cleanup failures are suppressed rather than replacing the original failure.
  • Add coverage for XLS/XLSX, file/stream templates, both in-memory modes, repeated fills with
    row shifting, append writes, source ownership, parser/I/O/callback failures and suppressed errors.
  • Document the remaining template-memory cost in Javadocs and the English/Chinese fill guides.

Allocation check

A local JDK 17 probe measured allocations on the current thread during WriteWorkbookHolder
construction, using a 32 MiB generated stream or a 32 MiB file. It warmed up five times and took
the minimum of three measured runs. The generated source does not allocate a 32 MiB input array.
Baseline classes were built from 0045070c1556d824e45851327e719867ec89a9d6.

Template source Before (bytes allocated) After (bytes allocated)
File 33,559,968 13,720
InputStream 100,668,808 13,400

These numbers measure template preparation only, not total workbook heap or POI parsing.

Validation

  • The new 32 MiB regression test failed before the change: template preparation eagerly read
    33,554,432 bytes instead of zero; it passes after the change.
  • The I/O failure regression also failed before initialization cleanup was added: the owned output
    stream was not closed; it passes with the cleanup in place.
  • ./mvnw -B -pl fesod-common,fesod-shaded,fesod-sheet -Dtest.excludedGroups=fuzz package:
    1,795 tests passed (60 common + 1,735 sheet), with no failures or skips.
  • Java 8: 35 targeted tests passed across TemplateStreamBufferingTest, WorkBookUtilTest and
    WriteWorkbookHolderOutputStreamTest.
  • ./mvnw -B -pl fesod-common,fesod-shaded,fesod-sheet spotless:check
  • ./mvnw -B -pl fesod-common,fesod-shaded,fesod-sheet -Dmaven.javadoc.skip=false javadoc:javadoc

Checklist

  • I have read the Contributor Guide.
  • I have written the necessary doc or comment.
  • I have added the necessary unit tests and all cases have passed.

Assertions.assertEquals(-1, source.read());
}
}
assertFilled(streamOutput);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Aias00, thanks for working on this.

I reproduced one remaining issue on Windows. With a file output and autoCloseStream(false), the caller-owned template stream stays open as intended, but the FileOutputStream created internally by Fesod also stays open. As a result, the generated XLS and XLSX files remain locked after finish().

Could you separate template-input ownership from internally created output ownership, and add a regression assertion that the generated file can be deleted or moved after finish()? Once this is covered, I think this will be ready to merge. Thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants