[Refactor] make faster construct_scorecard method for xbooster/lgb_constructor.py

[RektPunk]

Description

This PR optimizes the construct_scorecard method by replacing the iterative tree-by-tree processing with a vectorized approach using Pandas and NumPy.

Previously, the code used a Python loop to iterate through each tree, performing individual grouping and concatenation. This created significant overhead, especially as the number of trees increased. The new implementation leverages melt() and a single groupby() operation to process all trees simultaneously.

Key Changes

• Eliminated Python Loops: Removed for i in range(n_trees) and replaced it with pd.DataFrame.melt().
• Efficient Data Tiling: Used np.tile() for high-performance expansion of labels across all tree iterations.
• Unified Aggregation: Consolidated the calculation of Events, NonEvents, and Count into a single, optimized groupby operation.

Performance Benchmark

Measured using %%timeit on the example ipynb:

Method	Execution Time (Mean ± Std. Dev.)	Speedup
Original (Iterative)	18.6 ms ± 277 μs	1.0x
Improved (Vectorized)	2.55 ms ± 132 μs	~7x faster

Summary by Sourcery

Optimize LightGBM scorecard construction by replacing per-tree looping with a vectorized aggregation over all trees and merging results directly on tree/node identifiers.

Enhancements:

• Vectorize scorecard binning table construction using a single melt/groupby over all trees instead of iterative per-tree processing.
• Simplify the merge between leaf weights and binning statistics by aligning on common Tree/Node keys and removing redundant columns.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Optimizes construct_scorecard in xbooster/lgb_constructor.py by replacing per-tree looping and concatenation with a single vectorized Pandas/NumPy aggregation and updating the merge keys accordingly.

File-Level Changes

Change	Details	Files
Replace per-tree loop-based binning table construction with a single vectorized melt+groupby aggregation over all trees.	Remove the for-loop over n_trees that built index_and_label and binning_table per tree and concatenated into df_binning_table. Create a long-form DataFrame from tree_leaf_idx using pd.DataFrame(...).melt(var_name='Tree', value_name='Node'). Broadcast labels across all tree rows using np.tile(labels.values, tree_leaf_idx.shape[1]). Group by ['Tree', 'Node'] and aggregate label via sum and count, then compute NonEvents and EventRate via assign and column arithmetic. Standardize binning_table schema to ['Tree', 'Node', 'Events', 'NonEvents', 'Count', 'EventRate']. Update the merge with df_x_add_evidence to use on=['Tree', 'Node'] and drop the now-unused legacy columns.	xbooster/lgb_constructor.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location> `xbooster/lgb_constructor.py:297-298` </location>
<code_context>
-            # Aggregate indices, leafs, and counts of events and non-events
-            df_binning_table = pd.concat([df_binning_table, binning_table], axis=0)
+        # Aggregate indices, leafs, and counts of events and non-events
+        tree_leaf_idx_long = pd.DataFrame(tree_leaf_idx).melt(var_name="Tree", value_name="Node")
+        tree_leaf_idx_long["label"] = np.tile(labels.values, tree_leaf_idx.shape[1])
+        binning_table = (
+            tree_leaf_idx_long.groupby(["Tree", "Node"])["label"]
</code_context>

<issue_to_address>
**issue (bug_risk):** This refactor changes label alignment semantics from index-based to purely positional, which may subtly alter behavior if `labels` has a non-trivial index.

The previous `pd.concat(..., axis=1)` + groupby relied on index-aware alignment between `labels` and `tree_leaf_idx`. Swapping to `labels.values` + `np.tile` assumes pure positional alignment and ignores the label index, so any non-default or filtered index on `labels` could yield silently incorrect event/non-event counts. If index alignment matters, either reset the index on `labels` before using `.values`, or construct the long DataFrame via a concat that preserves the index instead of manual tiling.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[xRiskLab]

Great work on these performance optimizations!

All your changes from PRs #10, #11, #13, and #14 have been merged into the release/v0.2.8rc1 release candidate branch. Your optimizations are now part of the upcoming v0.2.8 release.

What's included:

• PR [Refactor] make faster construct_scorecard method for xbooster/lgb_constructor.py #10: Faster construct_scorecard() for LightGBM
• PR [Refactor] make faster _convert_tree_to_points use np.vectorize instead of merge in xbooster/lgb_constructor.py #11: Faster _convert_tree_to_points() using vectorized lookup
• PR [Refactor] faster get_leafs method for xbooster/lgb_constructor.py #13: Faster get_leafs() method for LightGBM
• PR [Refactor] make xgb_construct fast  #14: Faster construct_scorecard() for XGBoost

Impact:

• Replaced loop-based pd.concat() operations with vectorized melt() + groupby()
• Replaced pd.merge() in loops with dictionary mapping using .map()
• Significant performance improvements for models with many trees
• All changes maintain backward compatibility and numerical equivalence

The RC branch (release/v0.2.8rc1) combines your performance improvements with the new SHAP integration features. Once testing is complete, this will be released as v0.2.8.

Thank you for these excellent contributions!