Add custom prefix/adapter support for HuggingFace models and NDCG@k metric#2
Conversation
|
|
||
| class NDCG(Metric): | ||
| """ | ||
| Computes NDCG@k over all queries. |
There was a problem hiding this comment.
| Computes NDCG@k over all queries. | |
| Computes NDCG@k (Normalized Discounted Cumulative Gain) over all queries. |
chanberg
left a comment
chanberg
left a comment
There was a problem hiding this comment.
Hi @Andrian0s,
Thank you so much for these cool contributions! It's great to hear that this tool is useful for your research.
Before merging this, I would like to make the changes to the model class a bit more generalized for future model support. Are you ok with me pushing changes onto your branch or should I leave more detailed comments on the changes I envision?
Cheers,
Chantal
| super().__init__(name) | ||
| self.k = self._parse_k(name) | ||
|
|
||
| def _parse_k(self, name: str) -> int: |
There was a problem hiding this comment.
I suggest moving this method to the ABC class now that it will be used in multiple metrics.
|
|
||
| # Hardcoded Jina adapter configurations. | ||
| # Each maps to (query_task, query_prompt, doc_task, doc_prompt). | ||
| JINA_CONFIGS = { |
There was a problem hiding this comment.
I would prefer no hardcoded model settings. The prefixes and task options should be generalized and configurable in the yaml, so it's easier to support more models.
Hey Chantal, Of course, feel free to make changes directly to the branch before merging. If you want, once you are through with the changes, I can go ahead and test the branch as well before merging. Andrianos |
2 participants
Hi Chantal (or who is responsible for this repo)
I found this repo really useful and I plan to use it for my own research. I did have some slightly different needs, so in case someone else also has them too, I put them here as an issue with a PR that implements these changes.
Closes #1
All tests pass successfully (how I ran):
pip3 install uv
uv venv
source .venv/bin/activate
uv sync
uv run pytest
Note: I would also suggest another check from the original repo creators side to be sure this doesn't crash anything on their end.
List of changes:
semsearcheval/models.py
HuggingFaceModel.init accepts set_custom_query_prefix (str), set_custom_doc_prefix (str), and use_jina_adapters (string).
Validation rejects combining use_query_prompt/use_passage_prompt with custom prefixes.
compute_similarity routes through three paths: custom prefix → adapter switching → default prompt_name.
Added _prepend_prefix helper method.
semsearcheval/metrics.py
Added NDCG class with _parse_k and compute methods.
semsearcheval/constants.py
Imported NDCG, added "ndcg" key to metric_registry.
configs/example.yaml
Added ndcg@10 to metrics list.
Added commented-out examples for custom prefix and adapter config.
tests/test_metrics.py
Parametrized tests for NDCG (perfect ranking, partial ranking, out-of-top-k).
tests/test_models.py
test_huggingface_mutual_exclusivity — validates error on conflicting config.
test_huggingface_custom_prefix_init — checks prefix storage.
test_huggingface_adapters_init — checks adapter flag storage.
tests/test_evaluation.py and tests/test_config.yaml
Updated expected metrics list and count.