From 0cd5889620bf52be14f6836edea865563b0b4a1b Mon Sep 17 00:00:00 2001 From: youseihuayu-wonderful Date: Wed, 9 Sep 2026 23:45:33 -0400 Subject: [PATCH 1/2] fix: search files by directory path Refs IQSS/dataverse-frontend#1062 --- conf/solr/schema.xml | 2 + .../source/admin/solr-search-index.rst | 20 ++++ doc/sphinx-guides/source/api/changelog.rst | 1 + doc/sphinx-guides/source/api/native-api.rst | 6 +- doc/sphinx-guides/source/api/search.rst | 2 + .../source/user/find-use-data.rst | 4 +- .../DatasetVersionFilesServiceBean.java | 7 +- .../iq/dataverse/search/IndexServiceBean.java | 4 + .../iq/dataverse/search/SearchFields.java | 1 + .../harvard/iq/dataverse/api/DatasetsIT.java | 99 +++++++++++++++++ .../harvard/iq/dataverse/api/SearchIT.java | 105 ++++++++++++++++++ 11 files changed, 244 insertions(+), 7 deletions(-) diff --git a/conf/solr/schema.xml b/conf/solr/schema.xml index 4d65b378485..ec0b2a32488 100644 --- a/conf/solr/schema.xml +++ b/conf/solr/schema.xml @@ -179,6 +179,7 @@ + @@ -489,6 +490,7 @@ + diff --git a/doc/sphinx-guides/source/admin/solr-search-index.rst b/doc/sphinx-guides/source/admin/solr-search-index.rst index 27eb7fab6f4..623df1e8a98 100644 --- a/doc/sphinx-guides/source/admin/solr-search-index.rst +++ b/doc/sphinx-guides/source/admin/solr-search-index.rst @@ -6,6 +6,26 @@ A Dataverse installation requires Solr to be operational at all times. If you st .. contents:: Contents: :local: +.. _directory-name-search-index: + +Directory Name Search +--------------------- + +File directory paths are indexed in ``fileDirectoryLabel`` and copied into the ``_text_`` field used by basic search. When upgrading an installation that does not yet index directory paths, add the following definitions from ``conf/solr/schema.xml`` to the active core's ``schema.xml``, preserving any local metadata customizations: + +.. code-block:: xml + + + + +Reload the Solr core before deploying the application code that indexes directory paths. For the default core name: + +.. code-block:: bash + + curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1" + +After deploying the application, follow `Reindex in Place`_ below to index directory paths for existing files without clearing the search index. Reloading the schema alone does not add directory terms to existing documents; those files become searchable by directory name as they are reindexed. No PostgreSQL schema migration is required for this change. + Full Reindex ------------- diff --git a/doc/sphinx-guides/source/api/changelog.rst b/doc/sphinx-guides/source/api/changelog.rst index dc89c5db7e1..5feb372ae61 100644 --- a/doc/sphinx-guides/source/api/changelog.rst +++ b/doc/sphinx-guides/source/api/changelog.rst @@ -28,6 +28,7 @@ v6.12 - Dataset creation API calls may now behave differently when neither a license nor terms are provided, depending on the new :ref:`dataverse.feature.do-not-assume-default-license` feature flag. - Whether file extensions are included in the "Content-disposition" header returned when downloading auxiliary files depends on whether the relevant format is one of the content types supported in Tika. A recent update to the version of Tika has added new content types, including "text/markdown", and auxiliary files with these types now have a a file extension included (e.g. ".md" in this case) in the header. +- File text searches now also match directory paths, which can increase result counts and filtered download sizes. This affects ``searchText`` on dataset version ``files``, ``files/counts``, and ``downloadsize`` endpoints, as well as file results in the Search API. Existing installations must update their Solr schema and reindex existing files for Search API directory matches; see :ref:`directory-name-search-index`. Related request: `dataverse-frontend#1062 `_. v6.11 ----- diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index 1806772afbd..3f37d96ccc4 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -2458,7 +2458,7 @@ Usage example: curl "https://demo.dataverse.org/api/datasets/24/versions/1.0/files?contentType=image/png" -Filtering by search text is also optionally supported. The search will be applied to the labels and descriptions of the dataset files, to return the files that contain the text searched in one of such fields. +Filtering by search text is also optionally supported. The search matches case-insensitive substrings in the labels, descriptions, and directory paths (``directoryLabel``) of the dataset files. Files matching any of these fields are returned, subject to the other requested filters. Usage example: @@ -2562,7 +2562,7 @@ Usage example: curl "https://demo.dataverse.org/api/datasets/24/versions/1.0/files/counts?contentType=image/png" -Filtering by search text is also optionally supported. The search will be applied to the labels and descriptions of the dataset files, to return counts only for files that contain the text searched in one of such fields. +Filtering by search text is also optionally supported. The search matches case-insensitive substrings in the labels, descriptions, and directory paths (``directoryLabel``) of the dataset files. Only files matching the search and the other requested filters are counted. Usage example: @@ -3621,7 +3621,7 @@ Usage example: curl "https://demo.dataverse.org/api/datasets/24/versions/1.0/downloadsize?contentType=image/png" -Filtering by search text is also optionally supported. The search will be applied to the labels and descriptions of the dataset files, to return the size of all files available for download that contain the text searched in one of such fields. +Filtering by search text is also optionally supported. The search matches case-insensitive substrings in the labels, descriptions, and directory paths (``directoryLabel``) of the dataset files. The returned size includes only files available for download that match the search and the other requested filters. Usage example: diff --git a/doc/sphinx-guides/source/api/search.rst b/doc/sphinx-guides/source/api/search.rst index eea125b76b7..86e0a8cc3ea 100755 --- a/doc/sphinx-guides/source/api/search.rst +++ b/doc/sphinx-guides/source/api/search.rst @@ -8,6 +8,8 @@ The Search API supports the same searching, sorting, and faceting operations as To search unpublished content, you must pass in an API token as described in the :doc:`auth` section. +File searches include names within directory paths. For example, ``q=Figure1&type=file`` can return files in both ``Figure1`` and ``results/Figure1``, even when their filenames and descriptions do not contain the query term. Directory names follow the existing search tokenization and case-insensitive matching rules; the results remain files, not folders. + The parameters and JSON response are partly inspired by the `GitHub Search API `_. .. note:: |CORS| The search API can be used from scripts running in web browsers, as it allows cross-origin resource sharing (CORS). diff --git a/doc/sphinx-guides/source/user/find-use-data.rst b/doc/sphinx-guides/source/user/find-use-data.rst index 8e65108d680..09f46ec6d4d 100755 --- a/doc/sphinx-guides/source/user/find-use-data.rst +++ b/doc/sphinx-guides/source/user/find-use-data.rst @@ -16,6 +16,8 @@ Basic Search ------------ You can search the entire contents of the Dataverse installation, including Dataverse collections, datasets, and files. You can access the search by clicking the "Search" button in the header of every page. The search bar accepts search terms, queries, or exact phrases (in quotations). +File results can also match names in their directory paths. For example, searching for ``Figure1`` can find both ``Figure1/plot.txt`` and ``results/Figure1/nested.txt``, even when the filenames and descriptions do not contain ``Figure1``. The results are files, not separate folder entries. + Sorting and Viewing Search Results ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -78,7 +80,7 @@ Files in a Dataverse installation each have their own landing page that can be r File Search within Datasets --------------------------- -Datasets containing multiple files offer a file search function. On the Dataset page, under the Files tab, you'll see a search bar you can use to locate an individual file. It searches within the filename and file description. Performing a search will filter the file table to list only files matching your search. After you perform a search, if you'd like to return to the full list of files, just perform an empty search. +Datasets containing multiple files offer a file search function. On the Dataset page, under the Files tab, you'll see a search bar you can use to locate an individual file. It searches within the filename, file description, and directory path. Performing a search will filter the file table to list only files matching your search. After you perform a search, if you'd like to return to the full list of files, just perform an empty search. Under the search bar, you'll see file search facets you can use to filter the dataset's files by file type, access level, and file tags (see the example below). diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetVersionFilesServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetVersionFilesServiceBean.java index 74c3e833753..27d074d7f21 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetVersionFilesServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetVersionFilesServiceBean.java @@ -358,9 +358,10 @@ private Predicate createSearchCriteriaPredicate(DatasetVersion datasetVersion, } String searchText = searchCriteria.getSearchText(); if (searchText != null && !searchText.isEmpty()) { - searchText = searchText.trim().toLowerCase(); - predicates.add(criteriaBuilder.or(criteriaBuilder.like(criteriaBuilder.lower(fileMetadataRoot.get("label")), "%" + searchText + "%"), - criteriaBuilder.like(criteriaBuilder.lower(fileMetadataRoot.get("description")), "%" + searchText + "%"))); + String searchPattern = "%" + searchText.trim().toLowerCase() + "%"; + predicates.add(criteriaBuilder.or(criteriaBuilder.like(criteriaBuilder.lower(fileMetadataRoot.get("label")), searchPattern), + criteriaBuilder.like(criteriaBuilder.lower(fileMetadataRoot.get("description")), searchPattern), + criteriaBuilder.like(criteriaBuilder.lower(fileMetadataRoot.get("directoryLabel")), searchPattern))); } return criteriaBuilder.and(predicates.toArray(new Predicate[]{})); } diff --git a/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java index 8132c5e113d..b837a56fe54 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java @@ -1633,6 +1633,10 @@ public SolrInputDocuments toSolrDocs(IndexableDataset indexableDataset, Set Date: Thu, 10 Sep 2026 16:49:09 -0400 Subject: [PATCH 2/2] docs: add release note for directory search --- .../frontend-1062-directory-search.md | 34 +++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 doc/release-notes/frontend-1062-directory-search.md diff --git a/doc/release-notes/frontend-1062-directory-search.md b/doc/release-notes/frontend-1062-directory-search.md new file mode 100644 index 00000000000..3ebff579219 --- /dev/null +++ b/doc/release-notes/frontend-1062-directory-search.md @@ -0,0 +1,34 @@ +## Release Highlights + +### Search files by directory name + +You can now find files by directory name in dataset searches, site-wide searches, +and the Search API. + +See [IQSS/dataverse-frontend#1062](https://github.com/IQSS/dataverse-frontend/issues/1062) +and [PR #12685](https://github.com/IQSS/dataverse/pull/12685). + +## Upgrade Instructions + +1. Add the `fileDirectoryLabel` field and its `copyField` rule from + `conf/solr/schema.xml` to the active Solr core's `schema.xml`, keeping any + local customizations. + See [Directory Name Search](https://dataverse-guide--12685.org.readthedocs.build/en/12685/admin/solr-search-index.html#directory-name-search-index) + for details. + +2. Reload the Solr core before deploying the updated application. For the default + core name: + + ```bash + curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1" + ``` + +3. After deploying the application, reindex existing files using + [Reindex in Place](https://dataverse-guide--12685.org.readthedocs.build/en/12685/admin/solr-search-index.html#reindex-in-place): + + ```bash + curl -X DELETE http://localhost:8080/api/admin/index/timestamps + curl http://localhost:8080/api/admin/index/continue + ``` + +No PostgreSQL schema migration is required.