feat: improve API-ref extraction quality and coverage in the RAG by omkarjoshi0304 · Pull Request #83 · openstack-lightspeed/rag-content

omkarjoshi0304 · 2026-01-14T16:15:31Z

Description:-

This PR improves the quality of API reference data ingested into the RAG pipeline and ensures consistent coverage across OpenStack 2025.2 projects.

Use pandoc + html2text for cleaner API extraction (removes nav, headers, and boilerplate)

Apply heuristic filtering to index.txt files to keep meaningful single-page APIs and drop navigation-only indices

Auto-detect api-ref and api-guide sources to avoid missing project docs

Filter out placeholder/junk files using logo metadata detection

Akrog

As a rule the source of truth is the git repository, so we should try to include as much information as possible in the commit message.

In this case I see there is more information on the PR description than the commit message itself.

Akrog · 2026-01-15T08:32:55Z

scripts/get_openstack_plaintext_docs.sh

-    if [ "$OS_API_DOCS" = "true" ] && [ -d "./api-ref/source" ]; then
-        if ! grep -q "text-api-ref" tox.ini; then
-            echo "$tox_text_api_ref_target" >> tox.ini
+    # Build API-Ref


nit: Why remote the "if enabled" part of the comment?

My bad i accidentally removed it

Akrog · 2026-01-15T08:42:14Z

scripts/get_openstack_plaintext_docs.sh

+                    text_file="./$api_dir/build/text/${rel_path%.html}.txt"
+                    mkdir -p "$(dirname "$text_file")"
+
+                    if command -v pandoc &> /dev/null; then


Not sure it's a good idea to allow building things with 2 different tools, as they would create different outputs, so we may start having different DB contents and not realize. This would make it harder to find out WHY things don't behave like they did beefore.

I have removed the html2text fallback and standardized on Pandoc only.
The reasoning is that Pandoc produces significantly better structural output (especially for tables and code blocks) compared to html2text

Akrog · 2026-01-15T08:42:38Z

scripts/get_openstack_plaintext_docs.sh

+
+                    if command -v pandoc &> /dev/null; then
+                        # shellcheck disable=SC2015
+                        pandoc -f html -t plain --wrap=preserve "$html_file" -o "$text_file" 2>/dev/null && ((converted_count++)) || true


Why is it OK for conversion to fail?
This could hide problems with the process.

I removed the || true safety net. The script will now fail explicitly if a file cannot be converted, preventing us from silently indexing corrupted or missing data.

scripts/get_openstack_plaintext_docs.sh

Akrog · 2026-01-15T08:54:44Z

scripts/get_openstack_plaintext_docs.sh

+
+                if [ "$api_file_count" -gt 0 ]; then
+                    echo "API-Ref: Found $api_file_count content files for $project"
+                    # Only remove index.txt if other content files exist


I don't find this comment very helpful. I can see in the code at a glance that's what is happening, but it's not explained WHY we are deleting it. For example because when there are other files index.txt is a navigation page pointing to other pages, so we don't want it.

I have updated the comment

Akrog · 2026-01-15T08:57:08Z

scripts/get_openstack_plaintext_docs.sh

+                api_file_count=$(find "./$api_dir/build/text" -name "*.txt" -type f -size +1k 2>/dev/null | wc -l)
+
+                if [ "$api_file_count" -gt 0 ]; then
+                    echo "API-Ref: Found $api_file_count content files for $project"


Wouldn't this content calculation message wrong if we remove some files on L273?

moved the final file count calculation to the very end of the function so the log message accurately reflects the number of files

omkarjoshi0304 · 2026-01-15T10:03:36Z

As a rule the source of truth is the git repository, so we should try to include as much information as possible in the commit message.

In this case I see there is more information on the PR description than the commit message itself.

Thanks for addressing i will edit the commit message

lpiwowar

Just two comments that crossed my mind this morning.

lpiwowar · 2026-01-20T08:25:10Z

scripts/get_openstack_plaintext_docs.sh

    # neutron-lib is a library project that only generates API-Ref documentation.
    # Its regular doc build produces no usable output, but its API-Ref is needed by Neutron.
    if [ "$project" != "neutron-lib" ]; then
+        if ! grep -q "text-docs" tox.ini; then


question: Is this really needed? Aren't we adding the $tox_text_docs_target on line 195?

yes no need of this line

From your response I understand that we want to remove this line, but I see it still present in the code 🙈 . I'm keeping this open:). Let me know if I misinterpreted.

scripts/get_openstack_plaintext_docs.sh

Akrog · 2026-01-22T09:37:08Z

scripts/generate_embeddings_openstack.py

+            # Replace .txt with .html
+            remaining_path = remaining_path.replace(".txt", ".html")
+
+            # Build API-Ref URL: /api-ref/{service}/{remaining_path}


nit: The path in the comment doesn't look too useful to me. In my opinion the actual path doesn't add value since we can see it very clearly on the next line.
It is even misleading, as it has {service} but then we use {service_name}.

I have removed it

scripts/get_openstack_plaintext_docs.sh

Akrog · 2026-01-22T09:45:21Z

scripts/get_openstack_plaintext_docs.sh

-            find ./api-ref/build/text -mindepth 1 -depth -type d -empty -delete 2>/dev/null || true
+            if ! tox -etext-api-ref; then
+                echo "WARNING: API-Ref build failed for $project"
+                api_ref_failed="true"


-1: Why don't we fail the process if we fail to build the api-ref?
Wouldn't this hide errors?
I think we should fail, but maybe I'm missing something.

updated it to exit 1

Akrog · 2026-01-22T09:56:36Z

scripts/get_openstack_plaintext_docs.sh

-    [ -d "doc/build/text" ] && cp -r doc/build/text "$project_output_dir"/"$_output_version"_docs
+    # Copy regular docs if they were built (not for neutron-lib)
+    if [ -d "doc/build/text" ]; then
+        cp -r doc/build/text "$project_output_dir"/"$_output_version"_docs


nit: Why multiple double quotes? `"${project_output_dir}/${_output_version}_docs"

Akrog · 2026-01-22T09:57:37Z

scripts/get_openstack_plaintext_docs.sh

+    # Copy API-Ref documentation if it was built successfully
+    if [ "$OS_API_DOCS" = "true" ] && [ "$api_ref_failed" != "true" ] && \
+       [ -n "$api_dir" ] && [ -d "$api_dir/build/text" ]; then
+        cp -r "$api_dir/build/text" "$project_output_dir"/"$_output_version"_api-ref


nit: Same here about multiple double quotes

umago

I assume many part of bash scripting code has been AI generated ? I don't necessarily understand all of the commands there but the logic seems OK.

Assuming this has been tested and works I'm OK with it as we already have other parts of the code that has been assisted as well/.

scripts/get_openstack_plaintext_docs.sh

omkarjoshi0304 · 2026-02-06T11:32:20Z

I assume many part of bash scripting code has been AI generated ? I don't necessarily understand all of the commands there but the logic seems OK.

Assuming this has been tested and works I'm OK with it as we already have other parts of the code that has been assisted as well/.

index.txt
I have attached the one of the api ref doc file of cinder which has been parsed , quality seems good to me for other project as well . I took help of cursor to generate the script:) . just an brief of script

check for api directories (api-ref/source and api-guide/source)
configures tox
builds html with tox
coverts html to plain text using pandoc
deletes the junk files

lpiwowar · 2026-02-09T10:34:46Z

Code review

Found 1 issue:

converted_count is initialized to 0 but never incremented inside the while loop, so the log message at line 254 will always report "Converted 0 HTML files to text" regardless of how many files were actually converted. A ((converted_count++)) or equivalent is missing from the loop body.

rag-content/scripts/get_openstack_plaintext_docs.sh

Lines 240 to 255 in f5984b2

    
           converted_count=0 
        
           while IFS= read -r -d '' html_file; do 
        
               rel_path="${html_file#./"$api_dir"/build/html/}" 
        
               text_file="./$api_dir/build/text/${rel_path%.html}.txt" 
        
               mkdir -p "$(dirname "$text_file")" 
        
               # Convert HTML to plain text using pandoc (consistent output) 
        
               pandoc -f html -t plain --wrap=preserve "$html_file" -o "$text_file" || { 
        
                   echo "ERROR: Failed to convert $html_file" 
        
                   return 1 
        
               } 
        
           done < <(find "./$api_dir/build/html" -name "*.html" -type f -print0) 
        
           echo "Converted $converted_count HTML files to text for $project"

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

omkarjoshi0304 · 2026-02-09T13:39:54Z

Code review

Found 1 issue:

converted_count is initialized to 0 but never incremented inside the while loop, so the log message at line 254 will always report "Converted 0 HTML files to text" regardless of how many files were actually converted. A ((converted_count++)) or equivalent is missing from the loop body.

rag-content/scripts/get_openstack_plaintext_docs.sh

Lines 240 to 255 in f5984b2

converted_count=0

while IFS= read -r -d '' html_file; do

rel_path="${html_file#./"$api_dir"/build/html/}"

text_file="./$api_dir/build/text/${rel_path%.html}.txt"

mkdir -p "$(dirname "$text_file")"

# Convert HTML to plain text using pandoc (consistent output)

pandoc -f html -t plain --wrap=preserve "$html_file" -o "$text_file" || {

echo "ERROR: Failed to convert $html_file"

return 1

}

done < <(find "./$api_dir/build/html" -name "*.html" -type f -print0)

echo "Converted $converted_count HTML files to text for $project"

🤖 Generated with Claude Code

If this code review was useful, please react with 👍. Otherwise, react with 👎.

Thanks for address counter incremented

lpiwowar

Apologies, these are just small things, but I think it should be fixed.

lpiwowar · 2026-02-10T15:36:14Z

scripts/get_openstack_plaintext_docs.sh

-    # Only copy if text docs were built (skipped for neutron-lib)
-    [ -d "doc/build/text" ] && cp -r doc/build/text "$project_output_dir"/"$_output_version"_docs
+    # Copy regular docs if they were built (not for neutron-lib)
+    if [ -d "doc/build/text" ]; then


issue (blocking): I'm going to put this as a blocking because there is nothing from urging us to merge this PR immediately. As far as I can tell, this change is not related to this PR and does not modify the meaning of the code. Let's remove this change or move it to a separate commit. I know it is a nit, but I think it is a good practice to keep the PRs focused.

The line is still modified 🙈 as part of this PR. The change should be either:

part of a separate commit

or not part of this PR at all.

scripts/get_openstack_plaintext_docs.sh

omkarjoshi0304 · 2026-02-10T16:15:22Z

Apologies, these are just small things, but I think it should be fixed.

No worries I will fix it:)

lpiwowar · 2026-02-16T12:37:49Z

scripts/generate_embeddings_openstack.py

 class OpenStackDocsMetadataProcessor(MetadataProcessor):
    """Metadata processor for OpenStack documentation."""

+    API_REF_SERVICE_MAPPING = {


issue (blocking): The commit message should be polished. Also, there is an unwanted commit titled: Merge remote-tracking branch 'refs/remotes/upstream/main' into OSPRH-19255-api-ref

lpiwowar · 2026-02-16T12:40:35Z

scripts/get_openstack_plaintext_docs.sh

        echo "Using ${num_running_subproc}/${NUM_WORKERS} workers. Waiting ..."
        wait -n || log_and_die "Subprocess generating text documentation failed!"
-	echo "Using $(( --num_running_subproc ))/${NUM_WORKERS} workers."
+        echo "Using $(( --num_running_subproc ))/${NUM_WORKERS} workers."


issue (blocking): Thank you for fixing the line and the inconsistent tabs/spaces usage:). The thing is that this should be a separate commit ideally (because it is not related to the core of the PR).

lpiwowar · 2026-02-16T12:44:31Z

scripts/get_openstack_plaintext_docs.sh

-    # Only copy if text docs were built (skipped for neutron-lib)
-    [ -d "doc/build/text" ] && cp -r doc/build/text "$project_output_dir"/"$_output_version"_docs
+    # Copy regular docs if they were built (not for neutron-lib)
+    if [ -d "doc/build/text" ]; then


The line is still modified 🙈 as part of this PR. The change should be either:

part of a separate commit

or not part of this PR at all.

- Standardize on Pandoc for consistent RAG output; remove html2text fallback. - Add strict error handling for conversion failures and missing tools. - Fix "Url not reachable" errors by recursively pruning empty navigation index files. - Auto-detect api-ref and api-guide sources to avoid missing project docs. PR Comment fix

omkarjoshi0304 requested a review from a team as a code owner January 14, 2026 16:15

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch from ef662f1 to 9516e0e Compare January 14, 2026 16:33

omkarjoshi0304 marked this pull request as draft January 14, 2026 16:40

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch from 9516e0e to 55cdbad Compare January 14, 2026 18:39

omkarjoshi0304 marked this pull request as ready for review January 14, 2026 18:50

Akrog requested changes Jan 15, 2026

View reviewed changes

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch 2 times, most recently from 42f3a1d to ce60cfb Compare January 17, 2026 02:15

lpiwowar reviewed Jan 20, 2026

View reviewed changes

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch 2 times, most recently from 951ea86 to 212b8ed Compare January 21, 2026 03:41

Akrog requested changes Jan 22, 2026

View reviewed changes

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch from b8fedc6 to f5984b2 Compare February 2, 2026 20:08

umago previously approved these changes Feb 6, 2026

View reviewed changes

scripts/get_openstack_plaintext_docs.sh Show resolved Hide resolved

omkarjoshi0304 dismissed umago’s stale review via eb09cb5 February 9, 2026 13:38

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch from ece8977 to eb09cb5 Compare February 9, 2026 13:38

lpiwowar requested changes Feb 10, 2026

View reviewed changes

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch 5 times, most recently from e890d85 to 347b7d0 Compare February 11, 2026 12:55

lpiwowar requested changes Feb 16, 2026

View reviewed changes

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch 2 times, most recently from 69254f1 to 347b7d0 Compare February 16, 2026 13:11

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch 3 times, most recently from d4b1b21 to 16e2f3f Compare February 17, 2026 13:04

omkarjoshi0304 added 3 commits February 17, 2026 13:50

nit: fix for extra double inverted comma

b22cbee

nit: fixing inconsistent tabs/spaces

755363b

omkarjoshi0304 force-pushed the OSPRH-19255-api-ref branch from 78d3bf4 to 755363b Compare February 17, 2026 13:50

Conversation

omkarjoshi0304 commented Jan 14, 2026

Uh oh!

Akrog left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omkarjoshi0304 commented Jan 15, 2026

Uh oh!

lpiwowar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

umago left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

omkarjoshi0304 commented Feb 6, 2026

Uh oh!

lpiwowar commented Feb 9, 2026

Code review

Uh oh!

omkarjoshi0304 commented Feb 9, 2026

Code review

Uh oh!

lpiwowar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

omkarjoshi0304 commented Feb 10, 2026

Uh oh!

lpiwowar Feb 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

lpiwowar Feb 16, 2026 •

edited

Loading

lpiwowar Feb 16, 2026 •

edited

Loading