Initial auto config annual BEAGDP

[vishalworkdatacommon]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @vishalworkdatacommon, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes a new, automated data ingestion pipeline for annual Gross Domestic Product (GDP) data from the U.S. Bureau of Economic Analysis (BEA). It introduces a Python script to manage the download, extraction, and initial cleaning of the raw data, ensuring consistency and readiness for further processing. Complementing this, a comprehensive set of configuration files (manifest, pv_maps, metadata, and remap files) has been added to precisely define how various BEA GDP datasets are transformed and mapped into statistical variables within the Data Commons framework, enabling regular and structured updates of this critical economic indicator.

Highlights

• New Data Import Pipeline: An automated pipeline has been introduced for importing annual Gross Domestic Product (GDP) data from the U.S. Bureau of Economic Analysis (BEA).
• Data Download and Preprocessing Script: A new Python script (download_script.py) automates the download, extraction, filtering (for 'ALL_AREAS' files), lowercasing of filenames, and cleaning of CSV data by stripping spaces from key columns.
• Comprehensive Data Mapping: Extensive property-value map (pv_map) and metadata files have been added for multiple BEA GDP datasets (sagdp1-9, sagdp11), defining how raw data columns map to statistical variables and their properties.
• StatVar Remapping: Specific remapping files (_remap.csv) are included to standardize long or complex statistical variable names into more concise Data Commons IDs.
• Automated Scheduling: The manifest.json configures a cron schedule (0 5 3,17 * *) for the data import, ensuring automated execution on the 3rd and 17th of each month at 5 AM.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a new data import for BEA Annual GDP, including a download script, manifest, and configuration files. The overall structure is good, but there are several areas for improvement. The download script uses incorrect type hints (NoReturn instead of None) and has overly broad exception handling. The manifest.json is highly repetitive and could be greatly simplified by using a helper shell script. Additionally, there are potential correctness issues in the pvmap configuration files related to data filtering and inconsistencies across the metadata files that should be addressed to improve maintainability and correctness.

[vishalworkdatacommon]

-s in local got passed : https://pantheon.corp.google.com/batch/jobsDetail/regions/us-central1/jobs/beaannualgdp-kvishalll-20251214-173940?invt=Ab7M7w&mods=-monitoring_api_staging&project=datcom-infosys-dev

[SandeepTuniki]

Hi @vishalworkdatacommon, before I get into the detailed review - Can you help me understand why this import has so many pv-maps? For example, I see a lot of files with pattern sagdp. How do they differ (ex: sagdgp1 vs sagdp2) from each other?

[vishalworkdatacommon]

hi @SandeepTuniki

Please find the details below :

The reason for the high number of pv-maps is that while the files share a similar structure (the sagdp pattern), each one represents a distinct category. Since we cannot mix raw dollars, price indexes, and percentage growth in the same data property, each table requires its own mapping logic.

Here is a quick breakdown of how the files differ:

• SAGDP1 vs. SAGDP2: SAGDP1 provides the Total Summary (top-line GDP for a state), whereas SAGDP2 provides the Industry Breakdown (detailed contribution by sector).
• SAGDP1-5 (Current Dollars): These map to nominal dollar values and their components (like compensation or taxes).
• SAGDP6 & 9 (Real/Chained Dollars): These map to inflation-adjusted values (volume measures).
• SAGDP7-8 (Indexes): These map to Price and Quantity indexes rather than currency.
• SAGDP10-11 (Ratios/Growth): These map to percentage shares and percentage point contributions to growth.

By keeping these as separate pv-maps, we ensure that the data is ingested into the correct schema fields without "clashing" or overwriting different types of economic indicators.

[SandeepTuniki]

@vishalworkdatacommon Thanks for the info. Can you please document this in README file as well for future readers?

[vishalworkdatacommon]

Hi @SandeepTuniki

Sure adding it in readme file to.

[vishalworkdatacommon]

Hi @vishalworkdatacommon, before I get into the detailed review - Can you help me understand why this import has so many pv-maps? For example, I see a lot of files with pattern sagdp. How do they differ (ex: sagdgp1 vs sagdp2) from each other?

I have to take them as reference - but that is county level and but this state level as per understanding and knowledge