BLS_CPI changes made related to test cases,download and process

requirements.txt

@@ -21,14 +21,16 @@ pandas
 psutil
 pylint
 pytest
-requests==2.27.1
 requests_cache
 retry==0.9.2
 shapely==1.8.5
-urllib3==1.26.8
 xarray==0.19.0
 xlrd
 zipp
 beautifulsoup4
 ratelimit
 xlsxwriter==3.2.0
+websockets==12.0
+requests==2.31.0
+urllib3==1.26.17
+

requirements_all.txt

@@ -31,16 +31,17 @@ pylint
 pytest
 rasterio
 rdp==0.8
-requests==2.27.1
 retry==0.9.2
 s2sphere==0.2.5
 shapely==1.8.5
 tabula-py
-urllib3==1.26.8
 xarray==0.19.0
 xlrd
 yapf
 zipp
 beautifulsoup4
 ratelimit
 xlsxwriter==3.2.0
+websockets==12.0
+requests==2.31.0
+urllib3==1.26.17

scripts/us_bls/cpi/README.md

@@ -110,80 +110,11 @@ The license is available online at https://www.bls.gov/bls/linksite.htm.
   - Contains StatisticalVariables for C-CPI-U series.
 - [cpi_u.mcf](cpi_u.mcf)
 - [cpi_w.mcf](cpi_w.mcf)
-- [pop_type_enums.mcf](pop_type_enums.mcf)
-  - Contains populationType enums for all three types of series.
-- [unit_enums.mcf](unit_enums.mcf)
-  - Contains unit enums for all three types of series.
-
-#### Scripts
-
-- [generate_csv_mcf.py](generate_csv_mcf.py) generates the CSVs,
-  StatisticalVariable MCFs, and template MCFs. See module docstring.
-
-## Proposed manifest.json for adding product breakdown
-
-See About the Import > Status above for context.
-
-```
-{
-    "import_specifications": [
-        {
-            "import_name": "USBLS_CPI_U",
-            "curator_emails": [
-                "shijunjie@google.com"
-            ],
-            "provenance_url": "https://www.bls.gov/cpi/",
-            "provenance_description": "U.S. Bureau of Labor Statistics Consumer Price Index for All Urban Consumers",
-            "scripts": [
-                "generate_csv_mcf.py"
-            ],
-            "import_inputs": [
-                {
-                    "template_mcf": "cpi_u.tmcf",
-                    "cleaned_csv": "cpi_u.csv",
-                    "node_mcf": "cpi_u.mcf"
-                }
-            ],
-            "cron_schedule": "0 17 15 * *"
-        },
-        {
-            "import_name": "USBLS_CPI_W",
-            "curator_emails": [
-                "shijunjie@google.com"
-            ],
-            "provenance_url": "https://www.bls.gov/cpi/",
-            "provenance_description": "U.S. Bureau of Labor Statistics Consumer Price Index for Urban Wage Earners and Clerical Workers",
-            "scripts": [
-                "generate_csv_mcf.py"
-            ],
-            "import_inputs": [
-                {
-                    "template_mcf": "cpi_w.tmcf",
-                    "cleaned_csv": "cpi_w.csv",
-                    "node_mcf": "cpi_w.mcf"
-                }
-            ],
-            "cron_schedule": "15 17 15 * *"
-        },
-        {
-            "import_name": "USBLS_C_CPI_U",
-            "curator_emails": [
-                "shijunjie@google.com"
-            ],
-            "provenance_url": "https://www.bls.gov/cpi/",
-            "provenance_description": "U.S. Bureau of Labor Statistics Chained Consumer Price Index for All Urban Consumers",
-            "scripts": [
-                "generate_csv_mcf.py"
-            ],
-            "import_inputs": [
-                {
-                    "template_mcf": "c_cpi_u.tmcf",
-                    "cleaned_csv": "c_cpi_u.csv",
-                    "node_mcf": "c_cpi_u.mcf"
-                }
-            ],
-            "cron_schedule": "30 17 15 * *"
-        }
-    ]
-}
-```
+
+### Steps to download and process
+
+python3 generate_csv.py --mode=download
+python3 generate_csv.py --mode=process
+
+### Note: Data is available from 1947 in source , so the historical data is retained in GCP bucket (unresolved_mcf/bls/cpi/HistoricalData) from 1913 to 1946.
+### Auto-refresh is done only for 1947 and more year data.