PROJECT SENTRY
PROJECT SENTRY / DOCS / 02_data_dictionary_and_sources.md

Project SENTRY: Data Dictionary, Data Sources & Publication Calendar

1. Feature Store Overview

The Project SENTRY Feature Store encapsulates 93 candidate features across 46 Parquet tables backed up remotely in Kaggle Dataset soliculus/sentry-processed-features (Version 9). All feature tables are partitioned or indexed by geographical region (18 administrative regions), province/HUC (135 entities), and time period (monthly, quarterly, or annual).

data/processed/
├── psa_national_qna.parquet                  # National GDP, GNI, Expenditure aggregates (Quarterly)
├── psa_grdp.parquet                          # Regional GRDP (Annual ground truth, 2018–2024)
├── psa_gva_aff.parquet                       # Agriculture, Forestry & Fishing Regional GVA
├── psa_gva_industry.parquet                  # Industry Regional GVA (Mining, Manufacturing, Construction)
├── psa_gva_services.parquet                  # Services Regional GVA (Wholesale, Financial, Real Estate)
├── psa_ppa.parquet                           # Provincial Product Accounts (135 Jurisdictions, 2018–2024)
├── psa_regional_cpi.parquet                  # Regional Consumer Price Index (Monthly)
├── psa_regional_lfs.parquet                  # Regional Labor Force Survey (Monthly / Quarterly)
├── psa_cfs.parquet                           # Commodity Flow Survey bilateral trade values (Annual)
├── bsp_remittances.parquet                   # Overseas Filipino cash remittances by country/corridor (Monthly)
├── bsp_policy_events.parquet                 # Monetary Board policy decisions & target RRP rates (Event)
├── bsp_sentiment_scores.parquet              # FinBERT tone analysis of policy minutes (768-dim embeddings)
├── fred_macro.parquet                        # High-frequency macroeconomic & financial indicators (Monthly)
├── comtrade_trade.parquet                    # Bilateral trade flows across top 10 export/import partners (Monthly)
├── gee_ntl_monthly.parquet                   # NOAA-VIIRS nighttime lights radiance (Monthly)
├── gee_ndvi_monthly.parquet                  # Sentinel-2 surface reflectance NDVI (Monthly)
├── climate_shocks_monthly.parquet            # CHIRPS precipitation anomalies & Cyclone Disturbance Index (Monthly)
├── master_features.parquet                   # Consolidated multi-modal panel (88 raw + derived features)
├── pruned_features_gated.parquet             # Domain-gated 16 core indicators + 3 PLS latent factors
├── spatial_weight_cfs.parquet                # 18x18 hybrid economic-gravity spatial weight matrix W
├── spatial_disagg_regional.parquet           # Reconciled regional level and growth rate projections
└── operational_nowcast_provincial.parquet    # Reconciled provincial level nowcasts (135 entities)

2. Official Data Sources & Extraction Protocols

Data Source Domain Frequency API / Ingestion Method Latency / Publication Lag
PSA OpenSTAT National Accounts (QNA) Quarterly PxWeb REST API (fetch_psa.py) 45 days post-quarter
PSA OpenSTAT Regional Accounts (GRDP) Annual PxWeb REST API (fetch_psa.py) 4–5 months post-year (April release)
PSA OpenSTAT Provincial Accounts (PPA) Annual PxWeb REST API (fetch_psa.py) 12–16 months post-year (November release)
PSA OpenSTAT Regional CPI & Inflation Monthly PxWeb REST API (fetch_psa_cpi.py) 5 days post-month
PSA OpenSTAT Labor Force Survey (LFS) Monthly PxWeb REST API (fetch_psa_lfs.py) 35–40 days post-month
PSA OpenSTAT Commodity Flow (CFS) Annual PxWeb REST API (build_cfs_weights.py) 10–12 months post-year
Bangko Sentral ng Pilipinas (BSP) Cash Remittances Monthly Web Scraper & REST (fetch_remittances.py) 45 days post-month
Bangko Sentral ng Pilipinas (BSP) Policy Minutes & Statements Event-driven Scraper + FinBERT (nlp_inference.py) 2–3 weeks post-meeting
Federal Reserve (FRED) Global Macro & Commodity Prices Monthly/Daily fredapi Client (fetch_fred.py) 1–5 days post-period
UN Comtrade Bilateral Trade Flows Monthly Comtrade v2 REST (fetch_comtrade.py) 45–60 days post-month
Google Earth Engine (GEE) NOAA-VIIRS Nighttime Lights Monthly GEE Python API (extract_satellite.py) 15–20 days post-month
Google Earth Engine (GEE) Sentinel-2 NDVI Surface Vigor Monthly GEE Python API (extract_satellite.py) 5–10 days post-month
CHIRPS / UCSB Precipitation & Drought SPI Monthly Raster extraction (extract_climate_shocks.py) 15 days post-month
GDACS Tropical Cyclone Disturbance Index Event-driven Disaster Alert REST API Real-time (24 hours post-landfall)

3. Exhaustive Feature Dictionary (Core Candidate Variables)

3.1 Macroeconomic & Price Signals (fred_macro.parquet, psa_regional_cpi.parquet)

  • cpi_all_items: Regional Consumer Price Index (2018=1002018 = 100).
  • inflation_headline_yoy: Year-over-year percentage change in regional headline CPI.
  • inflation_food_yoy: Food and Non-Alcoholic Beverages inflation rate.
  • inflation_energy_yoy: Housing, Water, Electricity, Gas and Other Fuels inflation rate.
  • fx_usd_php: Daily average and end-of-month spot exchange rate (USD/PHP).
  • brent_crude_usd: Brent spot oil price ($USD/barrel), driving regional transport and electricity margins.
  • us_fed_funds_rate: Effective Federal Funds rate, capturing global monetary tightening spillovers.
  • us_10y_treasury: 10-Year Treasury constant maturity rate.
  • reer_bis_ph: Bank for International Settlements Real Effective Exchange Rate index for the Philippines.
  • us_indpro: US Industrial Production Index, proxying external goods demand.
  • china_indpro: China Industrial Production Index, driving East Asian bilateral trade flows.

3.2 Remittance & Financial Flows (bsp_remittances.parquet, bsp_finbert_embeddings.parquet)

  • remittance_total_usd: Total overseas Filipino cash remittances processed via banking channels (USD millions).
  • remittance_us_usd: Remittance inflows from the United States corridor.
  • remittance_me_usd: Remittance inflows from Middle East corridors (Saudi Arabia, UAE, Qatar).
  • remittance_asia_usd: Remittance inflows from East and Southeast Asia (Singapore, Japan, Hong Kong).
  • remittance_php_real: Real remittance purchasing power adjusted by regional CPI (RemittancePHP/CPIr,t\text{Remittance}{\text{PHP}} / \text{CPI}{r, t}).
  • finbert_sentiment_polarity: Domain-adapted FinBERT sentiment score of Monetary Board policy statements ([1.0,+1.0][-1.0, +1.0]).
  • finbert_hawkish_prob: Probability of hawkish policy stance extracted from statement text.
  • bank_deposit_liabilities: Total commercial and thrift bank deposit liabilities by province.

3.3 Bilateral Trade Channels (comtrade_trade.parquet)

  • trade_exports_total: Total bilateral goods export value (USD millions).
  • trade_imports_total: Total bilateral goods import value (USD millions).
  • trade_balance_usd: Goods trade balance (ExportsImports\text{Exports} - \text{Imports}).
  • trade_exports_electronics: HS-85 electrical machinery and equipment exports (over 55%55% of PH export basket).
  • trade_partner_share_us: Bilateral export share to the United States.
  • trade_partner_share_cn: Bilateral export share to mainland China.
  • trade_partner_share_jp: Bilateral export share to Japan.

3.4 Earth Observation & Physical Proxies (gee_ntl_monthly.parquet, gee_ndvi_monthly.parquet)

  • ntl_mean: Mean nocturnal cloud-free radiance (nWcm2sr1\text{nW} \cdot \text{cm}^{-2} \cdot \text{sr}^{-1}) from NOAA-VIIRS Day/Night Band (VCMCFG).
  • ntl_sum: Cumulative nocturnal radiance across all pixels within regional/provincial polygon boundaries.
  • ntl_growth_yoy: Year-over-year rate of change in mean nocturnal radiance.
  • ndvi_mean: Mean Normalized Difference Vegetation Index from Sentinel-2 Surface Reflectance (COPERNICUS/S2_SR_HARMONIZED), cloud-masked via QA60.
  • ndvi_p50: Median vegetative vigor across agricultural crop zones.
  • ndvi_anomaly_yoy: Deviation of vegetative vigor from 5-year historical monthly baseline.

3.5 Climate & Disaster Shock Indices (climate_shocks_monthly.parquet)

  • chirps_precip_mm: Mean monthly cumulative precipitation (mm) from CHIRPS high-resolution gridded datasets.
  • chirps_spi_3m: 3-Month Standardized Precipitation Index measuring regional drought (<1.5< -1.5) or extreme wetness (>+1.5> +1.5).
  • cyclone_disturbance_index: Weighted exposure score combining GDACS maximum sustained wind speed (knots) and distance from regional centroid to cyclone eye trajectory.
  • typhoon_days_count: Count of days per month where regional boundaries experienced tropical cyclone signal 1\ge 1.

3.6 Latent Projected Features (pls_reduction.py)

  • pls_factor_1: First orthogonal partial least squares projection capturing aggregate industrial-commercial co-movement.
  • pls_factor_2: Second PLS projection capturing terms-of-trade and currency devaluation shocks.
  • pls_factor_3: Third PLS projection capturing agricultural and meteorological disruption.

4. Publication Calendar, Lags & Ragged-Edge Timing

Official Philippine macroeconomic data releases follow a staggered release schedule creating an asymmetric "ragged edge":

[ NOWCAST TIMELINE: Reference Year T ]
  January 15: Q4 CPI, Trade, and Satellite Radiance (94% panel complete)
  February 10: Official PSA Q4 National QNA Release (GDP benchmark locked)
  April 25: Official PSA Annual Gross Regional Domestic Product (GRDP) Release
  November 30: Official PSA Annual Provincial Product Accounts (PPA) Release

Publication Asymmetry Matrix

  • Level 0 (National GDP): Released 45 days after quarter end (T+45dT+45\text{d}).
  • Level 1 (Regional GRDP): Released 115 days after year end (T+115dT+115\text{d} in late April). SENTRY nowcasts GRDP in real-time from T180dT-180\text{d} to T+0dT+0\text{d}.
  • Level 2 (Provincial PPA): Released 330 to 450 days after year end (T+330dT+330\text{d} to T+450dT+450\text{d}). SENTRY's constrained Denton downscaler bridges this 12-to-16 month information vacuum immediately upon GRDP nowcast convergence.

Revision Mechanics

PSA routinely revises preliminary estimates:

  1. First Vintage (V1V_1): Published in April of year T+1T+1.
  2. Second Vintage (V2V_2): Revised in April of year T+2T+2 upon full incorporation of the Annual Survey of Philippine Business and Industry (ASPBI).
  3. SENTRY trains on finalized V2V_2 vintages historically while evaluating walk-forward models on simulated real-time V1V_1 ragged edges to prevent look-ahead bias.
Mathematical typography rendered via KaTeX.
Research paper documentation compiled directly from /docs/.