data_downloader

Downloaders for the forecast tool’s external input data.

Provides two classes:

  • CFSDownloader retrieves Climate Forecast System v2 (CFSv2) GRIB2 forecast files from either the NOAA AWS public dataset or the NOAA NCEI archive, organising them into per-date local subdirectories.

  • SSTDownloader fetches Great Lakes surface water temperatures from NOAA GLSEA .dat files, used as initial conditions for forecasting.

Network access is required at run time; the heavy I/O dependencies (boto3, requests, BeautifulSoup) are mocked when building the documentation.

class src.data_downloader.CFSDownloader[source]

Bases: object

A utility class to download Climate Forecast System (CFSv2) GRIB2 forecast data from AWS or the NOAA NCEI archive.

This class manages local storage paths, S3 configurations for public AWS access, and sets up the environment for bulk data retrieval operations using threaded downloads.

base_download_dir

Root directory where all data will be downloaded.

Type:

str

cfs_dir

Subdirectory for storing CFS-specific data under the base download directory.

Type:

str

s3_config

Boto3 configuration with unsigned (public) access for AWS S3.

Type:

botocore.config.Config

s3

S3 client used to communicate with the NOAA AWS public dataset.

Type:

boto3.client

__init__()[source]

Initialize the DataDownloader class.

This method configures an S3 client for public access to download CFS data.

download(download_dir, start_date, end_date=None, hours=None, products=None, source=None, bucket_name=None)[source]

Download Climate Forecast System v2 (CFSv2) GRIB2 forecast files from AWS or NCEI over a specified date range.

This method automates the download of monthly mean CFSv2 GRIB2 forecast files for specified hours, products, and source (AWS or NCEI). Files are organized by date and stored in local subdirectories named after each date.

Parameters:

  • download_dir (str) – Path to the local root directory where all downloaded files will be stored.

  • start_date (str) – Start date for the download period, in ‘MM-DD-YYYY’ format.

  • end_date (str, optional) – End date for the download period, in ‘MM-DD-YYYY’ format. If not provided, only start_date will be used.

  • hours (list of str, optional) – List of forecast cycle hours to retrieve. Valid options are [‘00’, ‘06’, ‘12’, ‘18’]. If not provided, all hours are downloaded.

  • products (list of str, optional) – List of product prefixes to filter downloads. Typical values include: - ‘pgbf’: Pressure-level fields - ‘flxf’: Flux fields - ‘ocnf’: Ocean forecast fields - ‘ipvf’: Intermediate forecast variables If not provided, all default products are downloaded.

  • source (str, optional) – The data source to use. Must be either: - ‘aws’: Download from NOAA AWS public dataset (default) - ‘ncei’: Download from NOAA’s National Centers for Environmental Information

  • bucket_name (str, optional) – The name of the AWS S3 bucket. Defaults to ‘noaa-cfs-pds’ when using AWS. Not used when source=’ncei’.

Raises:

ValueError – If date formats are incorrect, if invalid product names are provided, or if an invalid source is specified.

Notes

  • If only start_date is provided, downloads only that day.

  • Otherwise, downloads all dates from start_date (inclusive) to end_date (exclusive).

  • Skips missing files or directories gracefully.

  • AWS downloads use S3 API calls; NCEI uses HTML scraping.

class src.data_downloader.SSTDownloader[source]

Bases: object

Download Great Lakes surface water temperatures from NOAA GLSEA.

GLSEA (Great Lakes Surface Environmental Analysis) publishes daily lake-averaged surface water temperatures as plain-text .dat files. This class parses those files into tidy, per-lake pandas.DataFrame objects (Superior, Michigan-Huron, Erie, Ontario) for use as forecast initial conditions, with optional unit conversion and climatology backfilling of missing dates.

__init__()[source]

Initialize the SSTDownloader. No configuration is required.

current_glsea(url, units='K')[source]

Fetch the most recent daily lake average surface water temperature from a NOAA GLSEA .dat file.

Parameters:

url (str) – The URL of the .dat file.

Returns:

A DataFrame with columns: Date, Superior, Michigan-Huron, Erie, Ontario or None if the URL is not accessible.

Return type:

pd.DataFrame

all_glsea(url, units='K', climatology_csv=None, start_date=None, end_date=None)[source]

Fetch daily lake surface water temperatures from a NOAA GLSEA .dat file URL. Checks for missing data within the specified period (default = last 9 months) and fills missing values using climatology if provided, otherwise with NaN.

Parameters:

  • url (str) – The URL of the .dat file (GLSEA daily data).

  • climatology_csv (str or None, optional) – Path to a CSV file with mean climatological SSTs for each day of year. Must contain columns: [‘dayofyear’, ‘superior_sst’, ‘michigan-huron_sst’, ‘erie_sst’, ‘ontario_sst’]. If None, missing values are filled with NaN.

  • units (str, optional) – Unit system for output. ‘C’ for Celsius, ‘K’ for Kelvin. Default is ‘K’.

  • start_date (str or pd.Timestamp, optional) – Start of the date range to verify. Format ‘YYYY-MM-DD’. If None, defaults to 9 months before yesterday.

  • end_date (str or pd.Timestamp, optional) – End of the date range to verify. Format ‘YYYY-MM-DD’. If None, defaults to yesterday.

Returns:

A DataFrame with a daily Date index and SST columns for each lake. Missing dates are filled using climatology if available, otherwise NaN.

Return type:

pd.DataFrame