data_loader

Loaders for observed and training datasets used by the forecast tool.

This module reads the various on-disk datasets the model is trained and validated against, returning each as a tidy, date-indexed pandas.DataFrame:

• GLCC observed Net Basin Supply (DataLoader.glcc())

• L2SWBM precipitation, evaporation, and runoff (DataLoader.l2swbm())

• GLSEA lake surface temperatures (DataLoader.glsea())

• SNODAS snow water equivalent (DataLoader.snodas())

• USACE probability-of-exceedance tables (DataLoader.lake_probabilities())

Several loaders accept a units argument and convert between cubic metres per second (cms) and millimetres per month (mm) using the per-lake surface areas in lake_areas.

class src.data_loader.DataLoader

Bases: object

Load the observed and training datasets used to fit and validate models.

Each method reads one dataset family from a local directory and returns a cleaned, date-indexed pandas.DataFrame. The loader is stateless; all paths and options are passed per call.

__init__()

Initialize the DataLoader. No configuration is required.

glcc(directory, units='cms')

Load observed GLCC NBS data for all four lakes in either ‘cms’ or ‘mm’ units.

l2swbm(directory)

Load L2SWBM runoff, evaporation, and precipitation for all lakes.

glsea(file_path, units='K')

Reads GLSEA SST data from a file and returns a cleaned DataFrame.

snodas(file)

Load and process SNODAS SWE data.

Parameters:

• local_path (str) – Base path to local data directory.

• X_df (pd.DataFrame, optional) – DataFrame to align SNODAS data with by index.

Returns:

• snodas (pd.DataFrame) – Processed SNODAS dataframe indexed by date.

• aligned_snodas (pd.DataFrame, optional) – SNODAS aligned to X_df index.

• aligned_X_df (pd.DataFrame, optional) – X_df aligned to SNODAS index.

lake_probabilities(file_dir, units='cms')

Reads probability of exceedance data for multiple Great Lakes from CSV files, converts units if requested, and combines them into a single tidy (long-format) DataFrame.

Each input CSV contains probability of exceedance values (0.99–0.01) for 12 monthly columns (Jan–Dec). The function reshapes and merges all lakes into one DataFrame suitable for analysis or plotting.

Parameters:

• file_dir (str) –

  Path to the directory containing the lake probability CSV files. Expected files are:

  • SUP.probs.csv (Lake Superior)

  • MIH.probs.csv (Lakes Michigan-Huron)

  • ERI.probs.csv (Lake Erie)

  • ONT.probs.csv (Lake Ontario)

• units (str, optional, default="cms") –

  Desired output units for flow values.

  • ”cms” : cubic meters per second (original units from source files)

  • ”mm” : millimeters per month, computed using lake surface area and number of days in each month.

Returns:

A tidy DataFrame containing columns:

• ”month” : str, calendar month (Jan–Dec)

• ”lake” : str, lake name (“superior”, “michigan-huron”, “erie”, “ontario”)

• ”prob_exceedance” : float, probability of exceedance (0.99–0.01)

• ”value” : float, flow or equivalent value in chosen units

Return type:

pandas.DataFrame

Notes

• When units=”mm”, values are converted from m³/s to mm/month using value_mm = value_cms * 1000 * 86400 * days_in_month / lake_area, where lake_area is the surface area of each lake in m².

• Input CSVs are assumed to have 7 header rows before the data block, and contain a column named “Probability Of Exceedance”.

Example

>>> df = combine_lake_probabilities("/path/to/files/", units="mm")
>>> df.query("lake == 'erie' and month == 'Jan'")