Find files on the local filesystem

Find files on the local filesystem.

Deprecated since version 2.14.0: This module has been moved to esmvalcore.io.local. Importing it as esmvalcore.local is deprecated and will be removed in version 2.16.0.

Classes:

DataSource(*args, **kwargs)	Data source for finding files on a local filesystem.
LocalDataSource(name, project, priority, ...)	Data source for finding files on a local filesystem.
LocalFile(*args, **kwargs)	File on the local filesystem.

Functions:

find_files(*[, debug])

Find files on the local filesystem.

class esmvalcore.local.DataSource(*args, **kwargs)

Bases: LocalDataSource

Data source for finding files on a local filesystem.

Deprecated since version 2.14.0: This class is deprecated and will be removed in version 2.16.0. Please use esmvalcore.local.LocalDataSource instead.

Methods:

find_files(**facets)	Find files.
get_glob_patterns(**facets)	Compose the globs that will be used to look for files.
path2facets(path, add_timerange)	Extract facets from path.

Attributes:

regex_pattern

Get regex pattern that can be used to extract facets from paths.

find_files(**facets: FacetValue) → list[LocalFile]

Find files.

Parameters:

facets (FacetValue)

Return type:

list[LocalFile]

get_glob_patterns(**facets: FacetValue) → list[Path]

Compose the globs that will be used to look for files.

Parameters:

facets (FacetValue)

Return type:

list[Path]

path2facets(path: Path, add_timerange: bool) → dict[str, str]

Extract facets from path.

Parameters:

• path (Path)

• add_timerange (bool)

Return type:

dict[str, str]

property regex_pattern: str

Get regex pattern that can be used to extract facets from paths.

class esmvalcore.local.LocalDataSource(name: str, project: str, priority: int, rootpath: ~pathlib.Path, dirname_template: str, filename_template: str, ignore_warnings: list[dict[str, ~typing.Any]] | None = <factory>)

Bases: DataSource

Data source for finding files on a local filesystem.

Attributes:

debug_info	A string containing debug information when no data is found.
dirname_template	The template for the directory names.
filename_template	The template for the file names.
ignore_warnings	Warnings to ignore when loading the data.
name	A name identifying the data source.
priority	The priority of the data source.
project	The project that the data source provides data for.
rootpath	The path where the directories are located.

Methods:

find_data(**facets)

Find data locally.

Parameters:

• name (str)

• project (str)

• priority (int)

• rootpath (Path)

• dirname_template (str)

• filename_template (str)

• ignore_warnings (list[dict[str, Any]] | None)

debug_info: str = ''

A string containing debug information when no data is found.

dirname_template: str

The template for the directory names.

filename_template: str

The template for the file names.

find_data(**facets: FacetValue) → list[LocalFile]

Find data locally.

Parameters:

**facets (FacetValue) – Find data matching these facets.

Returns:

A list of files.

Return type:

list[LocalFile]

ignore_warnings: list[dict[str, Any]] | None

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling LocalFile.to_iris().

name: str

A name identifying the data source.

priority: int

The priority of the data source. Lower values have priority.

project: str

The project that the data source provides data for.

rootpath: Path

The path where the directories are located.

class esmvalcore.local.LocalFile(*args, **kwargs)

Bases: PosixPath, DataElement

File on the local filesystem.

Attributes:

attributes	Attributes are key-value pairs describing the data.
facets	Facets are key-value pairs that were used to find this data.
ignore_warnings	Warnings to ignore when loading the data.

Methods:

prepare()	Prepare the data for access.
to_iris()	Load the data as Iris cubes.

property attributes: dict[str, Any]

Attributes are key-value pairs describing the data.

property facets: Facets

Facets are key-value pairs that were used to find this data.

property ignore_warnings: list[dict[str, Any]] | None

Warnings to ignore when loading the data.

The list should contain dict`s with keyword arguments that will be passed to the :func:`warnings.filterwarnings function when calling the to_iris method.

prepare() → None

Prepare the data for access.

Return type:

None

to_iris() → CubeList

Load the data as Iris cubes.

Returns:

The loaded data.

Return type:

iris.cube.CubeList

esmvalcore.local.find_files(*, debug: bool = False, **facets: FacetValue) → list[LocalFile] | tuple[list[LocalFile], list[Path]]

Find files on the local filesystem.

Deprecated since version 2.14.0: This function is deprecated and will be removed in version 2.16.0. Please use esmvalcore.local.LocalDataSource.find_data() instead.

The directories that are searched for files are defined in esmvalcore.config.CFG under the 'rootpath' key using the directory structure defined under the 'drs' key. If esmvalcore.config.CFG['rootpath'] contains a key that matches the value of the project facet, those paths will be used. If there is no project specific key, the directories in esmvalcore.config.CFG['rootpath']['default'] will be searched.

See Input data for extensive instructions on configuring ESMValCore so it can find files locally.

Parameters:

• debug (bool) – When debug is set to True, the function will return a tuple with the first element containing the files that were found and the second element containing the glob.glob() patterns that were used to search for files.

• **facets (FacetValue) – Facets used to search for files. An '*' can be used to match any value. By default, only the latest version of a file will be returned. To select all versions use version='*'. It is also possible to specify multiple values for a facet, e.g. exp=['historical', 'ssp585'] will match any file that belongs to either the historical or ssp585 experiment. The timerange facet can be specified in ISO 8601 format.

Return type:

list[LocalFile] | tuple[list[LocalFile], list[Path]]

Note

A value of timerange='*' is supported, but combining a '*' with a time or period as supported in the recipe is currently not supported and will return all found files.

Examples

Search for files containing surface air temperature from any CMIP6 model for the historical experiment:

>>> esmvalcore.local.find_files(
...     project='CMIP6',
...     activity='CMIP',
...     mip='Amon',
...     short_name='tas',
...     exp='historical',
...     dataset='*',
...     ensemble='*',
...     grid='*',
...     institute='*',
... )
[LocalFile('/home/bandela/climate_data/CMIP6/CMIP/BCC/BCC-ESM1/historical/r1i1p1f1/Amon/tas/gn/v20181214/tas_Amon_BCC-ESM1_historical_r1i1p1f1_gn_185001-201412.nc')]

Returns:

The files that were found.

Return type:

list[LocalFile]

Parameters:

• debug (bool)

• facets (FacetValue)