synthesizer.pipeline.pipeline_utils

A submodule with helpers for writing out Synthesizer pipeline results.

Functions

synthesizer.pipeline.pipeline_utils.cached_split(split_key)[source]

Split a key into a list of keys.

This is a cached version of the split function to avoid repeated splitting of the same key.

Parameters:

split_key (str) – The key to split in “key1/key2/…/keyN” format.

Returns:

A list of the split keys.

Return type:

list

synthesizer.pipeline.pipeline_utils.combine_list_of_dicts(dicts)[source]

Combine a list of dictionaries into a single dictionary.

Parameters:

dicts (list) – A list of dictionaries to combine.

Returns:

The combined dictionary.

Return type:

dict

synthesizer.pipeline.pipeline_utils.discover_dict_recursive(data, prefix='', output_set=None)[source]

Recursively discover all leaves in a dictionary.

Parameters:

  • obj (dict) – The dictionary to search.

  • prefix (str) – A prefix to add to the keys of the arrays.

  • output_set (set) – A set to store the output paths in.

Returns:

A dictionary of all the numpy arrays in the input dictionary.

Return type:

dict

synthesizer.pipeline.pipeline_utils.discover_dict_structure(data)[source]

Recursively discover the structure of a dictionary.

Parameters:

data (dict) – The dictionary to search.

Returns:

A dictionary of all the paths in the input dictionary.

Return type:

dict

synthesizer.pipeline.pipeline_utils.discover_outputs(galaxies)[source]

Recursively discover all outputs attached to a galaxy.

This function will collate all paths to attributes at any level within the input object.

If the object is a dictionary, we will loop over all keys and values recursing where appropriate.

If the object is a class instance (e.g. Galaxy, Stars, ImageCollection, etc.), we will loop over all attributes and recurse where appropriate.

If the object is a “value” (i.e. an array or a scalar), we will append the full path to the output list.

Parameters:

  • galaxy (dict) – The dictionary to search.

  • prefix (str) – A prefix to add to the keys of the arrays.

  • output (dict) – A dictionary to store the output paths in.

synthesizer.pipeline.pipeline_utils.discover_outputs_recursive(obj, prefix='', output_set=None)[source]

Recursively discover all outputs attached to a galaxy.

This function will collate all paths to attributes at any level within the input object.

If the object is a dictionary, we will loop over all keys and values recursing where appropriate.

If the object is a class instance (e.g. Galaxy, Stars, ImageCollection, etc.), we will loop over all attributes and recurse where appropriate.

If the object is a “value” (i.e. an array or a scalar), we will append the full path to the output list.

Parameters:

  • obj (dict) – The dictionary to search.

  • prefix (str) – A prefix to add to the keys of the arrays.

  • output_set (set) – A set to store the output paths in.

Returns:

A dictionary of all the numpy arrays in the input dictionary.

Return type:

dict

synthesizer.pipeline.pipeline_utils.get_dataset_properties(data, comm, root=0)[source]

Return the shapes, dtypes and units of all data arrays in a dictionary.

Parameters:

  • data (dict) – The data to get the shapes of.

  • comm (mpi.Comm) – The MPI communicator.

  • root (int) – The root rank to gather data to.

Returns:

A dictionary of the shapes of all data arrays. dict: A dictionary of the dtypes of all data arrays. dict: A dictionary of the units of all data arrays.

Return type:

dict

synthesizer.pipeline.pipeline_utils.unify_dict_structure_across_ranks(data, comm, root=0)[source]

Recursively unify the structure of a dictionary across all ranks.

This function will ensure that all ranks have the same structure in their dictionaries. This is necessary for writing out the data in parallel.

Parameters:

  • data (dict) – The data to unify.

  • comm (mpi.Comm) – The MPI communicator.

  • root (int) – The root rank to gather data to.