The Decl'NetCDF plugin

The Decl'NetCDF plugin was created to read from and write to NetCDF files in a declarative way.

Decl'NetCDF plugin allows you to:

1. Read/write data as NetCDF variables from/to NetCDF file.
2. Define groups in the NetCDF file. Groups can be nested.
3. Read/write variables and groups attributes. Automatically read and written when openning the file.
4. Execute input/output operations on event.
5. Execute input/output operations on data share (when the event is not defined).
6. Define UNLIMITED dimension and dimensions names.
7. Read/write only part of NetCDF variable by hyperslab definition.
8. Read/write in parallel mode using MPI.

Configuration elements

The root of Decl'NetCDF plugin configuration (named decl_netcdf), is a dictionary or a list of dictionaries that contains the following subtrees:

key	value
logging	logging	optional
file	file subtree	mandatory
communicator	communicator subtree	optional
on_event	on_event subtree	optional
groups	groups subtree	optional
variables	variables subtree	optional
write	write subtree	optional
read	read subtree	optional

Configuration examples:

plugins:
  decl_netcdf:
    file: ""file_name.nc""
    write: ...

plugins:
  decl_netcdf:
    - file: ""file_name.nc""
      write: ...
    - file: "file_name_2.nc"
      write: ...
    - file: ""file_name.nc""
      read: ...

file subtree

Defines name (path) of the input/output file.

key	value
file	string containing filename (can have $-expressions)

Configuration example:

plugins:
  decl_netcdf:
    file: "file_name_${i}.nc"

communicator subtree

Warning

Feature not tested yet

Enables parallel NetCDF input/output. Defines communicator to use on read/write (all processes must have the same communicator and filename).

key	value
communicator	$-expression to valid MPI_comm

Configuration example:

plugins:
  decl_netcdf:
    file: ""file_name.nc""
    communicator: $MPI_COMM_WORLD

on_event subtree

Defines on which events the plugin reads/writes data from/to NetCDF file. The value can be either single event name or the array of events names (both examples presented below).

key	value
on_event	string or array of string that cointain event names

Configuration examples:

plugins:
  decl_netcdf:
    file: "file_name.nc"
    on_event: "event"

plugins:
  decl_netcdf:
    file: "file_name.nc"
    on_event: ["event_1", "event_2"]

when subtree

Defines the condition on which plugin will execute input/output operation on the file.

key	value
when	$-expression condition evalueated to boolean value

Configuration example:

plugins:
  decl_netcdf:
    file: "file_name.nc"
    when: "$i < 10"

groups subtree

Defines groups in the NetCDF file. Mainly used to read/write attributes values of the groups.

If group won't have any attributes, this subtree can be omitted. Decl'NetCDF plugin will create group by default for every variable.

key	value
groups	Map of group name subtree

Configuration example:

plugins:
  decl_netcdf:
    file: "file_name.nc"
    groups:
      group1:
        attributes:
          attr1: 
            value: $value1
          attr2: 
            value: $value2
      group1/group2:
        attributes:
          attr1: 
            value: $value3
          attr2: 
            value: $value4

group name subtree

key	value
name (path) of the group	Map with attribute key with value as map of attribute subtree

variables subtree

Defines variables in the NetCDF file. Mainly used to define dimensions names and read/write attributes of the variable.

1. If no dimensions names are defined, the plugin will give them arbitrary names.
2. If the data type of variable differs from the one in descriptor this subtree is mandatory. It also allows to define UNLIMITED dimension (size of dimension must be then set to 0).
3. If the variable is in group it has to be define in the variable name (/ group notation: group_name/varaibale_name)
4. This subtree can be omitted. In this case, the name of NetCDF variable will be the same as the name of the descriptor and have the same type and no attributes.

key	value
variable path	variable definition subtree

Configuration example:

plugins:
  decl_netcdf:
    file: "file_name.nc"
    variables:
      group1/group2/variable_name:
        type: array
        subtype: double
        size: [0, $value, $value] # 0 -> UNLIMITED dimension
        dimensions: ["time", "height", "width"]
        attributes:
          attr1: $value

variable definition subtree

key	value
type	type of variable (defined the same way as other types in PDI)	optional
dimensions	array of dimensions names of variable	optional
attributes	Map of attribute subtree	optional

attribute subtree

key	value
attribute name	$-expression value to be written as attribute

write subtree

The write subtree can have 2 definitions:

1. Just a name of descriptor to write or the list of descriptors names to write to file. In this case the name of NetCDF variable will be the same as the name of the descriptor and have the same type and no attributes.

   Configuration examples:

   plugins:
     decl_netcdf:
       file: "file_name.nc"
       write: data_name
   #-------------------------------------
   plugins:
     decl_netcdf:
       file: "file_name.nc"
       write: [data_name_1, data_name_2]

2. Dictionary, with key as descriptor name and value as following subtree:

key	value
when	read/write when subtree	optional
variable	variable name to write (may be defined in variables subtree)	optional
variable_selection	subtree	optional

Configuration example:

data:
  int_submatrix_11:
    type: array
    subtype: int
    size: [4, 4]
 
plugins:
  decl_netcdf:
    file: "file_name.nc"
    variables:
        integer_matrix:
          type: array
          subtype: int
          size: [8, 8]
          dimensions: ["height", "width"]
    write:
      int_submatrix_11:
        when: $i < 10
        variable: integer_matrix
        variable_selection:  # select bottom-left submatrix 4x4
          start: [4, 4]
          subsize: [4, 4]

Warning

To write a record datatype, decl_netcdf.type type attribute must be defined with compound type name.

Record write configuration example:

data:
  particle:
    type: struct
    +decl_netcdf.type: particle_xyz
    members:
      - x: double
      - y: double
      - z: double
 
plugins:
  decl_netcdf:
    file: "file_name.nc"
    write: [particle]

subtree

Defines the part of file NetCDF variable where from read/to write the data. The hyperslab will be created from given start and subsize (count) lists.

key	value
start	list specifying start index for each dimension	optional
subsize	list specifying count for each dimension	optional

read/write when subtree

Defines the condition on which plugin will read/write specific variable.

key	value
when	$-expression condition evalueated to boolean value

read subtree

The read subtree can have 2 definitions:

1. Just a name of descriptor to read or the list of descriptors names to read to file. In this case the name of NetCDF variable must be the same as the name of the descriptor and have the same type and no attributes will be read.

   Configuration examples:

   plugins:
     decl_netcdf:
       file: "file_name.nc"
       read: data_name
   #-------------------------------------
   plugins:
     decl_netcdf:
       file: "file_name.nc"
       read: [data_name_1, data_name_2]

2. Dictionary, with key as descriptor name and value as following subtree:

key	value
when	read/write when subtree	optional
variable	variable name to read (may be defined in variables subtree)	optional
variable_selection	subtree	optional

Configuration example:

data:
  int_submatrix_11:
    type: array
    subtype: int
    size: [4, 4]
 
plugins:
  decl_netcdf:
    file: "file_name.nc"
    variables:
        integer_matrix:
          type: array
          subtype: int
          size: [8, 8]
    read:
      int_submatrix_11:
        when: $i < 10
        variable: integer_matrix
        variable_selection:  # select bottom-left submatrix 4x4
          start: [4, 4]
          subsize: [4, 4]

Full yaml example

metadata:
  var_attr: float
  group1_attr: float
  group1_data_attr: float
data:
  int_submatrix_top:
    type: array
    subtype: int
    size: [4, 8]
  int_submatrix_bottom:
    type: array
    subtype: int
    size: [4, 8]
plugins:
  decl_netcdf:
    - file: "example.nc"
      on_event: "write"
      groups:
        group_1:
          attributes:
            some_attr: $group1_attr
        group_1/data:
          attributes:
            some_attr: $group1_data_attr
      variables:
        group_1/data/int_matrix:
          type: array
          subtype: int
          size: [8, 8]
          dimensions: ["height", "width"]
          attributes:
            custom_attr: $var_attr
      write: 
        int_submatrix_top:
          variable: group_1/data/int_matrix
          variable_selection:
            start: [0, 0]
            subsize: [4, 8]
        int_submatrix_bottom:
          variable: group_1/data/int_matrix
          variable_selection:
            start: [4, 0]
            subsize: [4, 8]
    - file: "example.nc"
      on_event: "read"
      groups:
        group_1/data:
          attributes:
            some_attr: $custom_attr
      variables:
        group_1/data/int_matrix:
          type: array
          subtype: int
          size: [8, 8]
          dimensions: ["height", "width"]
          attributes:
            custom_attr: $var_attr
      read: 
        int_submatrix_top:
          variable: group_1/data/int_matrix
          variable_selection:
            start: [0, 0]
            subsize: [4, 8]
        int_submatrix_bottom:
          variable: group_1/data/int_matrix
          variable_selection:
            start: [4, 0]
            subsize: [4, 8]