In this tutorial, you will build a PDI-enabled application step-by-step from a PDI-free base. You will end-up building the C version of the example provided with PDI for the the Trace, Decl'HDF5, and Pycall plugins. Additional examples are available for the other plugins.
PDI installation is documented in a dedicated page.
Once PDI is installed, you can proceed with getting the sources for the hands-on tutorial from github:
Before compilation, configure the tutorial by detecting all dependencies:
Once you have correctly modified each exercise according to instructions, you can compile it by running:
Where ? is the number of the exercise.
You can run each exercise with the following command:
Where ? is the number of the exercise and 4 represents the number of MPI processes to use.
To store the logs for later comparison, you can use the following command (for example for ex2.):
Now you're ready to work, good luck!
Ex1. implements a simple heat equation solver using an explicit forward finite difference scheme parallelized with MPI. The code uses a block domain decomposition where each process holds a 2D block of data.
Locally, each process holds its local block of data with one additional element on each side for ghost zones.
In the following exercises however, PDI will only be used to decouple I/O operations. There is no need to fully dive in the core of the solver described in the PDI example algorithm and implemented in the iter and exchange functions.
The specification tree in the .yml files and the main function are the locations where all the I/O-related aspects will be handled and the only ones you will actually need to fully understand or modify.
This example gets its configuration from a file in the YAML format: ex1.yml file. If you're not familiar with YAML, please have a look at our quick YAML format documentation to understand it. The example uses the paraconf library to read this file.
Ex2. is the same code as that of ex1. with PDI calls added in main function. In our YAML file (ex2.yml), a new sub-tree has been added under the pdi key. This sub-tree is the PDI specification tree passed to PDI at initialization. Here, the PDI Trace plugin is used to trace PDI calls.
In this exercise, the code is the same as in ex2. No need to touch the C code here, modification of the YAML file (ex3.yml) should be enough.
The Trace plugin (trace) was replaced by the Decl'HDF5 plugin (decl_hdf5) in the specification tree. In its configuration, the dsize variable is written.
To achieve this result, you will need to fill 2 sections in the YAML file.
In this exercise each MPI process will write its local 2D array block contained in the main_field variable to a separate HDF5 file. Once again, this can be done by modifying the YAML file only, no nee to touch the C file.
Look at the number of blocks, you will have to use the correct number of MPI ranks to run the example.
Notice that in the YAML file, a list was used in the decl_hdf5 section with multiple write blocks instead of a single one as before in order to write to multiple files.
Also notice that this example now runs in parallel with two processes. Therefore it uses "$-expressions" to specify the file names and ensure we do not write to the same file from distinct ranks.
Unlike the other fields manipulated until now, the type of main_field is not fully known, its size is dynamic. By moving other fields in the metadata section, you can reference them from "$-expressions" in the configuration file.
Unlike the other fields manipulated until now, main_field is exposed multiple times along execution. In order not to overwrite it every time it is exposed, you need to add a when condition to restrict its output.
In ex4., two variables were written to ex4-data*.h5, but the file was opened and closed for each and every write. Since Decl'HDF5 only sees the data appear one after the other, it does not keep the file open. Since ii and main_field are shared in an interlaced way, they are both available to PDI at the same time and could be written without opening the file twice. You have to use events for that, you will modify both the C and YAML file.
As you can notice, the PDI code is quite redundant. In this exercise, you will use PDI_expose and PDI_multi_expose to simplify the code while keeping the exact same behaviour. For once, there is no need to modify the YAML file here, you only need to modify the C file in this exercise.
There are lots of matched PDI_share/PDI_reclaim in the code.
This replacement is not possible for interlaced PDI_share/PDI_reclaim with events in the middle. This case is however handled by PDI_multi_expose call that exposes all data, then triggers an event and finally does all the reclaim in reverse order.
In this exercise, you will only write a selection of the 2D array in memory excluding ghosts to the HDF5 file. Once again, you only need to modify the YAML file in this exercise, no need to touch the C file.
As you can notice, now the dataset is independently described in the file.
You can achieve this by using the memory_selection directive that specifies the selection of data from memory to write.
In this exercise, you will once again change the YAML file to handle a selection in the dataset in addition to the selection in memory from the previous exercise. You will write the 2D array from the previous exercise as a slice of 3D dataset including a dimension for time. Once again, you only need to modify the YAML file in this exercise, no need to touch the C file.
Notice how the dataset is now extended with an additional dimension for three time-steps.
You can achieve this by using the dataset_selection directive that specifies the selection where to write in the file dataset.
Running the code from the previous exercises in parallel should already work and yield one file per process containing the local data block. In this exercise you will write one single file with parallel HDF5 whose content should be independent from the number of processes used. Once again, you only need to modify the YAML file in this exercise, no need to touch the C file.
The mpi plugin was loaded to make sharing MPI communicators possible.
Match the output from ex9.h5dump, that should be independent from the number of processes used. You can easily check if the files are the same by running:
In this exercise, you will once again modify the YAML file only and use python to post-process the data in situ before writing it to HDF5. Here, you will write the square root of the raw data to HDF5 instead of the data itself.
Notice that the Decl'HDF5 configuration was simplified, no memory selection is applied, the when condition disappeared. The dataset name is however explicitly specified now because it does not match the PDI variable name anymore, you will instead write a new variable exposed from python.
The pycall section has been added to load the Pycall plugin. It executes the provided code when the "loop" event is triggered. The with section specifies the variables (parameters) to pass to Python as a set of "$-expressions". The provided code again exposes its result to PDI and multiple blocks can be chained this way.
In this tutorial, you used the C API of PDI and from YAML, you used the Trace, Decl'HDF5, and Pycall plugins.
If you want to try PDI from another language (Fortran, python, ...) or if you want to experiment with other PDI plugins, have a look at the examples provided with PDI.