diff --git a/README.md b/README.md index eaffeca..48140b1 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,45 @@ -# PnetCDF python +# PnetCDF for Python ![](https://img.shields.io/badge/python-v3.9-blue) ![](https://img.shields.io/badge/tests%20passed-49-brightgreen) ![](https://readthedocs.org/projects/pnetcdf-python/badge/?version=latest) -PnetCDF-python is a Python interface to -[PnetCDF](https://parallel-netcdf.github.io/), a high-performance parallel I/O -library for accessing netCDF files. -This package allows Python users to access netCDF data using the rich ecosystem -of Python's scientific computing libraries, making it a valuable tool for -applications that require parallel access to netCDF files. +PnetCDF-Python is a Python interface to +[PnetCDF](https://parallel-netcdf.github.io/), a high-performance I/O library +for accessing netCDF files in parallel. It can provide MPI-based parallel +python programs to achieve a scalable I/O performance. ### Software Dependencies -* Python 3.9 or above -* MPI libraries -* PnetCDF [C library](https://github.com/Parallel-netCDF/PnetCDF), built with shared libraries. -* Python library [mpi4py](https://mpi4py.readthedocs.io/en/stable/install.html) -* Python library [numpy](http://www.numpy.org/) +* Python 3.9 or later. +* [numpy](http://www.numpy.org/) Python package. +* MPI C library and Python package, [mpi4py](https://mpi4py.readthedocs.io/en/stable/install.html). +* [PnetCDF C library](https://github.com/Parallel-netCDF/PnetCDF), built with shared libraries. ### Developer Installation * Clone this GitHub repository -* Make sure the above dependent software are installed. -* In addition, [Cython](http://cython.org/), [packaging](https://pypi.org/project/packaging/), [setuptools>=65](https://pypi.org/project/setuptools/) and [wheel](https://pypi.org/project/wheel/) are required for developer installation. -* Set the environment variable `PNETCDF_DIR` to PnetCDF's installation path. -* Make sure utility program `pnetcdf-config` is available in `$PNETCDF_DIR/bin`. -* Run command below to install. +* Required software for developer installation: + + The above mentioned dependent software are installed and additionally, + + [Cython](http://cython.org/), [packaging](https://pypi.org/project/packaging/), [setuptools>=65](https://pypi.org/project/setuptools/) and [wheel](https://pypi.org/project/wheel/). +* Commands to install. ``` - CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir pip install --no-build-isolation -e . + export CC=/path/to/mpicc + export PNETCDF_DIR=/path/to/pnetcdf/dir + pip install --no-build-isolation -e . ``` -* Testing - + Run command `"make check"` to test all the programs available in folders - ["test/"](./test) and ["examples/"](./examples) in parallel on 4 MPI - processes. - + In addition, command `"make ptests"` runs the same tests using 3, 4, and 8 - MPI processes. +* Testing -- Command `"make check"` tests all the programs available in folders + ["test/"](./test) and ["examples/"](./examples). ### Additional Resources +* [Example python programs](./examples#pnetcdf-python-examples) available in + folder [./examples](./examples). * PnetCDF-python [User Guide](https://pnetcdf-python.readthedocs.io/en/latest) * [Data objects](docs/pnetcdf_objects.md) in PnetCDF python programming * [Comparison](docs/nc4_vs_pnetcdf.md) of NetCDF4-python and PnetCDF-python * [PnetCDF project home page](https://parallel-netcdf.github.io) -* [PnetCDF repository of C/Fortran library](https://github.com/Parallel-NetCDF/PnetCDF) +* [PnetCDF of C/Fortran library repository](https://github.com/Parallel-NetCDF/PnetCDF) ### Developer Team * Youjia Li <> -* Wei-keng Liao <> (Principle Investigator) +* Wei-keng Liao <> ### Acknowledgements Ongoing development and maintenance of PnetCDF-python is supported by the U.S. diff --git a/docs/copyright.rst b/docs/copyright.rst new file mode 100644 index 0000000..ef12b1c --- /dev/null +++ b/docs/copyright.rst @@ -0,0 +1,41 @@ +================ +Copyright Statement +================ + +:: + + Copyright (c) 2024 Northwestern University and Argonne National + Laboratory All rights reserved. + + Portions of this software were developed by the Unidata Program at the + University Corporation for Atmospheric Research. + + Access and use of this software shall impose the following obligations and + understandings on the user. The user is granted the right, without any fee + or cost, to use, copy, modify, alter, enhance and distribute this software, + and any derivative works thereof, and its supporting documentation for any + purpose whatsoever, provided that this entire notice appears in all copies + of the software, derivative works and supporting documentation. Further, + Northwestern University and Argonne National Laboratory request that the + user credit Northwestern University and Argonne National Laboratory in any + publications that result from the use of this software or in any product + that includes this software. The names Northwestern University and Argonne + National Laboratory, however, may not be used in any advertising or + publicity to endorse or promote any products or commercial entity unless + specific written permission is obtained from Northwestern University and + Argonne National Laboratory. The user also understands that Northwestern + University and Argonne National Laboratory are not obligated to provide the + user with any support, consulting, training or assistance of any kind with + regard to the use, operation and performance of this software nor to + provide the user with any updates, revisions, new versions or "bug fixes." + + THIS SOFTWARE IS PROVIDED BY NORTHWESTERN UNIVERSITY AND ARGONNE NATIONAL + LABORATORY "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT + NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A + PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL NORTHWESTERN + UNIVERSITY AND ARGONNE NATIONAL LABORATORY BE LIABLE FOR ANY SPECIAL, + INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM + LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE + OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE ACCESS, + USE OR PERFORMANCE OF THIS SOFTWARE. + diff --git a/docs/nc4_vs_pnetcdf.md b/docs/nc4_vs_pnetcdf.md index 507a5fb..b9f3c5e 100644 --- a/docs/nc4_vs_pnetcdf.md +++ b/docs/nc4_vs_pnetcdf.md @@ -1,7 +1,14 @@ -# Difference between NetCDF4-python and PnetCDF-python +# Comparison between PnetCDF-Python and NetCDF4-Python + +Programming using [NetCDF4-Python](http://unidata.github.io/netcdf4-python/) and +[PnetCDF-Python](https://pnetcdf-python.readthedocs.io) are very similar. +Below lists some of the differences, including the file format support and +operational modes. * [Supported File Formats](#supported-file-formats) * [Differences in Python Programming](#differences-in-python-programming) +* [Define Mode and Data Mode](#define-mode-and-data-mode) +* [Collective and Independent I/O Mode](#collective-and-independent-io-mode) * [Blocking vs. Nonblocking APIs](#blocking-vs-nonblocking-apis) --- @@ -59,6 +66,215 @@ | ... || | # close file
f.close() | ditto NetCDF4 | +--- +## Define Mode and Data Mode + +In PnetCDF, an opened file is in either define mode or data mode. Switching +between the modes is done by explicitly calling `"pnetcdf.File.enddef()"` and +`"pnetcdf.File.redef()"`. NetCDF4-Python has no such mode switching +requirement. The reason of PnetCDF enforcing such a requirement is to ensure +the metadata consistency across all the MPI processes and keep the overhead of +metadata synchronization small. + +* Define mode + + When calling constructor of python class `"pnetcdf.File()"` to create a new + file, the file is automatically put in the define mode. While in the + define mode, the python program can create new dimensions, i.e. instances + of class `"pnetcdf.Dimension"`, new variables, i.e. instances of class + `"pnetcdf.Variable"`, and netCDF attributes. Modification of these data + objects' metadata can only be done when the file is in the define mode. + + When opening an existing file, the opened file is automatically put in the + data mode. To add or modify the metadata, a python program must call + `"pnetcdf.File.redef()"`. + +* Data mode + + Once the creation or modification of metadata is complete, the python + program must call `"pnetcdf.File.enddef()"` to leave the define mode and + enter the data mode. + + While an open file is in data mode, the python program can make read and + write requests to that variables that have been created. + +
    +
  • A PnetCDF-Python example shows switching between define and data modes + after creating a new file.
    • +
  • + Example code fragment (click to expand) + +```python + import pnetcdf + ... + # Create the file + f = pnetcdf.File(filename, 'w', "NC_64BIT_DATA", MPI.COMM_WORLD) + ... + # Define dimensions + dim_y = f.def_dim("Y", 16) + dim_x = f.def_dim("X", 32) + + # Define a 2D variable of integer type + var = f.def_var("grid", pnetcdf.NC_INT, (dim_y, dim_x)) + + # Add an attribute of string type to the variable + var.str_att_name = "example attribute" + + # Exit the define mode + f.enddef() + + # Write to a subarray of the variable, var + var[4:8, 20:24] = buf + + # Re-enter the define mode + f.redef() + + # Define a new 2D variable of float type + var_flt = f.def_var("temperature", pnetcdf.NC_FLOAT, (dim_y, dim_x)) + + # Exit the define mode + f.enddef() + + # Write to a subarray of the variable, var_flt + var_flt[0:4, 16:20] = buf_flt + + # Close the file + f.close() +``` +
    • + +
  • An example shows switching between define and data modes after opening an existing file. +
    • +
  • + Example code fragment (click to expand) + +```python + import pnetcdf + ... + # Opening an existing file + f = pnetcdf.File(filename, 'r', MPI.COMM_WORLD) + ... + # get the python handler of variable named 'grid', a 2D variable of integer type + var = f.variables['grid'] + + # Read the variable's attribute named "str_att_name" + str_att = var.str_att_name + + # Read a subarray of the variable, var + r_buf = np.empty((4, 4), var.dtype) + r_buf = var[4:8, 20:24] + + # Re-enter the define mode + f.redef() + + # Define a new 2D variable of double type + var_dbl = f.def_var("precipitation", pnetcdf.NC_DOUBLE, (dim_y, dim_x)) + + # Add an attribute of string type to the variable + var_dbl.unit = "mm/s" + + # Exit the define mode + f.enddef() + + # Write to a subarray of the variable, temperature + var_dbl[0:4, 16:20] = buf_dbl + + # Close the file + f.close() +``` +
    • +
+ + +--- +## Collective and Independent I/O Mode + +The terminology of collective and independent I/O comes from MPI standard. A +collective I/O function call requires all the MPI processes opening the same +file to participate. On the other hand, an independent I/O function can be +called by an MPI process independently from others. + +For metadata I/O, both PnetCDF and NetCDF4 require the function calls to be +collective. + +* Mode Switch Mechanism + + PnetCDF-Python -- when a file is in the data mode, it can be put into + either collective or independent I/O mode. The default mode is collective + I/O mode. Switching to and exiting from the independent I/O mode is done + by explicitly calling `"pnetcdf.File.begin_indep()"` and + `"pnetcdf.File.end_indep()"`. + + + NetCDF4-Python -- collective and independent mode switching is done per + variable basis. Switching mode is done by explicitly calling + `"Variable.set_collective()"` before accessing the variable. + For more information, see + [NetCDF4-Python User Guide on Parallel I/O](https://unidata.github.io/netcdf4-python/#parallel-io) + +
    +
  • A PnetCDF-Python example shows switching between collective and + independent I/O modes.
    • +
  • + Example code fragment (click to expand) + +```python + import pnetcdf + ... + # Create the file + f = pnetcdf.File(filename, 'w', "NC_64BIT_DATA", MPI.COMM_WORLD) + ... + # Metadata operations to define dimensions and variables + ... + # Exit the define mode (by default, in the collective I/O mode) + f.enddef() + + # Write to variables collectively + var_flt[start_y:end_y, start_x:end_x] = buf_flt + var_dbl[start_y:end_y, start_x:end_x] = buf_dbl + + # Leaving collective I/O mode and entering independent I/O mode + f.begin_indep() + + # Write to variables independently + var_flt[start_y:end_y, start_x:end_x] = buf_flt + var_dbl[start_y:end_y, start_x:end_x] = buf_dbl + + # Close the file + f.close() +``` +
    • +
+ +
    +
  • A NetCDF4-Python example shows switching between collective and + independent I/O modes.
    • +
  • + Example code fragment (click to expand) + +```python + import netCDF4 + ... + # Create the file + f = netCDF4.File(filename, 'w', "NC_64BIT_DATA", MPI.COMM_WORLD, parallel=True) + ... + # Metadata operations to define dimensions and variables + ... + + # Write to variables collectively + var_flt.set_collective(True) + var_flt[start_y:end_y, start_x:end_x] = buf_flt + + var_dbl.set_collective(True) + var_dbl[start_y:end_y, start_x:end_x] = buf_dbl + + # Write to variables independently + var_flt.set_collective(False) + var_flt[start_y:end_y, start_x:end_x] = buf_flt + + var_dbl.set_collective(False) + var_dbl[start_y:end_y, start_x:end_x] = buf_dbl + + # Close the file + f.close() +``` +
    • +
+ --- ## Blocking vs Nonblocking APIs diff --git a/docs/source/api/dimension_api.rst b/docs/source/api/dimension_api.rst index fb22d2f..65e5c29 100644 --- a/docs/source/api/dimension_api.rst +++ b/docs/source/api/dimension_api.rst @@ -1,30 +1,30 @@ -.. currentmodule:: pnetcdf ============== Dimension ============== -Dimension defines the shape and structure of variables and stores -coordinate data for multidimensional arrays. The ``Dimension`` object, -which is also a key component of ``File`` class, provides an interface -to access dimensions. - -.. note:: - - ``Dimension`` instances should be created using the :meth:`File.def_dim` method of a ``File`` instance, - not using :meth:`Dimension.__init__` directly. +Dimension defines the shape and structure of variables and stores coordinate +data for multidimensional arrays. The ``Dimension`` object, which is also a key +component of ``File`` class, provides an interface to access dimensions. .. autoclass:: pnetcdf::Dimension :members: getfile, isunlimited :exclude-members: name, size -Dimension Attributes - The following class members are read-only and should not be modified by the user. +Read-only Python Attributes of Dimension Class + The following class members are read-only and should not be modified by the + user. .. attribute:: name String name of Dimension instance. This class member is read-only and - should not be modified by the user. To rename a dimension, use :meth:`File.rename_dim` method. + should not be modified by the user. To rename a dimension, use + :meth:`File.rename_dim` method. + + **Type:** `str` .. attribute:: size - - The current size of Dimension (calls ``len`` on Dimension instance). \ No newline at end of file + + The current size of Dimension (calls ``len`` on Dimension instance). + + **Type:** `int` + diff --git a/docs/source/api/file_api.rst b/docs/source/api/file_api.rst index ec80701..ca127c1 100644 --- a/docs/source/api/file_api.rst +++ b/docs/source/api/file_api.rst @@ -10,32 +10,43 @@ relations among data fields stored in a netCDF file. .. autoclass:: pnetcdf::File :members: __init__, close, filepath, redef, enddef, begin_indep, end_indep, - sync, flush, def_dim, rename_var, rename_dim, def_var, ncattrs, put_att, get_att, - del_att, rename_att, wait, wait_all, cancel, attach_buff, detach_buff, set_fill, - inq_buff_usage, inq_buff_size, inq_num_rec_vars, inq_num_fix_vars, inq_striping, - inq_recsize, inq_version, inq_info, inq_header_size, inq_put_size, inq_header_extent, - inq_nreqs - :exclude-members: indep_mode, path + sync, flush, def_dim, rename_var, rename_dim, def_var, ncattrs, put_att, + get_att, del_att, rename_att, wait, wait_all, cancel, attach_buff, + detach_buff, set_fill, inq_buff_usage, inq_buff_size, inq_num_rec_vars, + inq_num_fix_vars, inq_striping, inq_recsize, inq_version, inq_info, + inq_header_size, inq_put_size, inq_header_extent, inq_nreqs + :exclude-members: dimensions, variables, file_format, libver, indep_mode, + path -File Attributes - The following class members are read-only and should not be modified by the user. +Read-only Python Attributes of File Class + The following class members are read-only and should not be modified by the + user. - .. attribute:: dimensions + .. attribute:: dimensions - The dimensions dictionary maps the names of dimensions defined for the file - to instances of the ``pnetcdf.Dimension`` class. + The dimensions dictionary maps the names of dimensions defined in this + file as an instance of the ``pnetcdf.Dimension`` class. - .. attribute:: variables - - The variables dictionary maps the names of variables defined for this file - to instances of the ``pnetcdf.Variable`` class. + **Type:** `dict` - .. attribute:: file_format - - The file format in string of the netCDF file. Possible values include: "CLASSIC", "CDF2", - "64BIT_OFFSET", "64BIT", "CDF5", "64BIT_DATA", "NETCDF4" and "BP". + .. attribute:: variables - + The variables dictionary maps the names of variables defined in this file + as an instance of the ``pnetcdf.Variable`` class. + **Type:** `dict` + .. attribute:: file_format + + The file format of the netCDF file. Possible values are one of the + following strings. "CLASSIC", "CDF2", "64BIT_OFFSET", "64BIT", "CDF5", + "64BIT_DATA", "NETCDF4" and "BP". + + **Type:** `str` + + .. attribute:: libver + + The PnetCDF-Python version string. + + **Type:** `str` diff --git a/docs/source/api/function_api.rst b/docs/source/api/function_api.rst index 71206fb..e6111a6 100644 --- a/docs/source/api/function_api.rst +++ b/docs/source/api/function_api.rst @@ -9,5 +9,3 @@ Utility Functions .. autofunction:: pnetcdf::set_default_format .. autofunction:: pnetcdf::inq_default_format .. autofunction:: pnetcdf::inq_file_format -.. autofunction:: pnetcdf::inq_malloc_max_size -.. autofunction:: pnetcdf::inq_malloc_size diff --git a/docs/source/api/variable_api.rst b/docs/source/api/variable_api.rst index 80e567d..350906b 100644 --- a/docs/source/api/variable_api.rst +++ b/docs/source/api/variable_api.rst @@ -1,74 +1,76 @@ -.. currentmodule:: pnetcdf ========= Variable ========= -Variable is a core component of a netCDF file representing an array -of data values organized along one or more dimensions, with associated -metadata in the form of attributes. The ``Variable`` object in the library -provides operations to read and write the data and metadata of a variable -within a netCDF file. Particularly, data mode operations have a flexible -interface, where reads and writes can be done through either explicit -function-call style methods or indexer-style (numpy-like) syntax. +Variable is a core component of a netCDF file representing an array of data +values organized along one or more dimensions, with associated metadata in the +form of attributes. The ``Variable`` object in the library provides operations +to read and write the data and metadata of a variable within a netCDF file. +Particularly, data mode operations have a flexible interface, where reads and +writes can be done through either explicit function-call style methods or +indexer-style (numpy-like) syntax. .. autoclass:: pnetcdf::Variable - :members: ncattrs, put_att, get_att, del_att, rename_att, get_dims, def_fill, - inq_fill, fill_rec, set_auto_chartostring, put_var, put_var_all, get_var, - get_var_all, iput_var, bput_var iget_var, inq_offset - :exclude-members: name, shape, size + :members: ncattrs, put_att, get_att, del_att, rename_att, get_dims, + def_fill, inq_fill, fill_rec, set_auto_chartostring, put_var, put_var_all, + get_var, get_var_all, iput_var, bput_var iget_var, inq_offset + :exclude-members: name, dtype, datatype, shape, ndim, size, dimensions, + chartostring +Read-only Python Attributes of Variable Class + The following class members are read-only and should not be modified + directly by the user. -Variable Attributes - The following class members are read-only and should not be modified by the user. + .. attribute:: name - .. attribute:: name + The string name of Variable instance - The string name of Variable instance - **Type:** `str` + **Type:** `str` - .. attribute:: dtype + .. attribute:: dtype + A numpy data type of the variable. - Return the mapped numpy data type of the variable netCDF datatype. A numpy dtype object - describing the variable's data type. - **Type:** ``numpy.dtype`` + **Type:** ``numpy.dtype`` - .. attribute:: datatype + .. attribute:: datatype + Same as :meth:`Variable.dtype`. - Same as :meth:`Variable.dtype`. - **Type:** ``numpy.dtype`` + **Type:** ``numpy.dtype`` - .. attribute:: shape + .. attribute:: shape + The shape of the variable, which is the current sizes of all variable + dimensions - Return the shape of the variable, which is the current sizes of all variable - dimensions - **Type:** `Tuple[int, int]` + **Type:** `tuple of ints` - .. attribute:: ndim + .. attribute:: ndim + The number of variable dimensions. - The number of variable dimensions. - **Type:** `int` + **Type:** `int` - .. attribute:: size + .. attribute:: size + Return the number of stored elements - Return the number of stored elements - **Type:** `int` + **Type:** `int` - .. attribute:: dimensions + .. attribute:: dimensions + Return the variable's dimension names - Return the variable's dimension names - **Type:** `List[str]` + **Type:** `list of str` - .. attribute:: chartostring + .. attribute:: chartostring - If `True`, data is automatically converted to/from character - arrays to string arrays when the `_Encoding` variable attribute is set. - Default is `True`, can be reset using :meth:`Variable.set_auto_chartostring` method. - **Type:** `bool` + If `True`, data is automatically converted to/from character arrays to + string arrays when the `_Encoding` variable attribute is set. Default + is `True`, can be reset using :meth:`Variable.set_auto_chartostring` + method. + + **Type:** `bool` diff --git a/docs/source/conf.py b/docs/source/conf.py index 51f30aa..4fba7ba 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -7,8 +7,8 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information project = 'PnetCDF Python' -copyright = '2023, CUCIS' -author = 'CUCIS' +copyright = '2024, Northwestern University and Argonne National Laboratory' +author = 'PnetCDF-Python Developer Team' release = '0.1.0' # -- General configuration --------------------------------------------------- @@ -56,4 +56,4 @@ def autodoc_skip_member(app, what, name, obj, skip, options): return True if exclude else None def setup(app): - app.connect('autodoc-skip-member', autodoc_skip_member) \ No newline at end of file + app.connect('autodoc-skip-member', autodoc_skip_member) diff --git a/docs/source/index.rst b/docs/source/index.rst index c07ae6b..de5dfaf 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -28,7 +28,6 @@ PnetCDF-Python Documentation tutorial/non_blocking tutorial/compare_netcdf4 - .. toctree:: :maxdepth: 3 :caption: API Documentation @@ -38,6 +37,12 @@ PnetCDF-Python Documentation api/variable_api api/function_api +.. toctree:: + :maxdepth: 2 + :caption: Copyright Statement + + copyright + .. #.. toctree:: .. # :maxdepth: 2 .. # :caption: Development diff --git a/docs/source/tutorial/compare_netcdf4.rst b/docs/source/tutorial/compare_netcdf4.rst index 015efa8..30f00b9 100644 --- a/docs/source/tutorial/compare_netcdf4.rst +++ b/docs/source/tutorial/compare_netcdf4.rst @@ -1,6 +1,6 @@ .. currentmodule:: pnetcdf ================================= -Comparing with netCDF4-python API +Comparing with netCDF4-python ================================= PnetCDF-python programming in a way is very similar to netCDF4-python. diff --git a/src/pnetcdf/_Dimension.pyx b/src/pnetcdf/_Dimension.pyx index 7be4845..b0c12a6 100644 --- a/src/pnetcdf/_Dimension.pyx +++ b/src/pnetcdf/_Dimension.pyx @@ -16,16 +16,27 @@ include "PnetCDF.pxi" cdef class Dimension: def __init__(self, File file, name, size=-1, **kwargs): - self._dimid = 0 - self._file = file """ - **`__init__(self, File file, name, Offset size = None, **kwargs)`** - `Dimension` constructor. - **`file`**: `File` instance to associate with dimension. - **`name`**: Name of the dimension. - **`size`**: Size of the dimension. -1 means unlimited. (Default `-1`). - ***Note***: `Dimension` instances should be created using the - `Dataset.def_dim` method of a `File` instance, not using `Dimension.__init__` directly. + __init__(self, File file, name, size=-1, **kwargs) + + The constructor for :class:`pnetcdf.Dimension`. + + :param file: An :class:`pnetcdf.File` instance to associate with dimension. + :type file: :class:`pnetcdf.File` + + :param name: Name of the new dimension. + :type name: str + + :param size: Length of the dimension. ``-1`` means to create the + expandable dimension. (Default ``-1``). + :type size: int + + :return: The created dimension instance. + :rtype: :class:`pnetcdf.Dimension` + + .. note:: ``Dimension`` instances should be created using the + :meth:`File.def_dim` method of a ``File`` instance, not using + :meth:`Dimension.__init__` directly. """ cdef int ierr cdef char *dimname @@ -33,6 +44,8 @@ cdef class Dimension: self._file_id = file._ncid self._file_format = file.file_format self._name = name + self._dimid = 0 + self._file = file if 'id' in kwargs: self._dimid = kwargs['id'] @@ -97,7 +110,8 @@ cdef class Dimension: """ getfile(self) - Return the file that this ``Dimension`` is a member of. + :return: the ``pnetcdf.File`` instance that this ``Dimension`` is a + member of. :rtype: :class:`pnetcdf.File` """ @@ -107,7 +121,8 @@ cdef class Dimension: """ isunlimited(self) - Returns `True` if the ``Dimension`` instance is unlimited, ``False`` otherwise. + :return: ``True`` if this ``Dimension`` instance is unlimited, ``False`` + otherwise. :rtype: bool """ diff --git a/src/pnetcdf/_File.pyx b/src/pnetcdf/_File.pyx index c64694c..940e5e5 100644 --- a/src/pnetcdf/_File.pyx +++ b/src/pnetcdf/_File.pyx @@ -36,30 +36,39 @@ cdef class File: The constructor for :class:`pnetcdf.File`. - :param filename: Name of the new file. - :type filename: str + :param str filename: Name of the new file. - :param mode: Access mode. - - ``r``: Open a file for reading, error if the file does not exist. - - ``w``: Create a file, an existing file with the same name is deleted. - - ``x``: Create the file, returns an error if the file exists. - - ``a`` and ``r+``: append, creates the file if it does not exist. + :param str mode: Access mode. - :type mode: str + - ``r``: Open a file for read-only mode. An error will return if + the file does not exist. + - ``w``: Create a new file. If a file with the same name has + already existed, the file will be clobbered to zero size. + - ``x``: Create a new file, but return an error if the file has + already existed. + - ``a`` or ``r+``: append mode. It creates the file if it does not + exist. - :param format: underlying file format. Only relevant when creating file + :param str format: [Optional] underlying file format. Only relevant + when creating a new file. - ``NETCDF3_64BIT_OFFSET`` or ``NC_64BIT_OFFSET``: CDF-2 format - ``NETCDF3_64BIT_DATA`` or ``NC_64BIT_DATA``: CDF-5 format - - ``NETCDF3_CLASSIC`` or `None` defaults to default file format (CDF-1 format) + - ``NETCDF3_CLASSIC`` or `None` defaults to default file format + (CDF-1 format) - :type format: str - - :param comm: [Optional] MPI communicator to use for file access. `None` defaults to MPI_COMM_WORLD. + :param comm: [Optional] + MPI communicator to use for file access. `None` defaults to + ``MPI_COMM_WORLD``. :type comm: mpi4py.MPI.Comm or None - :param info: [Optional] MPI info object to use for file access. `None` defaults to MPI_INFO_NULL. + :param info: [Optional] + MPI info object to use for file access. `None` defaults to + ``MPI_INFO_NULL``. :type info: mpi4py.MPI.Info or None + + :return: The created file instance. + :rtype: :class:`pnetcdf.File` """ cdef int ncid encoding = sys.getfilesystemencoding() @@ -131,13 +140,19 @@ cdef class File: _check_err(ierr) self._isopen = 0 # indicates file already closed, checked by __dealloc__ - def filepath(self,encoding=None): + def filepath(self, encoding=None): """ - filepath(self,encoding=None) + filepath(self, encoding=None) + + Method to return the file system path which was used to open/create the + Dataset. The path is decoded into a string using + `sys.getfilesystemencoding()` by default. - Get the file system path which was used to open/create the Dataset. - The path is decoded into a string using `sys.getfilesystemencoding()` by default. + :param str encoding: [Optional] + character encoding of a string attribute (default is `utf-8`). + :return: The file path + :rtype: str """ cdef int ierr cdef int pathlen @@ -186,13 +201,16 @@ cdef class File: """ redef(self) - Enter define mode, so that dimensions, variables, and attributes can be added or - renamed and attributes can be deleted + Enter define mode, so that dimensions, variables, and attributes can be + added or renamed and attributes can be deleted - .. note:: This is a compulsory call before running define-mode operations. ``NetCDF4-python`` library automatically - switches between data and define mode for the user by calling ``redef`` and ``enddef`` internally at the define-mode - operations. This feature is NOT implemented in ``pnetcdf-python`` due to additional switching overheads between consecutive - define operations and potential hazards of hanging when performing independent I/O on multiple processes. + .. note:: In PnetCDF implementation, a file mode can be in either + define or data mode. While in the define mode, metadata can be + created and modified, for instance, creating new dimension, + variables, and attributes. Variables that have been defined can + only be read and written while in the data mode. This requirement + is to guarantee the data consistency when running application + programs in parallel. """ self._redef() @@ -202,17 +220,21 @@ cdef class File: with nogil: ierr = ncmpi_redef(fileid) _check_err(ierr) + def enddef(self): """ enddef(self) - Exit define mode. The netCDF file is then placed in data mode, so variable data can be - read or written. + Method to exit the current file's mode from define mode and place the + file in data mode, so variable data can be read or written. - .. note:: This is a compulsory call before data-mode operations. ``NetCDF4-python`` library automatically - switches between data and define mode for the user by calling ``redef`` and ``enddef`` internally at the define-mode - operations. This feature is NOT implemented in ``pnetcdf-python`` due to additional switching overheads between consecutive - define operations and potential hazards of hanging when performing independent I/O on multiple processes. + .. note:: In PnetCDF implementation, a file mode can be in either + define or data mode. While in the define mode, metadata can be + created and modified, for instance, creating new dimension, + variables, and attributes. Variables that have been defined can + only be read and written while in the data mode. This requirement + is to guarantee the data consistency when running application + programs in parallel. """ self._enddef() @@ -227,8 +249,9 @@ cdef class File: """ begin_indep(self) - The file leaves from collective data mode and enters into independent data mode. The default mode is collective data mode. - + Method to leave the file's current mode from collective data mode and + enter into independent data mode. The default mode is collective data + mode. """ cdef int ierr cdef int fileid = self._ncid @@ -241,7 +264,9 @@ cdef class File: """ end_indep(self) - This file leaves from independent data mode and enters into collective data mode. The default mode is collective data mode. + Method to leave the file's current mode from independent data mode and + enter into collective data mode. The default mode is collective data + mode. """ cdef int ierr cdef int fileid = self._ncid @@ -267,19 +292,17 @@ cdef class File: """ def_dim(self, dimname, size=-1) - Creates a new dimension with the given `dimname` and `size`. - `size` must be a positive integer or `-1`, which stands for - "unlimited" (default is `-1`). The return value is the `Dimension` - class instance describing the new dimension. To determine the current - maximum size of the dimension, use the `len` function on the `Dimension` - instance. To determine if a dimension is 'unlimited', use the - `Dimension.isunlimited` method of the `Dimension` instance. + Creates a new dimension with the given `dimname` and `size`. `size` + must be a positive integer or `-1`, which stands for "unlimited" + (default is `-1`). The return value is the `Dimension` class instance + describing the new dimension. To determine the current maximum size of + the dimension, use the `len` function on the `Dimension` instance. To + determine if a dimension is 'unlimited', use the + :meth:`Dimension.isunlimited` method of the `Dimension` instance. - :param dimname: Name of the new dimension. - :type dimname: str + :param str dimname: Name of the new dimension. - :param size: Size of the new dimension - :type size: int + :param int size: [Optional] Size of the new dimension. """ self.dimensions[dimname] = Dimension(self, dimname, size=size) return self.dimensions[dimname] @@ -288,7 +311,7 @@ cdef class File: """ createDimension(self, dimname, size=-1) - Same as ``pnetcdf.File.def_dim`` + Same as :meth:`File.def_dim` """ return self.def_dim(dimname, size) @@ -298,17 +321,17 @@ cdef class File: Rename a `Variable` named `oldname` to `newname` - :param oldname: Old name of the variable. - :type oldname: str + :param str oldname: Old name of the variable. - :param newname: New name of the variable. - :type newname: str + :param str newname: New name of the variable. - Operational mode: this method is collective subroutine, argument new name must - be consistent among all calling processes. If the new name is longer than the old name, - then the netCDF file must be in define mode. Otherwise, the netCDF file can be in either - define or data mode + :Operational mode: this method is collective subroutine, argument + new name must be consistent among all calling processes. If the new + name is longer than the old name, then the netCDF file must be in + define mode. Otherwise, the netCDF file can be in either define or + data mode. """ + cdef char *namstring cdef Variable var cdef int _file_id, _varid @@ -332,7 +355,7 @@ cdef class File: """ renameVariable(self, oldname, newname) - Same as ``pnetcdf.File.rename_var`` + Same as :meth:`File.rename_var` """ self.rename_var(oldname, newname) @@ -342,16 +365,15 @@ cdef class File: Rename a ``Dimension`` named `oldname` to `newname` - :param oldname: Old name of the dimension. - :type oldname: str + :param str oldname: Old name of the dimension. - :param newname: New name of the dimension. - :type newname: str + :param str newname: New name of the dimension. - Operational mode: this method is collective subroutine, argument new name must - be consistent among all calling processes. If the new name is longer than the old name, - then the netCDF file must be in define mode. Otherwise, the netCDF file can be in either - define or data mode + :Operational mode: this method is collective subroutine, argument new + name must be consistent among all calling processes. If the new + name is longer than the old name, then the netCDF file must be in + define mode. Otherwise, the netCDF file can be in either define or + data mode. """ cdef char *namstring cdef Variable var @@ -376,7 +398,7 @@ cdef class File: """ renameDimension(self, oldname, newname) - Same as ``pnetcdf.File.rename_dim`` + Same as :meth:`File.rename_dim` """ self.rename_dim(oldname, newname) @@ -386,44 +408,47 @@ cdef class File: Create a new variable with the given parameters. - :param varname: Name of the new variable. - :type varname: str - - :param datatype: The datatype of the new variable. Supported specifiers are: + :param str varname: Name of the new variable. - - ``pnetcdf.NC_CHAR`` for text data - - ``pnetcdf.NC_BYTE`` for 1-byte integer - - ``pnetcdf.NC_SHORT`` for 2-byte signed integer - - ``pnetcdf.NC_INT`` for 4-byte signed integer - - ``pnetcdf.NC_FLOAT`` for 4-byte floating point number - - ``pnetcdf.NC_DOUBLE`` for 8-byte real number in double precision + :param datatype: The data type of the new variable. + It can be a string that describes a numpy dtype object, a numpy + dtype object, or one of PnetCDF data type constant, as shown below. - The following are `CDF-5` format only + - ``pnetcdf.NC_CHAR`` for text data + - ``pnetcdf.NC_BYTE`` for 1-byte signed integer + - ``pnetcdf.NC_SHORT`` for 2-byte signed integer + - ``pnetcdf.NC_INT`` for 4-byte signed integer + - ``pnetcdf.NC_FLOAT`` for 4-byte floating point number + - ``pnetcdf.NC_DOUBLE`` for 8-byte real number in double precision - - ``pnetcdf.NC_UBYTE`` for unsigned 1-byte integer - - ``pnetcdf.NC_USHORT`` for unsigned 2-byte integer - - ``pnetcdf.NC_UINT`` for unsigned 4-byte intege - - ``pnetcdf.NC_INT64`` for signed 8-byte integer - - ``pnetcdf.NC_UINT64`` for unsigned 8-byte integer + The following are additional data types supported by `CDF-5` format. - :type datatype: int, str, or numpy.dtype - :param dimensions: [Optional] The dimensions of the new variable. Can be either dimension names - or dimension class instances. Default is an empty tuple which means the variable is a scalar - (and therefore has no dimensions). + - ``pnetcdf.NC_UBYTE`` for unsigned 1-byte integer + - ``pnetcdf.NC_USHORT`` for unsigned 2-byte integer + - ``pnetcdf.NC_UINT`` for unsigned 4-byte intege + - ``pnetcdf.NC_INT64`` for signed 8-byte integer + - ``pnetcdf.NC_UINT64`` for unsigned 8-byte integer + :param dimensions: [Optional] + The dimensions of the new variable. Can be either dimension names + or dimension class instances. Default is an empty tuple which means + the variable is a scalar (and therefore has no dimensions). :type dimensions: tuple of str or :class:`pnetcdf.Dimension` instances - :param fill_value: The fill value of the new variable. Accepted values are: + :param fill_value: [Optional] The fill value of the new variable. + Accepted values are shown below. - - ``None``: use the default fill value for the given datatype - - ``False``: fill mode is turned off - - any other value: use the given value as fill value + - ``None`` to use the default netCDF fill value for the given data type. + - ``False`` to turn off the fill mode. + - If specified with other value, the default netCDF `_FillValue` + (the value that the variable gets filled with before any data is + written to it) is replaced with this value. :return: The created variable :rtype: :class:`pnetcdf.Variable` """ - # the following should be added to explaination of variable class. + # the following should be added to explanation of variable class. # # A list of names corresponding to netCDF variable attributes can be # # obtained with the `Variable` method `Variable.ncattrs`. A dictionary # # containing all the netCDF attribute name/value pairs is provided by @@ -465,7 +490,7 @@ cdef class File: """ createVariable(self, varname, datatype, dimensions=(), fill_value=None) - Same as ``pnetcdf.File.def_var`` + Same as :meth:`File.def_var` """ return self.def_var(varname, datatype, dimensions, fill_value) @@ -494,36 +519,37 @@ cdef class File: :param value: Value of the new attribute. :type value: str, int, float or list of int and float - Operational mode: This method must be called while the file is in define mode. + :Operational mode: This method must be called while the file is in + define mode. """ cdef nc_type xtype xtype=-99 _set_att(self, NC_GLOBAL, name, value, xtype=xtype) - def get_att(self,name,encoding='utf-8'): + def get_att(self, name, encoding='utf-8'): """ - get_att(self,name,encoding='utf-8') + get_att(self, name, encoding='utf-8') - Retrieve a netCDF file attribute. - Useful when you need to get a netCDF attribute with the same - name as one of the reserved python attributes. - - option kwarg `encoding` can be used to specify the - character encoding of a string attribute (default is `utf-8`). + Retrieve a netCDF file attribute. Useful when you need to get a netCDF + attribute with the same name as one of the reserved python attributes. :param name: Name of the attribute. :type name: str - :rtype: str or numpy.ndarray + :param encoding: [Optional] + character encoding of a string attribute (default is `utf-8`). + :type encoding: str + + :rtype: str or ``numpy.ndarray`` - Operational mode: This method can be called while the file is in either - define or data mode (collective or independent). + :Operational mode: This method can be called while the file is in + either define or data mode (collective or independent). """ return _get_att(self, NC_GLOBAL, name, encoding=encoding) - def __delattr__(self,name): + def __delattr__(self, name): # if it's a netCDF attribute, remove it if name not in _private_atts: self.del_att(name) @@ -533,16 +559,15 @@ cdef class File: def del_att(self, name): """ - del_att(self,name,value) + del_att(self, name) - Delete a netCDF file attribute. Useful when you need to delete a - netCDF attribute with the same name as one of the reserved python - attributes. + Delete a netCDF attribute. :param name: Name of the attribute :type name: str - Operational mode: This method must be called while the file is in define mode. + :Operational mode: This method must be called while the file is in + define mode. """ cdef char *attname cdef int ierr @@ -591,9 +616,9 @@ cdef class File: :param oldname: Old name of the attribute. :type oldname: str - Operational mode: If the new name is longer than the original name, the netCDF file must be in define mode. - Otherwise, the netCDF file can be in either define or data mode. - + :Operational mode: If the new name is longer than the original name, + the netCDF file must be in define mode. Otherwise, the netCDF file + can be in either define or data mode. """ cdef char *oldnamec cdef char *newnamec @@ -613,7 +638,7 @@ cdef class File: """ renameAttribute(self, oldname, newname) - Same as ``pnetcdf.File.rename_att`` + Same as :meth:`File.rename_att` """ self.rename_att(oldname, newname) @@ -660,29 +685,32 @@ cdef class File: """ wait(self, num=None, requests=None, status=None) - This method is a blocking call that wait for the completion of nonblocking I/O requests made by ``Variable.iput_var``, - ``Variable.iget_var`` and ``Variable.bput_var`` + This method is a blocking call that wait for the completion of + nonblocking I/O requests made by :meth:`Variable.iput_var`, + :meth:`Variable.iget_var` and :meth:`Variable.bput_var` - :param num: [Optional] number of requests. It is also the array size of the next two arguments. Alternatively it - can be module-level constants: + :param int num: [Optional] + number of requests. It is also the array size of the next two + arguments. Alternatively it can be module-level constants: - - None or ``pnetcdf.NC_REQ_ALL``: flush all pending nonblocking requests + - ``pnetcdf.NC_REQ_ALL`` or ``None``: flush all pending nonblocking requests - ``pnetcdf.NC_GET_REQ_ALL``: flush all pending nonblocking GET requests - ``pnetcdf.NC_PUT_REQ_ALL``: flush all pending nonblocking PUT requests - :type num: int - - :param requests: [Optional] Integers specifying the nonblocking request IDs that were made earlier. + :param requests: [Optional] + Integers specifying the nonblocking request IDs returned from the + nonblocking requests posted earlier. :type requests: list of int - :param status: [Optional] List of `None` to hold returned error codes from the call, specifying the - statuses of corresponding nonblocking requests. The values can be used in a call to ``strerror()`` to - obtain the status messages. + :param status: [Optional] + List of integers to hold returned error codes from the call, each + specifying the status of corresponding nonblocking request. The + values can be used in a call to :meth:`pnetcdf.strerror` to obtain + the error messages. :type status: list - Optional mode: it is an independent subroutine and must be called while the file - is in independent data mode. - + :Operational mode: it is an independent subroutine and must be called + while the file is in independent data mode. """ return self._wait(num, requests, status, collective=False) @@ -690,12 +718,10 @@ cdef class File: """ wait_all(self, num=None, requests=None, status=None) - Same as ``File.wait`` but in collective data mode - - Optional mode: it is an collective subroutine and must be called while the file - is in collective data mode. - + Same as :meth:`File.wait` but in collective data mode + :Operational mode: it is an collective subroutine and must be called + while the file is in collective data mode. """ return self._wait(num, requests, status, collective=True) @@ -703,29 +729,32 @@ cdef class File: """ cancel(self, num=None, requests=None, status=None) - This method cancels a list of pending nonblocking requests made by ``Variable.iput_var``, ``Variable.iget_var``, - and ``Variable.bput_var`` - - :param num: [Optional] number of requests. It is also the array size of the next two arguments. Alternatively it - can be module-level constants: + This method cancels a list of pending nonblocking requests made by the + nonblocking methods, such as :meth:`Variable.iput_var`, + :meth:`Variable.iget_var`, and :meth:`Variable.bput_var` - - None or `pnetcdf.NC_REQ_ALL`: flush all pending nonblocking requests - - `pnetcdf.NC_GET_REQ_ALL`: flush all pending nonblocking GET requests - - `pnetcdf.NC_PUT_REQ_ALL`: flush all pending nonblocking PUT requests + :param int num: [Optional] + Number of requests. It is also the array size of the next two + arguments. Alternatively it can be module-level constants: - :type num: int + - ``pnetcdf.NC_REQ_ALL`` or ``None``: flush all pending nonblocking requests + - ``pnetcdf.NC_GET_REQ_ALL``: flush all pending nonblocking GET requests + - ``pnetcdf.NC_PUT_REQ_ALL``: flush all pending nonblocking PUT requests - :param requests: [Optional] Integers specifying the nonblocking request IDs that were made earlier. + :param requests: [Optional] + Integers specifying the nonblocking request IDs that were made + earlier. :type requests: list of int - :param status: [Optional] List of `None` to hold returned error codes from the call, specifying the - statuses of corresponding nonblocking requests. The values can be used in a call to ``strerror()`` to - obtain the status messages. + :param status: [Optional] + List of integers to hold returned error codes from the call, each + specifying the status of corresponding nonblocking request. The + values can be used in a call to :meth:`pnetcdf.strerror` to obtain + the status messages. :type status: list - - Optional mode: it can be called in either independent or collective data mode or define mode. - + :Operational mode: it can be called in either independent or collective + data mode or define mode. """ cdef int _file_id, ierr cdef int num_req @@ -759,10 +788,9 @@ cdef class File: """ inq_nreqs(self) - Reports the number of pending nonblocking requests. + Method to return the number of pending nonblocking requests. :rtype: int - """ cdef int _file_id, ierr cdef int num_req @@ -776,11 +804,14 @@ cdef class File: """ attach_buff(self, bufsize) - Allow PnetCDF to allocate an internal buffer for accommodating the write requests. This method call - is the prerequisite of buffered non-blocking write. A call to ``File.detach_buff()`` is required when + Allow PnetCDF to allocate an internal buffer for accommodating the + write requests. This method call is the prerequisite of buffered + non-blocking write. A call to :meth:`File.detach_buff` is required when this buffer is no longer needed. - :param bufsize: Size of the buffer in the unit of bytes. Can be obtained using ``numpy.ndarray.nbytes`` + :param bufsize: + Size of the buffer in the unit of bytes. Can be obtained using + ``numpy.ndarray.nbytes`` :type bufsize: int """ @@ -796,8 +827,8 @@ cdef class File: """ detach_buff(self) - Detach the write buffer previously attached for buffered non-blocking write - + Detach the write buffer previously attached for buffered non-blocking + write """ cdef int _file_id = self._ncid with nogil: @@ -808,10 +839,10 @@ cdef class File: """ inq_buff_usage(self) - Return the current usage of the internal buffer + Return the current usage of the internal attached buffer, set in the + call to :meth:`File.attach_buff`. :rtype: int - """ cdef int _file_id cdef MPI_Offset usage @@ -825,9 +856,9 @@ cdef class File: """ inq_buff_size(self) - Return the size (in number of bytes) of the attached buffer. This value is the same - as the one used in a call to ``File.attach_buff`` earlier. - + Return the size (in number of bytes) of the attached buffer. This value + is the same as the one used in a call to :meth:`File.attach_buff` + earlier. :rtype: int """ @@ -864,19 +895,23 @@ cdef class File: """ set_fill(self, fillmode) - Sets the fill mode for a netCDF file open for writing and returns the current fill mode. The fill mode - can be specified as either NC_FILL or NC_NOFILL. The default mode of PnetCDF is NC_NOFILL. The method call - will change the fill mode for all variables defined so far at the time this API is called. In other - words, it overwrites the fill mode for all variables previously defined. This method will also change the - default fill mode for new variables defined following this call. In PnetCDF, this API only affects non-record - variables. In addition, it can only be called while in the define mode. All non-record variables will be - filled with fill values (either default or user-defined) at the time ``File.enddef()`` is called. + Sets the fill mode for a netCDF file open for writing and returns the + current fill mode. The fill mode can be specified as either NC_FILL or + NC_NOFILL. The default mode of PnetCDF is NC_NOFILL. The method call + will change the fill mode for all variables defined so far at the time + this API is called. In other words, it overwrites the fill mode for all + variables previously defined. This method will also change the default + fill mode for new variables defined following this call. In PnetCDF, + this API only affects non-record variables. In addition, it can only be + called while in the define mode. All non-record variables will be + filled with fill values (either default or user-defined) at the time + :meth:`File.enddef` is called. :param fillmode: ``pnetcdf.NC_FILL`` or ``pnetcdf.NC_NOFILL`` :type fillmode: int - Operational mode: This method is a collective subroutine and must be called in define mode - + :Operational mode: This method is a collective subroutine and must be + called in define mode """ cdef int _file_id, _fillmode, _old_fillmode _file_id = self._ncid @@ -890,12 +925,15 @@ cdef class File: """ set_auto_chartostring(self, value) - Call ``Variable.set_auto_chartostring()`` for all variables contained in this `File`. Calling this function only affects - existing variables. Variables defined after calling this function will follow the default behaviour. + Call :meth:`Variable.set_auto_chartostring` for all variables + contained in this `File`. Calling this function only affects existing + variables. Variables defined after calling this function will follow + the default behaviour. :param value: True or False :type value: bool - Operational mode: Any + + :Operational mode: Any """ _vars = self.variables @@ -904,9 +942,8 @@ cdef class File: def inq_num_rec_vars(self): - #TODO: currently not working with PnetCDF-C version <= 1.12.3 """ - + inq_num_rec_vars(self) Returns the number of record variables defined for this netCDF file @@ -920,8 +957,9 @@ cdef class File: return num_rec_vars def inq_num_fix_vars(self): - #TODO: currently not working with PnetCDF-C version <= 1.12.3 """ + inq_num_fix_vars(self) + Return the number of fixed-size variables defined for this netCDF file :rtype: int @@ -951,8 +989,8 @@ cdef class File: """ inq_recsize(self) - Return the size of record block, sum of individual record sizes (one record each) of - all record variables, for this netCDF file. + Return the size of record block, sum of individual record sizes (one + record each) of all record variables, for this netCDF file. :rtype: int @@ -966,7 +1004,16 @@ cdef class File: def inq_version(self): """ + inq_version(self) + Return the file format version, one of the following PnetCDf constants. + + - ``pnetcdf.NC_CLASSIC_MODEL`` indicating the file is CDF-1 format + - ``pnetcdf.NC_64BIT_OFFSET`` indicating the file is CDF-2 format + - ``pnetcdf.NC_64BIT_DATA`` indicating the file is CDF-5 format + - ``pnetcdf.NC_NETCDF4`` indicating the file is HDF5 format + + :rtype: int """ cdef int ierr, nc_mode with nogil: @@ -979,7 +1026,8 @@ cdef class File: """ inq_info(self) - Returns an MPI info object containing all the file hints used by PnetCDF library. + Returns an MPI info object containing all the file hints used by + PnetCDF library. :rtype: mpi4py.MPI.Info @@ -997,11 +1045,10 @@ cdef class File: """ inq_header_size(self) - Reports the current file header size (in bytes) of an opened netCDF file. Note - this is the amount of space used by the metadata. + Reports the current file header size (in bytes) of an opened netCDF + file. Note this is the amount of space used by the metadata. :rtype: int - """ cdef int ierr cdef MPI_Offset size @@ -1014,8 +1061,8 @@ cdef class File: """ inq_put_size(self) - Reports the amount of data that has actually been written to the file since the - file is opened/created. + Reports the amount of data that has actually been written to the file + since the file is opened/created. :rtype: int """ @@ -1030,8 +1077,8 @@ cdef class File: """ inq_get_size(self) - Reports the amount of data that has actually the amount of data that has been actually read from the - file since the file is opened/created. + Reports the amount of data that has actually the amount of data that + has been actually read from the file since the file is opened/created. :rtype: int """ @@ -1046,8 +1093,8 @@ cdef class File: """ inq_header_extent(self) - Reports the current file header extent of an opened netCDF file. The amount - is the file space allocated for the file header. + Reports the current file header extent of an opened netCDF file. The + amount is the file space allocated for the file header. :rtype: int """ diff --git a/src/pnetcdf/_Variable.pyx b/src/pnetcdf/_Variable.pyx index 68c913f..25be239 100644 --- a/src/pnetcdf/_Variable.pyx +++ b/src/pnetcdf/_Variable.pyx @@ -37,8 +37,9 @@ cdef class Variable: analogous to numpy array objects. See :meth:`Variable.__init__` for more details. - .. note:: ``Variable`` instances should be created using the `File.def_var` - method of a `File` instance, not using this class constructor directly. + .. note:: ``Variable`` instances should be created using the + :meth:`File.def_var` method of a :meth:`File` instance, not using this + class constructor directly. """ def __init__(self, file, name, datatype, dimensions=(), fill_value=None, **kwargs): @@ -47,12 +48,11 @@ cdef class Variable: The constructor for :class:`pnetcdf.Variable`. - :param varname: Name of the new variable. - :type varname: str + :param str varname: Name of the new variable. - :param datatype: The datatype of the new variable. Can be specified by - providing a NC module constant, or numpy dtype object, or a string - that describes a numpy dtype object. Supported specifiers are: + :param datatype: The data type of the new variable. + It can be a string that describes a numpy dtype object, a numpy + dtype object, or one of PnetCDF data type constant, as shown below. - ``pnetcdf.NC_CHAR`` or ``S1`` for 1-character string - ``pnetcdf.NC_BYTE`` or ``i1`` for 1-byte integer @@ -61,7 +61,7 @@ cdef class Variable: - ``pnetcdf.NC_FLOAT`` or ``f4`` for 4-byte floating point number - ``pnetcdf.NC_DOUBLE`` or ``f8`` for 8-byte real number in double precision - The following datatypes are `CDF-5` format only + The following additional data types are available for `CDF-5` format. - ``pnetcdf.NC_UBYTE`` or ``u1`` for unsigned 1-byte integer - ``pnetcdf.NC_USHORT`` or ``u2`` for unsigned 2-byte integer @@ -69,22 +69,20 @@ cdef class Variable: - ``pnetcdf.NC_INT64`` or ``i8`` for signed 8-byte integer - ``pnetcdf.NC_UINT64`` or ``u8`` for unsigned 8-byte integer - :type datatype: int - :param dimensions: [Optional] The dimensions of the new variable. Can be either dimension names or dimension class instances. Default is an empty tuple which means the variable is a scalar (and therefore has no dimensions). :type dimensions: tuple of str or :class:`pnetcdf.Dimension` instances - :param fill_value: - The fill value of the new variable. Accepted values are: + :param fill_value: [Optional] The fill value of the new variable. + Accepted values are shown below. - - ``None``: use the default fill value for the given datatype - - ``False``: fill mode is turned off - - If specified with other value, the default netCDF `_FillValue` (the - value that the variable gets filled with before any data is - written to it) is replaced with this value. + - ``None`` to use the default netCDF fill value for the given data type. + - ``False`` to turn off the fill mode. + - If specified with other value, the default netCDF `_FillValue` + (the value that the variable gets filled with before any data is + written to it) is replaced with this value. :return: The created variable :rtype: :class:`pnetcdf.Variable` @@ -302,8 +300,8 @@ cdef class Variable: """ put_att(self,name,value) - Method to add or change a variable attribute. If this attribute is new, - or if the space required to store the attribute is greater than before, + Method add or change a variable attribute. If this attribute is new, or + if the space required to store the attribute is greater than before, the netCDF file must be in define mode. :param name: Name of the attribute. @@ -312,8 +310,8 @@ cdef class Variable: :param value: Value of the attribute. :type value: str or numpy.ndarray - Operational mode: This method must be called while the associated - netCDF file is in define mode. + :Operational mode: This method must be called while the associated + netCDF file is in define mode. """ cdef nc_type xtype xtype=-99 @@ -335,8 +333,8 @@ cdef class Variable: :rtype: str or numpy.ndarray - Operational mode: This method can be called while the file is in either - define or data mode (collective or independent). + :Operational mode: This method can be called while the file is in either + define or data mode (collective or independent). """ return _get_att(self._file, self._varid, name, encoding=encoding) @@ -349,8 +347,8 @@ cdef class Variable: :param name: Name of the attribute :type name: str - Operational mode: This method must be called while the associated - netCDF file is in define mode. + :Operational mode: This method must be called while the associated + netCDF file is in define mode. """ cdef char *attname bytestr = _strencode(name) @@ -407,9 +405,9 @@ cdef class Variable: :param oldname: Old name of the attribute. :type oldname: str - Operational mode: If the new name is longer than the original name, the - associated netCDF file must be in define mode. Otherwise, the netCDF - file can be in either define or data mode. + :Operational mode: If the new name is longer than the original name, the + associated netCDF file must be in define mode. Otherwise, the + netCDF file can be in either define or data mode. """ cdef char *oldnamec cdef char *newnamec @@ -472,8 +470,9 @@ cdef class Variable: :return: A tuple of two values which are ``no_fill`` mode and ``fill_value``, respectively. - - ``no_fill``: will be 1 if no_fill mode is set (else 0). + - ``no_fill``: 1 if no_fill mode is set (else 0). - ``fill_value``: the fill value for this variable. + :rtype: tuple """ @@ -838,28 +837,29 @@ cdef class Variable: """ put_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None) - Method call to write to one or more netCDF variables in independent I/O - mode. The behavior of the method varies depends on the pattern of - provided optional arguments - `starts`, and `counts`. This method is - equivalent to making multiple calls to :meth:`Variable.put_vara`, - same variable or more than one variable. Combining multiple `put_vara` - calls may achieve a better performance. + Method write multiple subarrays of a netCDF variable to the file. This + an independent I/O call and can only be called when the file is in the + independent I/O mode. This method is equivalent to making multiple + calls to :meth:`Variable.put_var`. Note, combining multiple `put_var` + calls into one can achieve a better performance. - - `data`, `num`,`starts`, `counts` - Write multiple subarrays of - values The part of the netCDF variable to write is specified by - giving multiple subarrays and each subarray is specified by a - corner and a vector of edge lengths that refer to an array section - of the netCDF variable. The example code and diagram below - illustrates a 4 subarray section in a 4 * 10 two-dimensional - variable ("-" means skip). + - `data`, `num`, `starts`, `counts` - Write multiple subarrays of values. + The part of the netCDF variable to write is specified by giving + multiple subarrays and each subarray is specified by a corner and a + vector of edge lengths that refer to an array section of the netCDF + variable. The example code and diagram below illustrates a 4 + subarray section in a 4 * 10 two-dimensional variable ("-" means + skip). - :: + :: num = 4 + starts[0][0] = 0; starts[0][1] = 5; counts[0][0] = 1; counts[0][1] = 2 starts[1][0] = 1; starts[1][1] = 0; counts[1][0] = 1; counts[1][1] = 1 starts[2][0] = 2; starts[2][1] = 6; counts[2][0] = 1; counts[2][1] = 2 starts[3][0] = 3; starts[3][1] = 0; counts[3][0] = 1; counts[3][1] = 3 + - - - - - a b - - - a b c d e f g h -> c - - - - - - - - - - - - - - - d e - - @@ -1091,118 +1091,152 @@ cdef class Variable: """ put_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method call to write in parallel to the netCDF variable in independent - I/O mode. The behavior of the method varies depends on the pattern of + Method to write in parallel to the netCDF variable in independent I/O + mode. The behavior of the method varies depends on the pattern of provided optional arguments - `start`, `count`, `stride`, `bufcount` and `buftype`. - - `data` - Write an entire variable - Write a netCDF variable entirely of an opened netCDF file, i.e. - calling this API with only argument `data`. This is the simplest - interface to use for writing a value in a scalar variable or whenever - all the values of a multidimensional variable can all be written at - once. + - `data` - Write an entire variable. + Write a netCDF variable entirely of an opened netCDF file, i.e. + calling this API with only argument `data`. This is the simplest + interface to use for writing a value in a scalar variable or + whenever all the values of a multidimensional variable can all be + written at once. - .. note:: Be careful when using the simplest forms of this interface - with record variables. When there is no record written into the file - yet, calling this API with only argument `data`, nothing will be - written. Similarly, when writing the entire record variable, one must - take the number of records into account. + .. note:: Be careful when using the simplest forms of this interface + with record variables. When there is no record written into the + file yet, calling this API with only argument `data`, nothing + will be written. Similarly, when writing the entire record + variable, one must take the number of records into account. - - `data`, `start` - Write a single data value (a single element) - Put a single data value specified by `start` into a variable of an - opened netCDF file. For example, start = [0,5] would specify the - following position in a 4 * 10 two-dimensional - variable ("-" means skip). + - `data`, `start` - Write a single data value (a single element). + Put a single data value specified by `start` into a variable of an + opened netCDF file. For example, start = [0,5] would specify the + following position in a 4 * 10 two-dimensional variable ("-" means + skip). - :: + :: - - - - - a - - - - a -> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - `data`, `start`, `count` - Write a subarray of values - The part of the netCDF variable to write is specified by giving a corner index and a vector of edge lengths that refer to - an array section of the netCDF variable. For example, start = [0,5] and count = [2,2] would specify the following array - section in a 4 * 10 two-dimensional variable ("-" means skip). + - `data`, `start`, `count` - Write a subarray of values. + The part of the netCDF variable to write is specified by giving a + corner index and a vector of edge lengths that refer to an array + section of the netCDF variable. For example, start = [0,5] and count + = [2,2] would specify the following array section in a 4 * 10 + two-dimensional variable ("-" means skip). - :: + :: - - - - - a b - - - a b -> - - - - - c d - - - c d - - - - - - - - - - - - - - - - - - - - - - `data`, `start`, `count`, `stride` - Write a subsampled array of values - The part of the netCDF variable to write is specified by giving a corner, a vector of edge lengths and stride vector that - refer to a subsampled array section of the netCDF variable. For example, start = [0,2], count = [2,4] and stride = [1,2] - would specify the following array section in a 4 * 10 two-dimensional variable ("-" means skip). - - :: - - - - a - b - c - d - - a b c d -> - - e - f - g - h - - e f g h - - - - - - - - - - - - - - - - - - - - - - - - `data`, `start`, `count`, `imap`, `stride` (optional) - Write a mapped array of values - The mapped array section is specified by giving a corner, a vector of counts, a stride vector, and an index mapping vector. - The index mapping vector (imap) is a vector of integers that specifies the mapping between the dimensions of a netCDF variable - and the in-memory structure of the internal data array. For example, imap = [3,8], start = [0,5] and count = [2,2] would specify the following - section in write butter and array section in a 4 * 10 two-dimensional variable ("-" means skip). - - :: + - `data`, `start`, `count`, `stride` - Write a subarray of values with strided space. + The part of the netCDF variable to write is specified by giving a + corner, a vector of edge lengths and stride vector that refer to a + subsampled array section of the netCDF variable. For example, start + = [0,2], count = [2,4] and stride = [1,2] would specify the + following array section in a 4 * 10 two-dimensional variable ("-" + means skip). + + :: + + - - a - b - c - d - + a b c d -> - - e - f - g - h - + e f g h - - - - - - - - - - + - - - - - - - - - - + + - `data`, `start`, `count`, `stride`, `imap` - Write a mapped array of values. + The mapped array section is specified by giving a corner, a vector + of counts, a stride vector, and an index mapping vector. The index + mapping vector (imap) is a vector of integers that specifies the + mapping between the dimensions of a netCDF variable and the + in-memory structure of the internal data array. For example, imap = + [3,8], start = [0,5] and count = [2,2] would specify the following + section in write butter and array section in a 4 * 10 + two-dimensional variable ("-" means skip). + + :: - - - - - a c - - - a - - b a c - - - - - b d - - - - - - - -> b d -> - - - - - - - - - - c - - d - - - - - - - - - - + distance from a to b is 3 in buffer => imap[0] = 3 distance from a to c is 8 in buffer => imap[1] = 8 - :param data: the numpy array that stores array values to be written, which serves as a write buffer. When writing a single data value, - it can also be a single numeric (e.g. np.int32) python variable. The datatype should match with the variable's datatype. Note this numpy array - write buffer can be in any shape as long as the number of elements (buffer size) is matched. - + :param data: the numpy array that stores array values to be written, + which serves as a write buffer. When writing a single data value, + it can also be a single numeric (e.g. np.int32) python variable. + The datatype should match with the variable's datatype. Note this + numpy array write buffer can be in any shape as long as the number + of elements (buffer size) is matched. :type data: numpy.ndarray - :param start: [Optional] Only relevant when writing a array of values, a subsampled array, a mapped array or a list of subarrays. - An array of integers specifying the index in the variable where the first of the data values will be written. The - elements of `start` must correspond to the variable’s dimensions in order. Hence, if the variable is a record variable, the first - index would correspond to the starting record number for writing the data values. When writing to a list of subarrays, `start` - is 2D array of size [num][ndims] and each start[i] is a vector specifying the index in the variable where the first of the data values - will be written. + :param start: [Optional] + Only relevant when writing a array of values, a subsampled array, a + mapped array or a list of subarrays. An array of integers + specifying the index in the variable where the first of the data + values will be written. The elements of `start` must correspond to + the variable’s dimensions in order. Hence, if the variable is a + record variable, the first index would correspond to the starting + record number for writing the data values. When writing to a list + of subarrays, `start` is 2D array of size [num][ndims] and each + start[i] is a vector specifying the index in the variable where the + first of the data values will be written. :type start: numpy.ndarray - :param count: [Optional] Only relevant when writing a array of values, a subsampled array, a mapped array or a list of subarrays. - An array of integers specifying the edge lengths along each dimension of the block of data values to be written. The - elements of `count` must correspond to the variable’s dimensions in order. Hence, if the variable is a record variable, the first - index would correspond to the starting record number for writing the data values. When writing to a list of subarrays, `count` - is 2D array of size [num][ndims] and each count[i] is a vector specifying the edge lengths along each dimension of the block of - data values to be written. - + :param count: [Optional] + Only relevant when writing a array of values, a subsampled array, a + mapped array or a list of subarrays. An array of integers + specifying the edge lengths along each dimension of the block of + data values to be written. The elements of `count` must correspond + to the variable’s dimensions in order. Hence, if the variable is a + record variable, the first index would correspond to the starting + record number for writing the data values. When writing to a list + of subarrays, `count` is 2D array of size [num][ndims] and each + count[i] is a vector specifying the edge lengths along each + dimension of the block of data values to be written. :type count: numpy.ndarray - :param stride: [Optional] Only relevant when writing a subsampled array or a mapped array. An array of integers specifying - the sampling interval along each dimension of the netCDF variable. The elements of the stride vector correspond, in order, to the - netCDF variable’s dimensions. + :param stride: [Optional] + Only relevant when writing a subsampled array or a mapped array. An + array of integers specifying the sampling interval along each + dimension of the netCDF variable. The elements of the stride vector + correspond, in order, to the netCDF variable’s dimensions. :type stride: numpy.ndarray - :param imap: [Optional] Only relevant when writing a subsampled array or a mapped array. An array of integers the mapping between - the dimensions of a netCDF variable and the in-memory structure of the internal data array. The elements of the index mapping vector - correspond, in order, to the netCDF variable’s dimensions. Each element value of imap should equal the memory location distance in write buffer between two adjacent elements along the corresponding dimension of netCDF variable. + :param imap: [Optional] + Only relevant when writing a subsampled array or a mapped array. An + array of integers the mapping between the dimensions of a netCDF + variable and the in-memory structure of the internal data array. + The elements of the index mapping vector correspond, in order, to + the netCDF variable’s dimensions. Each element value of imap should + equal the memory location distance in write buffer between two + adjacent elements along the corresponding dimension of netCDF + variable. :type imap: numpy.ndarray - :param bufcount: [Optional] Optional for all types of writing patterns. An integer indicates the number of MPI derived data type elements - in the write buffer to be written to the file. + :param bufcount: [Optional] + Optional for all types of writing patterns. An integer indicates + the number of MPI derived data type elements in the write buffer to + be written to the file. :type bufcount: int - :param buftype: [Optional] Optional for all types of writing patterns. An MPI derived data type that describes the memory layout of the - write buffer. + :param buftype: [Optional] + An MPI derived data type that describes the memory layout of the + write buffer. :type buftype: mpi4py.MPI.Datatype - Operational mode: This method must be called while the file is in independent data mode.""" + :Operational mode: This method must be called while the file is in + independent data mode.""" + if data is not None and all(arg is None for arg in [start, count, stride, imap]): self._put_var(data, collective = False, bufcount = bufcount, buftype = buftype) elif all(arg is not None for arg in [data, start]) and all(arg is None for arg in [count, stride, imap]): @@ -1220,13 +1254,13 @@ cdef class Variable: """ put_var_all(self, data, start=None, count=None, stride=None, num=None, imap=None, bufcount=None, buftype=None) - Method call to write in parallel to the netCDF variable in the - collective I/O mode. For the argument usage, please refer to method + Method to write in parallel to the netCDF variable in the collective + I/O mode. For the argument usage, please refer to method :meth:`Variable.put_var`. The only difference is this method is a collective operation. - Operational mode: This method must be called while the file is in - collective data mode. + :Operational mode: This method must be called while the file is in + collective data mode. """ if data is not None and all(arg is None for arg in [start, count, stride, num, imap]): self._put_var(data, collective = True, bufcount = bufcount, buftype = buftype) @@ -1563,73 +1597,84 @@ cdef class Variable: """ get_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method call to read in parallel from the netCDF variable in the - independent I/O mode. The behavior of the method varies depends on the - pattern of provided optional arguments - `start`, `count`, `stride`, - and `imap`. The method requires a empty array (`data`) as a read buffer - from caller to store returned array values. - - - `data` - Read an entire variable - Read all the values from a netCDF variable of an opened netCDF file. - This is the simplest interface to use for reading the value of a - scalar variable or when all the values of a multidimensional variable - can be read at once. - - .. note:: Be careful when using this simplest form to read a record - variable when you don’t specify how many records are to be read. If - you try to read all the values of a record variable into an array but - there are more records in the file than you assume, more data will be - read than you expect, which may cause a segmentation violation. - - - `data`, `start` - Read a single data value (a single element) - Put a single array element specified by `start` from a variable of an - opened netCDF file that is in data mode. For example, index = [0,5] - would specify the following position in a 4 * 10 two-dimensional - variable ("-" means skip). - - :: + Method to read in parallel from the netCDF variable in the independent + I/O mode. The behavior of the method varies depends on the pattern of + provided optional arguments - `start`, `count`, `stride`, and `imap`. + The method requires a empty array (`data`) as a read buffer from caller + to store returned array values. + + - `data` - Read an entire variable. + Read all the values from a netCDF variable of an opened netCDF file. + This is the simplest interface to use for reading the value of a + scalar variable or when all the values of a multidimensional + variable can be read at once. + + .. note:: Be careful when using this simplest form to read a record + variable when you don’t specify how many records are to be read. + If you try to read all the values of a record variable into an + array but there are more records in the file than you assume, + more data will be read than you expect, which may cause a + segmentation violation. + + - `data`, `start` - Read a single data value (a single element). + Put a single array element specified by `start` from a variable of + an opened netCDF file that is in data mode. For example, index = + [0,5] would specify the following position in a 4 * 10 + two-dimensional variable ("-" means skip). + + :: - - - - - a - - - - - - - - - - - - - - -> a - - - - - - - - - - - - - - - - - - - - - - `data`, `start`, `count` - Read a subarray of values - The part of the netCDF variable to read is specified by giving a corner index and a vector of edge lengths that refer to - an array section of the netCDF variable. For example, start = [0,5] and count = [2,2] would specify the following array - section in a 4 * 10 two-dimensional variable ("-" means skip). + - `data`, `start`, `count` - Read a subarray of values. + The part of the netCDF variable to read is specified by giving a + corner index and a vector of edge lengths that refer to an array + section of the netCDF variable. For example, start = [0,5] and count + = [2,2] would specify the following array section in a 4 * 10 + two-dimensional variable ("-" means skip). - :: + :: - - - - - a b - - - - - - - - c d - - - a b - - - - - - - - - - -> c d - - - - - - - - - - - - `data`, `start`, `count`, `stride` - Read a subsampled array of values - The part of the netCDF variable to read is specified by giving a corner, a vector of edge lengths and stride vector that - refer to a subsampled array section of the netCDF variable. For example, start = [0,2], count = [2,4] and stride = [1,2] - would specify the following array section in a 4 * 10 two-dimensional variable ("-" means skip). + - `data`, `start`, `count`, `stride` - Read a subarray of values with strided space. + The part of the netCDF variable to read is specified by giving a + corner, a vector of edge lengths and stride vector that refer to a + subsampled array section of the netCDF variable. For example, start + = [0,2], count = [2,4] and stride = [1,2] would specify the + following array section in a 4 * 10 two-dimensional variable ("-" + means skip). - :: + :: - - a - b - c - d - - - e - f - g - h - a b c d - - - - - - - - - - -> e f g h - - - - - - - - - - - - `data`, `start`, `count`, `imap`, `stride` (optional) - Read a mapped array of values - The mapped array section is specified by giving a corner, a vector of counts, a stride vector, and an index mapping vector. - The index mapping vector (imap) is a vector of integers that specifies the mapping between the dimensions of a netCDF variable - and the in-memory structure of the internal data array. For example, imap = [3,8], start = [0,5] and count = [2,2] would specify the following - section in read butter and array section in a 4 * 10 two-dimensional variable ("-" means skip). + - `data`, `start`, `count`, `stride`, `imap` - Read a mapped array of values. + The mapped array section is specified by giving a corner, a vector + of counts, a stride vector, and an index mapping vector. The index + mapping vector (imap) is a vector of integers that specifies the + mapping between the dimensions of a netCDF variable and the + in-memory structure of the internal data array. For example, imap = + [3,8], start = [0,5] and count = [2,2] would specify the following + section in read butter and array section in a 4 * 10 two-dimensional + variable ("-" means skip). - :: + :: - - - - - a c - - - a - - b a c - - - - - b d - - - - - - - <= b d <= - - - - - - - - - - c - - d - - - - - - - - - - + distance from a to b is 3 in buffer => imap[0] = 3 distance from a to c is 8 in buffer => imap[1] = 8 @@ -1639,7 +1684,6 @@ cdef class Variable: variable's datatype. Note this numpy array read buffer can be in any shape as long as the number of elements (buffer size) is matched. - :type data: numpy.ndarray :param start: [Optional] @@ -1686,15 +1730,18 @@ cdef class Variable: store data read from the file. :type bufcount: int - :param buftype: [Optional] Optional for all types of reading patterns. An MPI derived data type that describes the memory layout of the - read buffer. + :param buftype: [Optional] + An MPI derived data type that describes the memory layout of the + read buffer. :type buftype: mpi4py.MPI.Datatype - Operational mode: This method must be called while the file is in independent data mode. + :Operational mode: This method must be called while the file is in + independent data mode. """ - # Note that get_var requires a empty array as a buffer arg from caller to store returned array values. We understand this is - # against python API convention. But removing this or making this optional will create two layers of inconsistency that - # we don't desire: + # Note that get_var requires a empty array as a buffer arg from caller + # to store returned array values. We understand this is against python + # API convention. But removing this or making this optional will create + # two layers of inconsistency that we don't desire: # 1. Among all behaviors of get_var get_varm always requires a buffer argument # 2. Other i/o methods (iget/put/iput) all require buffer array as mandatory argument @@ -1715,13 +1762,13 @@ cdef class Variable: """ get_var_all(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method call to read in parallel from the netCDF variable in the - collective I/O mode. For the argument usage, please refer to method + Method to read in parallel from the netCDF variable in the collective + I/O mode. For the argument usage, please refer to method :meth:`Variable.get_var`. The only difference is this method is a collective operation. - Operational mode: This method must be called while the file is in - collective data mode. + :Operational mode: This method must be called while the file is in + collective data mode. """ if all(arg is None for arg in [start, count, stride, imap]): self._get_var(data, collective = True, bufcount = bufcount, buftype = buftype) @@ -1740,28 +1787,29 @@ cdef class Variable: """ get_varn(self, data, num, starts, counts=None, bufcount=None, buftype=None) - Method call to read in parallel from a netCDF variables. This an - independent I/O call. The behavior of the method varies depends on the - pattern of provided optional arguments - `starts`, and `counts`. This - method is equivalent to making multiple calls to - :meth:`Variable.get_vara`. Combining multiple - :meth:`Variable.get_vara` calls may achieve a better performance. + Method to read multiple subarrays of a netCDF variables from the file. + This an independent I/O call and can only be called when the file is in + the independent I/O mode. This method is equivalent to making multiple + calls to :meth:`Variable.get_var`. Note, combining multiple `get_var` + calls into one can achieve a better performance. - - `data`, `num`,`starts`, `counts` - Write multiple subarrays of - values The part of the netCDF variable to read is specified by - giving multiple subarrays and each subarray is specified by a - corner and a vector of edge lengths that refer to an array section - of the netCDF variable. The example code and diagram below - illustrates a 4 subarray section in a 4 * 10 two-dimensional - variable ("-" means skip). + - `data`, `num`,`starts`, `counts` - Write multiple subarrays of values. + The part of the netCDF variable to read is specified by giving + multiple subarrays and each subarray is specified by a corner and a + vector of edge lengths that refer to an array section of the netCDF + variable. The example code and diagram below illustrates a 4 + subarray section in a 4 * 10 two-dimensional variable ("-" means + skip). - :: + :: num = 4 + starts[0][0] = 0; starts[0][1] = 5; counts[0][0] = 1; counts[0][1] = 2 starts[1][0] = 1; starts[1][1] = 0; counts[1][0] = 1; counts[1][1] = 1 starts[2][0] = 2; starts[2][1] = 6; counts[2][0] = 1; counts[2][1] = 2 starts[3][0] = 3; starts[3][1] = 0; counts[3][0] = 1; counts[3][1] = 3 + - - - - - a b - - - a b c d e f g h <- c - - - - - - - - - - - - - - - d e - - @@ -2191,13 +2239,12 @@ cdef class Variable: """ bput_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method call to post a nonblocking, buffered write request to write to - the netCDF variable. The syntax is the same as - :meth:`Variable.put_var`. For the argument usage, please refer to - :meth:`Variable.put_var`. This method returns a request ID that can - be used in :meth:`File.wait` or :meth:`File.wait_all`. The posted - write request may not be committed until :meth:`File.wait` or - :meth:`File.wait_all` is called. + Method to post a nonblocking, buffered write request to write to the + netCDF variable. The syntax is the same as :meth:`Variable.put_var`. + For the argument usage, please refer to :meth:`Variable.put_var`. This + method returns a request ID that can be used in :meth:`File.wait` or + :meth:`File.wait_all`. The posted write request may not be committed + until :meth:`File.wait` or :meth:`File.wait_all` is called. .. note:: Note that this method requires a numpy array (`data`) as a write buffer from caller prepared for writing returned array values @@ -2215,8 +2262,8 @@ cdef class Variable: of the nonblocking operation. :rtype: int - Operational mode: This method can be called while the file is in either - collective or independent data mode. + :Operational mode: This method can be called while the file is in either + collective or independent data mode. """ if data is not None and all(arg is None for arg in [start, count, stride, imap]): @@ -2236,13 +2283,13 @@ cdef class Variable: """ iput_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method call to post a nonblocking request to write to the netCDF - variable. The syntax is the same as :meth:`Variable.put_var`. For the - argument usage, please refer to :meth:`Variable.put_var`. This method - returns a request ID that can This method returns a request ID that can - be used in :meth:`File.wait` or :meth:`File.wait_all`. The posted - write request may not be committed until :meth:`File.wait` or - :meth:`File.wait_all` is called. + Method to post a nonblocking request to write to the netCDF variable. + The syntax is the same as :meth:`Variable.put_var`. For the argument + usage, please refer to :meth:`Variable.put_var`. This method returns a + request ID that can This method returns a request ID that can be used + in :meth:`File.wait` or :meth:`File.wait_all`. The posted write request + may not be committed until :meth:`File.wait` or :meth:`File.wait_all` + is called. .. note:: Note that this method requires a numpy array (`data`) as a write buffer from caller prepared for writing returned array values @@ -2257,8 +2304,8 @@ cdef class Variable: of the nonblocking operation. :rtype: int - Operational mode: This method can be called while the file is in either - collective or independent data mode. + :Operational mode: This method can be called while the file is in either + collective or independent data mode. """ if data is not None and all(arg is None for arg in [start, count, stride, imap]): return self._iput_var(data, bufcount = bufcount, buftype = buftype) @@ -2494,10 +2541,10 @@ cdef class Variable: """ iget_var(self, data, start=None, count=None, stride=None, imap=None, bufcount=None, buftype=None) - Method call to post a nonblocking request to read from the netCDF - variable. The syntax is the same as :meth:`Variable.get_var`. For the - argument usage, please refer to :meth:`Variable.get_var`. This - method returns a request ID that can be used in :meth:`File.wait` or + Method to post a nonblocking request to read from the netCDF variable. + The syntax is the same as :meth:`Variable.get_var`. For the argument + usage, please refer to :meth:`Variable.get_var`. This method returns a + request ID that can be used in :meth:`File.wait` or :meth:`File.wait_all`. The posted read request may not be committed until :meth:`File.wait` or :meth:`File.wait_all` is called. @@ -2513,8 +2560,8 @@ cdef class Variable: of the nonblocking operation. :rtype: int - Operational mode: This method can be called in either define, - collective, or independent data mode. + :Operational mode: This method can be called in either define, + collective, or independent data mode. """ if data is not None and all(arg is None for arg in [start, count, stride, imap]): diff --git a/src/pnetcdf/_utils.pyx b/src/pnetcdf/_utils.pyx index 289d4d9..5b83a9b 100644 --- a/src/pnetcdf/_utils.pyx +++ b/src/pnetcdf/_utils.pyx @@ -335,71 +335,75 @@ cdef _safecast(a,b): is_safe = False return is_safe -cpdef chartostring(b,encoding='utf-8'): +cpdef chartostring(src, encoding='utf-8'): """ - chartostring(b,encoding='utf-8') + chartostring(src, encoding='utf-8') Convert a character array to a string array with one less dimension. - returns a numpy string array with datatype `'UN'` (or `'SN'`) and shape - `b.shape[:-1]` where where `N=b.shape[-1]`. + :param src: Input character array (numpy datatype `'S1'` or `'U1'`). + Will be converted to a array of strings, where each string has a fixed + length of `src.shape[-1]` characters. + :type src: numpy.ndarray - :param b: Input character array (numpy datatype `'S1'` or `'U1'`). - Will be converted to a array of strings, where each string has a fixed - length of `b.shape[-1]` characters. - :type b: numpy.ndarray - - :param encoding: [Optional] Can be used to specify character encoding (default - `utf-8`). If `encoding` is 'none' or 'bytes', a `np.string_` btye array is - returned. + :param encoding: [Optional] + Can be used to specify character encoding (default `utf-8`). If + `encoding` is 'none' or 'bytes', a `np.string_` btye array is returned. :type encoding: str - :rtype: numpy.ndarray + :return: A numpy string array with datatype `'UN'` (or `'SN'`) and shape + `src.shape[:-1]` where where `N=src.shape[-1]`. + + :rtype: ``numpy.ndarray`` """ - dtype = b.dtype.kind + dtype = src.dtype.kind if dtype not in ["S","U"]: raise ValueError("type must be string or unicode ('S' or 'U')") if encoding in ['none','None','bytes']: - bs = b.tobytes() + src_str = src.tobytes() else: - bs = b.tobytes().decode(encoding) - slen = int(b.shape[-1]) + src_str = src.tobytes().decode(encoding) + slen = int(src.shape[-1]) if encoding in ['none','None','bytes']: - a = np.array([bs[n1:n1+slen] for n1 in range(0,len(bs),slen)],'S'+repr(slen)) + out_str = np.array([src_str[n1:n1+slen] for n1 in range(0,len(src_str),slen)],'S'+repr(slen)) else: - a = np.array([bs[n1:n1+slen] for n1 in range(0,len(bs),slen)],'U'+repr(slen)) - a.shape = b.shape[:-1] - return a + out_str = np.array([src_str[n1:n1+slen] for n1 in range(0,len(src_str),slen)],'U'+repr(slen)) + out_str.shape = src.shape[:-1] + return out_str -cpdef stringtochar(a,encoding='utf-8'): +cpdef stringtochar(src, encoding='utf-8'): """ - stringtochar(a,encoding='utf-8') + stringtochar(src, encoding='utf-8') Convert a string array to a character array with one extra dimension. - Returns a numpy character array with datatype `'S1'` or `'U1'` - and shape `a.shape + (N,)`, where N is the length of each string in a. - - :param a: Input numpy string array with numpy datatype `'SN'` or `'UN'`, where N - is the number of characters in each string. Will be converted to - an array of characters (datatype `'S1'` or `'U1'`) of shape `a.shape + (N,)`. + :param src: Input numpy string array with numpy datatype `'SN'` or `'UN'`, + where N is the number of characters in each string. Will be converted + to an array of characters (datatype `'S1'` or `'U1'`) of shape + `src.shape + (N,)`. :type a: numpy.ndarray - :param encoding: [Optional] Can be used to specify character encoding (default - `utf-8`). If `encoding` is 'none' or 'bytes', a `numpy.string_` the input array - is treated a raw byte strings (`numpy.string_`). + :param encoding: [Optional] + Can be used to specify character encoding (default `utf-8`). If + `encoding` is 'none' or 'bytes', a `numpy.string_` the input array is + treated a raw byte strings (`numpy.string_`). :type encoding: str + + :return: A numpy character array with datatype `'S1'` or `'U1'` and shape + `src.shape + (N,)`, where N is the length of each string in src. + + :rtype: ``numpy.ndarray`` """ - dtype = a.dtype.kind + dtype = src.dtype.kind if dtype not in ["S","U"]: raise ValueError("type must string or unicode ('S' or 'U')") if encoding in ['none','None','bytes']: - b = np.array(tuple(a.tobytes()),'S1') + out_array = np.array(tuple(src.tobytes()),'S1') else: - b = np.array(tuple(a.tobytes().decode(encoding)),dtype+'1') - b.shape = a.shape + (a.itemsize,) - return b + out_array = np.array(tuple(src.tobytes().decode(encoding)),dtype+'1') + out_array.shape = src.shape + (src.itemsize,) + return out_array cdef _StartCountStride(elem, shape, dimensions=None, file=None, datashape=None,\ @@ -784,21 +788,22 @@ cdef _get_format(int ncid): if formatp not in _reverse_format_dict: raise ValueError('format not supported by python interface') return _reverse_format_dict[formatp] + + # external C functions. cpdef strerror(err_code): """ strerror(err_code) - The function strerror returns a static reference to an error message string corresponding - to an integer netCDF error status or to a system error number, presumably returned by a previous - call to some other PnetCDF function. + This function returns an error message string corresponding to an integer + netCDF error code or to a system error number, presumably returned by a + call to a PnetCDF function. - :param err_code: An error status that might have been returned from a previous call - to some PnetCDF function. - :type err_code: int. + :param err_code: An error code returned from a call to a PnetCDF function. + :type err_code: int + :return: error message :rtype: str - """ cdef int ierr ierr = err_code @@ -808,17 +813,17 @@ cpdef strerror(err_code): cpdef strerrno(err_code): """ - strerror(err_code) + strerrno(err_code) - The function strerrno returns a character string containing the name of the NC error code. - For instance, ncmpi_strerrno(NC_EBADID) returns string "NC_EBADID". + This function returns a character string containing the name of the NC + error code. For instance, ncmpi_strerrno(NC_EBADID) returns string + "NC_EBADID". - :param err_code: An error status that might have been returned from a previous call - to some PnetCDF function. - :type err_code: int. + :param err_code: An error code returned from a call to a PnetCDF function. + :type err_code: int + :return: name of the NC error code. :rtype: str - """ cdef int ierr ierr = err_code @@ -829,19 +834,22 @@ cpdef set_default_format(int new_format): """ set_default_format(int new_format) - The function allows the user to change the format of the netCDF file to be created by future - calls to ncmpi_create without specifying the file format. It returns the existing default - format while setting a new default format. + Thise function change the default format of the netCDF file to be created + in the successive file creations when no file format is explicitly passed + as a parameter. - :param new_format: ``pnetcdf.NC_FORMAT_CLASSIC`` (the default setting), ``pnetcdf.NC_FORMAT_CDF2`` - (``pnetcdf.NC_FORMAT_64BIT``), or ``pnetcdf.NC_FORMAT_CDF5`` (``pnetcdf.NC_FORMAT_64BIT_DATA``). + :param new_format: + - ``pnetcdf.NC_FORMAT_CLASSIC`` (the default setting), + - ``pnetcdf.NC_FORMAT_CDF2`` (``pnetcdf.NC_FORMAT_64BIT``), or + - ``pnetcdf.NC_FORMAT_CDF5`` (``pnetcdf.NC_FORMAT_64BIT_DATA``). :rtype: int - Operational mode: This function is an independent subroutine, but is expected to be called by all processes that - intend to create a file later. - + :return: The current default format before this call of setting a new + default format. + :Operational mode: This function can be called in either collective or + independent I/O mode, but is expected to be called by all processes. """ cdef int ierr, newformat, oldformat newformat = new_format @@ -854,12 +862,16 @@ cpdef inq_default_format(): """ inq_default_format() - The function returns the current setting for default file format. + Method to return the current setting for default file format, one of the + PnetCDF constants shown below. - :rtype: int + - ``pnetcdf.NC_FORMAT_CLASSIC`` (the default setting), + - ``pnetcdf.NC_FORMAT_CDF2`` (``pnetcdf.NC_FORMAT_64BIT``), or + - ``pnetcdf.NC_FORMAT_CDF5`` (``pnetcdf.NC_FORMAT_64BIT_DATA``). - Operational mode: This function is an independent subroutine. + :rtype: int + :Operational mode: This function is an independent subroutine. """ cdef int curformat with nogil: @@ -871,12 +883,16 @@ cpdef inq_file_format(str file_name): """ inq_file_format(str file_name) - The function returns the current setting for default file format. + Method to return the current setting for default file format, one of the + PnetCDF constants shown below. - :rtype: int + - ``pnetcdf.NC_FORMAT_CLASSIC`` (the default setting), + - ``pnetcdf.NC_FORMAT_CDF2`` (``pnetcdf.NC_FORMAT_64BIT``), or + - ``pnetcdf.NC_FORMAT_CDF5`` (``pnetcdf.NC_FORMAT_64BIT_DATA``). - Operational mode: This function is an independent subroutine. + :rtype: int + :Operational mode: This function is an independent subroutine. """ cdef char *filename cdef int ierr, curformat @@ -891,7 +907,8 @@ cpdef inq_malloc_max_size(): """ inq_malloc_max_size() - Return the maximum size in bytes of memory allocated internally + Return the maximum size in bytes of memory allocated internally in PnetCDF + :rtype: int """ cdef int ierr @@ -905,7 +922,8 @@ cpdef inq_malloc_size(): """ inq_malloc_size() - Return the size in bytes of current memory allocated internally + Return the size in bytes of current memory allocated internally in PnetCDF + :rtype: int """ cdef int ierr @@ -914,6 +932,7 @@ cpdef inq_malloc_size(): ierr = ncmpi_inq_malloc_size(&size) _check_err(ierr) return size + """ cpdef inq_files_opened(ncids=None): cdef int ierr, num