/[MITgcm]/manual/s_outp_pkgs/text/mdsio.tex
ViewVC logotype

Diff of /manual/s_outp_pkgs/text/mdsio.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.1 by molod, Mon Jul 18 20:45:28 2005 UTC revision 1.4 by edhill, Fri Sep 2 02:24:58 2005 UTC
# Line 2  Line 2 
2  % $Name$  % $Name$
3    
4    
5    \section{Fortran Binary I/O: MDSIO and RW}
6    \label{sec:mdsio_and_rw}
7    
8    
9  \subsection{MDSIO}  \subsection{MDSIO}
 \label{sec:mdsio}  
10  \label{sec:pkg:mdsio}  \label{sec:pkg:mdsio}
11  \begin{rawhtml}  \begin{rawhtml}
12  <!-- CMIREDIR:package_mdsio: -->  <!-- CMIREDIR:package_mdsio: -->
13  \end{rawhtml}  \end{rawhtml}
14    \label{sec:pkg:rw}
15    
16    \subsubsection{Introduction}
17  The \texttt{mdsio} package contains a group of Fortran routines  The \texttt{mdsio} package contains a group of Fortran routines
18  intended as a general interface for reading and writing direct-access  intended as a general interface for reading and writing direct-access
19  (``binary'') Fortran files.  The \texttt{mdsio} routines are used by  (``binary'') Fortran files.  The \texttt{mdsio} routines are used by
20  the \texttt{rw} package.  the \texttt{rw} package.
21    
22    The \texttt{mdsio} package is currently the primary method for MITgcm
23    I/O, but it is not being actively extended or enhanced.  Instead, the
24    \texttt{mnc} netCDF package (see Section \ref{sec:pkg:mnc}) is
25    expected to gain all of the current \texttt{mdsio} functionality and,
26    eventually, replace it.  For the short term, every effort has been
27    made to allow \texttt{mnc} and \texttt{mdsio} to peacefully co-exist.
28    In may cases, the model can read one format and write to the other.
29    This side-by-side functionality can be used to, for instance, help
30    convert pickup files or other data sets between the two formats.
31    
32    
33    \subsubsection{Using MDSIO}
34    The \texttt{mdsio} package is geared toward the reading and writing of
35    floating point (Fortran \texttt{REAL*4} or \texttt{REAL*8}) arrays.
36    It assumes that the in-memory layout of all arrays follows the per-tile
37    MITgcm convention
38    \begin{verbatim}
39    C     Example of a "2D" array
40          _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
41    
42    C     Example of a "3D" array
43          _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,1:Nr,nSx,nSy)
44    \end{verbatim}
45    where the first two dimensions are spatial or ``horizontal'' indicies
46    that include a ``halo'' or exchange region (please see
47    Chapters \ref{chap:sarch} and \ref{sec:exch2} which describe domain
48    decomposition), and the remaining indicies (\texttt{Nr},\texttt{nSx},
49    and \texttt{nSx}) are often present but not required.
50    
51    In order to write output, the \texttt{mdsio} package is called with a
52    function such as:
53    \begin{verbatim}
54          CALL MDSWRITEFIELD(fn,prec,lgf,typ,Nr,arr,irec,myIter,myThid)
55    \end{verbatim}
56    where:
57    \begin{quote}
58      \begin{description}
59      \item[\texttt{fn}] is a \texttt{CHARACTER} string containing a file
60        ``base'' name which will then be used to create file names that
61        contain tile and/or model iteration indicies
62      \item[\texttt{prec}] is an integer that contains one of two globally
63        defined values (\texttt{precFloat64} or \texttt{precFloat32})
64      \item[\texttt{lgf}] is a \texttt{LOGICAL} that typically contains
65        the globally defined \texttt{globalFile} option which specifies
66        the creation of globally (spatially) concatenated files
67      \item[\texttt{typ}] is a \texttt{CHARACTER} string that specifies
68        the type of the variable being written (\texttt{'RL'} or
69        \texttt{'RS'})
70      \item[\texttt{Nr}] is an integer that specifies the number of
71        vertical levels within the variable being written
72      \item[\texttt{arr}] is the variable (array) to be written
73      \item[\texttt{irec}] is the starting record within the output file
74        that will contain the array
75      \item[\texttt{myIter,myThid}] are integers containing, respectively,
76        the current model iteration count and the unique thread ID for the
77        current context of execution
78      \end{description}  
79    \end{quote}
80    As one can see from the above (generic) example, enough information is
81    made available (through both the argument list and through common blocks)
82    for the \texttt{mdsio} package to perform the following tasks:
83    \begin{enumerate}
84    \item open either a per-tile file such as:
85      \begin{center}
86        \texttt{uVel.0000302400.003.001.data}
87      \end{center}
88      or a ``global'' file such as
89      \begin{center}
90        \texttt{uVel.0000302400.data}
91      \end{center}
92    \item byte-swap (as necessary) the input array and write its contents
93      (minus any halo information) to the binary file -- or to the correct
94      location within the binary file if the globalfile option is used, and
95    \item create an ASCII--text metadata file (same name as the binary but
96      with a \texttt{.meta} extension) describing the binary file contents
97      (often, for later use with the MatLAB \texttt{rdmds()} utility).
98    \end{enumerate}
99    
100    Reading output with \texttt{mdsio} is very similar to writing it.  A
101    typical function call is
102    \begin{verbatim}
103          CALL MDSREADFIELD(fn,prec,typ,Nr,arr,irec,myThid)
104    \end{verbatim}
105    where variables are exactly the same as the \texttt{MDSWRITEFIELD}
106    example provided above.  It is important to note that the \texttt{lgf}
107    argument is missing from the \texttt{MDSREADFIELD} function.  By
108    default, \texttt{mdsio} will first try to read from an appropriately
109    named global file and, failing that, will try to read from a per-tile
110    file.
111    
112    
113    \subsubsection{Important Considerations}
114    When using \texttt{mdsio}, one should be aware of the following
115    package features and limitations:
116    \begin{description}
117    \item[Byte-swapping] is, for the most part, gracefully handled.  All
118      files intended for reading/writing by \texttt{mdsio} should contain
119      big-endian (sometimes called ``network byte order'') data.  By
120      handling byte-swapping within the model, MITgcm output is more
121      easily ported between different machines, architectures, compilers,
122      etc.  Byteswapping can be turned on/off at compile time within
123      \texttt{mdsio} using the \texttt{\_BYTESWAPIO} CPP macro which is
124      usually set within a \texttt{genmake2} options file or
125      ``\texttt{optfile}'' which are located in
126    \begin{verbatim}
127          MITgcm/tools/build_options
128    \end{verbatim}
129      Additionally, some compilers may have byte-swap options that are
130      speedier or more convenient to use.
131    
132    \item[Types] are currently limited to single-- or double--precision
133      floating point values.  These values can be converted, on-the-fly,
134      from one to the other so that any combination of either single-- or
135      double--precision variables can be read from or written to files
136      containing either single-- or double--precision data.
137    
138    \item[Array sizes] are limited.  The \texttt{mdsio} package is very
139      much geared towards the reading/writing of per-tile (that is,
140      domain-decomposed and halo-ed) arrays.  Data that cannot be made to
141      ``fit'' within these assumed sizes can be challenging to read or
142      write with \texttt{mdsio}.
143    
144    \item[Tiling] or domain decomposition is automatically handled by
145      \texttt{mdsio} for logically rectangular grid topologies
146      (\textit{eg.} lat-lon grids) and ``standard'' cubesphere topologies.
147      More complicated topologies will probably not be supported.  The
148      \texttt{mdsio} package can, without any coding changes, read and
149      write to/from files that were run on the same global grid but with
150      different tiling (grid decomposition) schemes.  For example,
151      \texttt{mdsio} can use and/or create identical input/output files
152      for a ``C32'' cube when the model is run with either 6, 12, or 24
153      tiles (corresponding to 1, 2 or 4 tiles per cubesphere face).
154      Currently, this is one of the primary advantages that the
155      \texttt{mdsio} package has over \texttt{mnc}.
156    
157    \item[Single-CPU I/O] can be specified with the flag
158    \begin{verbatim}
159           useSingleCpuIO = .TRUE.,
160    \end{verbatim}
161      in the \texttt{PARM01} namelist within the main \texttt{data} file.
162      Single--CPU I/O mode is appropriate for computers (\textit{eg.} some
163      SGI systems) where it can either speed overall I/O or solve problems
164      where the operating system or file systems cannot correctly handle
165      multiple threads or MPI processes simultaneously writing to the same
166      file.
167    
168    \item[Meta-data] is written on a per-file basis using a second file
169      with a \texttt{.meta} extension as described above.  One should be
170      careful not to delete the metadata files when using convenient
171      MatLAB post-processing scripts such as \texttt{rdmds()}.
172    
173    \item[Numerous files] can be written by \texttt{mdsio} due to its
174      typically per-time-step and per-variable orientation.  The creation of
175      both a binary (\texttt{*.data}) and ASCII text meta--data
176      (\texttt{*.meta}) file for each output type step tends to exacerbate
177      the problem.  Some (mostly, older) operating systems do not
178      gracefully handle large numbers (\textit{eg.} many thousands) of
179      files within one directory.  So care should be taken to split output
180      into smaller groups using subdirectories.
181    
182    \item[Overwriting] is the \textbf{default behavior} for
183      \texttt{mdsio}.  If a model tries to write to a file name that
184      already exists, the older file \textbf{will be deleted}.  For this
185      reason, MITgcm users should be careful to move output that that wish
186      to keep into, for instance, subdirectories before performing
187      subsequent runs that may over--lap in time or otherwise produce
188      files with identical names (\textit{eg.} Monte-Carlo simulations).
189    
190    \item[No ``halo'' information] is written or read by \texttt{mdsio}.
191      Along the horizontal dimensions, all variables are written in an
192      \texttt{sNx}--by--\texttt{sNy} fashion.  So, although variables
193      (arrays) may be defined at different locations on Arakawa grids [U
194      (right/left horizontal edges), V (top/bottom horizontal edges), M
195      (mass or cell center), or Z (vorticity or cell corner) points], they
196      are all written using only interior (\texttt{1:sNx} and
197      \texttt{1:sNy}) values.  For quantities defined at U, V, and M
198      points, writing \texttt{1:sNx} and \texttt{1:sNy} for every tile is
199      sufficient to ensure that all values are written globally for some
200      grids (eg. cubesphere, re-entrant channels, and doubly-periodic
201      rectangular regions).  For Z points, failing to write values at the
202      \texttt{sNx+1} and \texttt{sNy+1} locations means that, for some
203      tile topologies, not all values are written.  For instance, with a
204      cubesphere topology at least two corner values are ``lost'' (fail to
205      be written for any tile) if the \texttt{sNx+1} and \texttt{sNy+1}
206      values are ignored.  To fix this problem, the \texttt{mnc} package
207      writes the \texttt{sNx+1} and \texttt{sNy+1} grid values for the U,
208      V, and Z locations.  Also, the \texttt{mnc} package is capable of
209      reading and/or writing entire halo regions and more complicated
210      array shapes which can be helpful when debugging--features that
211      do not exist within \texttt{mdsio}.
212    \end{description}
213    
214    
215    \subsection{RW Basic binary I/O utilities}
216    \label{sec:pkg:rw}
217    \begin{rawhtml}
218    <!-- CMIREDIR:package_rw: -->
219    \end{rawhtml}
220    
221    The {\tt rw} package provides a very rudimentary binary I/O capability
222    for quickly writing {\it single record} direct-access Fortran binary files.
223    It is primarily used for writing diagnostic output.
224    
225    \subsubsection{Introduction}
226    Package {\tt rw} is an interface to the more general {\tt mdsio} package.
227    The {\tt rw} package can be used to write or read direct-access Fortran
228    binary files for two-dimensional XY and three-dimensional XYZ arrays.
229    The arrays are assumed to have been declared according to the standard
230    MITgcm two-dimensional or three-dimensional floating point array type:
231    \begin{verbatim}
232    C     Example of declaring a standard two dimensional "long"
233    C     floating point type array (the _RL macro is usually
234    C     mapped to 64-bit floats in most configurations)
235          _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
236    \end{verbatim}
237    
238    Each call to an {\tt rw} read or write routine will read (or write) to
239    the first record of a file. To write direct access Fortran files with
240    multiple records use the package {\tt mdsio} (see section
241    \ref{sec:pkg:mdsio}).  To write self-describing files that contain
242    embedded information describing the variables being written and the
243    spatial and temporal locations of those variables use the package {\tt
244      mnc} (see section \ref{sec:pkg:mnc}) which produces
245    \htlink{netCDF}{http://www.unidata.ucar.edu/packages/netcdf}
246    \cite{rew:97} based output.
247    
248    %% \subsubsection{Key subroutines, parameters and files}
249    %% \label{sec:pkg:rw:implementation_synopsis}
250    %% The {\tt rw} package has
251    
Legend:
Removed from v.1.1  
changed lines
  Added in v.1.4
  ViewVC Help
Powered by ViewVC 1.1.22