/[MITgcm]/manual/s_outp_pkgs/text/mdsio.tex
ViewVC logotype

Diff of /manual/s_outp_pkgs/text/mdsio.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.1 by molod, Mon Jul 18 20:45:28 2005 UTC revision 1.7 by jmc, Mon Aug 30 23:09:21 2010 UTC
# Line 2  Line 2 
2  % $Name$  % $Name$
3    
4    
5    \section{Fortran Native I/O: MDSIO and RW}
6    \label{sec:mdsio_and_rw}
7    
8    
9  \subsection{MDSIO}  \subsection{MDSIO}
 \label{sec:mdsio}  
10  \label{sec:pkg:mdsio}  \label{sec:pkg:mdsio}
11  \begin{rawhtml}  \begin{rawhtml}
12  <!-- CMIREDIR:package_mdsio: -->  <!-- CMIREDIR:package_mdsio: -->
13  \end{rawhtml}  \end{rawhtml}
14    
15    \subsubsection{Introduction}
16  The \texttt{mdsio} package contains a group of Fortran routines  The \texttt{mdsio} package contains a group of Fortran routines
17  intended as a general interface for reading and writing direct-access  intended as a general interface for reading and writing direct-access
18  (``binary'') Fortran files.  The \texttt{mdsio} routines are used by  (``binary'') Fortran files.  The \texttt{mdsio} routines are used by
19  the \texttt{rw} package.  the \texttt{rw} package.
20    
21    The \texttt{mdsio} package is currently the primary method for MITgcm
22    I/O, but it is not being actively extended or enhanced.  Instead, the
23    \texttt{mnc} netCDF package (see Section \ref{sec:pkg:mnc}) is
24    expected to gain all of the current \texttt{mdsio} functionality and,
25    eventually, replace it.  For the short term, every effort has been
26    made to allow \texttt{mnc} and \texttt{mdsio} to peacefully co-exist.
27    In may cases, the model can read one format and write to the other.
28    This side-by-side functionality can be used to, for instance, help
29    convert pickup files or other data sets between the two formats.
30    
31    
32    \subsubsection{Using MDSIO}
33    The \texttt{mdsio} package is geared toward the reading and writing of
34    floating point (Fortran \texttt{REAL*4} or \texttt{REAL*8}) arrays.
35    It assumes that the in-memory layout of all arrays follows the per-tile
36    MITgcm convention
37    \begin{verbatim}
38    C     Example of a "2D" array
39          _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
40    
41    C     Example of a "3D" array
42          _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,1:Nr,nSx,nSy)
43    \end{verbatim}
44    where the first two dimensions are spatial or ``horizontal'' indicies
45    that include a ``halo'' or exchange region (please see
46    Chapters \ref{chap:sarch} and \ref{sec:exch2} which describe domain
47    decomposition), and the remaining indicies (\texttt{Nr},\texttt{nSx},
48    and \texttt{nSx}) are often present but not required.
49    
50    In order to write output, the \texttt{mdsio} package is called with a
51    function such as:
52    \begin{verbatim}
53          CALL MDSWRITEFIELD(fn,prec,lgf,typ,Nr,arr,irec,myIter,myThid)
54    \end{verbatim}
55    where:
56    \begin{quote}
57      \begin{description}
58      \item[\texttt{fn}] is a \texttt{CHARACTER} string containing a file
59        ``base'' name which will then be used to create file names that
60        contain tile and/or model iteration indicies
61      \item[\texttt{prec}] is an integer that contains one of two globally
62        defined values (\texttt{precFloat64} or \texttt{precFloat32})
63      \item[\texttt{lgf}] is a \texttt{LOGICAL} that typically contains
64        the globally defined \texttt{globalFile} option which specifies
65        the creation of globally (spatially) concatenated files
66      \item[\texttt{typ}] is a \texttt{CHARACTER} string that specifies
67        the type of the variable being written (\texttt{'RL'} or
68        \texttt{'RS'})
69      \item[\texttt{Nr}] is an integer that specifies the number of
70        vertical levels within the variable being written
71      \item[\texttt{arr}] is the variable (array) to be written
72      \item[\texttt{irec}] is the starting record within the output file
73        that will contain the array
74      \item[\texttt{myIter,myThid}] are integers containing, respectively,
75        the current model iteration count and the unique thread ID for the
76        current context of execution
77      \end{description}  
78    \end{quote}
79    As one can see from the above (generic) example, enough information is
80    made available (through both the argument list and through common blocks)
81    for the \texttt{mdsio} package to perform the following tasks:
82    \begin{enumerate}
83    \item open either a per-tile file such as:
84      \begin{center}
85        \texttt{uVel.0000302400.003.001.data}
86      \end{center}
87      or a ``global'' file such as
88      \begin{center}
89        \texttt{uVel.0000302400.data}
90      \end{center}
91    \item byte-swap (as necessary) the input array and write its contents
92      (minus any halo information) to the binary file -- or to the correct
93      location within the binary file if the globalfile option is used, and
94    \item create an ASCII--text metadata file (same name as the binary but
95      with a \texttt{.meta} extension) describing the binary file contents
96      (often, for later use with the MatLAB \texttt{rdmds()} utility).
97    \end{enumerate}
98    
99    Reading output with \texttt{mdsio} is very similar to writing it.  A
100    typical function call is
101    \begin{verbatim}
102          CALL MDSREADFIELD(fn,prec,typ,Nr,arr,irec,myThid)
103    \end{verbatim}
104    where variables are exactly the same as the \texttt{MDSWRITEFIELD}
105    example provided above.  It is important to note that the \texttt{lgf}
106    argument is missing from the \texttt{MDSREADFIELD} function.  By
107    default, \texttt{mdsio} will first try to read from an appropriately
108    named global file and, failing that, will try to read from a per-tile
109    file.
110    
111    
112    \subsubsection{Important Considerations}
113    When using \texttt{mdsio}, one should be aware of the following
114    package features and limitations:
115    \begin{description}
116    \item[Byte-swapping] is, for the most part, gracefully handled.  All
117      files intended for reading/writing by \texttt{mdsio} should contain
118      big-endian (sometimes called ``network byte order'') data.  By
119      handling byte-swapping within the model, MITgcm output is more
120      easily ported between different machines, architectures, compilers,
121      etc.  Byteswapping can be turned on/off at compile time within
122      \texttt{mdsio} using the \texttt{\_BYTESWAPIO} CPP macro which is
123      usually set within a \texttt{genmake2} options file or
124      ``\texttt{optfile}'' which are located in
125    \begin{verbatim}
126          MITgcm/tools/build_options
127    \end{verbatim}
128      Additionally, some compilers may have byte-swap options that are
129      speedier or more convenient to use.
130    
131    \item[Types] are currently limited to single-- or double--precision
132      floating point values.  These values can be converted, on-the-fly,
133      from one to the other so that any combination of either single-- or
134      double--precision variables can be read from or written to files
135      containing either single-- or double--precision data.
136    
137    \item[Array sizes] are limited.  The \texttt{mdsio} package is very
138      much geared towards the reading/writing of per-tile (that is,
139      domain-decomposed and halo-ed) arrays.  Data that cannot be made to
140      ``fit'' within these assumed sizes can be challenging to read or
141      write with \texttt{mdsio}.
142    
143    \item[Tiling] or domain decomposition is automatically handled by
144      \texttt{mdsio} for logically rectangular grid topologies
145      (\textit{eg.} lat-lon grids) and ``standard'' cubesphere topologies.
146      More complicated topologies will probably not be supported.  The
147      \texttt{mdsio} package can, without any coding changes, read and
148      write to/from files that were run on the same global grid but with
149      different tiling (grid decomposition) schemes.  For example,
150      \texttt{mdsio} can use and/or create identical input/output files
151      for a ``C32'' cube when the model is run with either 6, 12, or 24
152      tiles (corresponding to 1, 2 or 4 tiles per cubesphere face).
153      Currently, this is one of the primary advantages that the
154      \texttt{mdsio} package has over \texttt{mnc}.
155    
156    \item[Single-CPU I/O] can be specified with the flag
157    \begin{verbatim}
158           useSingleCpuIO = .TRUE.,
159    \end{verbatim}
160      in the \texttt{PARM01} namelist within the main \texttt{data} file.
161      Single--CPU I/O mode is appropriate for computers (\textit{eg.} some
162      SGI systems) where it can either speed overall I/O or solve problems
163      where the operating system or file systems cannot correctly handle
164      multiple threads or MPI processes simultaneously writing to the same
165      file.
166    
167    \item[Meta-data] is written by MITgcm on a per-file basis using a
168      second file with a \texttt{.meta} extension as described above.
169      MITgcm itself does not read the \texttt{*.meta} files, they are
170      there primarly for convenience during post-processing.  One should
171      be careful not to delete the meta-data files when using MatLAB
172      post-processing scripts such as \texttt{rdmds()} since it relies
173      upon them.
174    
175    \item[Numerous files] can be written by \texttt{mdsio} due to its
176      typically per-time-step and per-variable orientation.  The creation of
177      both a binary (\texttt{*.data}) and ASCII text meta-data
178      (\texttt{*.meta}) file for each output type step tends to exacerbate
179      the problem.  Some (mostly, older) operating systems do not
180      gracefully handle large numbers (\textit{eg.} many thousands) of
181      files within one directory.  So care should be taken to split output
182      into smaller groups using subdirectories.
183    
184    \item[Overwriting] is the \textbf{default behavior} for
185      \texttt{mdsio}.  If a model tries to write to a file name that
186      already exists, the older file \textbf{will be deleted}.  For this
187      reason, MITgcm users should be careful to move output that that wish
188      to keep into, for instance, subdirectories before performing
189      subsequent runs that may over--lap in time or otherwise produce
190      files with identical names (\textit{eg.} Monte-Carlo simulations).
191    
192    \item[No ``halo'' information] is written or read by \texttt{mdsio}.
193      Along the horizontal dimensions, all variables are written in an
194      \texttt{sNx}--by--\texttt{sNy} fashion.  So, although variables
195      (arrays) may be defined at different locations on Arakawa grids [U
196      (right/left horizontal edges), V (top/bottom horizontal edges), M
197      (mass or cell center), or Z (vorticity or cell corner) points], they
198      are all written using only interior (\texttt{1:sNx} and
199      \texttt{1:sNy}) values.  For quantities defined at U, V, and M
200      points, writing \texttt{1:sNx} and \texttt{1:sNy} for every tile is
201      sufficient to ensure that all values are written globally for some
202      grids (eg. cubesphere, re-entrant channels, and doubly-periodic
203      rectangular regions).  For Z points, failing to write values at the
204      \texttt{sNx+1} and \texttt{sNy+1} locations means that, for some
205      tile topologies, not all values are written.  For instance, with a
206      cubesphere topology at least two corner values are ``lost'' (fail to
207      be written for any tile) if the \texttt{sNx+1} and \texttt{sNy+1}
208      values are ignored.  To fix this problem, the \texttt{mnc} package
209      writes the \texttt{sNx+1} and \texttt{sNy+1} grid values for the U,
210      V, and Z locations.  Also, the \texttt{mnc} package is capable of
211      reading and/or writing entire halo regions and more complicated
212      array shapes which can be helpful when debugging--features that
213      do not exist within \texttt{mdsio}.
214    \end{description}
215    
216    
217    \subsection{RW Basic binary I/O utilities}
218    \label{sec:pkg:rw}
219    \begin{rawhtml}
220    <!-- CMIREDIR:package_rw: -->
221    \end{rawhtml}
222    
223    The {\tt rw} package provides a very rudimentary binary I/O capability
224    for quickly writing {\it single record} direct-access Fortran binary files.
225    It is primarily used for writing diagnostic output.
226    
227    \subsubsection{Introduction}
228    Package {\tt rw} is an interface to the more general {\tt mdsio} package.
229    The {\tt rw} package can be used to write or read direct-access Fortran
230    binary files for two-dimensional XY and three-dimensional XYZ arrays.
231    The arrays are assumed to have been declared according to the standard
232    MITgcm two-dimensional or three-dimensional floating point array type:
233    \begin{verbatim}
234    C     Example of declaring a standard two dimensional "long"
235    C     floating point type array (the _RL macro is usually
236    C     mapped to 64-bit floats in most configurations)
237          _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
238    \end{verbatim}
239    
240    Each call to an {\tt rw} read or write routine will read (or write) to
241    the first record of a file. To write direct access Fortran files with
242    multiple records use the package {\tt mdsio} (see section
243    \ref{sec:pkg:mdsio}).  To write self-describing files that contain
244    embedded information describing the variables being written and the
245    spatial and temporal locations of those variables use the package {\tt
246      mnc} (see section \ref{sec:pkg:mnc}) which produces
247    \htlink{netCDF}{http://www.unidata.ucar.edu/packages/netcdf}
248    \cite{rew:97} based output.
249    
250    %% \subsubsection{Key subroutines, parameters and files}
251    %% \label{sec:pkg:rw:implementation_synopsis}
252    %% The {\tt rw} package has
253    
Legend:
Removed from v.1.1  
changed lines
  Added in v.1.7
  ViewVC Help
Powered by ViewVC 1.1.22