s_outp_pkgs/text/mdsio.tex

% $Header: /u/gcmpack/manual/part7/mdsio.tex,v 1.3 2005/09/01 19:00:43 edhill Exp $
% $Name:  $


\section{Fortran Binary I/O: MDSIO and RW}
\label{sec:mdsio_and_rw}


\subsection{MDSIO}
\label{sec:pkg:mdsio}
\begin{rawhtml}
<!-- CMIREDIR:package_mdsio: -->
\end{rawhtml}
\label{sec:pkg:rw}

\subsubsection{Introduction}
The \texttt{mdsio} package contains a group of Fortran routines
intended as a general interface for reading and writing direct-access
(``binary'') Fortran files.  The \texttt{mdsio} routines are used by
the \texttt{rw} package.

The \texttt{mdsio} package is currently the primary method for MITgcm
I/O, but it is not being actively extended or enhanced.  Instead, the
\texttt{mnc} netCDF package (see Section \ref{sec:pkg:mnc}) is
expected to gain all of the current \texttt{mdsio} functionality and,
eventually, replace it.  For the short term, every effort has been
made to allow \texttt{mnc} and \texttt{mdsio} to peacefully co-exist.
In may cases, the model can read one format and write to the other.
This side-by-side functionality can be used to, for instance, help
convert pickup files or other data sets between the two formats.


\subsubsection{Using MDSIO}
The \texttt{mdsio} package is geared toward the reading and writing of
floating point (Fortran \texttt{REAL*4} or \texttt{REAL*8}) arrays.
It assumes that the in-memory layout of all arrays follows the per-tile
MITgcm convention
\begin{verbatim}
C     Example of a "2D" array
      _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)

C     Example of a "3D" array
      _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,1:Nr,nSx,nSy)
\end{verbatim}
where the first two dimensions are spatial or ``horizontal'' indicies
that include a ``halo'' or exchange region (please see
Chapters \ref{chap:sarch} and \ref{sec:exch2} which describe domain
decomposition), and the remaining indicies (\texttt{Nr},\texttt{nSx},
and \texttt{nSx}) are often present but not required.

In order to write output, the \texttt{mdsio} package is called with a
function such as:
\begin{verbatim}
      CALL MDSWRITEFIELD(fn,prec,lgf,typ,Nr,arr,irec,myIter,myThid)
\end{verbatim}
where:
\begin{quote}
  \begin{description}
  \item[\texttt{fn}] is a \texttt{CHARACTER} string containing a file
    ``base'' name which will then be used to create file names that
    contain tile and/or model iteration indicies
  \item[\texttt{prec}] is an integer that contains one of two globally
    defined values (\texttt{precFloat64} or \texttt{precFloat32})
  \item[\texttt{lgf}] is a \texttt{LOGICAL} that typically contains
    the globally defined \texttt{globalFile} option which specifies
    the creation of globally (spatially) concatenated files
  \item[\texttt{typ}] is a \texttt{CHARACTER} string that specifies
    the type of the variable being written (\texttt{'RL'} or
    \texttt{'RS'})
  \item[\texttt{Nr}] is an integer that specifies the number of
    vertical levels within the variable being written
  \item[\texttt{arr}] is the variable (array) to be written
  \item[\texttt{irec}] is the starting record within the output file
    that will contain the array
  \item[\texttt{myIter,myThid}] are integers containing, respectively,
    the current model iteration count and the unique thread ID for the
    current context of execution
  \end{description}  
\end{quote}
As one can see from the above (generic) example, enough information is
made available (through both the argument list and through common blocks)
for the \texttt{mdsio} package to perform the following tasks:
\begin{enumerate}
\item open either a per-tile file such as:
  \begin{center}
    \texttt{uVel.0000302400.003.001.data}
  \end{center}
  or a ``global'' file such as
  \begin{center}
    \texttt{uVel.0000302400.data}
  \end{center}
\item byte-swap (as necessary) the input array and write its contents
  (minus any halo information) to the binary file -- or to the correct
  location within the binary file if the globalfile option is used, and 
\item create an ASCII--text metadata file (same name as the binary but
  with a \texttt{.meta} extension) describing the binary file contents
  (often, for later use with the MatLAB \texttt{rdmds()} utility).
\end{enumerate}

Reading output with \texttt{mdsio} is very similar to writing it.  A
typical function call is
\begin{verbatim}
      CALL MDSREADFIELD(fn,prec,typ,Nr,arr,irec,myThid)
\end{verbatim}
where variables are exactly the same as the \texttt{MDSWRITEFIELD}
example provided above.  It is important to note that the \texttt{lgf}
argument is missing from the \texttt{MDSREADFIELD} function.  By
default, \texttt{mdsio} will first try to read from an appropriately
named global file and, failing that, will try to read from a per-tile
file.


\subsubsection{Important Considerations}
When using \texttt{mdsio}, one should be aware of the following
package features and limitations:
\begin{description}
\item[Byte-swapping] is, for the most part, gracefully handled.  All
  files intended for reading/writing by \texttt{mdsio} should contain
  big-endian (sometimes called ``network byte order'') data.  By
  handling byte-swapping within the model, MITgcm output is more
  easily ported between different machines, architectures, compilers,
  etc.  Byteswapping can be turned on/off at compile time within
  \texttt{mdsio} using the \texttt{\_BYTESWAPIO} CPP macro which is
  usually set within a \texttt{genmake2} options file or
  ``\texttt{optfile}'' which are located in
\begin{verbatim}
      MITgcm/tools/build_options
\end{verbatim}
  Additionally, some compilers may have byte-swap options that are
  speedier or more convenient to use.

\item[Types] are currently limited to single-- or double--precision
  floating point values.  These values can be converted, on-the-fly,
  from one to the other so that any combination of either single-- or
  double--precision variables can be read from or written to files
  containing either single-- or double--precision data.

\item[Array sizes] are limited.  The \texttt{mdsio} package is very
  much geared towards the reading/writing of per-tile (that is,
  domain-decomposed and halo-ed) arrays.  Data that cannot be made to
  ``fit'' within these assumed sizes can be challenging to read or
  write with \texttt{mdsio}.

\item[Tiling] or domain decomposition is automatically handled by
  \texttt{mdsio} for logically rectangular grid topologies
  (\textit{eg.} lat-lon grids) and ``standard'' cubesphere topologies.
  More complicated topologies will probably not be supported.  The
  \texttt{mdsio} package can, without any coding changes, read and
  write to/from files that were run on the same global grid but with
  different tiling (grid decomposition) schemes.  For example,
  \texttt{mdsio} can use and/or create identical input/output files
  for a ``C32'' cube when the model is run with either 6, 12, or 24
  tiles (corresponding to 1, 2 or 4 tiles per cubesphere face).
  Currently, this is one of the primary advantages that the
  \texttt{mdsio} package has over \texttt{mnc}.

\item[Single-CPU I/O] can be specified with the flag
\begin{verbatim}
       useSingleCpuIO = .TRUE.,
\end{verbatim}
  in the \texttt{PARM01} namelist within the main \texttt{data} file.
  Single--CPU I/O mode is appropriate for computers (\textit{eg.} some
  SGI systems) where it can either speed overall I/O or solve problems
  where the operating system or file systems cannot correctly handle
  multiple threads or MPI processes simultaneously writing to the same
  file.

\item[Meta-data] is written on a per-file basis using a second file
  with a \texttt{.meta} extension as described above.  One should be
  careful not to delete the metadata files when using convenient
  MatLAB post-processing scripts such as \texttt{rdmds()}.

\item[Numerous files] can be written by \texttt{mdsio} due to its
  typically per-time-step and per-variable orientation.  The creation of
  both a binary (\texttt{*.data}) and ASCII text meta--data
  (\texttt{*.meta}) file for each output type step tends to exacerbate
  the problem.  Some (mostly, older) operating systems do not
  gracefully handle large numbers (\textit{eg.} many thousands) of
  files within one directory.  So care should be taken to split output
  into smaller groups using subdirectories.

\item[Overwriting] is the \textbf{default behavior} for
  \texttt{mdsio}.  If a model tries to write to a file name that
  already exists, the older file \textbf{will be deleted}.  For this
  reason, MITgcm users should be careful to move output that that wish
  to keep into, for instance, subdirectories before performing
  subsequent runs that may over--lap in time or otherwise produce
  files with identical names (\textit{eg.} Monte-Carlo simulations).

\item[No ``halo'' information] is written or read by \texttt{mdsio}.
  Along the horizontal dimensions, all variables are written in an
  \texttt{sNx}--by--\texttt{sNy} fashion.  So, although variables
  (arrays) may be defined at different locations on Arakawa grids [U
  (right/left horizontal edges), V (top/bottom horizontal edges), M
  (mass or cell center), or Z (vorticity or cell corner) points], they
  are all written using only interior (\texttt{1:sNx} and
  \texttt{1:sNy}) values.  For quantities defined at U, V, and M
  points, writing \texttt{1:sNx} and \texttt{1:sNy} for every tile is
  sufficient to ensure that all values are written globally for some
  grids (eg. cubesphere, re-entrant channels, and doubly-periodic
  rectangular regions).  For Z points, failing to write values at the
  \texttt{sNx+1} and \texttt{sNy+1} locations means that, for some
  tile topologies, not all values are written.  For instance, with a
  cubesphere topology at least two corner values are ``lost'' (fail to
  be written for any tile) if the \texttt{sNx+1} and \texttt{sNy+1}
  values are ignored.  To fix this problem, the \texttt{mnc} package
  writes the \texttt{sNx+1} and \texttt{sNy+1} grid values for the U,
  V, and Z locations.  Also, the \texttt{mnc} package is capable of
  reading and/or writing entire halo regions and more complicated
  array shapes which can be helpful when debugging--features that
  do not exist within \texttt{mdsio}.
\end{description}


\subsection{RW Basic binary I/O utilities}
\label{sec:pkg:rw}
\begin{rawhtml}
<!-- CMIREDIR:package_rw: -->
\end{rawhtml}

The {\tt rw} package provides a very rudimentary binary I/O capability
for quickly writing {\it single record} direct-access Fortran binary files.
It is primarily used for writing diagnostic output.

\subsubsection{Introduction}
Package {\tt rw} is an interface to the more general {\tt mdsio} package.
The {\tt rw} package can be used to write or read direct-access Fortran
binary files for two-dimensional XY and three-dimensional XYZ arrays.
The arrays are assumed to have been declared according to the standard
MITgcm two-dimensional or three-dimensional floating point array type:
\begin{verbatim}
C     Example of declaring a standard two dimensional "long"
C     floating point type array (the _RL macro is usually
C     mapped to 64-bit floats in most configurations)
      _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
\end{verbatim}

Each call to an {\tt rw} read or write routine will read (or write) to
the first record of a file. To write direct access Fortran files with
multiple records use the package {\tt mdsio} (see section
\ref{sec:pkg:mdsio}).  To write self-describing files that contain
embedded information describing the variables being written and the
spatial and temporal locations of those variables use the package {\tt
  mnc} (see section \ref{sec:pkg:mnc}) which produces
\htlink{netCDF}{http://www.unidata.ucar.edu/packages/netcdf}
\cite{rew:97} based output.

%% \subsubsection{Key subroutines, parameters and files}
%% \label{sec:pkg:rw:implementation_synopsis}
%% The {\tt rw} package has

1	edhill	1.4	% $Header: /u/gcmpack/manual/part7/mdsio.tex,v 1.3 2005/09/01 19:00:43 edhill Exp $
2	molod	1.1	% $Name: $
3
4
5	edhill	1.2	\section{Fortran Binary I/O: MDSIO and RW}
6			\label{sec:mdsio_and_rw}
7
8
9	molod	1.1	\subsection{MDSIO}
10			\label{sec:pkg:mdsio}
11			\begin{rawhtml}
12			<!-- CMIREDIR:package_mdsio: -->
13			\end{rawhtml}
14	edhill	1.2	\label{sec:pkg:rw}
15	molod	1.1
16	edhill	1.2	\subsubsection{Introduction}
17	molod	1.1	The \texttt{mdsio} package contains a group of Fortran routines
18			intended as a general interface for reading and writing direct-access
19			(``binary'') Fortran files. The \texttt{mdsio} routines are used by
20			the \texttt{rw} package.
21
22	edhill	1.4	The \texttt{mdsio} package is currently the primary method for MITgcm
23			I/O, but it is not being actively extended or enhanced. Instead, the
24			\texttt{mnc} netCDF package (see Section \ref{sec:pkg:mnc}) is
25			expected to gain all of the current \texttt{mdsio} functionality and,
26			eventually, replace it. For the short term, every effort has been
27			made to allow \texttt{mnc} and \texttt{mdsio} to peacefully co-exist.
28			In may cases, the model can read one format and write to the other.
29			This side-by-side functionality can be used to, for instance, help
30			convert pickup files or other data sets between the two formats.
31
32
33	edhill	1.2	\subsubsection{Using MDSIO}
34	edhill	1.3	The \texttt{mdsio} package is geared toward the reading and writing of
35			floating point (Fortran \texttt{REAL4} or \texttt{REAL8}) arrays.
36			It assumes that the in-memory layout of all arrays follows the per-tile
37			MITgcm convention
38			\begin{verbatim}
39			C Example of a "2D" array
40			_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
41
42			C Example of a "3D" array
43			_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,1:Nr,nSx,nSy)
44			\end{verbatim}
45			where the first two dimensions are spatial or ``horizontal'' indicies
46			that include a ``halo'' or exchange region (please see
47			Chapters \ref{chap:sarch} and \ref{sec:exch2} which describe domain
48			decomposition), and the remaining indicies (\texttt{Nr},\texttt{nSx},
49			and \texttt{nSx}) are often present but not required.
50	edhill	1.2
51	edhill	1.3	In order to write output, the \texttt{mdsio} package is called with a
52			function such as:
53			\begin{verbatim}
54			CALL MDSWRITEFIELD(fn,prec,lgf,typ,Nr,arr,irec,myIter,myThid)
55			\end{verbatim}
56			where:
57			\begin{quote}
58			\begin{description}
59			\item[\texttt{fn}] is a \texttt{CHARACTER} string containing a file
60			``base'' name which will then be used to create file names that
61			contain tile and/or model iteration indicies
62			\item[\texttt{prec}] is an integer that contains one of two globally
63			defined values (\texttt{precFloat64} or \texttt{precFloat32})
64			\item[\texttt{lgf}] is a \texttt{LOGICAL} that typically contains
65			the globally defined \texttt{globalFile} option which specifies
66			the creation of globally (spatially) concatenated files
67			\item[\texttt{typ}] is a \texttt{CHARACTER} string that specifies
68			the type of the variable being written (\texttt{'RL'} or
69			\texttt{'RS'})
70			\item[\texttt{Nr}] is an integer that specifies the number of
71			vertical levels within the variable being written
72			\item[\texttt{arr}] is the variable (array) to be written
73			\item[\texttt{irec}] is the starting record within the output file
74			that will contain the array
75			\item[\texttt{myIter,myThid}] are integers containing, respectively,
76			the current model iteration count and the unique thread ID for the
77			current context of execution
78			\end{description}
79			\end{quote}
80			As one can see from the above (generic) example, enough information is
81			made available (through both the argument list and through common blocks)
82			for the \texttt{mdsio} package to perform the following tasks:
83			\begin{enumerate}
84			\item open either a per-tile file such as:
85			\begin{center}
86			\texttt{uVel.0000302400.003.001.data}
87			\end{center}
88			or a ``global'' file such as
89			\begin{center}
90			\texttt{uVel.0000302400.data}
91			\end{center}
92			\item byte-swap (as necessary) the input array and write its contents
93			(minus any halo information) to the binary file -- or to the correct
94			location within the binary file if the globalfile option is used, and
95			\item create an ASCII--text metadata file (same name as the binary but
96			with a \texttt{.meta} extension) describing the binary file contents
97			(often, for later use with the MatLAB \texttt{rdmds()} utility).
98			\end{enumerate}
99
100			Reading output with \texttt{mdsio} is very similar to writing it. A
101			typical function call is
102			\begin{verbatim}
103			CALL MDSREADFIELD(fn,prec,typ,Nr,arr,irec,myThid)
104			\end{verbatim}
105			where variables are exactly the same as the \texttt{MDSWRITEFIELD}
106			example provided above. It is important to note that the \texttt{lgf}
107			argument is missing from the \texttt{MDSREADFIELD} function. By
108			default, \texttt{mdsio} will first try to read from an appropriately
109			named global file and, failing that, will try to read from a per-tile
110			file.
111
112
113			\subsubsection{Important Considerations}
114			When using \texttt{mdsio}, one should be aware of the following
115			package features and limitations:
116			\begin{description}
117			\item[Byte-swapping] is, for the most part, gracefully handled. All
118			files intended for reading/writing by \texttt{mdsio} should contain
119			big-endian (sometimes called ``network byte order'') data. By
120			handling byte-swapping within the model, MITgcm output is more
121			easily ported between different machines, architectures, compilers,
122			etc. Byteswapping can be turned on/off at compile time within
123			\texttt{mdsio} using the \texttt{\_BYTESWAPIO} CPP macro which is
124			usually set within a \texttt{genmake2} options file or
125			``\texttt{optfile}'' which are located in
126			\begin{verbatim}
127			MITgcm/tools/build_options
128			\end{verbatim}
129			Additionally, some compilers may have byte-swap options that are
130			speedier or more convenient to use.
131	edhill	1.2
132	edhill	1.3	\item[Types] are currently limited to single-- or double--precision
133			floating point values. These values can be converted, on-the-fly,
134			from one to the other so that any combination of either single-- or
135			double--precision variables can be read from or written to files
136			containing either single-- or double--precision data.
137
138			\item[Array sizes] are limited. The \texttt{mdsio} package is very
139			much geared towards the reading/writing of per-tile (that is,
140			domain-decomposed and halo-ed) arrays. Data that cannot be made to
141			``fit'' within these assumed sizes can be challenging to read or
142			write with \texttt{mdsio}.
143
144	edhill	1.4	\item[Tiling] or domain decomposition is automatically handled by
145			\texttt{mdsio} for logically rectangular grid topologies
146			(\textit{eg.} lat-lon grids) and ``standard'' cubesphere topologies.
147			More complicated topologies will probably not be supported. The
148			\texttt{mdsio} package can, without any coding changes, read and
149			write to/from files that were run on the same global grid but with
150			different tiling (grid decomposition) schemes. For example,
151			\texttt{mdsio} can use and/or create identical input/output files
152			for a ``C32'' cube when the model is run with either 6, 12, or 24
153			tiles (corresponding to 1, 2 or 4 tiles per cubesphere face).
154			Currently, this is one of the primary advantages that the
155			\texttt{mdsio} package has over \texttt{mnc}.
156
157			\item[Single-CPU I/O] can be specified with the flag
158			\begin{verbatim}
159			useSingleCpuIO = .TRUE.,
160			\end{verbatim}
161			in the \texttt{PARM01} namelist within the main \texttt{data} file.
162			Single--CPU I/O mode is appropriate for computers (\textit{eg.} some
163			SGI systems) where it can either speed overall I/O or solve problems
164			where the operating system or file systems cannot correctly handle
165			multiple threads or MPI processes simultaneously writing to the same
166			file.
167	edhill	1.3
168			\item[Meta-data] is written on a per-file basis using a second file
169			with a \texttt{.meta} extension as described above. One should be
170			careful not to delete the metadata files when using convenient
171			MatLAB post-processing scripts such as \texttt{rdmds()}.
172
173			\item[Numerous files] can be written by \texttt{mdsio} due to its
174			typically per-time-step and per-variable orientation. The creation of
175			both a binary (\texttt{*.data}) and ASCII text meta--data
176			(\texttt{*.meta}) file for each output type step tends to exacerbate
177			the problem. Some (mostly, older) operating systems do not
178			gracefully handle large numbers (\textit{eg.} many thousands) of
179			files within one directory. So care should be taken to split output
180			into smaller groups using subdirectories.
181
182			\item[Overwriting] is the \textbf{default behavior} for
183			\texttt{mdsio}. If a model tries to write to a file name that
184			already exists, the older file \textbf{will be deleted}. For this
185			reason, MITgcm users should be careful to move output that that wish
186			to keep into, for instance, subdirectories before performing
187			subsequent runs that may over--lap in time or otherwise produce
188			files with identical names (\textit{eg.} Monte-Carlo simulations).
189
190			\item[No ``halo'' information] is written or read by \texttt{mdsio}.
191			Along the horizontal dimensions, all variables are written in an
192			\texttt{sNx}--by--\texttt{sNy} fashion. So, although variables
193			(arrays) may be defined at different locations on Arakawa grids [U
194			(right/left horizontal edges), V (top/bottom horizontal edges), M
195			(mass or cell center), or Z (vorticity or cell corner) points], they
196			are all written using only interior (\texttt{1:sNx} and
197			\texttt{1:sNy}) values. For quantities defined at U, V, and M
198			points, writing \texttt{1:sNx} and \texttt{1:sNy} for every tile is
199			sufficient to ensure that all values are written globally for some
200			grids (eg. cubesphere, re-entrant channels, and doubly-periodic
201			rectangular regions). For Z points, failing to write values at the
202			\texttt{sNx+1} and \texttt{sNy+1} locations means that, for some
203			tile topologies, not all values are written. For instance, with a
204			cubesphere topology at least two corner values are ``lost'' (fail to
205			be written for any tile) if the \texttt{sNx+1} and \texttt{sNy+1}
206			values are ignored. To fix this problem, the \texttt{mnc} package
207			writes the \texttt{sNx+1} and \texttt{sNy+1} grid values for the U,
208			V, and Z locations. Also, the \texttt{mnc} package is capable of
209			reading and/or writing entire halo regions and more complicated
210			array shapes which can be helpful when debugging--features that
211			do not exist within \texttt{mdsio}.
212			\end{description}
213	edhill	1.2
214
215			\subsection{RW Basic binary I/O utilities}
216			\label{sec:pkg:rw}
217			\begin{rawhtml}
218			<!-- CMIREDIR:package_rw: -->
219			\end{rawhtml}
220
221			The {\tt rw} package provides a very rudimentary binary I/O capability
222			for quickly writing {\it single record} direct-access Fortran binary files.
223			It is primarily used for writing diagnostic output.
224
225			\subsubsection{Introduction}
226			Package {\tt rw} is an interface to the more general {\tt mdsio} package.
227			The {\tt rw} package can be used to write or read direct-access Fortran
228			binary files for two-dimensional XY and three-dimensional XYZ arrays.
229			The arrays are assumed to have been declared according to the standard
230			MITgcm two-dimensional or three-dimensional floating point array type:
231			\begin{verbatim}
232			C Example of declaring a standard two dimensional "long"
233			C floating point type array (the _RL macro is usually
234			C mapped to 64-bit floats in most configurations)
235			_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
236			\end{verbatim}
237
238			Each call to an {\tt rw} read or write routine will read (or write) to
239			the first record of a file. To write direct access Fortran files with
240			multiple records use the package {\tt mdsio} (see section
241			\ref{sec:pkg:mdsio}). To write self-describing files that contain
242			embedded information describing the variables being written and the
243			spatial and temporal locations of those variables use the package {\tt
244			mnc} (see section \ref{sec:pkg:mnc}) which produces
245			\htlink{netCDF}{http://www.unidata.ucar.edu/packages/netcdf}
246			\cite{rew:97} based output.
247
248			%% \subsubsection{Key subroutines, parameters and files}
249			%% \label{sec:pkg:rw:implementation_synopsis}
250			%% The {\tt rw} package has
251