s_outp_pkgs/text/mdsio.tex

% $Header: /u/gcmpack/manual/s_outp_pkgs/text/mdsio.tex,v 1.6 2006/04/04 20:51:09 molod Exp $
% $Name:  $


\section{Fortran Native I/O: MDSIO and RW}
\label{sec:mdsio_and_rw}


\subsection{MDSIO}
\label{sec:pkg:mdsio}
\begin{rawhtml}
<!-- CMIREDIR:package_mdsio: -->
\end{rawhtml}

\subsubsection{Introduction}
The \texttt{mdsio} package contains a group of Fortran routines
intended as a general interface for reading and writing direct-access
(``binary'') Fortran files.  The \texttt{mdsio} routines are used by
the \texttt{rw} package.

The \texttt{mdsio} package is currently the primary method for MITgcm
I/O, but it is not being actively extended or enhanced.  Instead, the
\texttt{mnc} netCDF package (see Section \ref{sec:pkg:mnc}) is
expected to gain all of the current \texttt{mdsio} functionality and,
eventually, replace it.  For the short term, every effort has been
made to allow \texttt{mnc} and \texttt{mdsio} to peacefully co-exist.
In may cases, the model can read one format and write to the other.
This side-by-side functionality can be used to, for instance, help
convert pickup files or other data sets between the two formats.


\subsubsection{Using MDSIO}
The \texttt{mdsio} package is geared toward the reading and writing of
floating point (Fortran \texttt{REAL*4} or \texttt{REAL*8}) arrays.
It assumes that the in-memory layout of all arrays follows the per-tile
MITgcm convention
\begin{verbatim}
C     Example of a "2D" array
      _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)

C     Example of a "3D" array
      _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,1:Nr,nSx,nSy)
\end{verbatim}
where the first two dimensions are spatial or ``horizontal'' indicies
that include a ``halo'' or exchange region (please see
Chapters \ref{chap:sarch} and \ref{sec:exch2} which describe domain
decomposition), and the remaining indicies (\texttt{Nr},\texttt{nSx},
and \texttt{nSx}) are often present but not required.

In order to write output, the \texttt{mdsio} package is called with a
function such as:
\begin{verbatim}
      CALL MDSWRITEFIELD(fn,prec,lgf,typ,Nr,arr,irec,myIter,myThid)
\end{verbatim}
where:
\begin{quote}
  \begin{description}
  \item[\texttt{fn}] is a \texttt{CHARACTER} string containing a file
    ``base'' name which will then be used to create file names that
    contain tile and/or model iteration indicies
  \item[\texttt{prec}] is an integer that contains one of two globally
    defined values (\texttt{precFloat64} or \texttt{precFloat32})
  \item[\texttt{lgf}] is a \texttt{LOGICAL} that typically contains
    the globally defined \texttt{globalFile} option which specifies
    the creation of globally (spatially) concatenated files
  \item[\texttt{typ}] is a \texttt{CHARACTER} string that specifies
    the type of the variable being written (\texttt{'RL'} or
    \texttt{'RS'})
  \item[\texttt{Nr}] is an integer that specifies the number of
    vertical levels within the variable being written
  \item[\texttt{arr}] is the variable (array) to be written
  \item[\texttt{irec}] is the starting record within the output file
    that will contain the array
  \item[\texttt{myIter,myThid}] are integers containing, respectively,
    the current model iteration count and the unique thread ID for the
    current context of execution
  \end{description}  
\end{quote}
As one can see from the above (generic) example, enough information is
made available (through both the argument list and through common blocks)
for the \texttt{mdsio} package to perform the following tasks:
\begin{enumerate}
\item open either a per-tile file such as:
  \begin{center}
    \texttt{uVel.0000302400.003.001.data}
  \end{center}
  or a ``global'' file such as
  \begin{center}
    \texttt{uVel.0000302400.data}
  \end{center}
\item byte-swap (as necessary) the input array and write its contents
  (minus any halo information) to the binary file -- or to the correct
  location within the binary file if the globalfile option is used, and 
\item create an ASCII--text metadata file (same name as the binary but
  with a \texttt{.meta} extension) describing the binary file contents
  (often, for later use with the MatLAB \texttt{rdmds()} utility).
\end{enumerate}

Reading output with \texttt{mdsio} is very similar to writing it.  A
typical function call is
\begin{verbatim}
      CALL MDSREADFIELD(fn,prec,typ,Nr,arr,irec,myThid)
\end{verbatim}
where variables are exactly the same as the \texttt{MDSWRITEFIELD}
example provided above.  It is important to note that the \texttt{lgf}
argument is missing from the \texttt{MDSREADFIELD} function.  By
default, \texttt{mdsio} will first try to read from an appropriately
named global file and, failing that, will try to read from a per-tile
file.


\subsubsection{Important Considerations}
When using \texttt{mdsio}, one should be aware of the following
package features and limitations:
\begin{description}
\item[Byte-swapping] is, for the most part, gracefully handled.  All
  files intended for reading/writing by \texttt{mdsio} should contain
  big-endian (sometimes called ``network byte order'') data.  By
  handling byte-swapping within the model, MITgcm output is more
  easily ported between different machines, architectures, compilers,
  etc.  Byteswapping can be turned on/off at compile time within
  \texttt{mdsio} using the \texttt{\_BYTESWAPIO} CPP macro which is
  usually set within a \texttt{genmake2} options file or
  ``\texttt{optfile}'' which are located in
\begin{verbatim}
      MITgcm/tools/build_options
\end{verbatim}
  Additionally, some compilers may have byte-swap options that are
  speedier or more convenient to use.

\item[Types] are currently limited to single-- or double--precision
  floating point values.  These values can be converted, on-the-fly,
  from one to the other so that any combination of either single-- or
  double--precision variables can be read from or written to files
  containing either single-- or double--precision data.

\item[Array sizes] are limited.  The \texttt{mdsio} package is very
  much geared towards the reading/writing of per-tile (that is,
  domain-decomposed and halo-ed) arrays.  Data that cannot be made to
  ``fit'' within these assumed sizes can be challenging to read or
  write with \texttt{mdsio}.

\item[Tiling] or domain decomposition is automatically handled by
  \texttt{mdsio} for logically rectangular grid topologies
  (\textit{eg.} lat-lon grids) and ``standard'' cubesphere topologies.
  More complicated topologies will probably not be supported.  The
  \texttt{mdsio} package can, without any coding changes, read and
  write to/from files that were run on the same global grid but with
  different tiling (grid decomposition) schemes.  For example,
  \texttt{mdsio} can use and/or create identical input/output files
  for a ``C32'' cube when the model is run with either 6, 12, or 24
  tiles (corresponding to 1, 2 or 4 tiles per cubesphere face).
  Currently, this is one of the primary advantages that the
  \texttt{mdsio} package has over \texttt{mnc}.

\item[Single-CPU I/O] can be specified with the flag
\begin{verbatim}
       useSingleCpuIO = .TRUE.,
\end{verbatim}
  in the \texttt{PARM01} namelist within the main \texttt{data} file.
  Single--CPU I/O mode is appropriate for computers (\textit{eg.} some
  SGI systems) where it can either speed overall I/O or solve problems
  where the operating system or file systems cannot correctly handle
  multiple threads or MPI processes simultaneously writing to the same
  file.

\item[Meta-data] is written by MITgcm on a per-file basis using a
  second file with a \texttt{.meta} extension as described above.
  MITgcm itself does not read the \texttt{*.meta} files, they are
  there primarly for convenience during post-processing.  One should
  be careful not to delete the meta-data files when using MatLAB
  post-processing scripts such as \texttt{rdmds()} since it relies
  upon them.

\item[Numerous files] can be written by \texttt{mdsio} due to its
  typically per-time-step and per-variable orientation.  The creation of
  both a binary (\texttt{*.data}) and ASCII text meta-data
  (\texttt{*.meta}) file for each output type step tends to exacerbate
  the problem.  Some (mostly, older) operating systems do not
  gracefully handle large numbers (\textit{eg.} many thousands) of
  files within one directory.  So care should be taken to split output
  into smaller groups using subdirectories.

\item[Overwriting] is the \textbf{default behavior} for
  \texttt{mdsio}.  If a model tries to write to a file name that
  already exists, the older file \textbf{will be deleted}.  For this
  reason, MITgcm users should be careful to move output that that wish
  to keep into, for instance, subdirectories before performing
  subsequent runs that may over--lap in time or otherwise produce
  files with identical names (\textit{eg.} Monte-Carlo simulations).

\item[No ``halo'' information] is written or read by \texttt{mdsio}.
  Along the horizontal dimensions, all variables are written in an
  \texttt{sNx}--by--\texttt{sNy} fashion.  So, although variables
  (arrays) may be defined at different locations on Arakawa grids [U
  (right/left horizontal edges), V (top/bottom horizontal edges), M
  (mass or cell center), or Z (vorticity or cell corner) points], they
  are all written using only interior (\texttt{1:sNx} and
  \texttt{1:sNy}) values.  For quantities defined at U, V, and M
  points, writing \texttt{1:sNx} and \texttt{1:sNy} for every tile is
  sufficient to ensure that all values are written globally for some
  grids (eg. cubesphere, re-entrant channels, and doubly-periodic
  rectangular regions).  For Z points, failing to write values at the
  \texttt{sNx+1} and \texttt{sNy+1} locations means that, for some
  tile topologies, not all values are written.  For instance, with a
  cubesphere topology at least two corner values are ``lost'' (fail to
  be written for any tile) if the \texttt{sNx+1} and \texttt{sNy+1}
  values are ignored.  To fix this problem, the \texttt{mnc} package
  writes the \texttt{sNx+1} and \texttt{sNy+1} grid values for the U,
  V, and Z locations.  Also, the \texttt{mnc} package is capable of
  reading and/or writing entire halo regions and more complicated
  array shapes which can be helpful when debugging--features that
  do not exist within \texttt{mdsio}.
\end{description}


\subsection{RW Basic binary I/O utilities}
\label{sec:pkg:rw}
\begin{rawhtml}
<!-- CMIREDIR:package_rw: -->
\end{rawhtml}

The {\tt rw} package provides a very rudimentary binary I/O capability
for quickly writing {\it single record} direct-access Fortran binary files.
It is primarily used for writing diagnostic output.

\subsubsection{Introduction}
Package {\tt rw} is an interface to the more general {\tt mdsio} package.
The {\tt rw} package can be used to write or read direct-access Fortran
binary files for two-dimensional XY and three-dimensional XYZ arrays.
The arrays are assumed to have been declared according to the standard
MITgcm two-dimensional or three-dimensional floating point array type:
\begin{verbatim}
C     Example of declaring a standard two dimensional "long"
C     floating point type array (the _RL macro is usually
C     mapped to 64-bit floats in most configurations)
      _RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
\end{verbatim}

Each call to an {\tt rw} read or write routine will read (or write) to
the first record of a file. To write direct access Fortran files with
multiple records use the package {\tt mdsio} (see section
\ref{sec:pkg:mdsio}).  To write self-describing files that contain
embedded information describing the variables being written and the
spatial and temporal locations of those variables use the package {\tt
  mnc} (see section \ref{sec:pkg:mnc}) which produces
\htlink{netCDF}{http://www.unidata.ucar.edu/packages/netcdf}
\cite{rew:97} based output.

%% \subsubsection{Key subroutines, parameters and files}
%% \label{sec:pkg:rw:implementation_synopsis}
%% The {\tt rw} package has

1	jmc	1.7	% $Header: /u/gcmpack/manual/s_outp_pkgs/text/mdsio.tex,v 1.6 2006/04/04 20:51:09 molod Exp $
2	molod	1.1	% $Name: $
3
4
5	molod	1.6	\section{Fortran Native I/O: MDSIO and RW}
6	edhill	1.2	\label{sec:mdsio_and_rw}
7
8
9	molod	1.1	\subsection{MDSIO}
10			\label{sec:pkg:mdsio}
11			\begin{rawhtml}
12			<!-- CMIREDIR:package_mdsio: -->
13			\end{rawhtml}
14
15	edhill	1.2	\subsubsection{Introduction}
16	molod	1.1	The \texttt{mdsio} package contains a group of Fortran routines
17			intended as a general interface for reading and writing direct-access
18			(``binary'') Fortran files. The \texttt{mdsio} routines are used by
19			the \texttt{rw} package.
20
21	edhill	1.4	The \texttt{mdsio} package is currently the primary method for MITgcm
22			I/O, but it is not being actively extended or enhanced. Instead, the
23			\texttt{mnc} netCDF package (see Section \ref{sec:pkg:mnc}) is
24			expected to gain all of the current \texttt{mdsio} functionality and,
25			eventually, replace it. For the short term, every effort has been
26			made to allow \texttt{mnc} and \texttt{mdsio} to peacefully co-exist.
27			In may cases, the model can read one format and write to the other.
28			This side-by-side functionality can be used to, for instance, help
29			convert pickup files or other data sets between the two formats.
30
31
32	edhill	1.2	\subsubsection{Using MDSIO}
33	edhill	1.3	The \texttt{mdsio} package is geared toward the reading and writing of
34			floating point (Fortran \texttt{REAL4} or \texttt{REAL8}) arrays.
35			It assumes that the in-memory layout of all arrays follows the per-tile
36			MITgcm convention
37			\begin{verbatim}
38			C Example of a "2D" array
39			_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
40
41			C Example of a "3D" array
42			_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,1:Nr,nSx,nSy)
43			\end{verbatim}
44			where the first two dimensions are spatial or ``horizontal'' indicies
45			that include a ``halo'' or exchange region (please see
46			Chapters \ref{chap:sarch} and \ref{sec:exch2} which describe domain
47			decomposition), and the remaining indicies (\texttt{Nr},\texttt{nSx},
48			and \texttt{nSx}) are often present but not required.
49	edhill	1.2
50	edhill	1.3	In order to write output, the \texttt{mdsio} package is called with a
51			function such as:
52			\begin{verbatim}
53			CALL MDSWRITEFIELD(fn,prec,lgf,typ,Nr,arr,irec,myIter,myThid)
54			\end{verbatim}
55			where:
56			\begin{quote}
57			\begin{description}
58			\item[\texttt{fn}] is a \texttt{CHARACTER} string containing a file
59			``base'' name which will then be used to create file names that
60			contain tile and/or model iteration indicies
61			\item[\texttt{prec}] is an integer that contains one of two globally
62			defined values (\texttt{precFloat64} or \texttt{precFloat32})
63			\item[\texttt{lgf}] is a \texttt{LOGICAL} that typically contains
64			the globally defined \texttt{globalFile} option which specifies
65			the creation of globally (spatially) concatenated files
66			\item[\texttt{typ}] is a \texttt{CHARACTER} string that specifies
67			the type of the variable being written (\texttt{'RL'} or
68			\texttt{'RS'})
69			\item[\texttt{Nr}] is an integer that specifies the number of
70			vertical levels within the variable being written
71			\item[\texttt{arr}] is the variable (array) to be written
72			\item[\texttt{irec}] is the starting record within the output file
73			that will contain the array
74			\item[\texttt{myIter,myThid}] are integers containing, respectively,
75			the current model iteration count and the unique thread ID for the
76			current context of execution
77			\end{description}
78			\end{quote}
79			As one can see from the above (generic) example, enough information is
80			made available (through both the argument list and through common blocks)
81			for the \texttt{mdsio} package to perform the following tasks:
82			\begin{enumerate}
83			\item open either a per-tile file such as:
84			\begin{center}
85			\texttt{uVel.0000302400.003.001.data}
86			\end{center}
87			or a ``global'' file such as
88			\begin{center}
89			\texttt{uVel.0000302400.data}
90			\end{center}
91			\item byte-swap (as necessary) the input array and write its contents
92			(minus any halo information) to the binary file -- or to the correct
93			location within the binary file if the globalfile option is used, and
94			\item create an ASCII--text metadata file (same name as the binary but
95			with a \texttt{.meta} extension) describing the binary file contents
96			(often, for later use with the MatLAB \texttt{rdmds()} utility).
97			\end{enumerate}
98
99			Reading output with \texttt{mdsio} is very similar to writing it. A
100			typical function call is
101			\begin{verbatim}
102			CALL MDSREADFIELD(fn,prec,typ,Nr,arr,irec,myThid)
103			\end{verbatim}
104			where variables are exactly the same as the \texttt{MDSWRITEFIELD}
105			example provided above. It is important to note that the \texttt{lgf}
106			argument is missing from the \texttt{MDSREADFIELD} function. By
107			default, \texttt{mdsio} will first try to read from an appropriately
108			named global file and, failing that, will try to read from a per-tile
109			file.
110
111
112			\subsubsection{Important Considerations}
113			When using \texttt{mdsio}, one should be aware of the following
114			package features and limitations:
115			\begin{description}
116			\item[Byte-swapping] is, for the most part, gracefully handled. All
117			files intended for reading/writing by \texttt{mdsio} should contain
118			big-endian (sometimes called ``network byte order'') data. By
119			handling byte-swapping within the model, MITgcm output is more
120			easily ported between different machines, architectures, compilers,
121			etc. Byteswapping can be turned on/off at compile time within
122			\texttt{mdsio} using the \texttt{\_BYTESWAPIO} CPP macro which is
123			usually set within a \texttt{genmake2} options file or
124			``\texttt{optfile}'' which are located in
125			\begin{verbatim}
126			MITgcm/tools/build_options
127			\end{verbatim}
128			Additionally, some compilers may have byte-swap options that are
129			speedier or more convenient to use.
130	edhill	1.2
131	edhill	1.3	\item[Types] are currently limited to single-- or double--precision
132			floating point values. These values can be converted, on-the-fly,
133			from one to the other so that any combination of either single-- or
134			double--precision variables can be read from or written to files
135			containing either single-- or double--precision data.
136
137			\item[Array sizes] are limited. The \texttt{mdsio} package is very
138			much geared towards the reading/writing of per-tile (that is,
139			domain-decomposed and halo-ed) arrays. Data that cannot be made to
140			``fit'' within these assumed sizes can be challenging to read or
141			write with \texttt{mdsio}.
142
143	edhill	1.4	\item[Tiling] or domain decomposition is automatically handled by
144			\texttt{mdsio} for logically rectangular grid topologies
145			(\textit{eg.} lat-lon grids) and ``standard'' cubesphere topologies.
146			More complicated topologies will probably not be supported. The
147			\texttt{mdsio} package can, without any coding changes, read and
148			write to/from files that were run on the same global grid but with
149			different tiling (grid decomposition) schemes. For example,
150			\texttt{mdsio} can use and/or create identical input/output files
151			for a ``C32'' cube when the model is run with either 6, 12, or 24
152			tiles (corresponding to 1, 2 or 4 tiles per cubesphere face).
153			Currently, this is one of the primary advantages that the
154			\texttt{mdsio} package has over \texttt{mnc}.
155
156			\item[Single-CPU I/O] can be specified with the flag
157			\begin{verbatim}
158			useSingleCpuIO = .TRUE.,
159			\end{verbatim}
160			in the \texttt{PARM01} namelist within the main \texttt{data} file.
161			Single--CPU I/O mode is appropriate for computers (\textit{eg.} some
162			SGI systems) where it can either speed overall I/O or solve problems
163			where the operating system or file systems cannot correctly handle
164			multiple threads or MPI processes simultaneously writing to the same
165			file.
166	edhill	1.3
167	edhill	1.5	\item[Meta-data] is written by MITgcm on a per-file basis using a
168			second file with a \texttt{.meta} extension as described above.
169			MITgcm itself does not read the \texttt{*.meta} files, they are
170			there primarly for convenience during post-processing. One should
171			be careful not to delete the meta-data files when using MatLAB
172			post-processing scripts such as \texttt{rdmds()} since it relies
173			upon them.
174	edhill	1.3
175			\item[Numerous files] can be written by \texttt{mdsio} due to its
176			typically per-time-step and per-variable orientation. The creation of
177	edhill	1.5	both a binary (\texttt{*.data}) and ASCII text meta-data
178	edhill	1.3	(\texttt{*.meta}) file for each output type step tends to exacerbate
179			the problem. Some (mostly, older) operating systems do not
180			gracefully handle large numbers (\textit{eg.} many thousands) of
181			files within one directory. So care should be taken to split output
182			into smaller groups using subdirectories.
183
184			\item[Overwriting] is the \textbf{default behavior} for
185			\texttt{mdsio}. If a model tries to write to a file name that
186			already exists, the older file \textbf{will be deleted}. For this
187			reason, MITgcm users should be careful to move output that that wish
188			to keep into, for instance, subdirectories before performing
189			subsequent runs that may over--lap in time or otherwise produce
190			files with identical names (\textit{eg.} Monte-Carlo simulations).
191
192			\item[No ``halo'' information] is written or read by \texttt{mdsio}.
193			Along the horizontal dimensions, all variables are written in an
194			\texttt{sNx}--by--\texttt{sNy} fashion. So, although variables
195			(arrays) may be defined at different locations on Arakawa grids [U
196			(right/left horizontal edges), V (top/bottom horizontal edges), M
197			(mass or cell center), or Z (vorticity or cell corner) points], they
198			are all written using only interior (\texttt{1:sNx} and
199			\texttt{1:sNy}) values. For quantities defined at U, V, and M
200			points, writing \texttt{1:sNx} and \texttt{1:sNy} for every tile is
201			sufficient to ensure that all values are written globally for some
202			grids (eg. cubesphere, re-entrant channels, and doubly-periodic
203			rectangular regions). For Z points, failing to write values at the
204			\texttt{sNx+1} and \texttt{sNy+1} locations means that, for some
205			tile topologies, not all values are written. For instance, with a
206			cubesphere topology at least two corner values are ``lost'' (fail to
207			be written for any tile) if the \texttt{sNx+1} and \texttt{sNy+1}
208			values are ignored. To fix this problem, the \texttt{mnc} package
209			writes the \texttt{sNx+1} and \texttt{sNy+1} grid values for the U,
210			V, and Z locations. Also, the \texttt{mnc} package is capable of
211			reading and/or writing entire halo regions and more complicated
212			array shapes which can be helpful when debugging--features that
213			do not exist within \texttt{mdsio}.
214			\end{description}
215	edhill	1.2
216
217			\subsection{RW Basic binary I/O utilities}
218			\label{sec:pkg:rw}
219			\begin{rawhtml}
220			<!-- CMIREDIR:package_rw: -->
221			\end{rawhtml}
222
223			The {\tt rw} package provides a very rudimentary binary I/O capability
224			for quickly writing {\it single record} direct-access Fortran binary files.
225			It is primarily used for writing diagnostic output.
226
227			\subsubsection{Introduction}
228			Package {\tt rw} is an interface to the more general {\tt mdsio} package.
229			The {\tt rw} package can be used to write or read direct-access Fortran
230			binary files for two-dimensional XY and three-dimensional XYZ arrays.
231			The arrays are assumed to have been declared according to the standard
232			MITgcm two-dimensional or three-dimensional floating point array type:
233			\begin{verbatim}
234			C Example of declaring a standard two dimensional "long"
235			C floating point type array (the _RL macro is usually
236			C mapped to 64-bit floats in most configurations)
237			_RL anArray(1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
238			\end{verbatim}
239
240			Each call to an {\tt rw} read or write routine will read (or write) to
241			the first record of a file. To write direct access Fortran files with
242			multiple records use the package {\tt mdsio} (see section
243			\ref{sec:pkg:mdsio}). To write self-describing files that contain
244			embedded information describing the variables being written and the
245			spatial and temporal locations of those variables use the package {\tt
246			mnc} (see section \ref{sec:pkg:mnc}) which produces
247			\htlink{netCDF}{http://www.unidata.ucar.edu/packages/netcdf}
248			\cite{rew:97} based output.
249
250			%% \subsubsection{Key subroutines, parameters and files}
251			%% \label{sec:pkg:rw:implementation_synopsis}
252			%% The {\tt rw} package has
253