/[MITgcm]/manual/s_phys_pkgs/mnc.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/mnc.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.11 by edhill, Wed Sep 22 15:11:32 2004 UTC revision 1.14 by edhill, Sat Dec 11 22:03:32 2004 UTC
# Line 3  Line 3 
3    
4  \section{NetCDF I/O Integration: MNC}  \section{NetCDF I/O Integration: MNC}
5  \label{sec:pkg:mnc}  \label{sec:pkg:mnc}
6    \begin{rawhtml}
7    <!-- CMIREDIR:package_mnc: -->
8    \end{rawhtml}
9    
10  The \texttt{mnc} package is a set of convenience routines written to  The \texttt{mnc} package is a set of convenience routines written to
11  expedite the process of creating, appending, and reading NetCDF files.  expedite the process of creating, appending, and reading NetCDF files.
# Line 18  http://www.unidata.ucar.edu/packages/net Line 21  http://www.unidata.ucar.edu/packages/net
21  \begin{rawhtml} </A> \end{rawhtml}  \begin{rawhtml} </A> \end{rawhtml}
22    
23    
24  \subsection{Introduction}  \subsection{Using MNC}
25    
26    \subsubsection{MNC Configuration}
27    
28    As with all MITgcm packages, MNC can be turned on or off at compile time
29    using the \texttt{packages.conf} file or the \texttt{genmake2}
30    \texttt{-enable=mnc} or \texttt{-disable=mnc} switches.
31    
32    While MNC is likely to work ``as is'', there are a few compile--time
33    constants that may need to be increased for simulations that employ
34    large numbers of tiles within each process.  Note that the important
35    quantity is the maximum number of tiles \textbf{per process}.  Since
36    MPI configurations tend to distribute large numbers of tiles over
37    relatively large numbers of MPI processes, these constants will rarely
38    need to be increased.
39    
40    If MNC runs out of space within its ``lookup'' tables during a
41    simulation, then it will provide an error message along with a
42    recommendation of which parameter to increase.  The parameters are all
43    located within \filelink{pkg/mnc/mnc\_common.h}{pkg-mnc-mnc_common.h}
44    and the ones that may need to be increased are:
45    
46    \begin{center}
47      {\footnotesize
48        \begin{tabular}[htb]{|l|r|l|}\hline
49          \textbf{Name}  &  
50          \textbf{Default}  &  \textbf{Description}  \\\hline
51          &  &  \\
52          \texttt{MNC\_MAX\_ID}  &  1000  &
53          \textbf{IDs for various low-level entities}  \\
54          \texttt{MNC\_MAX\_INFO}  &   400  &
55          \textbf{IDs (mostly for object sizes)}  \\
56          \texttt{MNC\_CW\_MAX\_I}  &  150  &
57          \textbf{IDs for the ``wrapper'' layer}  \\\hline
58        \end{tabular}
59      }
60    \end{center}
61    
62    In those rare cases where MNC ``out-of-memory'' error messages are
63    encountered, it is a good idea to increase the too-small parameter by
64    a factor of \textbf{2--10} in order to avoid wasting time on an
65    iterative compile--test sequence.
66    
67    
68    \subsubsection{MNC Inputs}
69    
70    For run-time configuration, most of the MNC--related model parameters
71    are contained within a Fortran namelist file called \texttt{data.mnc}.
72    If this file does not exist, then the MNC package will interpret that
73    as an indication that it is not to be used.  If the \texttt{data.mnc}
74    file does exist, then it may contain the following parameters:
75    
76    \begin{center}
77      {\footnotesize
78        \begin{tabular}[htb]{|l|c|l|l|}\hline
79          \textbf{Name}  &  \textbf{T}  &  
80          \textbf{Default}  &  \textbf{Description}  \\\hline
81          &  &  &  \\
82          \texttt{useMNC}  &  L  & \texttt{.FALSE.}  &  
83          \textbf{overall MNC ON/OFF switch}  \\
84          \texttt{mnc\_echo\_gvtypes}  &  L  & \texttt{.FALSE.}  &  
85          echo pre-defined ``types'' (debugging)   \\
86          \texttt{mnc\_use\_outdir}  &  L  & \texttt{.FALSE.}  &  
87          create a directory for output  \\
88          \texttt{mnc\_outdir\_str}  &  S  & \texttt{'mnc\_'}  &  
89          output directory name \\
90          \texttt{mnc\_outdir\_date}  &  L  & \texttt{.FALSE.}  &  
91          embed date in the output dir name  \\
92          \texttt{pickup\_write\_mnc}  &  L  & \texttt{.FALSE.}  &  
93          use MNC to write (create) pickup files  \\
94          \texttt{pickup\_read\_mnc}  &  L  & \texttt{.FALSE.}  &  
95          use MNC to read pickup files  \\
96          \texttt{mnc\_use\_indir}  &  L  & \texttt{.FALSE.}  &  
97          use a directory (path) for input  \\
98          \texttt{mnc\_indir\_str}  &  S  & \texttt{''}  &  
99          input directory (or path) name  \\
100          \texttt{snapshot\_mnc}  &  L  & \texttt{.FALSE.}  &  
101          write \texttt{snapshot} (instantaneous) w/MNC  \\
102          \texttt{monitor\_mnc}  &  L  & \texttt{.FALSE.}  &  
103          write \texttt{monitor} w/MNC  \\
104          \texttt{timeave\_mnc}  &  L  & \texttt{.FALSE.}  &  
105          write \texttt{timeave} w/MNC  \\
106          \texttt{autodiff\_mnc}  &  L  & \texttt{.FALSE.}  &  
107          write \texttt{autodiff} w/MNC  \\\hline
108        \end{tabular}
109      }
110    \end{center}
111    
112    Additional MNC--related parameters are contained within the main
113    \texttt{data} namelist file and in some of the namelist files for
114    individual packages.  These options are:
115    \begin{center}
116      {\footnotesize
117        \begin{tabular}[htb]{|l|c|l|l|}\hline
118          \textbf{Name}  &  \textbf{T}  &  
119          \textbf{Default}  &  \textbf{Description}  \\\hline
120          \multicolumn{4}{|c|}{\ }  \\
121          \multicolumn{4}{|c|}{Main namelist file:
122            ``\textbf{data}''}  \\\hline
123          \texttt{snapshot\_ioinc}  &  L  & \texttt{.FALSE.}  &  
124          write \texttt{snapshot} ``inclusively''  \\
125          \texttt{timeave\_ioinc}  &  L  & \texttt{.FALSE.}  &  
126          write \texttt{timeave} ``inclusively''  \\
127          \texttt{monitor\_ioinc}  &  L  & \texttt{.FALSE.}  &  
128          write \texttt{monitor} ``inclusively''  \\
129          \texttt{the\_run\_name}  &  C  & ``name...''  &  
130          name is included in all MNC output  \\\hline
131          \multicolumn{4}{|c|}{\ }  \\
132          \multicolumn{4}{|c|}{Diagnostics namelist file:
133            ``\textbf{data.diagnostics}''}  \\\hline
134          \texttt{diag\_mnc}  &  L  & \texttt{.FALSE.}  &  
135          write \texttt{diagnostics} w/MNC  \\
136          \texttt{diag\_ioinc}  &  L  & \texttt{.FALSE.}  &  
137          write \texttt{diagnostics} ``inclusively''  \\\hline
138        \end{tabular}
139      }
140    \end{center}
141    
142    By default, turning on MNC for a particular output type will result in
143    turning off all the corresponding (usually, default) MDSIO or STDOUT
144    output mechanisms.  In other words, output defaults to being an
145    exclusive selection.  To enable multiple kinds of simultaneous output,
146    flags of the form \texttt{NAME\_ioinc} have been created where
147    \texttt{NAME} corresponds to the various MNC output flags.  When a
148    \texttt{NAME\_ioinc} flag is set to \texttt{.TRUE.}, then multiple
149    simultaneous forms of output are allowed for the \texttt{NAME} output
150    mechanism.  The intent of this design is that typical users will only
151    want one kind of output while people debugging the code (particularly
152    the I/O routines) may want simultaneous types of output.
153    
154    This ``inclusive'' versus ``exclusive'' design is easily applied in
155    cases where three or more kinds of output may be generated.  Thus, it
156    can be readily extended to additional new output types (eg. HDF5).
157    
158    Input types are always exclusive.
159    
160    \subsubsection{MNC Output}
161    
162    While NetCDF files are supposed to be ``self-describing'', it is
163    helpful to note the following:
164    
165    \begin{itemize}
166    \item The constraints placed upon the ``unlimited'' (or ``record'')
167      dimension inherent with NetCDF v3.x make it very inefficient to put
168      variables written at potentially different intervals within the same
169      file.  For this reason, MNC output is split into a few file ``base
170      names'' which try to reflect the nature of their content.
171      
172    \item All MNC output is currently done in a ``tile-per-file'' fashion
173      since most NetCDF v3.x implementions cannot write safely within MPI
174      or multi-threaded environments.  This tiling is done in a global
175      fashion and the tile numbers are appended to the base names
176      described above.  Some scripts to ``assemble'' output are available
177      (\texttt{MITgcm/utils/matlab}).  More general manipulations can be
178      accomplished with the
179      \begin{rawhtml}
180        <A href="http://nco.sourceforge.net">
181      \end{rawhtml}
182    \begin{verbatim}
183    NetCDF Operators (or ``NCO'') at http://nco.sourceforge.net
184    \end{verbatim}
185      \begin{rawhtml} </A> \end{rawhtml}
186      which is a very powerful and convenient set of tools for working
187      with all NetCDF files.
188      
189    \item On many systems, NetCDF has practical file size limits on the
190      order of 2--4GB (the maximium memory addressable with 32bit
191      pointers) due to a lack of operating system, compiler, and/or
192      library support.  In cases where this limit is reached, it is
193      generally a good idea to reduce write frequencies or restart from
194      pickups.
195      
196    \item MNC does not (yet) provide a mechanism for reading information
197      from a single ``global'' file as can be done with the MDSIO
198      package.  This is in progress.
199    
200    \end{itemize}
201    
202    
203    \subsection{MNC Internals}
204    
205  The \texttt{mnc} package is a two-level convenience library (or  The \texttt{mnc} package is a two-level convenience library (or
206  ``wrapper'') for most of the NetCDF Fortran API.  Its purpose is to  ``wrapper'') for most of the NetCDF Fortran API.  Its purpose is to
# Line 66  The two levels of the \texttt{mnc} packa Line 248  The two levels of the \texttt{mnc} packa
248  \end{description}  \end{description}
249    
250    
251  \subsection{Using MNC}  \subsubsection{MNC Grid--Types and Variable--Types}
   
 \subsubsection{Grid--Types and Variable--Types}  
252    
253  As a convenience for users, the MNC package includes numerous routines  As a convenience for users, the MNC package includes numerous routines
254  to aid in the writing of data to NetCDF format.  Probably the biggest  to aid in the writing of data to NetCDF format.  Probably the biggest
# Line 127  and is used by the \texttt{mnc\_cw\_*\_[ Line 307  and is used by the \texttt{mnc\_cw\_*\_[
307  and writing variables.  and writing variables.
308    
309    
310  \subsubsection{An Example}  \subsubsection{Using MNC: Examples}
311    
312  Writing variables to NetCDF files can be accomplished in as few as two  Writing variables to NetCDF files can be accomplished in as few as two
313  function calls.  The first function call defines a variable type,  function calls.  The first function call defines a variable type,
# Line 183  C       Write dynvars using the MNC pack Line 363  C       Write dynvars using the MNC pack
363  \end{verbatim}  \end{verbatim}
364  }  }
365    
366    While it is easiest to write variables within typical 2D and 3D fields
367    where all data is known at a given time, it is also possible to write
368    fields where only a portion (\textit{eg.} a ``slab'' or ``slice'') is
369    known at a given instant.  An example is provided within
370    \filelink{pkg/mom\_vecinv/mom\_vecinv.F}{pkg-mom_vecinv-mom_vecinv.F}
371    where an offset vector is used: {\footnotesize
372    \begin{verbatim}
373           IF (useMNC .AND. snapshot_mnc) THEN
374             CALL MNC_CW_RL_W_OFFSET('D','mom_vi',bi,bj, 'fV', uCf,
375       &          offsets, myThid)
376             CALL MNC_CW_RL_W_OFFSET('D','mom_vi',bi,bj, 'fU', vCf,
377       &          offsets, myThid)
378           ENDIF
379    \end{verbatim}
380    }
381    to write a 3D field one depth slice at a time.
382    
383  \subsubsection{Parameters}  Each element in the offset vector corresponds (in order) to the
384    dimensions of the ``full'' (or virtual) array and specifies which are
385  Most of the MNC--related parameters are contained within a Fortran  known at the time of the call.  A zero within the offset array means
386  namelist file called \texttt{data.mnc}.  If this file does not exist,  that all values along that dimension are available while a positive
387  then the MNC package will interpret that as an indication that it is  integer means that only values along that index of the dimension are
388  not to be used.  If the \texttt{data.mnc} file does exist, then it may  available.  In all cases, the matrix passed is assumed to start (that
389  contain the following parameters:  is, have an in-memory structure) coinciding with the start of the
390    specified slice.  Thus, using this offset array mechanism, a slice
391  \begin{center}  can be written along any single dimension or combinations of
392    {\footnotesize  dimensions.
     \begin{tabular}[htb]{|l|c|l|l|}\hline  
       \textbf{Name}  &  \textbf{T}  &    
       \textbf{Default}  &  \textbf{Description}  \\\hline  
       &  &  &  \\  
       \texttt{useMNC}  &  L  & \texttt{.FALSE.}  &    
       \textbf{overall MNC ON/OFF switch}  \\  
       \texttt{mnc\_echo\_gvtypes}  &  L  & \texttt{.FALSE.}  &    
       echo pre-defined ``types'' (debugging)   \\  
       \texttt{mnc\_use\_outdir}  &  L  & \texttt{.FALSE.}  &    
       create a directory for output  \\  
       \texttt{mnc\_outdir\_str}  &  S  & \texttt{'mnc\_'}  &    
       output directory name \\  
       \texttt{mnc\_outdir\_date}  &  L  & \texttt{.FALSE.}  &    
       embed date in the output dir name  \\  
       \texttt{pickup\_write\_mnc}  &  L  & \texttt{.FALSE.}  &    
       use MNC to write (create) pickup files  \\  
       \texttt{pickup\_read\_mnc}  &  L  & \texttt{.FALSE.}  &    
       use MNC to read pickup files  \\  
       \texttt{mnc\_use\_indir}  &  L  & \texttt{.FALSE.}  &    
       use a directory (path) for input  \\  
       \texttt{mnc\_indir\_str}  &  S  & \texttt{''}  &    
       input directory (or path) name  \\  
       \texttt{snapshot\_mnc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{snapshot} (instantaneous) w/MNC  \\  
       \texttt{monitor\_mnc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{monitor} w/MNC  \\  
       \texttt{timeave\_mnc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{timeave} w/MNC  \\\hline  
     \end{tabular}  
   }  
 \end{center}  
   
 Additional MNC--related parameters are contained within the main  
 \texttt{data} namelist file and in some of the namelist files for  
 individual packages.  These options are:  
 \begin{center}  
   {\footnotesize  
     \begin{tabular}[htb]{|l|c|l|l|}\hline  
       \textbf{Name}  &  \textbf{T}  &    
       \textbf{Default}  &  \textbf{Description}  \\\hline  
       \multicolumn{4}{|c|}{\ }  \\  
       \multicolumn{4}{|c|}{Main namelist file:  
         ``\textbf{data}''}  \\\hline  
       \texttt{snapshot\_ioinc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{snapshot} ``inclusively''  \\  
       \texttt{timeave\_ioinc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{timeave} ``inclusively''  \\  
       \texttt{monitor\_ioinc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{monitor} ``inclusively''  \\\hline  
       \multicolumn{4}{|c|}{\ }  \\  
       \multicolumn{4}{|c|}{Diagnostics namelist file:  
         ``\textbf{data.diagnostics}''}  \\\hline  
       \texttt{diag\_mnc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{diagnostics} w/MNC  \\  
       \texttt{diag\_ioinc}  &  L  & \texttt{.FALSE.}  &    
       write \texttt{diagnostics} ``inclusively''  \\\hline  
     \end{tabular}  
   }  
 \end{center}  
   
 By default, turning on MNC for a particular output stream will result  
 in turning off all the corresponding (usually, default) MDSIO or  
 STDOUT output mechanisms.  In other words, output defaults to being an  
 exclusive selection.  To enable multiple kinds of simultaneous output,  
 flags of the form \texttt{NAME\_ioinc} can be used where \texttt{NAME}  
 corresponds to the various MNC output flags.  When a  
 \texttt{NAME\_ioinc} flag is set to \texttt{.TRUE.}, then multiple  
 forms of output are allowed for the \texttt{NAME} output mechanism.  
 The intent of this design is that typical users will only want one  
 kind of output while people debugging the code (particularly the I/O  
 routines) may want simultaneous types of output.  
   
 This ``inclusive'' versus ``exclusive'' design is easily applied in  
 cases where three or more kinds of output may be generated.  Thus, it  
 can be readily extended to additional new output types (eg. HDF5).  
393    
 Input types are always exclusive.  
Legend:
Removed from v.1.11  
changed lines
  Added in v.1.14
  ViewVC Help
Powered by ViewVC 1.1.22