/[MITgcm]/manual/s_getstarted/text/customization.tex
ViewVC logotype

Diff of /manual/s_getstarted/text/customization.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.5 by jmc, Sat Apr 15 14:43:59 2006 UTC revision 1.9 by jmc, Thu Aug 6 19:12:47 2009 UTC
# Line 1  Line 1 
1  \section[Customizing MITgcm]{Doing it yourself: customizing the code}  \section[Customizing MITgcm]{Doing it yourself: customizing the model configuration}
2  \label{sect:customize}  \label{sect:customize}
3  \begin{rawhtml}  \begin{rawhtml}
4  <!-- CMIREDIR:customizing_mitgcm: -->  <!-- CMIREDIR:customizing_mitgcm: -->
# Line 13  part of the code (the setup relative to Line 13  part of the code (the setup relative to
13  part is covered in the parallel implementation section) and on the  part is covered in the parallel implementation section) and on the
14  variables and parameters that you are likely to change.  variables and parameters that you are likely to change.
15    
   
 \subsection{Building/compiling the code elsewhere}  
   
 In the example above (section \ref{sect:buildingCode}) we built the  
 executable in the {\em input} directory of the experiment for  
 convenience. You can also configure and compile the code in other  
 locations, for example on a scratch disk with out having to copy the  
 entire source tree. The only requirement to do so is you have {\tt  
   genmake2} in your path or you know the absolute path to {\tt  
   genmake2}.  
   
 The following sections outline some possible methods of organizing  
 your source and data.  
   
 \subsubsection{Building from the {\em ../code directory}}  
   
 This is just as simple as building in the {\em input/} directory:  
 \begin{verbatim}  
 % cd verification/exp2/code  
 % ../../../tools/genmake2  
 % make depend  
 % make  
 \end{verbatim}  
 However, to run the model the executable ({\em mitgcmuv}) and input  
 files must be in the same place. If you only have one calculation to make:  
 \begin{verbatim}  
 % cd ../input  
 % cp ../code/mitgcmuv ./  
 % ./mitgcmuv > output.txt  
 \end{verbatim}  
 or if you will be making multiple runs with the same executable:  
 \begin{verbatim}  
 % cd ../  
 % cp -r input run1  
 % cp code/mitgcmuv run1  
 % cd run1  
 % ./mitgcmuv > output.txt  
 \end{verbatim}  
   
 \subsubsection{Building from a new directory}  
   
 Since the {\em input} directory contains input files it is often more  
 useful to keep {\em input} pristine and build in a new directory  
 within {\em verification/exp2/}:  
 \begin{verbatim}  
 % cd verification/exp2  
 % mkdir build  
 % cd build  
 % ../../../tools/genmake2 -mods=../code  
 % make depend  
 % make  
 \end{verbatim}  
 This builds the code exactly as before but this time you need to copy  
 either the executable or the input files or both in order to run the  
 model. For example,  
 \begin{verbatim}  
 % cp ../input/* ./  
 % ./mitgcmuv > output.txt  
 \end{verbatim}  
 or if you tend to make multiple runs with the same executable then  
 running in a new directory each time might be more appropriate:  
 \begin{verbatim}  
 % cd ../  
 % mkdir run1  
 % cp build/mitgcmuv run1/  
 % cp input/* run1/  
 % cd run1  
 % ./mitgcmuv > output.txt  
 \end{verbatim}  
   
 \subsubsection{Building on a scratch disk}  
   
 Model object files and output data can use up large amounts of disk  
 space so it is often the case that you will be operating on a large  
 scratch disk. Assuming the model source is in {\em ~/MITgcm} then the  
 following commands will build the model in {\em /scratch/exp2-run1}:  
 \begin{verbatim}  
 % cd /scratch/exp2-run1  
 % ~/MITgcm/tools/genmake2 -rootdir=~/MITgcm \  
   -mods=~/MITgcm/verification/exp2/code  
 % make depend  
 % make  
 \end{verbatim}  
 To run the model here, you'll need the input files:  
 \begin{verbatim}  
 % cp ~/MITgcm/verification/exp2/input/* ./  
 % ./mitgcmuv > output.txt  
 \end{verbatim}  
   
 As before, you could build in one directory and make multiple runs of  
 the one experiment:  
 \begin{verbatim}  
 % cd /scratch/exp2  
 % mkdir build  
 % cd build  
 % ~/MITgcm/tools/genmake2 -rootdir=~/MITgcm \  
   -mods=~/MITgcm/verification/exp2/code  
 % make depend  
 % make  
 % cd ../  
 % cp -r ~/MITgcm/verification/exp2/input run2  
 % cd run2  
 % ./mitgcmuv > output.txt  
 \end{verbatim}  
   
   
 \subsection{Using \texttt{genmake2}}  
 \label{sect:genmake}  
   
 To compile the code, first use the program \texttt{genmake2} (located  
 in the \texttt{tools} directory) to generate a Makefile.  
 \texttt{genmake2} is a shell script written to work with all  
 ``sh''--compatible shells including bash v1, bash v2, and Bourne.  
 Internally, \texttt{genmake2} determines the locations of needed  
 files, the compiler, compiler options, libraries, and Unix tools.  It  
 relies upon a number of ``optfiles'' located in the  
 \texttt{tools/build\_options} directory.  
   
 The purpose of the optfiles is to provide all the compilation options  
 for particular ``platforms'' (where ``platform'' roughly means the  
 combination of the hardware and the compiler) and code configurations.  
 Given the combinations of possible compilers and library dependencies  
 ({\it eg.}  MPI and NetCDF) there may be numerous optfiles available  
 for a single machine.  The naming scheme for the majority of the  
 optfiles shipped with the code is  
 \begin{center}  
   {\bf OS\_HARDWARE\_COMPILER }  
 \end{center}  
 where  
 \begin{description}  
 \item[OS] is the name of the operating system (generally the  
   lower-case output of the {\tt 'uname'} command)  
 \item[HARDWARE] is a string that describes the CPU type and  
   corresponds to output from the  {\tt 'uname -m'} command:  
   \begin{description}  
   \item[ia32] is for ``x86'' machines such as i386, i486, i586, i686,  
     and athlon  
   \item[ia64] is for Intel IA64 systems (eg. Itanium, Itanium2)  
   \item[amd64] is AMD x86\_64 systems  
   \item[ppc] is for Mac PowerPC systems  
   \end{description}  
 \item[COMPILER] is the compiler name (generally, the name of the  
   FORTRAN executable)  
 \end{description}  
   
 In many cases, the default optfiles are sufficient and will result in  
 usable Makefiles.  However, for some machines or code configurations,  
 new ``optfiles'' must be written. To create a new optfile, it is  
 generally best to start with one of the defaults and modify it to suit  
 your needs.  Like \texttt{genmake2}, the optfiles are all written  
 using a simple ``sh''--compatible syntax.  While nearly all variables  
 used within \texttt{genmake2} may be specified in the optfiles, the  
 critical ones that should be defined are:  
   
 \begin{description}  
 \item[FC] the FORTRAN compiler (executable) to use  
 \item[DEFINES] the command-line DEFINE options passed to the compiler  
 \item[CPP] the C pre-processor to use  
 \item[NOOPTFLAGS] options flags for special files that should not be  
   optimized  
 \end{description}  
   
 For example, the optfile for a typical Red Hat Linux machine (``ia32''  
 architecture) using the GCC (g77) compiler is  
 \begin{verbatim}  
 FC=g77  
 DEFINES='-D_BYTESWAPIO -DWORDLENGTH=4'  
 CPP='cpp  -traditional -P'  
 NOOPTFLAGS='-O0'  
 #  For IEEE, use the "-ffloat-store" option  
 if test "x$IEEE" = x ; then  
     FFLAGS='-Wimplicit -Wunused -Wuninitialized'  
     FOPTIM='-O3 -malign-double -funroll-loops'  
 else  
     FFLAGS='-Wimplicit -Wunused -ffloat-store'  
     FOPTIM='-O0 -malign-double'  
 fi  
 \end{verbatim}  
   
 If you write an optfile for an unrepresented machine or compiler, you  
 are strongly encouraged to submit the optfile to the MITgcm project  
 for inclusion.  Please send the file to the  
 \begin{rawhtml} <A href="mail-to:MITgcm-support@mitgcm.org"> \end{rawhtml}  
 \begin{center}  
   MITgcm-support@mitgcm.org  
 \end{center}  
 \begin{rawhtml} </A> \end{rawhtml}  
 mailing list.  
   
 In addition to the optfiles, \texttt{genmake2} supports a number of  
 helpful command-line options.  A complete list of these options can be  
 obtained from:  
 \begin{verbatim}  
 % genmake2 -h  
 \end{verbatim}  
   
 The most important command-line options are:  
 \begin{description}  
     
 \item[\texttt{--optfile=/PATH/FILENAME}] specifies the optfile that  
   should be used for a particular build.  
     
   If no "optfile" is specified (either through the command line or the  
   MITGCM\_OPTFILE environment variable), genmake2 will try to make a  
   reasonable guess from the list provided in {\em  
     tools/build\_options}.  The method used for making this guess is  
   to first determine the combination of operating system and hardware  
   (eg. "linux\_ia32") and then find a working FORTRAN compiler within  
   the user's path.  When these three items have been identified,  
   genmake2 will try to find an optfile that has a matching name.  
     
 \item[\texttt{--pdefault='PKG1 PKG2 PKG3 ...'}] specifies the default  
   set of packages to be used.  The normal order of precedence for  
   packages is as follows:  
   \begin{enumerate}  
   \item If available, the command line (\texttt{--pdefault}) settings  
     over-rule any others.  
   
   \item Next, \texttt{genmake2} will look for a file named  
     ``\texttt{packages.conf}'' in the local directory or in any of the  
     directories specified with the \texttt{--mods} option.  
       
   \item Finally, if neither of the above are available,  
     \texttt{genmake2} will use the \texttt{/pkg/pkg\_default} file.  
   \end{enumerate}  
     
 \item[\texttt{--pdepend=/PATH/FILENAME}] specifies the dependency file  
   used for packages.  
     
   If not specified, the default dependency file {\em pkg/pkg\_depend}  
   is used.  The syntax for this file is parsed on a line-by-line basis  
   where each line containes either a comment ("\#") or a simple  
   "PKGNAME1 (+|-)PKGNAME2" pairwise rule where the "+" or "-" symbol  
   specifies a "must be used with" or a "must not be used with"  
   relationship, respectively.  If no rule is specified, then it is  
   assumed that the two packages are compatible and will function  
   either with or without each other.  
     
 \item[\texttt{--adof=/path/to/file}] specifies the "adjoint" or  
   automatic differentiation options file to be used.  The file is  
   analogous to the ``optfile'' defined above but it specifies  
   information for the AD build process.  
     
   The default file is located in {\em  
     tools/adjoint\_options/adjoint\_default} and it defines the "TAF"  
   and "TAMC" compilers.  An alternate version is also available at  
   {\em tools/adjoint\_options/adjoint\_staf} that selects the newer  
   "STAF" compiler.  As with any compilers, it is helpful to have their  
   directories listed in your {\tt \$PATH} environment variable.  
     
 \item[\texttt{--mods='DIR1 DIR2 DIR3 ...'}] specifies a list of  
   directories containing ``modifications''.  These directories contain  
   files with names that may (or may not) exist in the main MITgcm  
   source tree but will be overridden by any identically-named sources  
   within the ``MODS'' directories.  
     
   The order of precedence for this "name-hiding" is as follows:  
   \begin{itemize}  
   \item ``MODS'' directories (in the order given)  
   \item Packages either explicitly specified or provided by default  
     (in the order given)  
   \item Packages included due to package dependencies (in the order  
     that that package dependencies are parsed)  
   \item The "standard dirs" (which may have been specified by the  
     ``-standarddirs'' option)  
   \end{itemize}  
     
 \item[\texttt{--mpi}] This option enables certain MPI features (using  
   CPP \texttt{\#define}s) within the code and is necessary for MPI  
   builds (see Section \ref{sect:mpi-build}).  
     
 \item[\texttt{--make=/path/to/gmake}] Due to the poor handling of  
   soft-links and other bugs common with the \texttt{make} versions  
   provided by commercial Unix vendors, GNU \texttt{make} (sometimes  
   called \texttt{gmake}) should be preferred.  This option provides a  
   means for specifying the make executable to be used.  
     
 \item[\texttt{--bash=/path/to/sh}] On some (usually older UNIX)  
   machines, the ``bash'' shell is unavailable.  To run on these  
   systems, \texttt{genmake2} can be invoked using an ``sh'' (that is,  
   a Bourne, POSIX, or compatible) shell.  The syntax in these  
   circumstances is:  
   \begin{center}  
     \texttt{\%  /bin/sh genmake2 -bash=/bin/sh [...options...]}  
   \end{center}  
   where \texttt{/bin/sh} can be replaced with the full path and name  
   of the desired shell.  
   
 \end{description}  
   
   
 \subsection{Building with MPI}  
 \label{sect:mpi-build}  
   
 Building MITgcm to use MPI libraries can be complicated due to the  
 variety of different MPI implementations available, their dependencies  
 or interactions with different compilers, and their often ad-hoc  
 locations within file systems.  For these reasons, its generally a  
 good idea to start by finding and reading the documentation for your  
 machine(s) and, if necessary, seeking help from your local systems  
 administrator.  
   
 The steps for building MITgcm with MPI support are:  
 \begin{enumerate}  
     
 \item Determine the locations of your MPI-enabled compiler and/or MPI  
   libraries and put them into an options file as described in Section  
   \ref{sect:genmake}.  One can start with one of the examples in:  
   \begin{rawhtml} <A  
     href="http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm/tools/build_options/">  
   \end{rawhtml}  
   \begin{center}  
     \texttt{MITgcm/tools/build\_options/}  
   \end{center}  
   \begin{rawhtml} </A> \end{rawhtml}  
   such as \texttt{linux\_ia32\_g77+mpi\_cg01} or  
   \texttt{linux\_ia64\_efc+mpi} and then edit it to suit the machine at  
   hand.  You may need help from your user guide or local systems  
   administrator to determine the exact location of the MPI libraries.  
   If libraries are not installed, MPI implementations and related  
   tools are available including:  
   \begin{itemize}  
   \item \begin{rawhtml} <A  
       href="http://www-unix.mcs.anl.gov/mpi/mpich/">  
     \end{rawhtml}  
     MPICH  
     \begin{rawhtml} </A> \end{rawhtml}  
   
   \item \begin{rawhtml} <A  
       href="http://www.lam-mpi.org/">  
     \end{rawhtml}  
     LAM/MPI  
     \begin{rawhtml} </A> \end{rawhtml}  
   
   \item \begin{rawhtml} <A  
       href="http://www.osc.edu/~pw/mpiexec/">  
     \end{rawhtml}  
     MPIexec  
     \begin{rawhtml} </A> \end{rawhtml}  
   \end{itemize}  
     
 \item Build the code with the \texttt{genmake2} \texttt{-mpi} option  
   (see Section \ref{sect:genmake}) using commands such as:  
 {\footnotesize \begin{verbatim}  
   %  ../../../tools/genmake2 -mods=../code -mpi -of=YOUR_OPTFILE  
   %  make depend  
   %  make  
 \end{verbatim} }  
     
 \item Run the code with the appropriate MPI ``run'' or ``exec''  
   program provided with your particular implementation of MPI.  
   Typical MPI packages such as MPICH will use something like:  
 \begin{verbatim}  
   %  mpirun -np 4 -machinefile mf ./mitgcmuv  
 \end{verbatim}  
   Sightly more complicated scripts may be needed for many machines  
   since execution of the code may be controlled by both the MPI  
   library and a job scheduling and queueing system such as PBS,  
   LoadLeveller, Condor, or any of a number of similar tools.  A few  
   example scripts (those used for our \begin{rawhtml} <A  
     href="http://mitgcm.org/testing.html"> \end{rawhtml}regular  
   verification runs\begin{rawhtml} </A> \end{rawhtml}) are available  
   at:  
   \begin{rawhtml} <A  
     href="http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm_contrib/test_scripts/">  
   \end{rawhtml}  
   {\footnotesize \tt  
     http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm\_contrib/test\_scripts/ }  
   \begin{rawhtml} </A> \end{rawhtml}  
   
 \end{enumerate}  
   
 An example of the above process on the MITgcm cluster (``cg01'') using  
 the GNU g77 compiler and the mpich MPI library is:  
   
 {\footnotesize \begin{verbatim}  
   %  cd MITgcm/verification/exp5  
   %  mkdir build  
   %  cd build  
   %  ../../../tools/genmake2 -mpi -mods=../code \  
        -of=../../../tools/build_options/linux_ia32_g77+mpi_cg01  
   %  make depend  
   %  make  
   %  cd ../input  
   %  /usr/local/pkg/mpi/mpi-1.2.4..8a-gm-1.5/g77/bin/mpirun.ch_gm \  
        -machinefile mf --gm-kill 5 -v -np 2  ../build/mitgcmuv  
 \end{verbatim} }  
   
 \subsection{Configuration and setup}  
   
16  The CPP keys relative to the ``numerical model'' part of the code are  The CPP keys relative to the ``numerical model'' part of the code are
17  all defined and set in the file \textit{CPP\_OPTIONS.h }in the  all defined and set in the file \textit{CPP\_OPTIONS.h }in the
18  directory \textit{ model/inc }or in one of the \textit{code  directory \textit{ model/inc }or in one of the \textit{code
# Line 414  values can be modified in the namelist f Line 24  values can be modified in the namelist f
24  to be located in the directory where you will run the model. The  to be located in the directory where you will run the model. The
25  parameters are initialized in the routine  parameters are initialized in the routine
26  \textit{model/src/ini\_parms.F}.  Look at this routine to see in what  \textit{model/src/ini\_parms.F}.  Look at this routine to see in what
27  part of the namelist the parameters are located.  part of the namelist the parameters are located. Here is a complete list
28    of the model parameters related to the main model (namelist parameters
29    for the packages are located in the package descriptions), their meaning,
30    and their default values:
31    
32    \input{./part3/main-parms.tex}
33    
34  In what follows the parameters are grouped into categories related to  In what follows the parameters are grouped into categories related to
35  the computational domain, the equations solved in the model, and the  the computational domain, the equations solved in the model, and the
36  simulation controls.  simulation controls.
37    
38  \subsection{Computational domain, geometry and time-discretization}  \subsection{Parameters: Computational domain, geometry and time-discretization}
39    
40  \begin{description}  \begin{description}
41  \item[dimensions] \  \item[dimensions] \
# Line 439  simulation controls. Line 54  simulation controls.
54    through the logical variables \textbf{usingCartesianGrid},    through the logical variables \textbf{usingCartesianGrid},
55    \textbf{usingSphericalPolarGrid}, and \textbf{usingCurvilinearGrid}.    \textbf{usingSphericalPolarGrid}, and \textbf{usingCurvilinearGrid}.
56    In the case of spherical and curvilinear grids, the southern    In the case of spherical and curvilinear grids, the southern
57    boundary is defined through the variable \textbf{phiMin} which    boundary is defined through the variable \textbf{ygOrigin} which
58    corresponds to the latitude of the southern most cell face (in    corresponds to the latitude of the southern most cell face (in
59    degrees). The resolution along the x and y directions is controlled    degrees). The resolution along the x and y directions is controlled
60    by the 1D arrays \textbf{delx} and \textbf{dely} (in meters in the    by the 1D arrays \textbf{delx} and \textbf{dely} (in meters in the
# Line 513  simulation controls. Line 128  simulation controls.
128  \end{description}  \end{description}
129    
130    
131  \subsection{Equation of state}  \subsection{Parameters: Equation of state}
132    
133  First, because the model equations are written in terms of  First, because the model equations are written in terms of
134  perturbations, a reference thermodynamic state needs to be specified.  perturbations, a reference thermodynamic state needs to be specified.
# Line 568  There there are also higher polynomials Line 183  There there are also higher polynomials
183  For none of these options an reference profile of temperature or  For none of these options an reference profile of temperature or
184  salinity is required.  salinity is required.
185    
186  \subsection{Momentum equations}  \subsection{Parameters: Momentum equations}
187    
188  In this section, we only focus for now on the parameters that you are  In this section, we only focus for now on the parameters that you are
189  likely to change, i.e. the ones relative to forcing and dissipation  likely to change, i.e. the ones relative to forcing and dissipation
# Line 587  variables. Line 202  variables.
202  \begin{description}  \begin{description}
203  \item[initialization] \  \item[initialization] \
204        
205    The velocity components are initialized to 0 unless the simulation    The initial horizontal velocity components can be specified from
206    is starting from a pickup file (see section on simulation control    binary files \textbf{uVelInitFile} and \textbf{vVelInitFile}.
207    parameters).    These files should contain 3D data ordered in an (x,y,r) fashion with
208      k=1 as the first vertical level (surface level).
209      If no file names are provided, the velocity is initialised to zero.
210      The initial vertical velocity is always derived from the horizontal velocity
211      using the continuity equation, even in the case of non-hydrostatic simulation
212      (see, e.g.: {\it tutorial\_deep\_convection/input/data}).
213    
214      In the case of a restart (from the end of a previous simulation),
215      the velocity field is read from a pickup file
216      (see section on simulation control parameters)
217      and the initial velocity files are ignored.
218    
219  \item[forcing] \  \item[forcing] \
220        
# Line 673  variables. Line 298  variables.
298    
299  \end{description}  \end{description}
300    
301  \subsection{Tracer equations}  \subsection{Parameters: Tracer equations}
302    
303  This section covers the tracer equations i.e. the potential  This section covers the tracer equations i.e. the potential
304  temperature equation and the salinity (for the ocean) or specific  temperature equation and the salinity (for the ocean) or specific
# Line 765  for a precise definition. Line 390  for a precise definition.
390    
391  \end{description}  \end{description}
392    
393  \subsection{Simulation controls}  \subsection{Parameters: Simulation controls}
394    
395  The model ''clock'' is defined by the variable \textbf{deltaTClock}  The model ''clock'' is defined by the variable \textbf{deltaTClock}
396  (in s) which determines the IO frequencies and is used in tagging  (in s) which determines the IO frequencies and is used in tagging
Legend:
Removed from v.1.5  
changed lines
  Added in v.1.9
  ViewVC Help
Powered by ViewVC 1.1.22