% $Header: /home/ubuntu/mnt/e9_copy/manual/s_getstarted/text/getting_started.tex,v 1.28 2004/10/14 14:54:24 cnh Exp $ % $Name: $ %\section{Getting started} In this section, we describe how to use the model. In the first section, we provide enough information to help you get started with the model. We believe the best way to familiarize yourself with the model is to run the case study examples provided with the base version. Information on how to obtain, compile, and run the code is found there as well as a brief description of the model structure directory and the case study examples. The latter and the code structure are described more fully in chapters \ref{chap:discretization} and \ref{chap:sarch}, respectively. Here, in this section, we provide information on how to customize the code when you are ready to try implementing the configuration you have in mind. \section{Where to find information} \label{sect:whereToFindInfo} A web site is maintained for release 2 (``Pelican'') of MITgcm: \begin{rawhtml} \end{rawhtml} \begin{verbatim} http://mitgcm.org/pelican \end{verbatim} \begin{rawhtml} \end{rawhtml} Here you will find an on-line version of this document, a ``browsable'' copy of the code and a searchable database of the model and site, as well as links for downloading the model and documentation, to data-sources, and other related sites. There is also a web-archived support mailing list for the model that you can email at \texttt{MITgcm-support@mitgcm.org} or browse at: \begin{rawhtml} \end{rawhtml} \begin{verbatim} http://mitgcm.org/mailman/listinfo/mitgcm-support/ http://mitgcm.org/pipermail/mitgcm-support/ \end{verbatim} \begin{rawhtml} \end{rawhtml} Essentially all of the MITgcm web pages can be searched using a popular web crawler such as Google or through our own search facility: \begin{rawhtml} \end{rawhtml} \begin{verbatim} http://mitgcm.org/htdig/ \end{verbatim} \begin{rawhtml} \end{rawhtml} %%% http://www.google.com/search?q=hydrostatic+site%3Amitgcm.org \section{Obtaining the code} \label{sect:obtainingCode} MITgcm can be downloaded from our system by following the instructions below. As a courtesy we ask that you send e-mail to us at \begin{rawhtml} \end{rawhtml} MITgcm-support@mitgcm.org \begin{rawhtml} \end{rawhtml} to enable us to keep track of who's using the model and in what application. You can download the model two ways: \begin{enumerate} \item Using CVS software. CVS is a freely available source code management tool. To use CVS you need to have the software installed. Many systems come with CVS pre-installed, otherwise good places to look for the software for a particular platform are \begin{rawhtml} \end{rawhtml} cvshome.org \begin{rawhtml} \end{rawhtml} and \begin{rawhtml} \end{rawhtml} wincvs.org \begin{rawhtml} \end{rawhtml} . \item Using a tar file. This method is simple and does not require any special software. However, this method does not provide easy support for maintenance updates. \end{enumerate} \subsection{Method 1 - Checkout from CVS} \label{sect:cvs_checkout} If CVS is available on your system, we strongly encourage you to use it. CVS provides an efficient and elegant way of organizing your code and keeping track of your changes. If CVS is not available on your machine, you can also download a tar file. Before you can use CVS, the following environment variable(s) should be set within your shell. For a csh or tcsh shell, put the following \begin{verbatim} % setenv CVSROOT :pserver:cvsanon@mitgcm.org:/u/gcmpack \end{verbatim} in your .cshrc or .tcshrc file. For bash or sh shells, put: \begin{verbatim} % export CVSROOT=':pserver:cvsanon@mitgcm.org:/u/gcmpack' \end{verbatim} in your \texttt{.profile} or \texttt{.bashrc} file. To get MITgcm through CVS, first register with the MITgcm CVS server using command: \begin{verbatim} % cvs login ( CVS password: cvsanon ) \end{verbatim} You only need to do a ``cvs login'' once. To obtain the latest sources type: \begin{verbatim} % cvs co MITgcm \end{verbatim} or to get a specific release type: \begin{verbatim} % cvs co -P -r checkpoint52i_post MITgcm \end{verbatim} The MITgcm web site contains further directions concerning the source code and CVS. It also contains a web interface to our CVS archive so that one may easily view the state of files, revisions, and other development milestones: \begin{rawhtml} \end{rawhtml} \begin{verbatim} http://mitgcm.org/source_code.html \end{verbatim} \begin{rawhtml} \end{rawhtml} As a convenience, the MITgcm CVS server contains aliases which are named subsets of the codebase. These aliases can be especially helpful when used over slow internet connections or on machines with restricted storage space. Table \ref{tab:cvsModules} contains a list of CVS aliases \begin{table}[htb] \centering \begin{tabular}[htb]{|lp{3.25in}|}\hline \textbf{Alias Name} & \textbf{Information (directories) Contained} \\\hline \texttt{MITgcm\_code} & Only the source code -- none of the verification examples. \\ \texttt{MITgcm\_verif\_basic} & Source code plus a small set of the verification examples (\texttt{global\_ocean.90x40x15}, \texttt{aim.5l\_cs}, \texttt{hs94.128x64x5}, \texttt{front\_relax}, and \texttt{plume\_on\_slope}). \\ \texttt{MITgcm\_verif\_atmos} & Source code plus all of the atmospheric examples. \\ \texttt{MITgcm\_verif\_ocean} & Source code plus all of the oceanic examples. \\ \texttt{MITgcm\_verif\_all} & Source code plus all of the verification examples. \\\hline \end{tabular} \caption{MITgcm CVS Modules} \label{tab:cvsModules} \end{table} The checkout process creates a directory called \textit{MITgcm}. If the directory \textit{MITgcm} exists this command updates your code based on the repository. Each directory in the source tree contains a directory \textit{CVS}. This information is required by CVS to keep track of your file versions with respect to the repository. Don't edit the files in \textit{CVS}! You can also use CVS to download code updates. More extensive information on using CVS for maintaining MITgcm code can be found \begin{rawhtml} \end{rawhtml} here \begin{rawhtml} \end{rawhtml} . It is important to note that the CVS aliases in Table \ref{tab:cvsModules} cannot be used in conjunction with the CVS \texttt{-d DIRNAME} option. However, the \texttt{MITgcm} directories they create can be changed to a different name following the check-out: \begin{verbatim} % cvs co MITgcm_verif_basic % mv MITgcm MITgcm_verif_basic \end{verbatim} \subsection{Method 2 - Tar file download} \label{sect:conventionalDownload} If you do not have CVS on your system, you can download the model as a tar file from the web site at: \begin{rawhtml} \end{rawhtml} \begin{verbatim} http://mitgcm.org/download/ \end{verbatim} \begin{rawhtml} \end{rawhtml} The tar file still contains CVS information which we urge you not to delete; even if you do not use CVS yourself the information can help us if you should need to send us your copy of the code. If a recent tar file does not exist, then please contact the developers through the \begin{rawhtml} \end{rawhtml} MITgcm-support@mitgcm.org \begin{rawhtml} \end{rawhtml} mailing list. \subsubsection{Upgrading from an earlier version} If you already have an earlier version of the code you can ``upgrade'' your copy instead of downloading the entire repository again. First, ``cd'' (change directory) to the top of your working copy: \begin{verbatim} % cd MITgcm \end{verbatim} and then issue the cvs update command such as: \begin{verbatim} % cvs -q update -r checkpoint52i_post -d -P \end{verbatim} This will update the ``tag'' to ``checkpoint52i\_post'', add any new directories (-d) and remove any empty directories (-P). The -q option means be quiet which will reduce the number of messages you'll see in the terminal. If you have modified the code prior to upgrading, CVS will try to merge your changes with the upgrades. If there is a conflict between your modifications and the upgrade, it will report that file with a ``C'' in front, e.g.: \begin{verbatim} C model/src/ini_parms.F \end{verbatim} If the list of conflicts scrolled off the screen, you can re-issue the cvs update command and it will report the conflicts. Conflicts are indicated in the code by the delimites ``$<<<<<<<$'', ``======='' and ``$>>>>>>>$''. For example, {\small \begin{verbatim} <<<<<<< ini_parms.F & bottomDragLinear,myOwnBottomDragCoefficient, ======= & bottomDragLinear,bottomDragQuadratic, >>>>>>> 1.18 \end{verbatim} } means that you added ``myOwnBottomDragCoefficient'' to a namelist at the same time and place that we added ``bottomDragQuadratic''. You need to resolve this conflict and in this case the line should be changed to: {\small \begin{verbatim} & bottomDragLinear,bottomDragQuadratic,myOwnBottomDragCoefficient, \end{verbatim} } and the lines with the delimiters ($<<<<<<$,======,$>>>>>>$) be deleted. Unless you are making modifications which exactly parallel developments we make, these types of conflicts should be rare. \paragraph*{Upgrading to the current pre-release version} We don't make a ``release'' for every little patch and bug fix in order to keep the frequency of upgrades to a minimum. However, if you have run into a problem for which ``we have already fixed in the latest code'' and we haven't made a ``tag'' or ``release'' since that patch then you'll need to get the latest code: \begin{verbatim} % cvs -q update -A -d -P \end{verbatim} Unlike, the ``check-out'' and ``update'' procedures above, there is no ``tag'' or release name. The -A tells CVS to upgrade to the very latest version. As a rule, we don't recommend this since you might upgrade while we are in the processes of checking in the code so that you may only have part of a patch. Using this method of updating also means we can't tell what version of the code you are working with. So please be sure you understand what you're doing. \section{Model and directory structure} The ``numerical'' model is contained within a execution environment support wrapper. This wrapper is designed to provide a general framework for grid-point models. MITgcmUV is a specific numerical model that uses the framework. Under this structure the model is split into execution environment support code and conventional numerical model code. The execution environment support code is held under the \textit{eesupp} directory. The grid point model code is held under the \textit{model} directory. Code execution actually starts in the \textit{eesupp} routines and not in the \textit{model} routines. For this reason the top-level \textit{MAIN.F} is in the \textit{eesupp/src} directory. In general, end-users should not need to worry about this level. The top-level routine for the numerical part of the code is in \textit{model/src/THE\_MODEL\_MAIN.F}. Here is a brief description of the directory structure of the model under the root tree (a detailed description is given in section 3: Code structure). \begin{itemize} \item \textit{bin}: this directory is initially empty. It is the default directory in which to compile the code. \item \textit{diags}: contains the code relative to time-averaged diagnostics. It is subdivided into two subdirectories \textit{inc} and \textit{src} that contain include files (*.\textit{h} files) and Fortran subroutines (*.\textit{F} files), respectively. \item \textit{doc}: contains brief documentation notes. \item \textit{eesupp}: contains the execution environment source code. Also subdivided into two subdirectories \textit{inc} and \textit{src}. \item \textit{exe}: this directory is initially empty. It is the default directory in which to execute the code. \item \textit{model}: this directory contains the main source code. Also subdivided into two subdirectories \textit{inc} and \textit{src}. \item \textit{pkg}: contains the source code for the packages. Each package corresponds to a subdirectory. For example, \textit{gmredi} contains the code related to the Gent-McWilliams/Redi scheme, \textit{aim} the code relative to the atmospheric intermediate physics. The packages are described in detail in section 3. \item \textit{tools}: this directory contains various useful tools. For example, \textit{genmake2} is a script written in csh (C-shell) that should be used to generate your makefile. The directory \textit{adjoint} contains the makefile specific to the Tangent linear and Adjoint Compiler (TAMC) that generates the adjoint code. The latter is described in details in part V. \item \textit{utils}: this directory contains various utilities. The subdirectory \textit{knudsen2} contains code and a makefile that compute coefficients of the polynomial approximation to the knudsen formula for an ocean nonlinear equation of state. The \textit{matlab} subdirectory contains matlab scripts for reading model output directly into matlab. \textit{scripts} contains C-shell post-processing scripts for joining processor-based and tiled-based model output. \item \textit{verification}: this directory contains the model examples. See section \ref{sect:modelExamples}. \end{itemize} \section[MITgcm Example Experiments]{Example experiments} \label{sect:modelExamples} %% a set of twenty-four pre-configured numerical experiments The MITgcm distribution comes with more than a dozen pre-configured numerical experiments. Some of these example experiments are tests of individual parts of the model code, but many are fully fledged numerical simulations. A few of the examples are used for tutorial documentation in sections \ref{sect:eg-baro} - \ref{sect:eg-global}. The other examples follow the same general structure as the tutorial examples. However, they only include brief instructions in a text file called {\it README}. The examples are located in subdirectories under the directory \textit{verification}. Each example is briefly described below. \subsection{Full list of model examples} \begin{enumerate} \item \textit{exp0} - single layer, ocean double gyre (barotropic with free-surface). This experiment is described in detail in section \ref{sect:eg-baro}. \item \textit{exp1} - Four layer, ocean double gyre. This experiment is described in detail in section \ref{sect:eg-baroc}. \item \textit{exp2} - 4x4 degree global ocean simulation with steady climatological forcing. This experiment is described in detail in section \ref{sect:eg-global}. \item \textit{exp4} - Flow over a Gaussian bump in open-water or channel with open boundaries. \item \textit{exp5} - Inhomogenously forced ocean convection in a doubly periodic box. \item \textit{front\_relax} - Relaxation of an ocean thermal front (test for Gent/McWilliams scheme). 2D (Y-Z). \item \textit{internal wave} - Ocean internal wave forced by open boundary conditions. \item \textit{natl\_box} - Eastern subtropical North Atlantic with KPP scheme; 1 month integration \item \textit{hs94.1x64x5} - Zonal averaged atmosphere using Held and Suarez '94 forcing. \item \textit{hs94.128x64x5} - 3D atmosphere dynamics using Held and Suarez '94 forcing. \item \textit{hs94.cs-32x32x5} - 3D atmosphere dynamics using Held and Suarez '94 forcing on the cubed sphere. \item \textit{aim.5l\_zon-ave} - Intermediate Atmospheric physics. Global Zonal Mean configuration, 1x64x5 resolution. \item \textit{aim.5l\_XZ\_Equatorial\_Slice} - Intermediate Atmospheric physics, equatorial Slice configuration. 2D (X-Z). \item \textit{aim.5l\_Equatorial\_Channel} - Intermediate Atmospheric physics. 3D Equatorial Channel configuration. \item \textit{aim.5l\_LatLon} - Intermediate Atmospheric physics. Global configuration, on latitude longitude grid with 128x64x5 grid points ($2.8^\circ{\rm degree}$ resolution). \item \textit{adjustment.128x64x1} Barotropic adjustment problem on latitude longitude grid with 128x64 grid points ($2.8^\circ{\rm degree}$ resolution). \item \textit{adjustment.cs-32x32x1} Barotropic adjustment problem on cube sphere grid with 32x32 points per face ( roughly $2.8^\circ{\rm degree}$ resolution). \item \textit{advect\_cs} Two-dimensional passive advection test on cube sphere grid. \item \textit{advect\_xy} Two-dimensional (horizontal plane) passive advection test on Cartesian grid. \item \textit{advect\_yz} Two-dimensional (vertical plane) passive advection test on Cartesian grid. \item \textit{carbon} Simple passive tracer experiment. Includes derivative calculation. Described in detail in section \ref{sect:eg-carbon-ad}. \item \textit{flt\_example} Example of using float package. \item \textit{global\_ocean.90x40x15} Global circulation with GM, flux boundary conditions and poles. \item \textit{global\_ocean\_pressure} Global circulation in pressure coordinate (non-Boussinesq ocean model). Described in detail in section \ref{sect:eg-globalpressure}. \item \textit{solid-body.cs-32x32x1} Solid body rotation test for cube sphere grid. \end{enumerate} \subsection{Directory structure of model examples} Each example directory has the following subdirectories: \begin{itemize} \item \textit{code}: contains the code particular to the example. At a minimum, this directory includes the following files: \begin{itemize} \item \textit{code/CPP\_EEOPTIONS.h}: declares CPP keys relative to the ``execution environment'' part of the code. The default version is located in \textit{eesupp/inc}. \item \textit{code/CPP\_OPTIONS.h}: declares CPP keys relative to the ``numerical model'' part of the code. The default version is located in \textit{model/inc}. \item \textit{code/SIZE.h}: declares size of underlying computational grid. The default version is located in \textit{model/inc}. \end{itemize} In addition, other include files and subroutines might be present in \textit{code} depending on the particular experiment. See Section 2 for more details. \item \textit{input}: contains the input data files required to run the example. At a minimum, the \textit{input} directory contains the following files: \begin{itemize} \item \textit{input/data}: this file, written as a namelist, specifies the main parameters for the experiment. \item \textit{input/data.pkg}: contains parameters relative to the packages used in the experiment. \item \textit{input/eedata}: this file contains ``execution environment'' data. At present, this consists of a specification of the number of threads to use in $X$ and $Y$ under multithreaded execution. \end{itemize} In addition, you will also find in this directory the forcing and topography files as well as the files describing the initial state of the experiment. This varies from experiment to experiment. See section 2 for more details. \item \textit{results}: this directory contains the output file \textit{output.txt} produced by the simulation example. This file is useful for comparison with your own output when you run the experiment. \end{itemize} Once you have chosen the example you want to run, you are ready to compile the code. \section[Building MITgcm]{Building the code} \label{sect:buildingCode} To compile the code, we use the {\em make} program. This uses a file ({\em Makefile}) that allows us to pre-process source files, specify compiler and optimization options and also figures out any file dependencies. We supply a script ({\em genmake2}), described in section \ref{sect:genmake}, that automatically creates the {\em Makefile} for you. You then need to build the dependencies and compile the code. As an example, let's assume that you want to build and run experiment \textit{verification/exp2}. The are multiple ways and places to actually do this but here let's build the code in \textit{verification/exp2/input}: \begin{verbatim} % cd verification/exp2/input \end{verbatim} First, build the {\em Makefile}: \begin{verbatim} % ../../../tools/genmake2 -mods=../code \end{verbatim} The command line option tells {\em genmake} to override model source code with any files in the directory {\em ./code/}. On many systems, the {\em genmake2} program will be able to automatically recognize the hardware, find compilers and other tools within the user's path (``echo \$PATH''), and then choose an appropriate set of options from the files contained in the {\em tools/build\_options} directory. Under some circumstances, a user may have to create a new ``optfile'' in order to specify the exact combination of compiler, compiler flags, libraries, and other options necessary to build a particular configuration of MITgcm. In such cases, it is generally helpful to read the existing ``optfiles'' and mimic their syntax. Through the MITgcm-support list, the MITgcm developers are willing to provide help writing or modifing ``optfiles''. And we encourage users to post new ``optfiles'' (particularly ones for new machines or architectures) to the \begin{rawhtml} \end{rawhtml} MITgcm-support@mitgcm.org \begin{rawhtml} \end{rawhtml} list. To specify an optfile to {\em genmake2}, the syntax is: \begin{verbatim} % ../../../tools/genmake2 -mods=../code -of /path/to/optfile \end{verbatim} Once a {\em Makefile} has been generated, we create the dependencies: \begin{verbatim} % make depend \end{verbatim} This modifies the {\em Makefile} by attaching a [long] list of files upon which other files depend. The purpose of this is to reduce re-compilation if and when you start to modify the code. The {\tt make depend} command also creates links from the model source to this directory. Next compile the code: \begin{verbatim} % make \end{verbatim} The {\tt make} command creates an executable called \textit{mitgcmuv}. Additional make ``targets'' are defined within the makefile to aid in the production of adjoint and other versions of MITgcm. Now you are ready to run the model. General instructions for doing so are given in section \ref{sect:runModel}. Here, we can run the model with: \begin{verbatim} ./mitgcmuv > output.txt \end{verbatim} where we are re-directing the stream of text output to the file {\em output.txt}. \section[Running MITgcm]{Running the model in prognostic mode} \label{sect:runModel} If compilation finished succesfuully (section \ref{sect:buildingCode}) then an executable called \texttt{mitgcmuv} will now exist in the local directory. To run the model as a single process (ie. not in parallel) simply type: \begin{verbatim} % ./mitgcmuv \end{verbatim} The ``./'' is a safe-guard to make sure you use the local executable in case you have others that exist in your path (surely odd if you do!). The above command will spew out many lines of text output to your screen. This output contains details such as parameter values as well as diagnostics such as mean Kinetic energy, largest CFL number, etc. It is worth keeping this text output with the binary output so we normally re-direct the {\em stdout} stream as follows: \begin{verbatim} % ./mitgcmuv > output.txt \end{verbatim} For the example experiments in {\em verification}, an example of the output is kept in {\em results/output.txt} for comparison. You can compare your {\em output.txt} with this one to check that the set-up works. \subsection{Output files} The model produces various output files. At a minimum, the instantaneous ``state'' of the model is written out, which is made of the following files: \begin{itemize} \item \textit{U.00000nIter} - zonal component of velocity field (m/s and $> 0 $ eastward). \item \textit{V.00000nIter} - meridional component of velocity field (m/s and $> 0$ northward). \item \textit{W.00000nIter} - vertical component of velocity field (ocean: m/s and $> 0$ upward, atmosphere: Pa/s and $> 0$ towards increasing pressure i.e. downward). \item \textit{T.00000nIter} - potential temperature (ocean: $^{0}$C, atmosphere: $^{0}$K). \item \textit{S.00000nIter} - ocean: salinity (psu), atmosphere: water vapor (g/kg). \item \textit{Eta.00000nIter} - ocean: surface elevation (m), atmosphere: surface pressure anomaly (Pa). \end{itemize} The chain \textit{00000nIter} consists of ten figures that specify the iteration number at which the output is written out. For example, \textit{% U.0000000300} is the zonal velocity at iteration 300. In addition, a ``pickup'' or ``checkpoint'' file called: \begin{itemize} \item \textit{pickup.00000nIter} \end{itemize} is written out. This file represents the state of the model in a condensed form and is used for restarting the integration. If the C-D scheme is used, there is an additional ``pickup'' file: \begin{itemize} \item \textit{pickup\_cd.00000nIter} \end{itemize} containing the D-grid velocity data and that has to be written out as well in order to restart the integration. Rolling checkpoint files are the same as the pickup files but are named differently. Their name contain the chain \textit{ckptA} or \textit{ckptB} instead of \textit{00000nIter}. They can be used to restart the model but are overwritten every other time they are output to save disk space during long integrations. \subsection{Looking at the output} All the model data are written according to a ``meta/data'' file format. Each variable is associated with two files with suffix names \textit{.data} and \textit{.meta}. The \textit{.data} file contains the data written in binary form (big\_endian by default). The \textit{.meta} file is a ``header'' file that contains information about the size and the structure of the \textit{.data} file. This way of organizing the output is particularly useful when running multi-processors calculations. The base version of the model includes a few matlab utilities to read output files written in this format. The matlab scripts are located in the directory \textit{utils/matlab} under the root tree. The script \textit{rdmds.m} reads the data. Look at the comments inside the script to see how to use it. Some examples of reading and visualizing some output in {\em Matlab}: \begin{verbatim} % matlab >> H=rdmds('Depth'); >> contourf(H');colorbar; >> title('Depth of fluid as used by model'); >> eta=rdmds('Eta',10); >> imagesc(eta');axis ij;colorbar; >> title('Surface height at iter=10'); >> eta=rdmds('Eta',[0:10:100]); >> for n=1:11; imagesc(eta(:,:,n)');axis ij;colorbar;pause(.5);end \end{verbatim}