/[MITgcm]/manual/s_getstarted/text/getting_started.tex
ViewVC logotype

Diff of /manual/s_getstarted/text/getting_started.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.34 by edhill, Sat Apr 8 15:29:05 2006 UTC revision 1.35 by molod, Thu Apr 20 22:09:08 2006 UTC
# Line 15  structure are described more fully in ch Line 15  structure are described more fully in ch
15  this section, we provide information on how to customize the code when  this section, we provide information on how to customize the code when
16  you are ready to try implementing the configuration you have in mind.  you are ready to try implementing the configuration you have in mind.
17    
   
18  \section{Where to find information}  \section{Where to find information}
19  \label{sect:whereToFindInfo}  \label{sect:whereToFindInfo}
20  \begin{rawhtml}  \begin{rawhtml}
# Line 603  and then calling the executable with: Line 602  and then calling the executable with:
602  where we are re-directing the stream of text output to the file  where we are re-directing the stream of text output to the file
603  \texttt{output.txt}.  \texttt{output.txt}.
604    
605    \subsection{Building/compiling the code elsewhere}
606    
607    In the example above (section \ref{sect:buildingCode}) we built the
608    executable in the {\em input} directory of the experiment for
609    convenience. You can also configure and compile the code in other
610    locations, for example on a scratch disk with out having to copy the
611    entire source tree. The only requirement to do so is you have {\tt
612      genmake2} in your path or you know the absolute path to {\tt
613      genmake2}.
614    
615    The following sections outline some possible methods of organizing
616    your source and data.
617    
618    \subsubsection{Building from the {\em ../code directory}}
619    
620    This is just as simple as building in the {\em input/} directory:
621    \begin{verbatim}
622    % cd verification/exp2/code
623    % ../../../tools/genmake2
624    % make depend
625    % make
626    \end{verbatim}
627    However, to run the model the executable ({\em mitgcmuv}) and input
628    files must be in the same place. If you only have one calculation to make:
629    \begin{verbatim}
630    % cd ../input
631    % cp ../code/mitgcmuv ./
632    % ./mitgcmuv > output.txt
633    \end{verbatim}
634    or if you will be making multiple runs with the same executable:
635    \begin{verbatim}
636    % cd ../
637    % cp -r input run1
638    % cp code/mitgcmuv run1
639    % cd run1
640    % ./mitgcmuv > output.txt
641    \end{verbatim}
642    
643    \subsubsection{Building from a new directory}
644    
645    Since the {\em input} directory contains input files it is often more
646    useful to keep {\em input} pristine and build in a new directory
647    within {\em verification/exp2/}:
648    \begin{verbatim}
649    % cd verification/exp2
650    % mkdir build
651    % cd build
652    % ../../../tools/genmake2 -mods=../code
653    % make depend
654    % make
655    \end{verbatim}
656    This builds the code exactly as before but this time you need to copy
657    either the executable or the input files or both in order to run the
658    model. For example,
659    \begin{verbatim}
660    % cp ../input/* ./
661    % ./mitgcmuv > output.txt
662    \end{verbatim}
663    or if you tend to make multiple runs with the same executable then
664    running in a new directory each time might be more appropriate:
665    \begin{verbatim}
666    % cd ../
667    % mkdir run1
668    % cp build/mitgcmuv run1/
669    % cp input/* run1/
670    % cd run1
671    % ./mitgcmuv > output.txt
672    \end{verbatim}
673    
674    \subsubsection{Building on a scratch disk}
675    
676    Model object files and output data can use up large amounts of disk
677    space so it is often the case that you will be operating on a large
678    scratch disk. Assuming the model source is in {\em ~/MITgcm} then the
679    following commands will build the model in {\em /scratch/exp2-run1}:
680    \begin{verbatim}
681    % cd /scratch/exp2-run1
682    % ~/MITgcm/tools/genmake2 -rootdir=~/MITgcm \
683      -mods=~/MITgcm/verification/exp2/code
684    % make depend
685    % make
686    \end{verbatim}
687    To run the model here, you'll need the input files:
688    \begin{verbatim}
689    % cp ~/MITgcm/verification/exp2/input/* ./
690    % ./mitgcmuv > output.txt
691    \end{verbatim}
692    
693    As before, you could build in one directory and make multiple runs of
694    the one experiment:
695    \begin{verbatim}
696    % cd /scratch/exp2
697    % mkdir build
698    % cd build
699    % ~/MITgcm/tools/genmake2 -rootdir=~/MITgcm \
700      -mods=~/MITgcm/verification/exp2/code
701    % make depend
702    % make
703    % cd ../
704    % cp -r ~/MITgcm/verification/exp2/input run2
705    % cd run2
706    % ./mitgcmuv > output.txt
707    \end{verbatim}
708    
709    
710    \subsection{Using \texttt{genmake2}}
711    \label{sect:genmake}
712    
713    To compile the code, first use the program \texttt{genmake2} (located
714    in the \texttt{tools} directory) to generate a Makefile.
715    \texttt{genmake2} is a shell script written to work with all
716    ``sh''--compatible shells including bash v1, bash v2, and Bourne.
717    Internally, \texttt{genmake2} determines the locations of needed
718    files, the compiler, compiler options, libraries, and Unix tools.  It
719    relies upon a number of ``optfiles'' located in the
720    \texttt{tools/build\_options} directory.
721    
722    The purpose of the optfiles is to provide all the compilation options
723    for particular ``platforms'' (where ``platform'' roughly means the
724    combination of the hardware and the compiler) and code configurations.
725    Given the combinations of possible compilers and library dependencies
726    ({\it eg.}  MPI and NetCDF) there may be numerous optfiles available
727    for a single machine.  The naming scheme for the majority of the
728    optfiles shipped with the code is
729    \begin{center}
730      {\bf OS\_HARDWARE\_COMPILER }
731    \end{center}
732    where
733    \begin{description}
734    \item[OS] is the name of the operating system (generally the
735      lower-case output of the {\tt 'uname'} command)
736    \item[HARDWARE] is a string that describes the CPU type and
737      corresponds to output from the  {\tt 'uname -m'} command:
738      \begin{description}
739      \item[ia32] is for ``x86'' machines such as i386, i486, i586, i686,
740        and athlon
741      \item[ia64] is for Intel IA64 systems (eg. Itanium, Itanium2)
742      \item[amd64] is AMD x86\_64 systems
743      \item[ppc] is for Mac PowerPC systems
744      \end{description}
745    \item[COMPILER] is the compiler name (generally, the name of the
746      FORTRAN executable)
747    \end{description}
748    
749    In many cases, the default optfiles are sufficient and will result in
750    usable Makefiles.  However, for some machines or code configurations,
751    new ``optfiles'' must be written. To create a new optfile, it is
752    generally best to start with one of the defaults and modify it to suit
753    your needs.  Like \texttt{genmake2}, the optfiles are all written
754    using a simple ``sh''--compatible syntax.  While nearly all variables
755    used within \texttt{genmake2} may be specified in the optfiles, the
756    critical ones that should be defined are:
757    
758    \begin{description}
759    \item[FC] the FORTRAN compiler (executable) to use
760    \item[DEFINES] the command-line DEFINE options passed to the compiler
761    \item[CPP] the C pre-processor to use
762    \item[NOOPTFLAGS] options flags for special files that should not be
763      optimized
764    \end{description}
765    
766    For example, the optfile for a typical Red Hat Linux machine (``ia32''
767    architecture) using the GCC (g77) compiler is
768    \begin{verbatim}
769    FC=g77
770    DEFINES='-D_BYTESWAPIO -DWORDLENGTH=4'
771    CPP='cpp  -traditional -P'
772    NOOPTFLAGS='-O0'
773    #  For IEEE, use the "-ffloat-store" option
774    if test "x$IEEE" = x ; then
775        FFLAGS='-Wimplicit -Wunused -Wuninitialized'
776        FOPTIM='-O3 -malign-double -funroll-loops'
777    else
778        FFLAGS='-Wimplicit -Wunused -ffloat-store'
779        FOPTIM='-O0 -malign-double'
780    fi
781    \end{verbatim}
782    
783    If you write an optfile for an unrepresented machine or compiler, you
784    are strongly encouraged to submit the optfile to the MITgcm project
785    for inclusion.  Please send the file to the
786    \begin{rawhtml} <A href="mail-to:MITgcm-support@mitgcm.org"> \end{rawhtml}
787    \begin{center}
788      MITgcm-support@mitgcm.org
789    \end{center}
790    \begin{rawhtml} </A> \end{rawhtml}
791    mailing list.
792    
793    In addition to the optfiles, \texttt{genmake2} supports a number of
794    helpful command-line options.  A complete list of these options can be
795    obtained from:
796    \begin{verbatim}
797    % genmake2 -h
798    \end{verbatim}
799    
800    The most important command-line options are:
801    \begin{description}
802      
803    \item[\texttt{--optfile=/PATH/FILENAME}] specifies the optfile that
804      should be used for a particular build.
805      
806      If no "optfile" is specified (either through the command line or the
807      MITGCM\_OPTFILE environment variable), genmake2 will try to make a
808      reasonable guess from the list provided in {\em
809        tools/build\_options}.  The method used for making this guess is
810      to first determine the combination of operating system and hardware
811      (eg. "linux\_ia32") and then find a working FORTRAN compiler within
812      the user's path.  When these three items have been identified,
813      genmake2 will try to find an optfile that has a matching name.
814      
815    \item[\texttt{--pdefault='PKG1 PKG2 PKG3 ...'}] specifies the default
816      set of packages to be used.  The normal order of precedence for
817      packages is as follows:
818      \begin{enumerate}
819      \item If available, the command line (\texttt{--pdefault}) settings
820        over-rule any others.
821    
822      \item Next, \texttt{genmake2} will look for a file named
823        ``\texttt{packages.conf}'' in the local directory or in any of the
824        directories specified with the \texttt{--mods} option.
825        
826      \item Finally, if neither of the above are available,
827        \texttt{genmake2} will use the \texttt{/pkg/pkg\_default} file.
828      \end{enumerate}
829      
830    \item[\texttt{--pdepend=/PATH/FILENAME}] specifies the dependency file
831      used for packages.
832      
833      If not specified, the default dependency file {\em pkg/pkg\_depend}
834      is used.  The syntax for this file is parsed on a line-by-line basis
835      where each line containes either a comment ("\#") or a simple
836      "PKGNAME1 (+|-)PKGNAME2" pairwise rule where the "+" or "-" symbol
837      specifies a "must be used with" or a "must not be used with"
838      relationship, respectively.  If no rule is specified, then it is
839      assumed that the two packages are compatible and will function
840      either with or without each other.
841      
842    \item[\texttt{--adof=/path/to/file}] specifies the "adjoint" or
843      automatic differentiation options file to be used.  The file is
844      analogous to the ``optfile'' defined above but it specifies
845      information for the AD build process.
846      
847      The default file is located in {\em
848        tools/adjoint\_options/adjoint\_default} and it defines the "TAF"
849      and "TAMC" compilers.  An alternate version is also available at
850      {\em tools/adjoint\_options/adjoint\_staf} that selects the newer
851      "STAF" compiler.  As with any compilers, it is helpful to have their
852      directories listed in your {\tt \$PATH} environment variable.
853      
854    \item[\texttt{--mods='DIR1 DIR2 DIR3 ...'}] specifies a list of
855      directories containing ``modifications''.  These directories contain
856      files with names that may (or may not) exist in the main MITgcm
857      source tree but will be overridden by any identically-named sources
858      within the ``MODS'' directories.
859      
860      The order of precedence for this "name-hiding" is as follows:
861      \begin{itemize}
862      \item ``MODS'' directories (in the order given)
863      \item Packages either explicitly specified or provided by default
864        (in the order given)
865      \item Packages included due to package dependencies (in the order
866        that that package dependencies are parsed)
867      \item The "standard dirs" (which may have been specified by the
868        ``-standarddirs'' option)
869      \end{itemize}
870      
871    \item[\texttt{--mpi}] This option enables certain MPI features (using
872      CPP \texttt{\#define}s) within the code and is necessary for MPI
873      builds (see Section \ref{sect:mpi-build}).
874      
875    \item[\texttt{--make=/path/to/gmake}] Due to the poor handling of
876      soft-links and other bugs common with the \texttt{make} versions
877      provided by commercial Unix vendors, GNU \texttt{make} (sometimes
878      called \texttt{gmake}) should be preferred.  This option provides a
879      means for specifying the make executable to be used.
880      
881    \item[\texttt{--bash=/path/to/sh}] On some (usually older UNIX)
882      machines, the ``bash'' shell is unavailable.  To run on these
883      systems, \texttt{genmake2} can be invoked using an ``sh'' (that is,
884      a Bourne, POSIX, or compatible) shell.  The syntax in these
885      circumstances is:
886      \begin{center}
887        \texttt{\%  /bin/sh genmake2 -bash=/bin/sh [...options...]}
888      \end{center}
889      where \texttt{/bin/sh} can be replaced with the full path and name
890      of the desired shell.
891    
892    \end{description}
893    
894    
895    \subsection{Building with MPI}
896    \label{sect:mpi-build}
897    
898    Building MITgcm to use MPI libraries can be complicated due to the
899    variety of different MPI implementations available, their dependencies
900    or interactions with different compilers, and their often ad-hoc
901    locations within file systems.  For these reasons, its generally a
902    good idea to start by finding and reading the documentation for your
903    machine(s) and, if necessary, seeking help from your local systems
904    administrator.
905    
906    The steps for building MITgcm with MPI support are:
907    \begin{enumerate}
908      
909    \item Determine the locations of your MPI-enabled compiler and/or MPI
910      libraries and put them into an options file as described in Section
911      \ref{sect:genmake}.  One can start with one of the examples in:
912      \begin{rawhtml} <A
913        href="http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm/tools/build_options/">
914      \end{rawhtml}
915      \begin{center}
916        \texttt{MITgcm/tools/build\_options/}
917      \end{center}
918      \begin{rawhtml} </A> \end{rawhtml}
919      such as \texttt{linux\_ia32\_g77+mpi\_cg01} or
920      \texttt{linux\_ia64\_efc+mpi} and then edit it to suit the machine at
921      hand.  You may need help from your user guide or local systems
922      administrator to determine the exact location of the MPI libraries.
923      If libraries are not installed, MPI implementations and related
924      tools are available including:
925      \begin{itemize}
926      \item \begin{rawhtml} <A
927          href="http://www-unix.mcs.anl.gov/mpi/mpich/">
928        \end{rawhtml}
929        MPICH
930        \begin{rawhtml} </A> \end{rawhtml}
931    
932      \item \begin{rawhtml} <A
933          href="http://www.lam-mpi.org/">
934        \end{rawhtml}
935        LAM/MPI
936        \begin{rawhtml} </A> \end{rawhtml}
937    
938      \item \begin{rawhtml} <A
939          href="http://www.osc.edu/~pw/mpiexec/">
940        \end{rawhtml}
941        MPIexec
942        \begin{rawhtml} </A> \end{rawhtml}
943      \end{itemize}
944      
945    \item Build the code with the \texttt{genmake2} \texttt{-mpi} option
946      (see Section \ref{sect:genmake}) using commands such as:
947    {\footnotesize \begin{verbatim}
948      %  ../../../tools/genmake2 -mods=../code -mpi -of=YOUR_OPTFILE
949      %  make depend
950      %  make
951    \end{verbatim} }
952      
953    \item Run the code with the appropriate MPI ``run'' or ``exec''
954      program provided with your particular implementation of MPI.
955      Typical MPI packages such as MPICH will use something like:
956    \begin{verbatim}
957      %  mpirun -np 4 -machinefile mf ./mitgcmuv
958    \end{verbatim}
959      Sightly more complicated scripts may be needed for many machines
960      since execution of the code may be controlled by both the MPI
961      library and a job scheduling and queueing system such as PBS,
962      LoadLeveller, Condor, or any of a number of similar tools.  A few
963      example scripts (those used for our \begin{rawhtml} <A
964        href="http://mitgcm.org/testing.html"> \end{rawhtml}regular
965      verification runs\begin{rawhtml} </A> \end{rawhtml}) are available
966      at:
967      \begin{rawhtml} <A
968        href="http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm_contrib/test_scripts/">
969      \end{rawhtml}
970      {\footnotesize \tt
971        http://mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm\_contrib/test\_scripts/ }
972      \begin{rawhtml} </A> \end{rawhtml}
973    
974    \end{enumerate}
975    
976    An example of the above process on the MITgcm cluster (``cg01'') using
977    the GNU g77 compiler and the mpich MPI library is:
978    
979    {\footnotesize \begin{verbatim}
980      %  cd MITgcm/verification/exp5
981      %  mkdir build
982      %  cd build
983      %  ../../../tools/genmake2 -mpi -mods=../code \
984           -of=../../../tools/build_options/linux_ia32_g77+mpi_cg01
985      %  make depend
986      %  make
987      %  cd ../input
988      %  /usr/local/pkg/mpi/mpi-1.2.4..8a-gm-1.5/g77/bin/mpirun.ch_gm \
989           -machinefile mf --gm-kill 5 -v -np 2  ../build/mitgcmuv
990    \end{verbatim} }
991    
992  \section[Running MITgcm]{Running the model in prognostic mode}  \section[Running MITgcm]{Running the model in prognostic mode}
993  \label{sect:runModel}  \label{sect:runModel}
Legend:
Removed from v.1.34  
changed lines
  Added in v.1.35
  ViewVC Help
Powered by ViewVC 1.1.22