/[MITgcm]/MITgcm/doc/devel_HOWTO.sgml
ViewVC logotype

Diff of /MITgcm/doc/devel_HOWTO.sgml

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.2 by edhill, Wed Aug 20 19:47:14 2003 UTC revision 1.4 by edhill, Thu Nov 20 00:01:42 2003 UTC
# Line 128  Build commands: Line 128  Build commands:
128         doc                      tags -- connect to real docs?         doc                      tags -- connect to real docs?
129         eesupp                   cnh?         eesupp                   cnh?
130         exe                      ecco user build         exe                      ecco user build
131      *- jobs                     runtime shell scripts for      ,- jobs                     runtime shell scripts for
132      |                             various platforms      |                             various platforms
133      |  lsopt                    line search      |  lsopt                    line search
134     m|  model                    main dynamics (core)     m|  model                    main dynamics (core)
135     e|    optimization_drivers   ?     e|    optimization_drivers   ?
136     r|  optim                    line search interface     r|  optim                    line search interface
137     g|  pkg                      alternate and optional numerics, etc.     g|  pkg                      alternate and optional numerics, etc.
138     e*- tools     e|- tools
139     ?|  tutorial_examples        documented tests     ?|  tutorial_examples        documented tests
140      |                             only populated on release1 branch      |                             only populated on release1 branch
141      |                             and not validated during "testscript"      |                             and not validated during "testscript"
142      *- utils      '- utils
143         verification             std tests         verification             std tests
144    
145    
# Line 316  $ cvs co MITgcm manual mitgcm.org Line 316  $ cvs co MITgcm manual mitgcm.org
316      </sect2>      </sect2>
317    
318      <sect2>      <sect2>
319        <title>Editing</title>        <title>Editing the Documentation</title>
320    
321        <para>The documentation is contained in the        <para>The documentation is contained in the
322        <filename>manual</filename> directory in a raw LaTeX format.        <filename>manual</filename> directory in a raw LaTeX format.
# Line 364  $ cvs co MITgcm manual mitgcm.org Line 364  $ cvs co MITgcm manual mitgcm.org
364      </sect2>      </sect2>
365    
366      <sect2>      <sect2>
367        <title>Building</title> <para>Given the directory structure of        <title>Building the Documentation</title>
368        <xref linkend="documentation_getting">, the entire documentation        
369        for the web site can be built using:</para>        <para>Given the directory structure of <xref
370          linkend="documentation_getting">, the entire documentation for the web
371          site can be built using:</para>
372    
373  <screen>  <screen>
374  $ cd mitgcm.org/devel/buildweb  $ cd mitgcm.org/devel/buildweb
# Line 398  $ mv scratch/dev_docs /u/u0/httpd/html Line 400  $ mv scratch/dev_docs /u/u0/httpd/html
400    </sect1>    </sect1>
401    
402    <sect1 id="coding">    <sect1 id="coding">
403      <title>Coding</title>      <title>Coding for MITgcm</title>
404    
405        <sect2 id="build_tools">
406          <title>Build Tools</title>
407    
408          <para>Many Open Source projects use the "GNU Autotools" to help streamline
409            the build process for various Unix and Unix-like architectures.  For a
410            user, the result is the common "configure" (that is,
411            "<filename>./configure && make && make install</filename>") commands.
412            For MITgcm, the process is similar.  Typical commands are:</para>
413    
414    <screen>
415    $ genmake -mods=../code
416    $ make depend
417    $ make
418    </screen>
419    
420          <para>The following sections describe the individual steps in the build
421            process.</para>
422          
423          <sect3 id="genmake">
424            <title>The <filename>genmake2</> Utility</title>
425    
426            <para><emphasis>Please note that the older <filename>genmake</> is
427                deprecated and will eventually be replaced by <filename>genmake2</>.
428                This HOWTO only describes the newer tool.</emphasis></para>
429    
430            <para>The first step in any MITgcm build is to create a Unix-style
431              <filename>Makefile</filename> which will be parsed by
432              <filename>make</filename> to specify how to compile the MITgcm source
433              files.  For more detailed descriptions of what the make tools are and
434              how they are used, please see:</para>
435    
436            <itemizedlist>
437              <listitem>
438                <para><ulink url="http://www.gnu.org/software/make/make.html">
439                    http://www.gnu.org/software/make/make.html</></para>
440              </listitem>
441              <listitem>
442                <para><ulink url="http://www.oreilly.com/catalog/make2/">
443                    http://www.oreilly.com/catalog/make2/</></para>
444              </listitem>
445            </itemizedlist>
446    
447            <para>Genmake can often be invoked successfully with a command line as
448            simple as:</para>
449    
450    <screen>
451    $ genmake2 -mods=../code
452    </screen>
453    
454            <para>However, some systems (particularly commercial Unixes that lack a
455              more modern "/bin/sh" implementation or that have shells installed in
456              odd locations) may require an explicit shell invocation such as one of
457              the following: </para>
458    
459    <screen>
460    $ /usr/bin/sh genmake2 -make=gmake  -mods=../code
461    $ /opt/gnu/bin/bash genmake2 -ieee -make=/usr/local/bin/gmake -mods=../code
462    </screen>
463    
464            <para>The genmake2 code has been written in a Bourne and BASH (v1)
465            compatible syntax so it should work with most "sh" and all recent "bash"
466            implementations.</para>
467    
468            <para>As the name implies, <filename>genmake2</filename> generates a
469              <filename>Makefile</filename>.  It does so by first parsing the
470              information supplied from the following sources</para>
471    
472            <orderedlist>
473              <listitem>
474                <para>a <filename>gm_local</filename> file in the current
475                  directory</para>
476              </listitem>
477              <listitem>
478                <para>directly from command-line options</para>
479              </listitem>
480              <listitem>
481                <para>an "options file" as specified by the command-line option
482                  <filename>-optfile='FILENAME'</filename></para>
483              </listitem>
484            </orderedlist>
485    
486            <para>then checking certain dependency rules (the package dependencies),
487              and finally writing a <filename>Makefile</filename> based upon the
488              source code that it finds.  For convenience within various Unix
489              shells, <filename>genmake2</> supports both "long"- and "short"-style
490              options.  A complete list of the available options can be obtained
491              from:</para>
492    
493    <screen>
494    $ genmake2 -help
495    </screen>
496    
497            <para>The most important options for <filename>genmake2</> are:</para>
498    
499            <variablelist>
500    
501              <varlistentry>
502                <term><filename>--optfile=/PATH/FILENAME</></term>
503    
504                <listitem>
505                  <para>This specifies the "options file" that should be used for a
506                    particular build.  The options file is a convenient and
507                    machine-indepenent way of specifying parameters such as the
508                    FORTRAN compiler (<filename>FC=</>), FORTRAN compiler
509                    optimization flags (<filename>FFLAGS=</>), and the locations of
510                    various platform- and/or machine-specific tools
511                    (eg. <filename>MAKEDEPEND=</>).  As with <filename>genmake2</>,
512                    all options files should be written to be compatible with
513                    Bourne--shell ("sh" or "BASH v1") syntax.  Examples of various
514                    options files can be found in
515                    <filename>$ROOTDIR/tools/build_options</>.</para>
516    
517                  <para>If no "optfile" is specified (either through the command lin
518                    or the environment variable), genmake2 will try to make a
519                    reasonable guess from the list provided in
520                    <filename>$ROOTDIR/tools/build_options</>.  The method used for
521                    making this guess is to first determine the combination of
522                    operating system and hardware (eg. "linux_ia32") and then find a
523                    working Fortran compiler within the user's path.  When these
524                    three items have been identified, genmake2 will try to find an
525                    optfile that has a matching name. </para>
526    
527                  <para>Everyone is encouraged to submit their options files to the
528                    MITgcm project for inclusion (please send to
529                    <email>MITgcm-support@mitgcm.org</email>).  We are particularly
530                    grateful for options files tested on new or unique
531                    platforms!</para>
532                </listitem>
533    
534              </varlistentry>
535    
536              <varlistentry>
537                <term><filename>-pdepend=/PATH/FILENAME</></term>
538    
539                <listitem>
540                  <para>This specifies the dependency file used for packages.  If
541                  not specified, the default dependency file is
542                  <filename>$ROOTDIR/pkg/pkg_depend</>.  The syntax for this file is
543                  parsed on a line-by-line basis where each line containes either a
544                  comment ("#") or a simple "PKGNAME1 (+|-)PKGNAME2" pairwise rule
545                  where the "+" or "-" symbol specifies a "must be used with" or a
546                  "must not be used with" relationship, respectively.  If no rule is
547                  specified, then it is assumed that the two packages are compatible
548                  and will function either with or without each other.</para>
549                </listitem>
550    
551              </varlistentry>
552    
553              <varlistentry>
554                <term><filename>-pdefault=PKG</></term>
555                <term><filename>-pdefault='PKG1 [PKG2 PKG3 ...]'</></term>
556                <listitem>
557                  <para>This option specifies the default set of packages
558                  to be used.  If not set, the default package list will
559                  be read from
560                  <filename>$ROOTDIR/pkg/pkg_default</>.</para>
561                </listitem>
562              </varlistentry>
563    
564              <varlistentry>
565                <term><filename>-mods=DIR</></term>
566                <term><filename>-mods='DIR1 [DIR2 ...]'</></term>
567                <listitem>
568                  <para>This option specifies a list of directories containing
569                    "modifications".  These directories contain files with names
570                    that may (or may not) exist in the main MITgcm source tree but
571                    will be overridden by any identically-named sources within the
572                    "MODS" directories.  The order of precedence for this
573                    "name-hiding" is as follows:</para>
574                  
575                  <itemizedlist>
576                    <listitem><para>"MODS" directories (in the order given)
577                      </para></listitem>
578                    <listitem><para>Packages either explicitly specified or
579                        provided by default (in the order given)</para></listitem>
580                    <listitem><para>Packages included due to package dependencies
581                        (in the order that that package dependencies are
582                        parsed)</para></listitem>
583                    <listitem><para>The "standard dirs" (which may have been
584                        specified by the "-standarddirs" option)</para></listitem>
585                  </itemizedlist>
586    
587                </listitem>
588              </varlistentry>
589    
590              <varlistentry>
591                <term><filename>-make=/path/to/gmake</></term>
592                <listitem>
593                  <para>Due to the poor handling of soft-links and other bugs common
594                    with the <filename>make</> versions provided by commercial Unix
595                    vendors, GNU <filename>make</filename> (sometimes called
596                    <filename>gmake</filename>) should be preferred.  This option
597                    provides a means for specifying the make program to be
598                    used.</para>
599                </listitem>
600              </varlistentry>
601    
602            </variablelist>
603            
604            <para>A successful run of <filename>genmake2</> will produce a
605              <filename>Makefile</>, a <filename>PACKAGES_CONFIG.h</> file, and
606              various convenience files used for the automatic differentiation
607              process.</para>
608    
609            <para>In general, it is best to use <filename>genmake2</> on a "clean"
610              directory that is free of all source (*.[F,f],*.[F,f]90) and header
611              (*.h,*.inc) files.  Generally, this can be accomplished in an
612              "un-clean" directory by running "make CLEAN" followed by "make
613              makefile".</para>
614    
615          </sect3>
616    
617          <sect3 id="makefile_use">
618            <title>Using <filename>Makefile</></title>
619    
620            <para>Once a <filename>Makefile</> has been created, one can
621            build an executable using:</para>
622    
623    <screen>
624    $ make CLEAN
625    $ make depend
626    $ make
627    </screen>
628    
629            <para>The "make CLEAN" step will remove any stale source files, include
630              files, and links.  It is strongly recommended for "un-clean"
631              directories which may contain the (perhaps partial) results of
632              previous builds.  Such "debris" can interfere with the next stage of
633              the build.</para>
634    
635            <para>The "make depend" step will create a large number of symbolic
636              links from the local directory to the source file locations.  It also
637              parses these files and creates an extensive list of dependencies
638              within the <filename>Makefile</> itself.  The links that exist at this
639              stage are mostly "large F" files (*.F and *.F90) that need to be
640              processed by a C preprocessor ("CPP").  Since "make depend" edits the
641              <filename>Makefile</>, it is important not to skip this step!</para>
642    
643            <para>The final "make" invokes the C preprocessor to produce the "little
644              f" files (*.f and *.f90) and then compiles them to object code using
645              the specified FORTRAN compiler and options.  An intermediate script is
646              often used during this stage to further process (usually, make simple
647              substitutions) custom definitions such as variable types within the
648              source files.  This additional stage is necessary in order to overcome
649              some of the inconsistencies in the sizes of objects (bytes) between
650              different compilers.</para>
651    
652            <para>Please report compilation failures or other problems to
653              <email>MITgcm-support@mitgcm.org</email>.</para>
654    
655          </sect3>
656    
657        </sect2>
658    
659        <sect2 id="verification">
660          <title>The Verification Suite</title>
661    
662          <para>The MITgcm CVS tree (within the <filename>$ROOTDIR/verification/</>
663            directory) includes more than a dozen examples intended for regression
664            testing.  Each one of these example directories contains "known-good"
665            output files along with all the input (including both code and data
666            files) required for their re-calculation.  These example directories are
667            further broken down into sets of subdirectories
668            (eg. <filename>/input</>, <filename>/code</>) intended to expedite the
669            testing process.</para>
670    
671          <sect3 id="testreport">
672            <title>The <filename>testreport</> Utility</title>
673    
674            <para>Also included in <filename>$ROOTDIR/verification/</> are shell
675              scripts for automated testing.  The newest script (which was written
676              to work with <filename>genmake2</>) is called <filename>testreport</>.
677              This script can be used to build different versions of the MITgcm
678              code, run the various examples, compare the output, and (if specified)
679              email the results of each one of these tests to a central
680              repository.</para>
681    
682            <para>On some systems, the testreport script can be run with a command
683            line as simple as:</para>
684    
685    <screen>
686    $ cd verification
687    $ ./testreport -ieee
688    </screen>
689    
690            <para>However, some systems (those lacking or wiht a broken "/bin/sh")
691              may require an explicit shell invocation such as:</para>
692    
693    <screen>
694    $ sh ./testreport -ieee -t 'exp0 exp4'
695    $ /some/path/to/bash ./testreport -ieee -t 'ideal_2D_oce lab_sea natl_box'
696    </screen>
697    
698            <para>The <filename>testreport</> script accepts a number of
699              command-line options which can be listed using the <filename>-help</>
700              option.  The most important ones are:</para>
701    
702            <variablelist>
703    
704              <varlistentry>
705                <term><filename>-ieee</></term>
706                <listitem>
707                  <para>If allowed by the compiler (as defined in the "optfile"),
708                    use IEEE arithmetic.  This option, along with the GCC compiler,
709                    is how the standard results were produced.</para>
710                </listitem>
711              </varlistentry>
712    
713              <varlistentry>
714                <term><filename>-tdir TESTDIR</></term>
715                <term><filename>-tdir 'TDIR1 TDIR2 [...]'</></term>
716                <listitem>
717                  <para>This option specifies the test directory or list of test
718                    directories that should be used.  Each of these entries should
719                    exactly (note: they are case sensitive!) match the names of
720                    directries in <filename>$ROOTDIR/verification/</>.  If this
721                    option is omitted, then all directories that are properly
722                    formatted (that is, containing an <filename>input</>
723                    sub-directory and a <filename>results/output.txt</> file) will
724                    be used.</para>
725                </listitem>
726              </varlistentry>
727    
728              <varlistentry>
729                <term><filename>-optfile=/PATH/FILENAME</></term>
730                <term><filename>-optfile '/PATH/F1 [/PATH/F2 ...]'</></term>
731                <listitem>
732                  <para>This specifies a list of "options files" that will be passed
733                    to <filename>genmake2</>.  If multiple options files are used
734                    (say, to test different compilers or different sets of options
735                    for the same compiler), then each options file will be used with
736                    each of the test directories.</para>
737                </listitem>
738              </varlistentry>
739    
740              <varlistentry>
741                <term><filename>-addr EMAIL</></term>
742                <term><filename>-addr 'EMAIL1 EMAIL2 [...]'</></term>
743                <listitem>
744                  <para>Send the results (namely, <filename>output.txt</>,
745                    <filename>genmake_local</>, <filename>genmake_state</>, and
746                    <filename>Makefile</>) to the specified email addresses.  The
747                    results are gzipped, placed in a tar file, MIME encoded, and
748                    sent to the specified address.  If no email addresses are
749                    specified, no mail is sent.</para>
750                </listitem>
751              </varlistentry>
752    
753              <varlistentry>
754                <term><filename>-mpi</></term>
755                <listitem>
756                  <para>If the necessary files
757                  (<filename>TESTDIR/code/CPP_EEOPTIONS.h_mpi</> and
758                  <filename>TESTDIR/code/SIZE.h_mpi</>) exist, then use them for an
759                  MPI--enabled run.  Note that the use of MPI typically requires a
760                  special command option (see "-command" below) to invoke the MPI
761                  executable.  Examples of PBS scripts using MPI with testreport can be
762                  found in the <ulink
763                  url="http://dev.mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm_contrib/test_scripts/">
764                  MITgcm-contrib area</ulink></para>
765                </listitem>
766              </varlistentry>
767    
768              <varlistentry>
769                <term><filename>-command='some command to run'</></term>
770                <listitem>
771                  <para>For some tests, particularly MPI runs, the default "make
772                  output.txt" is not sufficient.  This option allows a more general
773                  command (or shell script) to be invoked.  Examples of PBS scripts
774                  using MPI with testreport can be found in the <ulink
775                  url="http://dev.mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm_contrib/test_scripts/">
776                  MITgcm-contrib area</ulink></para>
777                </listitem>
778              </varlistentry>
779    
780            </variablelist>
781    
782            <para>The <filename>testreport</> script will write progress to the
783              screen (stdout) as it runs.  In addition, it will create a
784              <filename>tr_out.txt</> file that contains a brief comparison of the
785              current output with the "known-good" output.</para>
786    
787          </sect3>
788    
789        </sect2>
790    
791    
792      <sect2 id="packages">      <sect2 id="packages">
793        <title>Coding Packages</title>        <title>Creating MITgcm Packages</title>
794    
795        <para>Optional parts of code have been separated from the MITgcmUV        <para>Optional parts of code have been separated from the MITgcmUV core
796        core driver code and organised into packages. The packaging          driver code and organised into packages.  The packaging structure
797        structure provides a mechanism for maintaining suites of code,          provides a mechanism for maintaining suites of code, specific to
798        specific to particular classes of problems, in a way that is          particular classes of problems, in a way that is cleanly separated from
799        cleanly separated from the generic fluid dynamical          the generic fluid dynamical engine.</para>
800        engine.</para>  
801          <para>The MITgcmUV packaging structure is described below using generic
802        <para>The MITgcmUV packaging structure is described below using          package names ${pkg}.  A concrete examples of a package is the code for
803        generic package names ${pkg}.  A concrete examples of a package          implementing GM/Redi mixing. This code uses the package name</para>
       is the code for implementing GM/Redi mixing. This code uses the  
       package name</para>  
804    
805      </sect2>      </sect2>
806    

Legend:
Removed from v.1.2  
changed lines
  Added in v.1.4

  ViewVC Help
Powered by ViewVC 1.1.22