MITgcm/doc/devel_HOWTO.sgml

<!DOCTYPE ARTICLE PUBLIC "-//OASIS//DTD DocBook V4.1//EN">

<article id="MITgcm-Development-HOWTO">

<!--
Build commands:
  db2ps -d ldp.dsl devel_HOWTO.sgml
  db2pdf -d ldp.dsl devel_HOWTO.sgml
  db2html -d ./ldp.dsl devel_HOWTO.sgml
  db2html -u -d ./ldp.dsl devel_HOWTO.sgml
-->

  <articleinfo>
    <title>MITgcm Development HOWTO</title>

    <author>
      <firstname>Ed</firstname>
      <surname>Hill III</surname>
      <affiliation>
        <address><email>eh3@mit.edu</email></address>
      </affiliation>
    </author>

    <revhistory>
      <revision>
	<revnumber>0.01</revnumber>
	<date>2003-08-07</date>
	<authorinitials>eh3</authorinitials>
	<revremark>
	  Initial version.
        </revremark>
      </revision>
    </revhistory>

    <abstract>
      <para>This document describes how to develop software for the
      MITgcm project.</para>
    </abstract>
  </articleinfo>

  <sect1 id="intro">
    <title>Introduction</title> <para>The purpose of this document is
    to help new developers get "up to speed" with MITgcm
    development.</para>
    <sect2>
      <title>New Versions of This Document</title> <para>You can
      obtain the latest version of this document <ulink
      url="http://mitgcm.org/dev_docs/devel_HOWTO/">online</ulink> in
      various formats.</para>
    </sect2>
    <sect2>
      <title>Feedback and corrections</title> <para>If you have
      questions or comments about this document, please feel free to
      <ulink url="mailto:MITgcm-support@mitgcm.org">contact the
      authors</ulink>.
      </para>
    </sect2>
  </sect1>

  <sect1 id="background">
    <title>Background</title>

    <sect2>
      <title>User Manual</title>

      <para>Before jumping into
      development, please familiarize yourself with the MITgcm user
      manual which is available <ulink
      url="http://mitgcm.org/">on the main web page</ulink>.  This
      document contains volumes of useful information and is included
      here by reference.</para>

      <para>Also, a "snapshot" or<ulink
      url="http://mitgcm.org/dev_docs/">development version</ulink> of
      the user manual may be available, though this is only put on the
      web for testing purposes.</para>
    </sect2>

    <sect2>
      <title>Prerequisites</title> <para>To develop for MITgcm project
      you will need a UNIX or UNIX-like set of build tools including
      the following:</para>
      <blockquote>
	<simplelist type="inline">
	  <member>CVS client</member>
	  <member>make or (preferably) GNU make</member>
	  <member>FORTRAN compiler</member>
	  <member>C compiler</member>
	  <member>[ba]sh and [t]csh shells</member>
	  <member>PERL</member>
	  <member>LaTeX and LaTeX2HTML</member>
	</simplelist>
      </blockquote>
      <para>Essentially all of the work described here has been tested
      on recent versions of Red Hat Linux (eg. 7.3 through 9).  Except
      where noted, all shell commands will be provided using bash
      syntax.
      </para>
    </sect2>

  </sect1>

  <sect1 id="cvs">
    <title>CVS Repository</title>
    <sect2>
      <title>Layout</title>

      <para>Unlike many open source projects, the MITgcm CVS tree does
      not follow a simple "src", "docs", "share", and "test" directory
      layout.  Instead, there are multiple higher-level directories
      that each, to some extent, depend upon the presence of the
      others.  The tree currently resembles:</para>

<programlisting>gcmpack/
  MITgcm-contrib        contributed code
  CS-regrid             goes into utils
  cvspolicy.html        -save-
  CVSROOT               -save-
  development           experimental stuff
  manual                -save-
  misc                  -?-

  MITgcm                code
       adjoint                  fold into genmake
       bin                      stub for ecco build
       compare01                old from 20th century
       diags                    timeave f77 in pkgs now
       doc                      tags -- connect to real docs?
       eesupp                   cnh?
       exe                      ecco user build
    *- jobs                     runtime shell scripts for 
    |                             various platforms
    |  lsopt                    line search
   m|  model                    main dynamics (core)
   e|    optimization_drivers   ?
   r|  optim                    line search interface
   g|  pkg                      alternate and optional numerics, etc.
   e*- tools
   ?|  tutorial_examples        documented tests
    |                             only populated on release1 branch 
    |                             and not validated during "testscript"
    *- utils
       verification             std tests


  mitgcmdoc -> manual   -remove-
  mitgcm.org            build web site
  models                -?-
  packages              -?-
  preprocess            -?-
  tmp                   -?-
</programlisting>

      <para>Efforts are underway to reduce the complexity.</para>

    </sect2>

 <!--
    <sect2>
      <title>Releases</title> <para>Currently, there are two main
      branches:</para>
      <itemizedlist mark="bullet">
        <listitem>
          <para>Development</para>
	  <itemizedlist mark="bullet">
	    <listitem>
	      <para>MAIN</para>
	    </listitem>
	    <listitem>
	      <para>ecco-branch</para>
	    </listitem>
	  </itemizedlist>
        </listitem>
        <listitem>
          <para>Production</para>
	  <itemizedlist mark="bullet">
	    <listitem>
	      <para>Release1</para>
	    </listitem>
	    <listitem>
	      <para>Release2</para>
	    </listitem>
	  </itemizedlist>
        </listitem>
      </itemizedlist>
    </sect2>
-->

    <sect2>
      <title>Branches</title>

      <para>As shown in the online <ulink
      url="http://dev.mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm/doc/tag-index?graph=1.174">ViewCVS-generated
      tree</ulink>, the MITgcm codebase is split into to two branches
      or "lines" under which development proceeds.  These two lines
      are referred to as the "MAIN" and "ecco" versions of the code.
      While not identical, the bulk of the MAIN and ecco lines are
      composed of files from the same codebase.
      </para>

      <para>Periodically, a "Release" branch is formed from the "MAIN"
      development branch.  This is done in order to create a
      relatively stable reference point for both users and developers.
      The intent is that once a relese branch has been created, only
      bug-fixes will be added to it.  Meanwhile, development (which
      might "break" or otherwise render invalid the documentation,
      tutorials, and/or examples contained within a release branch) is
      allowed to continue along the MAIN and ecco lines.</para>
    </sect2>

    <sect2>
      <title>Tagging</title>

      <para>The intent of tagging is to create "known-good"
      checkpoints that developers can use as references.
      Traditionally, MITgcm tagging has maintained the following
      conventions:</para>

      <orderedlist>
	<listitem>
	  <para>Developer checks out code into a local CVS-managed
	  directory, makes various changes/additions, tests these
	  edits, and eventually reaches a point where (s)he is
	  satisfied that the changes form a new "useful" point in the
	  evolution of the code.</para>
	</listitem>

	<listitem>
	  <para>The developer then runs the <ulink
          url="http://dev.mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm/verification/testscript">testscript</ulink>
          shell script to see if any problems are introduced.  While
          not intended to be exhaustive, the test cases within the
          verification directory do provide some indication whether
          gross errors have been introduced.
          </para>
	</listitem>

	<listitem>
	  <para>Having satisfied him- or herself that the changes are
	  ready to be committed to the CVS repository, the developer
	  then:</para>
	  <orderedlist>
	    <listitem>
	      <para>adds a "checkpointXY_pre" comment (where X is a
	      checkpoint number and Y is a letter) to the <ulink
	      url="http://dev.mitgcm.org/cgi-bin/viewcvs.cgi/MITgcm/doc/tag-index">tag-index</ulink>
	      file and checks it into the CVS repository</para>
	    </listitem>
	    <listitem>
	      <para>submits the set of changes to the CVS repository
	      and adds comments to <filename>tag-index</filename>
	      describing what the changes are along with a matching
	      "checkpointXY_post" entry</para>
	    </listitem>
	  </orderedlist>
	</listitem>
      </orderedlist>

      <para>The result of this tagging procedure is a sequence of
      development checkpoints with comments which resembles:</para>

<programlisting>
checkpoint50e_post
o make KPP work with PTRACERS
 - fix gad_calc_rhs to call new routine kpp_transport_ptr, which is
   nearly a copy of kpp_transport_s
 - there is no analogue to SurfaceTendencyS, so I have to use 
   gPtr(of the surface layer) instead
o add a new platform SunFire+mpi (SunFire 15000) to genmake
checkpoint50e_pre

checkpoint50d_post
o change kpp output from multiple-record state files to single-record state 
  files analogous to write_state.F
o reduce the output frequency of cg3d-related stuff to the monitor frequency, 
  analogous to the cg2d-related output. 
o fix small problem with in ptracers_write_checkpoint.F: len(suff)=512, 
  so that writing to internal file fn (with length 512) fails.
checkpoint50d_pre
</programlisting>

      <para>This information can be used to refer to various stages of
      the code development.  For example, bugs can be traced to
      individual sets of CVS checkins based upon their first
      appearance when comparing the results from different
      checkpoints.</para>

    </sect2>
  </sect1>


  <sect1 id="documentation">
    <title>Editing the Documentation</title>

    <sect2 id="documentation_getting">
      <title>Getting the Docs and Code</title>

      <para>The first step towards editing the documentation is to
      checkout a copy of code, docs, and build scripts from the CVS
      server using:</para>

<screen>
$ export CVS_RSH=ssh
$ export CVSROOT=':ext:auden.lcs.mit.edu:/u/u3/gcmpack'
$ mkdir scratch
$ cvs co MITgcm manual mitgcm.org
</screen>

      <para>These commands extract the necessary information from the
      CVS server and create a temporary (called
      <filename>scratch</filename>) directory for the storage of the
      HTML and other files that will be created.  Please note that you
      must either create <filename>scratch</filename> as shown or edit
      the various <filename>Makefile</filename>s and scripts used to
      create the documentation.</para>
    </sect2>

    <sect2>
      <title>Editing</title>

      <para>The documentation is contained in the
      <filename>manual</filename> directory in a raw LaTeX format.
      The main document is <filename>manual.tex</filename> and it uses
      <command>\input{}</command>s to include the chapters and
      subsections.</para>

      <para>Since the same LaTeX source is used to produce PostScript,
      PDF, and HTML output, care should be taken to follow certain
      conventions.  Two of the most important are the usage of the
      <command>\filelink{}{}</command> and
      <command>\varlink{}{}</command> commands.  Both of these
      commands have been defined to simplify the connection between
      the automatically generated ("code browser") HTML and the HTML
      version of the manual produced by LaTeX2HTML.  They each take
      two arguments (corresponding to the contents of the two sets of
      curly braces) which are the text that the author wishes to be
      "wrapped" within the link, and a specially formatted link thats
      relative to the <filename>MITgcm</filename> directory within the
      CVS tree.</para>

      <para>The result is a command that resembles either</para>
      
      <orderedlist>
	<listitem>
	  <para>a reference to a variable or subroutine name such as
	  <command>\varlink{tRef}{tRef}</command>, or </para>
	</listitem>

	<listitem>
	  <para>a reference to a file such as
	      <command>\varlink{tRef}{path-to-the-file_name.F}</command>
	      where the absolute path to the file is of the form
	      <filename>/foo/MITgcm/path/to/the/file_name.F</filename></para>
	      <para>(please note how the leading "/foo/MITgcm"
	      component of the path is dropped leaving the path
	      <emphasis>relative</emphasis> to the head of the code
	      directory and each directory separator "/" is turned
	      into a "-")</para>
	</listitem>
      </orderedlist>
	  

    </sect2>

    <sect2>
      <title>Building</title> <para>Given the directory structure of
      <xref linkend="documentation_getting">, the entire documentation
      for the web site can be built using:</para>

<screen>
$ cd mitgcm.org/devel/buildweb
$ make All
</screen>

      <para>Which builds the PDF from the LaTeX source, creates the
      HTML output from the LaTeX source, parses the FORTRAN code base
      to produce a hyperlinked HTML version of the source, and then
      determines the cross-linking between the various HTML
      components.</para>

      <para>If there are no errors, the result of the build process
      (which can take 30+ minutes on a P4/2.5Ghz) will be contained
      within a single directory called
      <filename>scratch/dev_docs</filename>.  This is a freshly built
      version of the entire on-line users manual.  If you have the
      correct permissions, it can be directly copied to the web server
      area:</para>

<screen>
$ mv scratch/dev_docs /u/u0/httpd/html
</screen>

      <para>and the update is complete.</para>

    </sect2>

  </sect1>

  <sect1 id="coding">
    <title>Coding for MITgcm</title>

    <sect2 id="build_tools">
      <title>Build Tools</title>

      <para>Many Open Source projects use the "GNU Autotools" to help
      streamline the build process for various Unix and Unix-like
      architectures.  For a user, the result is the common "configure"
      (that is, "<filename>./configure && make && make
      install</filename>") commands.  For MITgcm, the process is
      similar.  Typical commands are:</para>

<screen>
$ genmake -mods=../code
$ make depend
$ make
</screen>

      <para>The following sections describe the individual steps in
      the build process.</para>

      <sect3 id="genmake">
	<title>The <filename>genmake2</> Utility</title>

	<para><emphasis>Please note that the older
	<filename>genmake</> is deprecated and will eventually
	be replaced by <filename>genmake2</>.  This HOWTO only
	describes the newer tool.</emphasis></para>

	<para>The first step in any MITgcm build is to create a
	Unix-style <filename>Makefile</filename> which will be parsed
	by <filename>make</filename> to specify how to compile the
	MITgcm source files.  For more detailed descriptions of what
	the make tools are and how they are used, please see:</para>

	<itemizedlist>
	  <listitem>
	    <para><ulink url="http://www.gnu.org/software/make/make.html">http://www.gnu.org/software/make/make.html</></para>
	  </listitem>
	  <listitem>
	    <para><ulink url="http://www.oreilly.com/catalog/make2/">http://www.oreilly.com/catalog/make2/</></para>
	  </listitem>
	</itemizedlist>

	<para>Due to the poor handling of soft-links and other bugs
	common with the <filename>make</filename> versions provided by
	commercial Unix vendors, GNU <filename>make</filename>
	(sometimes called <filename>gmake</filename>) should be
	preferred.</para>

	<para>As the name implies, <filename>genmake2</filename>
	generates a <filename>Makefile</filename>.  It does so by
	first parsing the information supplied from the following
	sources</para>

	<orderedlist>
	  <listitem>
	    <para>a <filename>gm_local</filename> file in the current
	    directory</para>
	  </listitem>
	  <listitem>
	    <para>directly from command-line options</para>
	  </listitem>
	  <listitem>
	    <para>an "options file" as specified by the command-line
	    option <filename>-optfile='FILENAME'</filename></para>
	  </listitem>
	</orderedlist>

	<para>then checking certain dependency rules (the package
	dependencies), and then writing a
	<filename>Makefile</filename> based upon the source code that
	it finds.  For convenience with the various Unix shells,
	<filename>genmake2</> supports both "long"- and "shor"-style
	options.  A complete list of the available options can be
	obtained from:</para>

<screen>
$ genmake2 -help
</screen>

	<para>The most important options for <filename>genmake2</>
	are:</para>

	<variablelist>

	  <varlistentry>
	    <term><filename>--optfile=/PATH/FILENAME</></term>
	    <listitem>
	      <para>This specifies the "options file" that should be
	      used for a particular build.  The options file is a
	      convenient and machine-indepenent way of specifying
	      parameters such as the FORTRAN compiler
	      (<filename>FC=</>), FORTRAN compiler optimization flags
	      (<filename>FFLAGS=</>), and the locations of various
	      platform- and/or machine-specific tools
	      (eg. <filename>MAKEDEPEND=</>).  As with
	      <filename>genmake2</>, all options files should be
	      written a BASH v1-compatible syntax.  Examples of
	      various options files can be found in
	      <filename>$ROOTDIR/tools/build_options</>.  Everyone is
	      encouraged to submit their options files to the MITgcm
	      project for inclusion (please send to
	      <email>MITgcm-support@mitgcm.org</email>).  We are
	      particularly grateful for options files tested on new or
	      unique platforms!</para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><filename>-pdepend=/PATH/FILENAME</></term>
	    <listitem>
	      <para>This specifies the dependency file used for
	      packages.  If not specified, the default dependency file
	      is <filename>$ROOTDIR/pkg/pkg_depend</>.  The syntax for
	      this file is parsed on a line-by-line basis where each
	      line containes either a comment ("#") or a simple
	      "PKGNAME1 (+|-)PKGNAME2" pairwise rule where the "+" or
	      "-" symbol specifies a "must be used with" or a "must
	      not be used with" relationship, respectively.  If no
	      rule is specified, then it is assumed that the two
	      packages are compatible and will function either with or
	      without each other.</para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><filename>-pdefault=PKG</></term>
	    <term><filename>-pdefault='PKG1 [PKG2 PKG3 ...]'</></term>
	    <listitem>
	      <para>This option specifies the default set of packages
	      to be used.  If not set, the default package list will
	      be read from
	      <filename>$ROOTDIR/pkg/pkg_default</>.</para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><filename></>-mods=DIR</term>
	    <term><filename></>-mods='DIR1 [DIR2 ...]'</term>
	    <listitem>
	      <para>This option specifies a list of directories
	      containing "modifications".  These are files that may
	      (or may not) exist in the main MITgcm source tree but
	      will be overridden by any identically-named sources
	      within the "MODS" directories.</para>
	    </listitem>
	  </varlistentry>

	</variablelist>
	
	<para>A successful run of <filename>genmake2</> will produce
	both a <filename>Makefile</> and a locally modified copy of
	the specified <filename>CPP_OPTIONS.h</> file.  The local copy
	of <filename>CPP_OPTIONS.h</> will contain a list of
	<filename>genmake2</>-created #DEFINE and #UNDEF statements
	that reflect the list of packages that will be compiled into
	the code (either directly through enable/disable/defaults
	options or indirectly through dependencies).</para>

	<para>In general, it is best to use <filename>genmake2</> on a
	"clean" directory that is free of all source
	(*.[F,f],*.[F,f]90) and header (*.h,*.inc) files.  Generally,
	this can be accomplished in an "un-clean" directory by running
	"make CLEAN" followed by "make makefile".</para>

      </sect3>

      <sect3 id="makefile_use">
	<title>Using <filename>Makefile</></title>

	<para>Once a <filename>Makefile</> has been created, one can
	build the executable using:</para>

<screen>
$ make CLEAN
$ make depend
$ make
</screen>

	<para>The "make CLEAN" step will remove any local source
	files, include files, and links.  It is strongly recommended
	for "un-clean" directories which may contain the (partial?)
	results of previous builds.  Such "debris" can interfere with
	the next stage of the build.</para>

	<para>The "make depend" step will create a large number of
	symbolic links from the local directory to the source file
	locations.  It also parses these files and creates an
	extensive list of dependencies within the
	<filename>Makefile</> itself.  The links that exist at this
	stage are mostly "large F" files (*.F and *.F90) that need to
	be processed by a C preprocessor ("CPP").
	</para>

	<para>The final "make" invokes the C preprocessor to produce
	the "little f" files (*.f and *.f90) and then compiles them to
	object code using the specified FORTRAN compiler and options.
	An intermediate script is often used during this stage to
	further process (usually, make simple substitutions) custom
	definitions such as variable types within the source files.
	This additional stage is necessary in order to overcome some
	of the inconsistencies in the sizes of objects (bytes) between
	different compilers.</para>

	<para>Please report compilation failures or other problems to
	<email>MITgcm-support@mitgcm.org</email>.</para>

      </sect3>

    </sect2>

    <sect2 id="verification">
      <title>The Verification Suite</title>

      <para>The MITgcm CVS tree (within the
      <filename>$ROOTDIR/verification/</> directory) includes more
      than a dozen examples intended for regression testing.  Each one
      of these example directories contains "known-good" output files
      along with all the input (including both code and data files)
      required for their re-calculation.  These example directories
      are further broken down into sets of subdirectories
      (eg. <filename>/input</>, <filename>/code</>) intended to
      expedite the testing process.</para>

      <sect3 id="testreport">
	<title>The <filename>testreport</> Utility</title>

	<para>Also included in <filename>$ROOTDIR/verification/</> are
	shell scripts for automated testing.  The newest script (which
	was written to work with <filename>genmake2</>) is called
	<filename>testreport</>.  Ths script can be used to build the
	different versions of the MITgcm code, run the various
	examples, compare the output, and (if specified) email the
	results of each one of these tests to a central
	repository.</para>

	<para>The <filename>testreport</> script accepts a number of
	command-line options which can be listed using the
	<filename>-help</> option.  The most important ones are:</para>

	<variablelist>

	  <varlistentry>
	    <term><filename>-tdir TESTDIR</></term>
	    <term><filename>-tdir 'TDIR1 TDIR2 [...]'</></term>
	    <listitem>
	      <para>This option specifies the test directory or list
	      of test directories that should be used.  Each of these
	      entries should exactly (note: they're case sensitive!)
	      match the names of directries in
	      <filename>$ROOTDIR/verification/</>.  If this option is
	      omitted, then all directories that are properly
	      formatted (that is, containing an <filename>input</>
	      sub-directory and example output) will be used.</para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><filename>-optfile=/PATH/FILENAME</></term>
	    <term><filename>-optfile '/PATH/F1 [/PATH/F2 ...]'</></term>
	    <listitem>
	      <para>This specifies a list of "options files" that will
	      be passed to <filename>genmake2</>.  If multiple options
	      files are used (say, to test different compilers or
	      different sets of options for the same compiler), then
	      each options file will be used with each of the test
	      directories.</para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><filename>-addr EMAIL</></term>
	    <term><filename>-addr 'EMAIL1 EMAIL2 [...]'</></term>
	    <listitem>
	      <para>Send the results (namely, <filename>output.txt</>,
	      <filename>gm_local</>, <filename>gm_state</>, and
	      <filename>Makefile</>) to the specified email addresses.
	      The results are gzipped, placed in a tar file, MIME
	      encoded, and sent to an @mitgcm.org address.  If no
	      email addresses are specified, no mail is sent.</para>
	    </listitem>
	  </varlistentry>

	</variablelist>

	<para>The <filename>testreport</> script will write progress
	to the screen (stdout) as it runs.  In addition, it will
	create a <filename>summary.txt</> file that contains a brief
	comparison of the current output with the "known-good"
	output.</para>

      </sect3>

    </sect2>

    <sect2 id="packages">
      <title>Creating MITgcm Packages</title>

      <para>Optional parts of code have been separated from the MITgcmUV
      core driver code and organised into packages. The packaging
      structure provides a mechanism for maintaining suites of code,
      specific to particular classes of problems, in a way that is
      cleanly separated from the generic fluid dynamical
      engine.</para>

      <para>The MITgcmUV packaging structure is described below using
      generic package names ${pkg}.  A concrete examples of a package
      is the code for implementing GM/Redi mixing. This code uses the
      package name</para>

    </sect2>

  </sect1>

  <sect1>
    <title>Chris's Notes...</title>

<programlisting>
           MITgcmUV Packages
           =================

  Optional parts of code are separated from
the MITgcmUV core driver code and organised into
packages. The packaging structure provides a mechanism for
maintaining suites of code, specific to particular
classes of problem, in a way that is cleanly
separated from the generic fluid dynamical engine.

 The MITgcmUV packaging structure is describe
below using generic package names ${pkg}.
A concrete examples of a package is the code
for implementing GM/Redi mixing. This code uses
the package name 
*   ${PKG} = GMREDI
*   ${pkg} = gmredi
*   ${Pkg} = gmRedi

Package states
==============

 Packages can be any one of four states, included,
 excluded, enabled, disabled as follows:

 included(excluded) compile time state which
                    includes(excludes) package
                    code and routine calls from
                    compilation/linking etc...

 enabled(disabled)  run-time state which
                    enables(disables) package code
                    execution.

 Every call to a ${pkg}_... routine from outside the package
 should be placed within both a
 #ifdef ALLOW_${PKG} ... block and a
 if ( use${Pkg} ) ... then block.
 Package states are generally not expected to change during
 a model run.

Package structure
=================

o  Each package gets its runtime configuration
   parameters from a file named "data.${pkg}"
   Package runtime config. options are imported
   into a common block held in a header file
   called "${PKG}.h".

o  The core driver part of the model can check
   for runtime enabling or disabling of individual packages
   through logical flags use${Pkg}.
   The information is loaded from a
   global package setup file called "data.pkg".
   The use${Pkg} flags are not used within
   individual packages.

o  Included in "${PKG}.h" is a logical flag
   called ${Pkg}IsOn. The "${PKG}.h" header file can be imported
   by other packages to check dependencies and requirements
   from other packages ( see "Package Boot Sequence" section).
   NOTE: This procedure is not presently implemented,
   ----- neither for kpp nor for gmRedi.

CPP Flags
=========

    1. Within the core driver code flags of the form
       ALLOW_${PKG} are used to include or exclude
       whole packages. The ALLOW_${PKG} flags are included
       from a PKG_CPP_OPTIONS block which is currently
       held in-line in the CPP_OPTIONS.h header file.
       e.g.

       Core model code .....

       #include "CPP_OPTIONS.h"
         :
         :
         :

       #ifdef ALLOW_${PKG}
         if ( use${Pkg} ) CALL ${PKG}_DO_SOMETHING(...)
       #endif

    2. Within an individual package a header file,
       "${PKG}_OPTIONS.h", is used to set CPP flags
       specific to that package. It is not recommended
       to include this file in "CPP_OPTIONS.h".


Package Boot Sequence
=====================

    Calls to package routines within the core code timestepping
    loop can vary. However, all packages follow a required
    "boot" sequence outlined here:

    1. S/R PACKAGES_BOOT()
            :
        CALL OPEN_COPY_DATA_FILE( 'data.pkg', 'PACKAGES_BOOT', ... )
 

    2. S/R PACKAGES_READPARMS()
            :
        #ifdef ALLOW_${PKG}
          if ( use${Pkg} )
     &       CALL ${PKG}_READPARMS( retCode )
        #endif

    2. S/R PACKAGES_CHECK()
            :
        #ifdef ALLOW_${PKG}
          if ( use${Pkg} )
     &       CALL ${PKG}_CHECK( retCode )
        #else
          if ( use${Pkg} )
     &       CALL PACKAGES_CHECK_ERROR('${PKG}')
        #endif

    3. S/R PACKAGES_INIT()
            :
        #ifdef ALLOW_${PKG}
          if ( use${Pkg} )
     &       CALL ${PKG}_INIT( retCode )
        #endif


Description
===========

      - ${PKG}_READPARMS() 
    is responsible for reading
    in the package parameters file data.${pkg}, and storing
    the package parameters in "${PKG}.h".
    -> called in INITIALISE_FIXED

     - ${PKG}_CHECK() 
    is responsible for validating
    basic package setup and inter-package dependencies.
    ${PKG}_CHECK can import other package parameters it may
    need to check. This is done through header files "${PKG}.h".
    It is assumed that parameters owned by other packages
    will not be reset during ${PKG}_CHECK().
    -> called in INITIALISE_FIXED

     - ${PKG}_INIT() 
    is responsible for completing the
    internal setup of a package. This routine is called after
    the core model state has been completely initialised
    but before the core model timestepping starts.
    -> called in INITIALISE_VARIA

Summary
=======

- CPP options:
  -----------------------
  * ALLOW_${PKG}         include/exclude package for compilation

- FORTRAN logical:
  -----------------------
  * use${Pkg}            enable package for execution at runtime
                         -> declared in PARAMS.h
  * ${Pkg}IsOn           for package cross-dependency check
                         -> declared in ${PKG}.h
                         N.B.: Not presently used!

- header files
  -----------------------
  * ${PKG}_OPTIONS.h     has further package-specific CPP options
  * ${PKG}.h             package-specific common block variables, fields

- FORTRAN source files
  -----------------------
  * ${pkg}_readparms.F   reads parameters from file data.${pkg}
  * ${pkg}_check.F       checks package dependencies and consistencies
  * ${pkg}_init.F        initialises package-related fields
  * ${pkg}_... .F        package source code

- parameter file
  -----------------------
  * data.${pkg}          parameter file
</programlisting>

  </sect1>


</article>