--- MITgcm/doc/devel_HOWTO.sgml 2003/08/28 22:44:00 1.3 +++ MITgcm/doc/devel_HOWTO.sgml 2003/11/20 00:01:42 1.4 @@ -128,18 +128,18 @@ doc tags -- connect to real docs? eesupp cnh? exe ecco user build - *- jobs runtime shell scripts for + ,- jobs runtime shell scripts for | various platforms | lsopt line search m| model main dynamics (core) e| optimization_drivers ? r| optim line search interface g| pkg alternate and optional numerics, etc. - e*- tools + e|- tools ?| tutorial_examples documented tests | only populated on release1 branch | and not validated during "testscript" - *- utils + '- utils verification std tests @@ -316,7 +316,7 @@ - Editing + Editing the Documentation The documentation is contained in the manual directory in a raw LaTeX format. @@ -364,9 +364,11 @@ - Building Given the directory structure of - , the entire documentation - for the web site can be built using: + Building the Documentation + + Given the directory structure of , the entire documentation for the web + site can be built using: $ cd mitgcm.org/devel/buildweb @@ -403,12 +405,11 @@ Build Tools - Many Open Source projects use the "GNU Autotools" to help - streamline the build process for various Unix and Unix-like - architectures. For a user, the result is the common "configure" - (that is, "./configure && make && make - install") commands. For MITgcm, the process is - similar. Typical commands are: + Many Open Source projects use the "GNU Autotools" to help streamline + the build process for various Unix and Unix-like architectures. For a + user, the result is the common "configure" (that is, + "./configure && make && make install") commands. + For MITgcm, the process is similar. Typical commands are: $ genmake -mods=../code @@ -416,112 +417,137 @@ $ make - The following sections describe the individual steps in - the build process. - + The following sections describe the individual steps in the build + process. + The <filename>genmake2</> Utility - Please note that the older - genmake is deprecated and will eventually - be replaced by genmake2. This HOWTO only - describes the newer tool. - - The first step in any MITgcm build is to create a - Unix-style Makefile which will be parsed - by make to specify how to compile the - MITgcm source files. For more detailed descriptions of what - the make tools are and how they are used, please see: + Please note that the older genmake is + deprecated and will eventually be replaced by genmake2. + This HOWTO only describes the newer tool. + + The first step in any MITgcm build is to create a Unix-style + Makefile which will be parsed by + make to specify how to compile the MITgcm source + files. For more detailed descriptions of what the make tools are and + how they are used, please see: - http://www.gnu.org/software/make/make.html + + http://www.gnu.org/software/make/make.html - http://www.oreilly.com/catalog/make2/ + + http://www.oreilly.com/catalog/make2/ - Due to the poor handling of soft-links and other bugs - common with the make versions provided by - commercial Unix vendors, GNU make - (sometimes called gmake) should be - preferred. - - As the name implies, genmake2 - generates a Makefile. It does so by - first parsing the information supplied from the following - sources + Genmake can often be invoked successfully with a command line as + simple as: + + +$ genmake2 -mods=../code + + + However, some systems (particularly commercial Unixes that lack a + more modern "/bin/sh" implementation or that have shells installed in + odd locations) may require an explicit shell invocation such as one of + the following: + + +$ /usr/bin/sh genmake2 -make=gmake -mods=../code +$ /opt/gnu/bin/bash genmake2 -ieee -make=/usr/local/bin/gmake -mods=../code + + + The genmake2 code has been written in a Bourne and BASH (v1) + compatible syntax so it should work with most "sh" and all recent "bash" + implementations. + + As the name implies, genmake2 generates a + Makefile. It does so by first parsing the + information supplied from the following sources a gm_local file in the current - directory + directory directly from command-line options - an "options file" as specified by the command-line - option -optfile='FILENAME' + an "options file" as specified by the command-line option + -optfile='FILENAME' - then checking certain dependency rules (the package - dependencies), and then writing a - Makefile based upon the source code that - it finds. For convenience with the various Unix shells, - genmake2 supports both "long"- and "shor"-style - options. A complete list of the available options can be - obtained from: + then checking certain dependency rules (the package dependencies), + and finally writing a Makefile based upon the + source code that it finds. For convenience within various Unix + shells, genmake2 supports both "long"- and "short"-style + options. A complete list of the available options can be obtained + from: $ genmake2 -help - The most important options for genmake2 - are: + The most important options for genmake2 are: --optfile=/PATH/FILENAME + - This specifies the "options file" that should be - used for a particular build. The options file is a - convenient and machine-indepenent way of specifying - parameters such as the FORTRAN compiler - (FC=), FORTRAN compiler optimization flags - (FFLAGS=), and the locations of various - platform- and/or machine-specific tools - (eg. MAKEDEPEND=). As with - genmake2, all options files should be - written a BASH v1-compatible syntax. Examples of - various options files can be found in - $ROOTDIR/tools/build_options. Everyone is - encouraged to submit their options files to the MITgcm - project for inclusion (please send to - MITgcm-support@mitgcm.org). We are - particularly grateful for options files tested on new or - unique platforms! + This specifies the "options file" that should be used for a + particular build. The options file is a convenient and + machine-indepenent way of specifying parameters such as the + FORTRAN compiler (FC=), FORTRAN compiler + optimization flags (FFLAGS=), and the locations of + various platform- and/or machine-specific tools + (eg. MAKEDEPEND=). As with genmake2, + all options files should be written to be compatible with + Bourne--shell ("sh" or "BASH v1") syntax. Examples of various + options files can be found in + $ROOTDIR/tools/build_options. + + If no "optfile" is specified (either through the command lin + or the environment variable), genmake2 will try to make a + reasonable guess from the list provided in + $ROOTDIR/tools/build_options. The method used for + making this guess is to first determine the combination of + operating system and hardware (eg. "linux_ia32") and then find a + working Fortran compiler within the user's path. When these + three items have been identified, genmake2 will try to find an + optfile that has a matching name. + + Everyone is encouraged to submit their options files to the + MITgcm project for inclusion (please send to + MITgcm-support@mitgcm.org). We are particularly + grateful for options files tested on new or unique + platforms! + -pdepend=/PATH/FILENAME + - This specifies the dependency file used for - packages. If not specified, the default dependency file - is $ROOTDIR/pkg/pkg_depend. The syntax for - this file is parsed on a line-by-line basis where each - line containes either a comment ("#") or a simple - "PKGNAME1 (+|-)PKGNAME2" pairwise rule where the "+" or - "-" symbol specifies a "must be used with" or a "must - not be used with" relationship, respectively. If no - rule is specified, then it is assumed that the two - packages are compatible and will function either with or - without each other. + This specifies the dependency file used for packages. If + not specified, the default dependency file is + $ROOTDIR/pkg/pkg_depend. The syntax for this file is + parsed on a line-by-line basis where each line containes either a + comment ("#") or a simple "PKGNAME1 (+|-)PKGNAME2" pairwise rule + where the "+" or "-" symbol specifies a "must be used with" or a + "must not be used with" relationship, respectively. If no rule is + specified, then it is assumed that the two packages are compatible + and will function either with or without each other. + @@ -536,33 +562,55 @@ - -mods=DIR - -mods='DIR1 [DIR2 ...]' + -mods=DIR + -mods='DIR1 [DIR2 ...]' - This option specifies a list of directories - containing "modifications". These are files that may - (or may not) exist in the main MITgcm source tree but - will be overridden by any identically-named sources - within the "MODS" directories. + This option specifies a list of directories containing + "modifications". These directories contain files with names + that may (or may not) exist in the main MITgcm source tree but + will be overridden by any identically-named sources within the + "MODS" directories. The order of precedence for this + "name-hiding" is as follows: + + + "MODS" directories (in the order given) + + Packages either explicitly specified or + provided by default (in the order given) + Packages included due to package dependencies + (in the order that that package dependencies are + parsed) + The "standard dirs" (which may have been + specified by the "-standarddirs" option) + + + + + + + -make=/path/to/gmake + + Due to the poor handling of soft-links and other bugs common + with the make versions provided by commercial Unix + vendors, GNU make (sometimes called + gmake) should be preferred. This option + provides a means for specifying the make program to be + used. - A successful run of genmake2 will produce - both a Makefile and a locally modified copy of - the specified CPP_OPTIONS.h file. The local copy - of CPP_OPTIONS.h will contain a list of - genmake2-created #DEFINE and #UNDEF statements - that reflect the list of packages that will be compiled into - the code (either directly through enable/disable/defaults - options or indirectly through dependencies). - - In general, it is best to use genmake2 on a - "clean" directory that is free of all source - (*.[F,f],*.[F,f]90) and header (*.h,*.inc) files. Generally, - this can be accomplished in an "un-clean" directory by running - "make CLEAN" followed by "make makefile". + A successful run of genmake2 will produce a + Makefile, a PACKAGES_CONFIG.h file, and + various convenience files used for the automatic differentiation + process. + + In general, it is best to use genmake2 on a "clean" + directory that is free of all source (*.[F,f],*.[F,f]90) and header + (*.h,*.inc) files. Generally, this can be accomplished in an + "un-clean" directory by running "make CLEAN" followed by "make + makefile". @@ -570,7 +618,7 @@ Using <filename>Makefile</> Once a Makefile has been created, one can - build the executable using: + build an executable using: $ make CLEAN @@ -578,33 +626,31 @@ $ make - The "make CLEAN" step will remove any local source - files, include files, and links. It is strongly recommended - for "un-clean" directories which may contain the (partial?) - results of previous builds. Such "debris" can interfere with - the next stage of the build. - - The "make depend" step will create a large number of - symbolic links from the local directory to the source file - locations. It also parses these files and creates an - extensive list of dependencies within the - Makefile itself. The links that exist at this - stage are mostly "large F" files (*.F and *.F90) that need to - be processed by a C preprocessor ("CPP"). - - - The final "make" invokes the C preprocessor to produce - the "little f" files (*.f and *.f90) and then compiles them to - object code using the specified FORTRAN compiler and options. - An intermediate script is often used during this stage to - further process (usually, make simple substitutions) custom - definitions such as variable types within the source files. - This additional stage is necessary in order to overcome some - of the inconsistencies in the sizes of objects (bytes) between - different compilers. + The "make CLEAN" step will remove any stale source files, include + files, and links. It is strongly recommended for "un-clean" + directories which may contain the (perhaps partial) results of + previous builds. Such "debris" can interfere with the next stage of + the build. + + The "make depend" step will create a large number of symbolic + links from the local directory to the source file locations. It also + parses these files and creates an extensive list of dependencies + within the Makefile itself. The links that exist at this + stage are mostly "large F" files (*.F and *.F90) that need to be + processed by a C preprocessor ("CPP"). Since "make depend" edits the + Makefile, it is important not to skip this step! + + The final "make" invokes the C preprocessor to produce the "little + f" files (*.f and *.f90) and then compiles them to object code using + the specified FORTRAN compiler and options. An intermediate script is + often used during this stage to further process (usually, make simple + substitutions) custom definitions such as variable types within the + source files. This additional stage is necessary in order to overcome + some of the inconsistencies in the sizes of objects (bytes) between + different compilers. Please report compilation failures or other problems to - MITgcm-support@mitgcm.org. + MITgcm-support@mitgcm.org. @@ -613,46 +659,69 @@ The Verification Suite - The MITgcm CVS tree (within the - $ROOTDIR/verification/ directory) includes more - than a dozen examples intended for regression testing. Each one - of these example directories contains "known-good" output files - along with all the input (including both code and data files) - required for their re-calculation. These example directories - are further broken down into sets of subdirectories - (eg. /input, /code) intended to - expedite the testing process. + The MITgcm CVS tree (within the $ROOTDIR/verification/ + directory) includes more than a dozen examples intended for regression + testing. Each one of these example directories contains "known-good" + output files along with all the input (including both code and data + files) required for their re-calculation. These example directories are + further broken down into sets of subdirectories + (eg. /input, /code) intended to expedite the + testing process. The <filename>testreport</> Utility - Also included in $ROOTDIR/verification/ are - shell scripts for automated testing. The newest script (which - was written to work with genmake2) is called - testreport. Ths script can be used to build the - different versions of the MITgcm code, run the various - examples, compare the output, and (if specified) email the - results of each one of these tests to a central - repository. + Also included in $ROOTDIR/verification/ are shell + scripts for automated testing. The newest script (which was written + to work with genmake2) is called testreport. + This script can be used to build different versions of the MITgcm + code, run the various examples, compare the output, and (if specified) + email the results of each one of these tests to a central + repository. + + On some systems, the testreport script can be run with a command + line as simple as: + + +$ cd verification +$ ./testreport -ieee + + + However, some systems (those lacking or wiht a broken "/bin/sh") + may require an explicit shell invocation such as: + + +$ sh ./testreport -ieee -t 'exp0 exp4' +$ /some/path/to/bash ./testreport -ieee -t 'ideal_2D_oce lab_sea natl_box' + The testreport script accepts a number of - command-line options which can be listed using the - -help option. The most important ones are: + command-line options which can be listed using the -help + option. The most important ones are: + -ieee + + If allowed by the compiler (as defined in the "optfile"), + use IEEE arithmetic. This option, along with the GCC compiler, + is how the standard results were produced. + + + + -tdir TESTDIR -tdir 'TDIR1 TDIR2 [...]' - This option specifies the test directory or list - of test directories that should be used. Each of these - entries should exactly (note: they're case sensitive!) - match the names of directries in - $ROOTDIR/verification/. If this option is - omitted, then all directories that are properly - formatted (that is, containing an input - sub-directory and example output) will be used. + This option specifies the test directory or list of test + directories that should be used. Each of these entries should + exactly (note: they are case sensitive!) match the names of + directries in $ROOTDIR/verification/. If this + option is omitted, then all directories that are properly + formatted (that is, containing an input + sub-directory and a results/output.txt file) will + be used. @@ -660,12 +729,11 @@ -optfile=/PATH/FILENAME -optfile '/PATH/F1 [/PATH/F2 ...]' - This specifies a list of "options files" that will - be passed to genmake2. If multiple options - files are used (say, to test different compilers or - different sets of options for the same compiler), then - each options file will be used with each of the test - directories. + This specifies a list of "options files" that will be passed + to genmake2. If multiple options files are used + (say, to test different compilers or different sets of options + for the same compiler), then each options file will be used with + each of the test directories. @@ -674,40 +742,65 @@ -addr 'EMAIL1 EMAIL2 [...]' Send the results (namely, output.txt, - gm_local, gm_state, and - Makefile) to the specified email addresses. - The results are gzipped, placed in a tar file, MIME - encoded, and sent to an @mitgcm.org address. If no - email addresses are specified, no mail is sent. + genmake_local, genmake_state, and + Makefile) to the specified email addresses. The + results are gzipped, placed in a tar file, MIME encoded, and + sent to the specified address. If no email addresses are + specified, no mail is sent. + + + + + -mpi + + If the necessary files + (TESTDIR/code/CPP_EEOPTIONS.h_mpi and + TESTDIR/code/SIZE.h_mpi) exist, then use them for an + MPI--enabled run. Note that the use of MPI typically requires a + special command option (see "-command" below) to invoke the MPI + executable. Examples of PBS scripts using MPI with testreport can be + found in the + MITgcm-contrib area + + + + + -command='some command to run' + + For some tests, particularly MPI runs, the default "make + output.txt" is not sufficient. This option allows a more general + command (or shell script) to be invoked. Examples of PBS scripts + using MPI with testreport can be found in the + MITgcm-contrib area - The testreport script will write progress - to the screen (stdout) as it runs. In addition, it will - create a summary.txt file that contains a brief - comparison of the current output with the "known-good" - output. + The testreport script will write progress to the + screen (stdout) as it runs. In addition, it will create a + tr_out.txt file that contains a brief comparison of the + current output with the "known-good" output. + Creating MITgcm Packages - Optional parts of code have been separated from the MITgcmUV - core driver code and organised into packages. The packaging - structure provides a mechanism for maintaining suites of code, - specific to particular classes of problems, in a way that is - cleanly separated from the generic fluid dynamical - engine. - - The MITgcmUV packaging structure is described below using - generic package names ${pkg}. A concrete examples of a package - is the code for implementing GM/Redi mixing. This code uses the - package name + Optional parts of code have been separated from the MITgcmUV core + driver code and organised into packages. The packaging structure + provides a mechanism for maintaining suites of code, specific to + particular classes of problems, in a way that is cleanly separated from + the generic fluid dynamical engine. + + The MITgcmUV packaging structure is described below using generic + package names ${pkg}. A concrete examples of a package is the code for + implementing GM/Redi mixing. This code uses the package name