--- MITgcm/doc/devel_HOWTO.sgml 2003/12/10 20:44:39 1.5 +++ MITgcm/doc/devel_HOWTO.sgml 2010/05/28 01:47:32 1.14 @@ -1,4 +1,8 @@ +
@@ -22,6 +26,22 @@ Initial version. + + 0.02 + 2010-01-21 + jmc + + update links. + + + + 0.03 + 2010-04-25 + jmc + + Add subsection "Developer settings" (under CVS Repository). + + @@ -37,7 +57,7 @@ New Versions of This Document You can obtain the latest version of this document online in + url="http://mitgcm.org/public/docs.html">online in various formats. @@ -56,7 +76,7 @@ User Manual Before jumping into development, please familiarize yourself with - the MITgcm user manual + the MITgcm user manual . This document contains volumes of useful information and is included here by reference. @@ -94,6 +114,7 @@ CVS Repository + Layout @@ -104,46 +125,45 @@ others. The tree currently resembles: gcmpack/ - MITgcm-contrib contributed code - CS-regrid goes into utils - cvspolicy.html -save- - CVSROOT -save- - development experimental stuff - manual -save- - misc -?- + CVSROOT -hidden- MITgcm code - adjoint fold into genmake - bin stub for ecco build - compare01 old from 20th century - diags timeave f77 in pkgs now - doc tags -- connect to real docs? - eesupp cnh? - exe ecco user build - ,- jobs runtime shell scripts for - | various platforms - | lsopt line search - m| model main dynamics (core) - e| optimization_drivers ? - r| optim line search interface - g| pkg alternate and optional numerics, etc. - e|- tools - ?| tutorial_examples documented tests - | only populated on release1 branch - | and not validated during "testscript" - '- utils - verification std tests + bin empty + doc basic developpment documentation + eesupp execution environment support code (wrapper) + exe empty + jobs runtime shell scripts for + various platforms (not maintained) + lsopt line search + model main dynamics (core) + optim line search interface + pkg alternate and optional numerics, etc. + tools scripts to build (and test) + utils pre/post processing tools (matlab, ..) + verification standard regression tests + examples + + documented examples (tutorials) + tutorial_examples (only in release1 branch) + MITgcm_contrib contributed code - mitgcmdoc -> manual -remove- + acesgrid.org build acesgrid web site + development experimental stuff + gfd_lab -?- + manual source of MITgcm documentation mitgcm.org build web site - models -?- - packages -?- - preprocess -?- - tmp -?- + old_develop old and early development source + misc -?- + models -?- + packages -?- + preprocess -?- + pdfs some pdfs + planetinabottle.org unfinished web site + www.ecco-group.org build ecco web site ? + @@ -182,48 +202,129 @@ Branches As shown in the online ViewCVS-generated - tree, the MITgcm codebase is split into to two branches - or "lines" under which development proceeds. These two lines - are referred to as the "MAIN" and "ecco" versions of the code. - While not identical, the bulk of the MAIN and ecco lines are - composed of files from the same codebase. + url="http://mitgcm.org/viewvc/MITgcm/MITgcm/model/src/forward_step.F?view=graph"> + ViewCVS-generated tree, the MITgcm codebase is split into + branches or "lines" under which development proceeds. The main line + of development is referred to as the "MAIN" version of the code. Periodically, a "Release" branch is formed from the "MAIN" - development branch. This is done in order to create a - relatively stable reference point for both users and developers. - The intent is that once a relese branch has been created, only - bug-fixes will be added to it. Meanwhile, development (which - might "break" or otherwise render invalid the documentation, - tutorials, and/or examples contained within a release branch) is - allowed to continue along the MAIN and ecco lines. + development branch. This is done in order to create a relatively stable + reference point for both users and developers. The intent is that once a + release branch has been created, only bug-fixes will be added to it. + Meanwhile, development (which might "break" or otherwise render invalid + the documentation, tutorials, and/or examples contained within a release + branch) is allowed to continue along the MAIN line. - Tagging + Developer settings + + CVS is a convenient tool to keep up-to-date a personal copy of the + MITgcm code (see: + using CVS ). The same tool is used by developers to + incorporate any change into the repository. However, this later + function requires specific settings, as detailed here after: + + + You will need an account (loggin access) to the server + "mitgcm.org" with the proper group setting (e.g., + group "gcmctrb" to add/modify code into MITgcm_contrib). + This kind of account is granted only upon well motivated request. + The access to the server mitgcm.org is through ssh-key authorization + which will need to be set properly on both side (on your local machine + and on your server account). You need to be able to + to ssh to mitgcm.org (or ssh MY_USER_NAME@mitgcm.org + in case of different user-name on both sides) to proceed further. + + + + You need to register to the + + mitgcm-cvs mailing list. + This ensures that other developers will receive email notification + when you make changes; you will also receive as well such email + when others make changes to the repository. + + + + + It is highly recommended that you register also to the + + mitgcm-devel mailing list (expect a short delay for + this request to be processed). + This list is intended for developer discussions. + + + + + The standard anonymous mode (using "cvsanon", as mentionned + + here ) does not allow check-in ("cvs commit") permission. + Instead, you will need to set our CVS environment as follow: + + $ export CVS_RSH=ssh + $ export CVSROOT=':ext:MY_USER_NAME@mitgcm.org:/u/gcmpack' + + After downloading a directory, e.g.: myCopy, + from the CVS repository (e.g., + MITgcm_contrib/thisPart) using the command: + + $ cvs co -P -d myCopy MITgcm_contrib/thisPart + + the type of CVS environment which has been used + is stored in the file myCopy/CVS/Root + and makes it difficult to re-use, for cvs-commit purpose, + a cvs local copy (myCopy) which was obtained + using the CVS anonymous mode. + - The intent of tagging is to create "known-good" - checkpoints that developers can use as references. - Traditionally, MITgcm tagging has maintained the following - conventions: + + At this stage, you should be able to send your modified source + file (e.g., src_file) from your local copy directory + (myCopy) to the CVS repository + (MITgcm_contrib/thisPart) using the command + "cvs commit": + + $ cd myCopy + $ cvs -n update (optional; check if new changes have been made) + $ cvs diff src_file (optional; list your changes) + $ cvs commit src_file + + It is essential that you provide a short description of the + changes you made to src_file as you check-in + this file (the "cvs commit" command automatically opens your standard + editor for this purpose). + + + + + + + + Main code development + (formerly named "Tagging" ; this section needs an update) + + + The intent of tagging is to create "known-good" checkpoints that + developers can use as references. Traditionally, MITgcm tagging has + maintained the following conventions: - Developer checks out code into a local CVS-managed - directory, makes various changes/additions, tests these - edits, and eventually reaches a point where (s)he is - satisfied that the changes form a new "useful" point in the - evolution of the code. + Developer checks out code into a local CVS-managed directory, + makes various changes/additions, tests these edits, and eventually + reaches a point where (s)he is satisfied that the changes form a new + "useful" point in the evolution of the code. The developer then runs the testscript - shell script to see if any problems are introduced. While - not intended to be exhaustive, the test cases within the - verification directory do provide some indication whether - gross errors have been introduced. + url="http://mitgcm.org/viewvc/MITgcm/MITgcm/verification/testreport"> + testreport shell script to see if any problems are introduced. + While not intended to be exhaustive, the test cases within the + verification directory do provide some indication whether gross errors + have been introduced. @@ -233,23 +334,23 @@ then: - adds a "checkpointXY_pre" comment (where X is a - checkpoint number and Y is a letter) to the tag-index - file and checks it into the CVS repository + adds a "checkpointXY_pre" comment (where X is a checkpoint + number and Y is a letter) to the + tag-index file and checks it into the CVS + repository - submits the set of changes to the CVS repository - and adds comments to tag-index - describing what the changes are along with a matching - "checkpointXY_post" entry + submits the set of changes to the CVS repository and adds + comments to tag-index describing what the + changes are along with a matching "checkpointXY_post" entry - The result of this tagging procedure is a sequence of - development checkpoints with comments which resembles: + The result of this tagging procedure is a sequence of development + checkpoints with comments which resembles: checkpoint50e_post @@ -271,126 +372,15 @@ checkpoint50d_pre - This information can be used to refer to various stages of - the code development. For example, bugs can be traced to - individual sets of CVS checkins based upon their first - appearance when comparing the results from different - checkpoints. + This information can be used to refer to various stages of the code + development. For example, bugs can be traced to individual sets of CVS + checkins based upon their first appearance when comparing the results from + different checkpoints. - - Editing the Documentation - - - Getting the Docs and Code - - The first step towards editing the documentation is to - checkout a copy of code, docs, and build scripts from the CVS - server using: - - - $ export CVS_RSH=ssh - $ export CVSROOT=':ext:auden.lcs.mit.edu:/u/u3/gcmpack' - $ mkdir scratch - $ cvs co MITgcm manual mitgcm.org - - - These commands extract the necessary information from the - CVS server and create a temporary (called - scratch) directory for the storage of the - HTML and other files that will be created. Please note that you - must either create scratch as shown or edit - the various Makefiles and scripts used to - create the documentation. - - - - Editing the Documentation - - The documentation is contained in the - manual directory in a raw LaTeX format. - The main document is manual.tex and it uses - \input{}s to include the chapters and - subsections. - - Since the same LaTeX source is used to produce PostScript, - PDF, and HTML output, care should be taken to follow certain - conventions. Two of the most important are the usage of the - \filelink{}{} and - \varlink{}{} commands. Both of these - commands have been defined to simplify the connection between - the automatically generated ("code browser") HTML and the HTML - version of the manual produced by LaTeX2HTML. They each take - two arguments (corresponding to the contents of the two sets of - curly braces) which are the text that the author wishes to be - "wrapped" within the link, and a specially formatted link thats - relative to the MITgcm directory within the - CVS tree. - - The result is a command that resembles either - - - - a reference to a variable or subroutine name such as - \varlink{tRef}{tRef}, or - - - - a reference to a file such as - \varlink{tRef}{path-to-the-file_name.F} - where the absolute path to the file is of the form - /foo/MITgcm/path/to/the/file_name.F - (please note how the leading "/foo/MITgcm" - component of the path is dropped leaving the path - relative to the head of the code - directory and each directory separator "/" is turned - into a "-") - - - - - - - - - Building the Documentation - - Given the directory structure of , the entire documentation for the web - site can be built using: - - - $ cd mitgcm.org/devel/buildweb - $ make All - - - Which builds the PDF from the LaTeX source, creates the - HTML output from the LaTeX source, parses the FORTRAN code base - to produce a hyperlinked HTML version of the source, and then - determines the cross-linking between the various HTML - components. - - If there are no errors, the result of the build process - (which can take 30+ minutes on a P4/2.5Ghz) will be contained - within a single directory called - scratch/dev_docs. This is a freshly built - version of the entire on-line users manual. If you have the - correct permissions, it can be directly copied to the web server - area: - - - $ mv scratch/dev_docs /u/u0/httpd/html - - - and the update is complete. - - - - - Coding for MITgcm @@ -404,7 +394,7 @@ For MITgcm, the process is similar. Typical commands are: - $ genmake -mods=../code + $ genmake2 -mods=../code $ make depend $ make @@ -415,9 +405,8 @@ The <filename>genmake2</> Utility - Please note that the older genmake is - deprecated and will eventually be replaced by genmake2. - This HOWTO only describes the newer tool. + (Note: the older genmake + has been replaced by genmake2) The first step in any MITgcm build is to create a Unix-style Makefile which will be parsed by @@ -463,7 +452,7 @@ - a gm_local file in the current + a gemake_local file in the current directory @@ -473,6 +462,12 @@ an "options file" as specified by the command-line option -optfile='FILENAME' + + a packages.conf file (in the current + directory or in one of the "MODS" directories, see below) + which contains the specific list of packages to compile + + then checking certain dependency rules (the package dependencies), @@ -526,33 +521,6 @@ - -pdepend=/PATH/FILENAME - - - This specifies the dependency file used for packages. If - not specified, the default dependency file is - $ROOTDIR/pkg/pkg_depend. The syntax for this file is - parsed on a line-by-line basis where each line containes either a - comment ("#") or a simple "PKGNAME1 (+|-)PKGNAME2" pairwise rule - where the "+" or "-" symbol specifies a "must be used with" or a - "must not be used with" relationship, respectively. If no rule is - specified, then it is assumed that the two packages are compatible - and will function either with or without each other. - - - - - -pdefault=PKG - -pdefault='PKG1 [PKG2 PKG3 ...]' - - This option specifies the default set of packages - to be used. If not set, the default package list will - be read from - $ROOTDIR/pkg/pkg_default. - - - - -adof=/path/to/file -adoptfile=/path/to/file @@ -580,7 +548,6 @@ will be overridden by any identically-named sources within the "MODS" directories. The order of precedence for this "name-hiding" is as follows: - "MODS" directories (in the order given) @@ -592,7 +559,38 @@ The "standard dirs" (which may have been specified by the "-standarddirs" option) + + + + -pgroups=/PATH/FILENAME + + This option specifies the file where package groups are + defined. If not set, the package-groups definition will + be read from + $ROOTDIR/pkg/pkg_groups. + + It also contains the default list of packages (defined + as the group "default_pkg_list") which is used + when no specific package list (file: packages.conf) + is found in current directory or in any "MODS" directory. + + + + + + -pdepend=/PATH/FILENAME + + + This specifies the dependency file used for packages. If + not specified, the default dependency file is + $ROOTDIR/pkg/pkg_depend. The syntax for this file is + parsed on a line-by-line basis where each line containes either a + comment ("#") or a simple "PKGNAME1 (+|-)PKGNAME2" pairwise rule + where the "+" or "-" symbol specifies a "must be used with" or a + "must not be used with" relationship, respectively. If no rule is + specified, then it is assumed that the two packages are compatible + and will function either with or without each other. @@ -618,7 +616,7 @@ In general, it is best to use genmake2 on a "clean" directory that is free of all source (*.[F,f],*.[F,f]90) and header (*.h,*.inc) files. Generally, this can be accomplished in an - "un-clean" directory by running "make CLEAN" followed by "make + "un-clean" directory by running "make Clean" followed by "make makefile". @@ -631,16 +629,18 @@ simulator) executable using: - $ make CLEAN + $ make Clean $ make depend $ make - The "make CLEAN" step will remove any stale source files, include + The "make Clean" step will remove any stale source files, include files, and links. It is strongly recommended for "un-clean" directories which may contain the (perhaps partial) results of - previous builds. Such "debris" can interfere with the next stage of - the build. + previous builds. Such "debris" can interfere with the next stage of + the build. + A more agressive cleaning option, "make CLEAN", can be used to also + remove the executable and output files from a previous run. The "make depend" step will create a large number of symbolic links from the local directory to the source file locations. It also @@ -799,7 +799,7 @@ special command option (see "-command" below) to invoke the MPI executable. Examples of PBS scripts using MPI with testreport can be found in the + url="http://mitgcm.org/viewvc/MITgcm/MITgcm_contrib/test_scripts/"> MITgcm-contrib area @@ -811,7 +811,7 @@ output.txt" is not sufficient. This option allows a more general command (or shell script) to be invoked. Examples of PBS scripts using MPI with testreport can be found in the + url="http://mitgcm.org/viewvc/MITgcm/MITgcm_contrib/test_scripts/"> MITgcm-contrib area @@ -838,7 +838,7 @@ the generic fluid dynamical engine. The MITgcmUV packaging structure is described below using generic - package names ${pkg}. A concrete examples of a package is the code for + package names ${pkg}. A concrete examples of a package is the code for implementing GM/Redi mixing. This code uses the package name @@ -898,6 +898,9 @@ Package runtime config. options are imported into a common block held in a header file called "${PKG}.h". + Note: In some packages, the header file "${PKG}.h" is splitted + into "${PKG}_PARAMS.h" that contains the package parameters and + ${PKG}_VARS.h" for the field arrays. o The core driver part of the model can check for runtime enabling or disabling of individual packages @@ -920,12 +923,14 @@ 1. Within the core driver code flags of the form ALLOW_${PKG} are used to include or exclude whole packages. The ALLOW_${PKG} flags are included - from a PKG_CPP_OPTIONS block which is currently + from a PACKAGES_CONFIG.h file that is automatically + generated by genmake2 (see genmake2 section). held in-line in the CPP_OPTIONS.h header file. e.g. Core model code ..... + #include "PACKAGES_CONFIG.h" #include "CPP_OPTIONS.h" : : @@ -937,8 +942,8 @@ 2. Within an individual package a header file, "${PKG}_OPTIONS.h", is used to set CPP flags - specific to that package. It is not recommended - to include this file in "CPP_OPTIONS.h". + specific to that package. It also includes + "PACKAGES_CONFIG.h" and "CPP_OPTIONS.h". Package Boot Sequence @@ -960,7 +965,14 @@ & CALL ${PKG}_READPARMS( retCode ) #endif - 2. S/R PACKAGES_CHECK() + 3. S/R PACKAGES_INIT_FIXED() + : + #ifdef ALLOW_${PKG} + if ( use${Pkg} ) + & CALL ${PKG}_INIT_FIXED( retCode ) + #endif + + 4. S/R PACKAGES_CHECK() : #ifdef ALLOW_${PKG} if ( use${Pkg} ) @@ -970,13 +982,28 @@ & CALL PACKAGES_CHECK_ERROR('${PKG}') #endif - 3. S/R PACKAGES_INIT() + 5. S/R PACKAGES_INIT_VARIABLES() : #ifdef ALLOW_${PKG} if ( use${Pkg} ) - & CALL ${PKG}_INIT( retCode ) + & CALL ${PKG}_INIT_VARIA( ) + #endif + +Package Output +============== + 6. S/R DO_THE_MODEL_IO + + #ifdef ALLOW_${PKG} + if ( use${Pkg} ) + & CALL ${PKG}_OUTPUT( ) #endif + 7. S/R PACKAGES_WRITE_PICKUP() + + #ifdef ALLOW_${PKG} + if ( use${Pkg} ) + & CALL ${PKG}_WRITE_PICKUP( ) + #endif Description =========== @@ -984,8 +1011,15 @@ - ${PKG}_READPARMS() is responsible for reading in the package parameters file data.${pkg}, and storing - the package parameters in "${PKG}.h". - -> called in INITIALISE_FIXED + the package parameters in "${PKG}.h" (or in "${PKG}_PARAMS.h"). + -> called from INITIALISE_FIXED in PACKAGES_READPARMS + + - ${PKG}_INIT_FIXED() + is responsible for completing the internal setup of a package. + -> called from INITIALISE_FIXED in PACKAGES_INIT_FIXED + note: 1) some pkg use instead: + CALL ${PKG}_INITIALISE ( or the old form CALL ${PKG}_INIT ) + 2) for simple pkg setup, this part is done inside ${PKG}_READPARMS - ${PKG}_CHECK() is responsible for validating @@ -994,14 +1028,36 @@ need to check. This is done through header files "${PKG}.h". It is assumed that parameters owned by other packages will not be reset during ${PKG}_CHECK(). - -> called in INITIALISE_FIXED + -> called from INITIALISE_FIXED in PACKAGES_CHECK - - ${PKG}_INIT() - is responsible for completing the - internal setup of a package. This routine is called after - the core model state has been completely initialised - but before the core model timestepping starts. - -> called in INITIALISE_VARIA + - ${PKG}_INIT_VARIA() + is responsible for fill-in all package variables with an initial value. + Contains eventually a call to ${PKG}_READ_PICKUP that will read + from a pickup file the package variables required to restart the model. + This routine is called after the core model state has been completely + initialised but before the core model timestepping starts. + -> called from INITIALISE_VARIA in PACKAGES_INIT_VARIABLES + note: the name ${PKG}_INIT_VARIA is not yet standard and some pkg + use for e.g. ${PKG}_INI_VARS, ${PKG}_INIT_VARIABLES, or the old + form ${PKG}_INIT + + - ${PKG}_OUTPUT( ) + is responsible for writing time-average fields to output files + (but the cumulating step is done within the package main S/R). + Can also contain other diagnostics (.e.g. CALL ${PKG}_MONITOR) + and write snap-shot fields that are hold in common blocks. Other + temporary fields are directly dump to file where they are available. + NOTE: 1) the S/R old name ${PKG}_DIAGS is used in some packages + but is beeing replaced by ${PKG}_OUTPUT + to avoid confusion with pkg/diagnostics functionality. + 2) the output part is not yet in a standard form and might still + evolve a lot. + -> called within DO_THE_MODEL_IO + + - ${PKG}_WRITE_PICKUP() + is responsible for writing a package pickup file when necessary for + a restart. (found also the old name: ${PKG}_WRITE_CHECKPOINT ) + -> called from FORWARD_STEP and THE_MODEL_MAIN in PACKAGES_WRITE_PICKUP Summary ======= @@ -1022,13 +1078,22 @@ ----------------------- * ${PKG}_OPTIONS.h has further package-specific CPP options * ${PKG}.h package-specific common block variables, fields + or ${PKG}_PARAMS.h package-specific common block parameters + and ${PKG}_VARS.h package-specific common block fields - FORTRAN source files ----------------------- - * ${pkg}_readparms.F reads parameters from file data.${pkg} - * ${pkg}_check.F checks package dependencies and consistencies - * ${pkg}_init.F initialises package-related fields - * ${pkg}_... .F package source code + * ${pkg}_readparms.F reads parameters from file data.${pkg} + * ${pkg}_init_fixed.F complete the package setup + * ${pkg}_check.F checks package dependencies and consistencies + * ${pkg}_init_varia.F initialises package-related fields + * ${pkg}_... .F package source code + * ${pkg}_output.F write output to file. + * ${pkg}_write_pickup.F write a package pickup file to restart the model + + New: Subroutine in one package (pkgA) that only contains code which + is connected to a 2nd package (pkgB) (e.g.: gmredi_diagnostics_init.F) + will be named: pkgA_pkgB_something.F - parameter file ----------------------- @@ -1038,6 +1103,108 @@ + + Editing the Documentation + + + Getting the Docs and Code + + The first step towards editing the documentation is to checkout a + copy of code, docs, and build scripts from the CVS server using: + + + $ export CVS_RSH=ssh + $ export CVSROOT=':ext:NAME@mitgcm.org:/u/gcmpack' + $ mkdir scratch + $ cvs co -P MITgcm manual mitgcm.org + + + These commands extract the necessary information from the CVS server + and create a temporary (called scratch) directory for + the storage of the HTML and other files that will be created. Please note + that you must either create scratch as shown or edit + the various Makefiles and scripts used to create the + documentation. + + + + Editing the Documentation + + The documentation is contained in the manual + directory in a raw LaTeX format. The main document is + manual.tex and it uses \input{}s + to include the chapters and subsections. + + Since the same LaTeX source is used to produce PostScript, PDF, and + HTML output, care should be taken to follow certain conventions. Two of + the most important are the usage of the \filelink{}{} + and \varlink{}{} commands. Both of these commands have + been defined to simplify the connection between the automatically + generated ("code browser") HTML and the HTML version of the manual + produced by LaTeX2HTML. They each take two arguments (corresponding to + the contents of the two sets of curly braces) which are the text that the + author wishes to be "wrapped" within the link, and a specially formatted + link thats relative to the MITgcm directory within + the CVS tree. + + The result is a command that resembles either + + + + a reference to a variable or subroutine name such as + \varlink{tRef}{tRef}, or + + + + a reference to a file such as + \varlink{tRef}{path-to-the-file_name.F} + where the absolute path to the file is of the form + /foo/MITgcm/path/to/the/file_name.F + (please note how the leading "/foo/MITgcm" + component of the path is dropped leaving the path + relative to the head of the code + directory and each directory separator "/" is turned + into a "-") + + + + + + + + + Building the Documentation + + Given the directory structure of , the entire documentation for the web + site can be built using: + + + $ cd mitgcm.org/devel/buildweb + $ make All + + + Which builds the PDF from the LaTeX source, creates the HTML output + from the LaTeX source, parses the FORTRAN code base to produce a + hyperlinked HTML version of the source, and then determines the + cross-linking between the various HTML components. + + If there are no errors, the result of the build process (which can + take 30+ minutes on a P4/2.5Ghz) will be contained within a single + directory called scratch/dev_docs. This is a freshly + built version of the entire on-line users manual. If you have the correct + permissions, it can be directly copied to the web server area: + + + $ mv scratch/dev_docs /u/u0/httpd/html + + + and the update is complete. + + + + +