--- MITgcm/doc/devel_HOWTO.sgml 2003/12/10 20:44:39 1.5 +++ MITgcm/doc/devel_HOWTO.sgml 2010/05/15 01:44:50 1.13 @@ -1,4 +1,8 @@ +
@@ -22,6 +26,22 @@ Initial version. + + 0.02 + 2010-01-21 + jmc + + update links. + + + + 0.03 + 2010-04-25 + jmc + + Add subsection "Developer settings" (under CVS Repository). + + @@ -37,7 +57,7 @@ New Versions of This Document You can obtain the latest version of this document online in + url="http://mitgcm.org/public/docs.html">online in various formats. @@ -56,7 +76,7 @@ User Manual Before jumping into development, please familiarize yourself with - the MITgcm user manual + the MITgcm user manual . This document contains volumes of useful information and is included here by reference. @@ -94,6 +114,7 @@ CVS Repository + Layout @@ -104,46 +125,45 @@ others. The tree currently resembles: gcmpack/ - MITgcm-contrib contributed code - CS-regrid goes into utils - cvspolicy.html -save- - CVSROOT -save- - development experimental stuff - manual -save- - misc -?- + CVSROOT -hidden- MITgcm code - adjoint fold into genmake - bin stub for ecco build - compare01 old from 20th century - diags timeave f77 in pkgs now - doc tags -- connect to real docs? - eesupp cnh? - exe ecco user build - ,- jobs runtime shell scripts for - | various platforms - | lsopt line search - m| model main dynamics (core) - e| optimization_drivers ? - r| optim line search interface - g| pkg alternate and optional numerics, etc. - e|- tools - ?| tutorial_examples documented tests - | only populated on release1 branch - | and not validated during "testscript" - '- utils - verification std tests + bin empty + doc basic developpment documentation + eesupp execution environment support code (wrapper) + exe empty + jobs runtime shell scripts for + various platforms (not maintained) + lsopt line search + model main dynamics (core) + optim line search interface + pkg alternate and optional numerics, etc. + tools scripts to build (and test) + utils pre/post processing tools (matlab, ..) + verification standard regression tests + examples + + documented examples (tutorials) + tutorial_examples (only in release1 branch) + MITgcm_contrib contributed code - mitgcmdoc -> manual -remove- + acesgrid.org build acesgrid web site + development experimental stuff + gfd_lab -?- + manual source of MITgcm documentation mitgcm.org build web site - models -?- - packages -?- - preprocess -?- - tmp -?- + old_develop old and early development source + misc -?- + models -?- + packages -?- + preprocess -?- + pdfs some pdfs + planetinabottle.org unfinished web site + www.ecco-group.org build ecco web site ? + @@ -182,48 +202,128 @@ Branches As shown in the online ViewCVS-generated - tree, the MITgcm codebase is split into to two branches - or "lines" under which development proceeds. These two lines - are referred to as the "MAIN" and "ecco" versions of the code. - While not identical, the bulk of the MAIN and ecco lines are - composed of files from the same codebase. + url="http://mitgcm.org/viewvc/MITgcm/MITgcm/model/src/forward_step.F?view=graph"> + ViewCVS-generated tree, the MITgcm codebase is split into + branches or "lines" under which development proceeds. The main line + of development is referred to as the "MAIN" version of the code. Periodically, a "Release" branch is formed from the "MAIN" - development branch. This is done in order to create a - relatively stable reference point for both users and developers. - The intent is that once a relese branch has been created, only - bug-fixes will be added to it. Meanwhile, development (which - might "break" or otherwise render invalid the documentation, - tutorials, and/or examples contained within a release branch) is - allowed to continue along the MAIN and ecco lines. + development branch. This is done in order to create a relatively stable + reference point for both users and developers. The intent is that once a + release branch has been created, only bug-fixes will be added to it. + Meanwhile, development (which might "break" or otherwise render invalid + the documentation, tutorials, and/or examples contained within a release + branch) is allowed to continue along the MAIN line. - Tagging + Developer settings + + CVS is a convenient tool to keep up-to-date a personal copy of the + MITgcm code (see: + using CVS ). The same tool is used by developers to + incorporate any change into the repository. However, this later + function requires specific settings, as detailed here after: + + + You will need an account (loggin access) to the server + "mitgcm.org" with the proper group setting (e.g., + group "gcmctrb" to add/modify code into MITgcm_contrib). + This kind of account is granted only upon well motivated request. + The access to the server mitgcm.org is through ssh-key authorization + which will need to be set properly on both side (on your local machine + and on your server account). You need to be able to + to ssh to mitgcm.org (or ssh MY_USER_NAME@mitgcm.org + in case of different user-name on both sides) to proceed further. + + + + You need to register to the + + mitgcm-cvs mailing list. + This ensures that other developers will receive email notification + when you make changes; you will also receive as well such email + when others make changes to the repository. + + - The intent of tagging is to create "known-good" - checkpoints that developers can use as references. - Traditionally, MITgcm tagging has maintained the following - conventions: + + It is highly recommended that you register also to the + + mitgcm-devel mailing list (expect a short delay for + this request to be processed). + This list is intended for developer discussions. + + + + + The standard anonymous mode (using "cvsanon", as mentionned + + here ) does not allow check-in ("cvs commit") permission. + Instead, you will need to set our CVS environment as follow: + + $ export CVS_RSH=ssh + $ export CVSROOT=':ext:MY_USER_NAME@mitgcm.org:/u/gcmpack' + + After downloading a directory, e.g.: myCopy, + from the CVS repository (e.g., + MITgcm_contrib/thisPart) using the command: + + $ cvs co -P -d myCopy MITgcm_contrib/thisPart + + the type of CVS environment which has been used + is stored in the file myCopy/CVS/Root + and makes it difficult to re-use, for cvs-commit purpose, + a cvs local copy (myCopy) which was obtained + using the CVS anonymous mode. + + + + At this stage, you should be able to send your modified source + file (e.g., src_file) from your local copy directory + (myCopy) to the CVS repository + (MITgcm_contrib/thisPart) using the command + "cvs commit": + + $ cd myCopy + $ cvs -n update (optional; check if new changes have been made) + $ cvs diff src_file (optional; list your changes) + $ cvs commit src_file + + It is essential that you provide a short description of the + changes you made to src_file as you check-in + this file (the "cvs commit" command automatically opens your standard + editor for this purpose). + + + + + + + + Main code development + (formerly named "Tagging" ; this section needs an update) + + The intent of tagging is to create "known-good" checkpoints that + developers can use as references. Traditionally, MITgcm tagging has + maintained the following conventions: - Developer checks out code into a local CVS-managed - directory, makes various changes/additions, tests these - edits, and eventually reaches a point where (s)he is - satisfied that the changes form a new "useful" point in the - evolution of the code. + Developer checks out code into a local CVS-managed directory, + makes various changes/additions, tests these edits, and eventually + reaches a point where (s)he is satisfied that the changes form a new + "useful" point in the evolution of the code. The developer then runs the testscript - shell script to see if any problems are introduced. While - not intended to be exhaustive, the test cases within the - verification directory do provide some indication whether - gross errors have been introduced. + url="http://mitgcm.org/viewvc/MITgcm/MITgcm/verification/testreport"> + testreport shell script to see if any problems are introduced. + While not intended to be exhaustive, the test cases within the + verification directory do provide some indication whether gross errors + have been introduced. @@ -233,23 +333,23 @@ then: - adds a "checkpointXY_pre" comment (where X is a - checkpoint number and Y is a letter) to the tag-index - file and checks it into the CVS repository + adds a "checkpointXY_pre" comment (where X is a checkpoint + number and Y is a letter) to the + tag-index file and checks it into the CVS + repository - submits the set of changes to the CVS repository - and adds comments to tag-index - describing what the changes are along with a matching - "checkpointXY_post" entry + submits the set of changes to the CVS repository and adds + comments to tag-index describing what the + changes are along with a matching "checkpointXY_post" entry - The result of this tagging procedure is a sequence of - development checkpoints with comments which resembles: + The result of this tagging procedure is a sequence of development + checkpoints with comments which resembles: checkpoint50e_post @@ -271,126 +371,15 @@ checkpoint50d_pre - This information can be used to refer to various stages of - the code development. For example, bugs can be traced to - individual sets of CVS checkins based upon their first - appearance when comparing the results from different - checkpoints. + This information can be used to refer to various stages of the code + development. For example, bugs can be traced to individual sets of CVS + checkins based upon their first appearance when comparing the results from + different checkpoints. - - Editing the Documentation - - - Getting the Docs and Code - - The first step towards editing the documentation is to - checkout a copy of code, docs, and build scripts from the CVS - server using: - - - $ export CVS_RSH=ssh - $ export CVSROOT=':ext:auden.lcs.mit.edu:/u/u3/gcmpack' - $ mkdir scratch - $ cvs co MITgcm manual mitgcm.org - - - These commands extract the necessary information from the - CVS server and create a temporary (called - scratch) directory for the storage of the - HTML and other files that will be created. Please note that you - must either create scratch as shown or edit - the various Makefiles and scripts used to - create the documentation. - - - - Editing the Documentation - - The documentation is contained in the - manual directory in a raw LaTeX format. - The main document is manual.tex and it uses - \input{}s to include the chapters and - subsections. - - Since the same LaTeX source is used to produce PostScript, - PDF, and HTML output, care should be taken to follow certain - conventions. Two of the most important are the usage of the - \filelink{}{} and - \varlink{}{} commands. Both of these - commands have been defined to simplify the connection between - the automatically generated ("code browser") HTML and the HTML - version of the manual produced by LaTeX2HTML. They each take - two arguments (corresponding to the contents of the two sets of - curly braces) which are the text that the author wishes to be - "wrapped" within the link, and a specially formatted link thats - relative to the MITgcm directory within the - CVS tree. - - The result is a command that resembles either - - - - a reference to a variable or subroutine name such as - \varlink{tRef}{tRef}, or - - - - a reference to a file such as - \varlink{tRef}{path-to-the-file_name.F} - where the absolute path to the file is of the form - /foo/MITgcm/path/to/the/file_name.F - (please note how the leading "/foo/MITgcm" - component of the path is dropped leaving the path - relative to the head of the code - directory and each directory separator "/" is turned - into a "-") - - - - - - - - - Building the Documentation - - Given the directory structure of , the entire documentation for the web - site can be built using: - - - $ cd mitgcm.org/devel/buildweb - $ make All - - - Which builds the PDF from the LaTeX source, creates the - HTML output from the LaTeX source, parses the FORTRAN code base - to produce a hyperlinked HTML version of the source, and then - determines the cross-linking between the various HTML - components. - - If there are no errors, the result of the build process - (which can take 30+ minutes on a P4/2.5Ghz) will be contained - within a single directory called - scratch/dev_docs. This is a freshly built - version of the entire on-line users manual. If you have the - correct permissions, it can be directly copied to the web server - area: - - - $ mv scratch/dev_docs /u/u0/httpd/html - - - and the update is complete. - - - - - Coding for MITgcm @@ -463,7 +452,7 @@ - a gm_local file in the current + a gemake_local file in the current directory @@ -799,7 +788,7 @@ special command option (see "-command" below) to invoke the MPI executable. Examples of PBS scripts using MPI with testreport can be found in the + url="http://mitgcm.org/viewvc/MITgcm/MITgcm_contrib/test_scripts/"> MITgcm-contrib area @@ -811,7 +800,7 @@ output.txt" is not sufficient. This option allows a more general command (or shell script) to be invoked. Examples of PBS scripts using MPI with testreport can be found in the + url="http://mitgcm.org/viewvc/MITgcm/MITgcm_contrib/test_scripts/"> MITgcm-contrib area @@ -838,7 +827,7 @@ the generic fluid dynamical engine. The MITgcmUV packaging structure is described below using generic - package names ${pkg}. A concrete examples of a package is the code for + package names ${pkg}. A concrete examples of a package is the code for implementing GM/Redi mixing. This code uses the package name @@ -898,6 +887,9 @@ Package runtime config. options are imported into a common block held in a header file called "${PKG}.h". + Note: In some packages, the header file "${PKG}.h" is splitted + into "${PKG}_PARAMS.h" that contains the package parameters and + ${PKG}_VARS.h" for the field arrays. o The core driver part of the model can check for runtime enabling or disabling of individual packages @@ -920,12 +912,14 @@ 1. Within the core driver code flags of the form ALLOW_${PKG} are used to include or exclude whole packages. The ALLOW_${PKG} flags are included - from a PKG_CPP_OPTIONS block which is currently + from a PACKAGES_CONFIG.h file that is automatically + generated by genmake2 (see genmake2 section). held in-line in the CPP_OPTIONS.h header file. e.g. Core model code ..... + #include "PACKAGES_CONFIG.h" #include "CPP_OPTIONS.h" : : @@ -937,8 +931,8 @@ 2. Within an individual package a header file, "${PKG}_OPTIONS.h", is used to set CPP flags - specific to that package. It is not recommended - to include this file in "CPP_OPTIONS.h". + specific to that package. It also includes + "PACKAGES_CONFIG.h" and "CPP_OPTIONS.h". Package Boot Sequence @@ -960,7 +954,14 @@ & CALL ${PKG}_READPARMS( retCode ) #endif - 2. S/R PACKAGES_CHECK() + 3. S/R PACKAGES_INIT_FIXED() + : + #ifdef ALLOW_${PKG} + if ( use${Pkg} ) + & CALL ${PKG}_INIT_FIXED( retCode ) + #endif + + 4. S/R PACKAGES_CHECK() : #ifdef ALLOW_${PKG} if ( use${Pkg} ) @@ -970,13 +971,28 @@ & CALL PACKAGES_CHECK_ERROR('${PKG}') #endif - 3. S/R PACKAGES_INIT() + 5. S/R PACKAGES_INIT_VARIABLES() : #ifdef ALLOW_${PKG} if ( use${Pkg} ) - & CALL ${PKG}_INIT( retCode ) + & CALL ${PKG}_INIT_VARIA( ) + #endif + +Package Output +============== + 6. S/R DO_THE_MODEL_IO + + #ifdef ALLOW_${PKG} + if ( use${Pkg} ) + & CALL ${PKG}_OUTPUT( ) #endif + 7. S/R PACKAGES_WRITE_PICKUP() + + #ifdef ALLOW_${PKG} + if ( use${Pkg} ) + & CALL ${PKG}_WRITE_PICKUP( ) + #endif Description =========== @@ -984,8 +1000,15 @@ - ${PKG}_READPARMS() is responsible for reading in the package parameters file data.${pkg}, and storing - the package parameters in "${PKG}.h". - -> called in INITIALISE_FIXED + the package parameters in "${PKG}.h" (or in "${PKG}_PARAMS.h"). + -> called from INITIALISE_FIXED in PACKAGES_READPARMS + + - ${PKG}_INIT_FIXED() + is responsible for completing the internal setup of a package. + -> called from INITIALISE_FIXED in PACKAGES_INIT_FIXED + note: 1) some pkg use instead: + CALL ${PKG}_INITIALISE ( or the old form CALL ${PKG}_INIT ) + 2) for simple pkg setup, this part is done inside ${PKG}_READPARMS - ${PKG}_CHECK() is responsible for validating @@ -994,14 +1017,36 @@ need to check. This is done through header files "${PKG}.h". It is assumed that parameters owned by other packages will not be reset during ${PKG}_CHECK(). - -> called in INITIALISE_FIXED + -> called from INITIALISE_FIXED in PACKAGES_CHECK - - ${PKG}_INIT() - is responsible for completing the - internal setup of a package. This routine is called after - the core model state has been completely initialised - but before the core model timestepping starts. - -> called in INITIALISE_VARIA + - ${PKG}_INIT_VARIA() + is responsible for fill-in all package variables with an initial value. + Contains eventually a call to ${PKG}_READ_PICKUP that will read + from a pickup file the package variables required to restart the model. + This routine is called after the core model state has been completely + initialised but before the core model timestepping starts. + -> called from INITIALISE_VARIA in PACKAGES_INIT_VARIABLES + note: the name ${PKG}_INIT_VARIA is not yet standard and some pkg + use for e.g. ${PKG}_INI_VARS, ${PKG}_INIT_VARIABLES, or the old + form ${PKG}_INIT + + - ${PKG}_OUTPUT( ) + is responsible for writing time-average fields to output files + (but the cumulating step is done within the package main S/R). + Can also contain other diagnostics (.e.g. CALL ${PKG}_MONITOR) + and write snap-shot fields that are hold in common blocks. Other + temporary fields are directly dump to file where they are available. + NOTE: 1) the S/R old name ${PKG}_DIAGS is used in some packages + but is beeing replaced by ${PKG}_OUTPUT + to avoid confusion with pkg/diagnostics functionality. + 2) the output part is not yet in a standard form and might still + evolve a lot. + -> called within DO_THE_MODEL_IO + + - ${PKG}_WRITE_PICKUP() + is responsible for writing a package pickup file when necessary for + a restart. (found also the old name: ${PKG}_WRITE_CHECKPOINT ) + -> called from FORWARD_STEP and THE_MODEL_MAIN in PACKAGES_WRITE_PICKUP Summary ======= @@ -1022,13 +1067,22 @@ ----------------------- * ${PKG}_OPTIONS.h has further package-specific CPP options * ${PKG}.h package-specific common block variables, fields + or ${PKG}_PARAMS.h package-specific common block parameters + and ${PKG}_VARS.h package-specific common block fields - FORTRAN source files ----------------------- - * ${pkg}_readparms.F reads parameters from file data.${pkg} - * ${pkg}_check.F checks package dependencies and consistencies - * ${pkg}_init.F initialises package-related fields - * ${pkg}_... .F package source code + * ${pkg}_readparms.F reads parameters from file data.${pkg} + * ${pkg}_init_fixed.F complete the package setup + * ${pkg}_check.F checks package dependencies and consistencies + * ${pkg}_init_varia.F initialises package-related fields + * ${pkg}_... .F package source code + * ${pkg}_output.F write output to file. + * ${pkg}_write_pickup.F write a package pickup file to restart the model + + New: Subroutine in one package (pkgA) that only contains code which + is connected to a 2nd package (pkgB) (e.g.: gmredi_diagnostics_init.F) + will be named: pkgA_pkgB_something.F - parameter file ----------------------- @@ -1038,6 +1092,108 @@ + + Editing the Documentation + + + Getting the Docs and Code + + The first step towards editing the documentation is to checkout a + copy of code, docs, and build scripts from the CVS server using: + + + $ export CVS_RSH=ssh + $ export CVSROOT=':ext:NAME@mitgcm.org:/u/gcmpack' + $ mkdir scratch + $ cvs co -P MITgcm manual mitgcm.org + + + These commands extract the necessary information from the CVS server + and create a temporary (called scratch) directory for + the storage of the HTML and other files that will be created. Please note + that you must either create scratch as shown or edit + the various Makefiles and scripts used to create the + documentation. + + + + Editing the Documentation + + The documentation is contained in the manual + directory in a raw LaTeX format. The main document is + manual.tex and it uses \input{}s + to include the chapters and subsections. + + Since the same LaTeX source is used to produce PostScript, PDF, and + HTML output, care should be taken to follow certain conventions. Two of + the most important are the usage of the \filelink{}{} + and \varlink{}{} commands. Both of these commands have + been defined to simplify the connection between the automatically + generated ("code browser") HTML and the HTML version of the manual + produced by LaTeX2HTML. They each take two arguments (corresponding to + the contents of the two sets of curly braces) which are the text that the + author wishes to be "wrapped" within the link, and a specially formatted + link thats relative to the MITgcm directory within + the CVS tree. + + The result is a command that resembles either + + + + a reference to a variable or subroutine name such as + \varlink{tRef}{tRef}, or + + + + a reference to a file such as + \varlink{tRef}{path-to-the-file_name.F} + where the absolute path to the file is of the form + /foo/MITgcm/path/to/the/file_name.F + (please note how the leading "/foo/MITgcm" + component of the path is dropped leaving the path + relative to the head of the code + directory and each directory separator "/" is turned + into a "-") + + + + + + + + + Building the Documentation + + Given the directory structure of , the entire documentation for the web + site can be built using: + + + $ cd mitgcm.org/devel/buildweb + $ make All + + + Which builds the PDF from the LaTeX source, creates the HTML output + from the LaTeX source, parses the FORTRAN code base to produce a + hyperlinked HTML version of the source, and then determines the + cross-linking between the various HTML components. + + If there are no errors, the result of the build process (which can + take 30+ minutes on a P4/2.5Ghz) will be contained within a single + directory called scratch/dev_docs. This is a freshly + built version of the entire on-line users manual. If you have the correct + permissions, it can be directly copied to the web server area: + + + $ mv scratch/dev_docs /u/u0/httpd/html + + + and the update is complete. + + + + +