/[MITgcm]/manual/s_autodiff/text/doc_ad_2.tex

Diff of /manual/s_autodiff/text/doc_ad_2.tex

Parent Directory | Revision Log | View Revision Graph Revision Graph | View Patch Patch

-revision 1.14 by cnh,
Thu Feb 28 19:32:20 2002 UTC
+revision 1.16 by heimbach,
Tue May 11 21:55:14 2004 UTC
 Line 557 
 Because of the local character of the de
  (a derivative is defined w.r.t. a point along the trajectory),
  the intermediate results of the model trajectory
  $\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$
- are needed to evaluate the intermediate Jacobian
+ may be required to evaluate the intermediate Jacobian
  $M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $.
+ This is the case e.g. for nonlinear expressions
+ (momentum advection, nonlinear equation of state), state-dependent
+ conditional statements (parameterization schemes).
  In the forward mode, the intermediate results are required
  in the same order as computed by the full forward model ${\cal M}$,
  but in the reverse mode they are required in the reverse order.
-Line 569 
 point of evaluation has to be recomputed
+Line 572 
 point of evaluation has to be recomputed
  A method to balance the amount of recomputations vs.
  storage requirements is called {\sf checkpointing}
- (e.g. \cite{res-eta:98}).
+ (e.g. \cite{gri:92}, \cite{res-eta:98}).
  It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing
  [as an example, we give explicit numbers for a 3-day
  integration with a 1-hourly timestep in square brackets].
-Line 580 
 In a first step, the model trajectory is
+Line 583 
 In a first step, the model trajectory is
  $ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals],
  with the label $lev3$ for this outermost loop.
  The model is then integrated along the full trajectory,
- and the model state stored only at every $ k_{i}^{lev3} $-th timestep
+ and the model state stored to disk only at every $ k_{i}^{lev3} $-th timestep
  [i.e. 3 times, at
  $ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $].
+ In addition, the cost function is computed, if needed.
  %
  \item [$lev2$]
  In a second step each subsection itself is divided into
- $ {n}^{lev2} $ sub-subsections
+ $ {n}^{lev2} $ subsections
  [$ {n}^{lev2} $=4 6-hour intervals per subsection].
  The model picks up at the last outermost dumped state
  $ v_{k_{n}^{lev3}} $ and is integrated forward in time along
  the last subsection, with the label $lev2$ for this
  intermediate loop.
- The model state is now stored at every $ k_{i}^{lev2} $-th
+ The model state is now stored to disk at every $ k_{i}^{lev2} $-th
  timestep
  [i.e. 4 times, at
  $ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $].
-Line 600 
 $ i = 0,1,2,3 $ corresponding to $ k_{i}
+Line 604 
 $ i = 0,1,2,3 $ corresponding to $ k_{i}
  \item [$lev1$]
  Finally, the model picks up at the last intermediate dump state
  $ v_{k_{n}^{lev2}} $ and is integrated forward in time along
- the last sub-subsection, with the label $lev1$ for this
+ the last subsection, with the label $lev1$ for this
  intermediate loop.
- Within this sub-subsection only, the model state is stored
+ Within this sub-subsection only, parts of the model state is stored
- at every timestep
+ to memory at every timestep
  [i.e. every hour $ i=0,...,5$ corresponding to
  $ k_{i}^{lev1} = 66, 67, \ldots, 71 $].
- Thus, the  final state $ v_n = v_{k_{n}^{lev1}} $ is reached
+ The  final state $ v_n = v_{k_{n}^{lev1}} $ is reached
- and the model state of all proceeding timesteps along the last
+ and the model state of all preceding timesteps along the last
- sub-subsections are available, enabling integration backwards
+ innermost subsection are available, enabling integration backwards
- in time along the last sub-subsection.
+ in time along the last subsection.
- Thus, the adjoint can be computed along this last
+ The adjoint can thus be computed along this last
- sub-subsection $k_{n}^{lev2}$.
+ subsection $k_{n}^{lev2}$.
  %
  \end{itemize}
  %
  This procedure is repeated consecutively for each previous
- sub-subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $
+ subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $
  carrying the adjoint computation to the initial time
  of the subsection $k_{n}^{lev3}$.
  Then, the procedure is repeated for the previous subsection
-Line 627 
 $k_{1}^{lev3}$.
+Line 631 
 $k_{1}^{lev3}$.
  For the full model trajectory of
  $ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps
  the required storing of the model state was significantly reduced to
- $ n^{lev1} + n^{lev2} + n^{lev3} $
+ $ n^{lev2} + n^{lev3} $ to disk and roughly $ n^{lev1} $ to memory
  [i.e. for the 3-day integration with a total oof 72 timesteps
- the model state was stored 13 times].
+ the model state was stored 7 times to disk and roughly 6 times
+ to memory].
  This saving in memory comes at a cost of a required
 full forward integrations of the model (one for each
  checkpointing level).
- The balance of storage vs. recomputation certainly depends
+ The optimal balance of storage vs. recomputation certainly depends
- on the computing resources available.
+ on the computing resources available and may be adjusted by
+ adjusting the partitioning among the
+ $ n^{lev3}, \,\, n^{lev2}, \,\, n^{lev1} $.
  \begin{figure}[t!]
  \begin{center}
-Line 673 
 Schematic view of intermediate dump and
+Line 680 
 Schematic view of intermediate dump and
  In this section we describe in a general fashion
  the parts of the code that are relevant for automatic
- differentiation using the software tool TAMC.
+ differentiation using the software tool TAF.
  \input{part5/doc_ad_the_model}
  The basic flow is depicted in \ref{fig:adthemodel}.
- If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine
+ If CPP option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine
  {\it the\_model\_main}, instead of calling {\it the\_main\_loop},
  invokes the adjoint of this routine, {\it adthe\_main\_loop},
- which is the toplevel routine in terms of reverse mode computation.
+ which is the toplevel routine in terms of automatic differentiation.
- The routine {\it adthe\_main\_loop} has been generated using TAMC.
+ The routine {\it adthe\_main\_loop} has been generated by TAF.
- It contains both the forward integration of the full model,
+ It contains both the forward integration of the full model, the
+ cost function calculation,
  any additional storing that is required for efficient checkpointing,
  and the reverse integration of the adjoint model.
- The structure of {\it adthe\_main\_loop} has been strongly
- simplified for clarification; in particular, no checkpointing
+ [DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM]
+ In Fig. \ref{fig:adthemodel}
+ the structure of {\it adthe\_main\_loop} has been strongly
+ simplified to focus on the essentials; in particular, no checkpointing
  procedures are shown here.
  Prior to the call of {\it adthe\_main\_loop}, the routine
- {\it ctrl\_unpack} is invoked to unpack the control vector,
+ {\it ctrl\_unpack} is invoked to unpack the control vector
- and following that call, the routine {\it ctrl\_pack}
+ or initialise the control variables.
+ Following the call of {\it adthe\_main\_loop},
+ the routine {\it ctrl\_pack}
  is invoked to pack the control vector
  (cf. Section \ref{section_ctrl}).
  If gradient checks are to be performed, the option
-Line 700 
 the driver routine {\it grdchk\_main} is
+Line 714 
 the driver routine {\it grdchk\_main} is
  the gradient has been computed via the adjoint
  (cf. Section \ref{section_grdchk}).
+ %------------------------------------------------------------------
+ \subsection{General setup
+ \label{section_ad_setup}}
+ In order to configure AD-related setups the following packages need
+ to be enabled:
+ {\it
+ \begin{table}[h!]
+ \begin{tabular}{l}
+ autodiff \\
+ ctrl \\
+ cost \\
+ grdchk \\
+ \end{tabular}
+ \end{table}
+ }
+ The packages are enabled by adding them to your experiment-specific
+ configuration file
+ {\it packages.conf} (see Section ???).
+ The following AD-specific CPP option files need to be customized:
+ %
+ \begin{itemize}
+ %
+ \item {\it ECCO\_CPPOPTIONS.h} \\
+ This header file collects CPP options for the packages
+ {\it autodiff, cost, ctrl} as well as AD-unrelated options for
+ the external forcing package {\it exf}.
+ \footnote{NOTE: These options are not set in their package-specific
+ headers such as {\it COST\_CPPOPTIONS.h}, but are instead collected
+ in the single header file {\it ECCO\_CPPOPTIONS.h}.
+ The package-specific header files serve as simple
+ placeholders at this point.}
+ %
+ \item {\it tamc.h} \\
+ This header configures the splitting of the time stepping loop
+ w.r.t. the 3-level checkpointing (see section ???).
+ %
+ \end{itemize}
+ %------------------------------------------------------------------
+ \subsection{Building the AD code
+ \label{section_ad_build}}
+ The build process of an AD code is very similar to building
+ the forward model. However, depending on which AD code one wishes
+ to generate, and on which AD tool is available (TAF or TAMC),
+ the following {\tt make} targets are available:
+ \begin{table}[h!]
+ {\footnotesize
+ \begin{tabular}{ccll}
+ ~ & {\it AD-target} & {\it output} & {\it description} \\
+ \hline
+ \hline
+ (1) & {\tt <MODE><TOOL>only} & {\tt <MODE>\_<TOOL>\_output.f}  &
+ generates code for $<$MODE$>$ using $<$TOOL$>$ \\
+ ~ & ~ & ~ & no {\tt make} dependencies on {\tt .F .h} \\
+ ~ & ~ & ~ & useful for compiling on remote platforms \\
+ \hline
+ (2) & {\tt <MODE><TOOL>} & {\tt <MODE>\_<TOOL>\_output.f}  &
+ generates code for $<$MODE$>$ using $<$TOOL$>$ \\
+ ~ & ~ & ~ & includes {\tt make} dependencies on {\tt .F .h} \\
+ ~ & ~ & ~ & i.e. input for $<$TOOL$>$ may be re-generated \\
+ \hline
+ (3) & {\tt <MODE>all} & {\tt mitgcmuv\_<MODE>}  &
+ generates code for $<$MODE$>$ using $<$TOOL$>$ \\
+ ~ & ~ & ~ & and compiles all code \\
+ ~ & ~ & ~ & (use of TAF is set as default) \\
+ \hline
+ \hline
+ \end{tabular}
+ }
+ \end{table}
+ %
+ Here, the following placeholders are used
+ %
+ \begin{itemize}
+ %
+ \item [$<$TOOL$>$]
+ %
+ \begin{itemize}
+ %
+ \item {\tt TAF}
+ \item {\tt TAMC}
+ %
+ \end{itemize}
+ %
+ \item [$<$MODE$>$]
+ %
+ \begin{itemize}
+ %
+ \item {\tt ad} generates the adjoint model (ADM)
+ \item {\tt ftl} generates the tangent linear model (TLM)
+ \item {\tt svd} generates both ADM and TLM for \\
+ singular value decomposition (SVD) type calculations
+ %
+ \end{itemize}
+ %
+ \end{itemize}
+ For example, to generate the adjoint model using TAF after routines ({\tt .F})
+ or headers ({\tt .h}) have been modified, but without compilation,
+ type {\tt make adtaf};
+ or, to generate the tangent linear model using TAMC without
+ re-generating the input code, type {\tt make ftltamconly}.
+ A typical full build process to generate the ADM via TAF would
+ look like follows:
+ \begin{verbatim}
+ % mkdir build
+ % cd build
+ % ../../../tools/genmake2 -mods=../code_ad
+ % make depend
+ % make adall
+ \end{verbatim}
+ %------------------------------------------------------------------
+ \subsection{The AD build process in detail
+ \label{section_ad_build_detail}}
+ The {\tt make <MODE>all} target consists of the following procedures:
+ \begin{enumerate}
+ %
+ \item
+ A header file {\tt AD\_CONFIG.h} is generated which contains a CPP option
+ on which code ought to be generated. Depending on the {\tt make} target,
+ the contents is
+ \begin{itemize}
+ \item
+ {\tt \#define ALLOW\_ADJOINT\_RUN}
+ \item
+ {\tt \#define ALLOW\_TANGENTLINEAR\_RUN}
+ \item
+ {\tt \#define ALLOW\_ECCO\_OPTIMIZATION}
+ \end{itemize}
+ %
+ \item
+ A single file {\tt <MODE>\_input\_code.f} is concatenated
+ consisting of all {\tt .f} files that are part of the list {\bf AD\_FILES}
+ and all {\tt .flow} files that are part of the list {\bf AD\_FLOW\_FILES}.
+ %
+ \item
+ The AD tool is invoked with the {\bf <MODE>\_<TOOL>\_FLAGS}.
+ The default AD tool flags in {\tt genmake2} can be overrwritten by
+ an {\tt adjoint\_options} file (similar to the platform-specific
+ {\tt build\_options}, see Section ???.
+ The AD tool writes the resulting AD code into the file
+ {\tt <MODE>\_input\_code\_ad.f}
+ %
+ \item
+ A short sed script {\tt adjoint\_sed} is applied to
+ {\tt <MODE>\_input\_code\_ad.f}
+ to reinstate {\bf myThid} into the CALL argument list of active file I/O.
+ The result is written to file {\tt <MODE>\_<TOOL>\_output.f}.
+ %
+ \item
+ All routines are compiled and an executable is generated
+ (see Table ???).
+ %
+ \end{enumerate}
+ \subsubsection{The list AD\_FILES and {\tt .list} files}
+ Not all routines are presented to the AD tool.
+ Routines typically hidden are diagnostics routines which
+ do not influence the cost function, but may create
+ artificial flow dependencies such as I/O of active variables.
+ {\tt genmake2} generates a list (or variable) {\bf AD\_FILES}
+ which contains all routines that are shown to the AD tool.
+ This list is put together from all files with suffix {\tt .list}
+ that {\tt genmake2} finds in its search directories.
+ The list file for the core MITgcm routines is in {\tt model/src/}
+ is called {\tt model\_ad\_diff.list}.
+ Note that no wrapper routine is shown to TAF. These are either
+ not visible at all to the AD code, or hand-written AD code
+ is available (see next section).
+ Each package directory contains its package-specific
+ list file {\tt <PKG>\_ad\_diff.list}. For example,
+ {\tt pkg/ptracers/} contains the file {\tt ptracers\_ad\_diff.list}.
+ Thus, enabling a package will automatically extend the
+ {\bf AD\_FILES} list of {\tt genmake2} to incorporate the
+ package-specific routines.
+ Note that you will need to regenerate the {\tt Makefile} if
+ you enable a package (e.g. by adding it to {\tt packages.conf})
+ and a {\tt Makefile} already exists.
+ \subsubsection{The list AD\_FLOW\_FILES and {\tt .flow} files}
+ TAMC and TAF can evaluate user-specified directives
+ that start with a specific syntax ({\tt CADJ}, {\tt C\$TAF}, {\tt !\$TAF}).
+ The main categories of directives are STORE directives and
+ FLOW directives. Here, we are concerned with flow directives,
+ store directives are treated elsewhere.
+ Flow directives enable the AD tool to evaluate how it should treat
+ routines that are 'hidden' by the user, i.e. routines which are
+ not contained in the {\bf AD\_FILES} list (see previous section),
+ but which are called in part of the code that the AD tool does see.
+ The flow directive tell the AD tool
+ %
+ \begin{itemize}
+ %
+ \item which subroutine arguments are input/output
+ \item which subroutine arguments are active
+ \item which subroutine arguments are required to compute the cost
+ \item which subroutine arguments are dependent
+ %
+ \end{itemize}
+ %
+ The syntax for the flow directives can be found in the
+ AD tool manuals.
+ {\tt genmake2} generates a list (or variable) {\bf AD\_FLOW\_FILES}
+ which contains all files with suffix{\tt .flow} that it finds
+ in its search directories.
+ The flow directives for the core MITgcm routines of
+ {\tt eesupp/src/} and {\tt model/src/}
+ reside in {\tt pkg/autodiff/}.
+ This directory also contains hand-written adjoint code
+ for the MITgcm WRAPPER (see Section ???).
+ Flow directives for package-specific routines are contained in
+ the corresponding package directories in the file
+ {\tt <PKG>\_ad.flow}, e.g. ptracers-specific directives are in
+ {\tt ptracers\_ad.flow}.
+ \subsubsection{Store directives for 3-level checkpointing}
+ The storing that is required at each period of the
+-level checkpointing is controled by three
+ top-level headers.
+ \begin{verbatim}
+ do ilev_3 = 1, nchklev_3
+ #  include ``checkpoint_lev3.h''
+    do ilev_2 = 1, nchklev_2
+ #     include ``checkpoint_lev2.h''
+       do ilev_1 = 1, nchklev_1
+ #        include ``checkpoint_lev1.h''
+ ...
+       end do
+    end do
+ end do
+ \end{verbatim}
+ All files {\tt checkpoint\_lev?.h} are contained in directory
+ {\tt pkg/autodiff/}.
+ \subsubsection{Changing the default AD tool flags: ad\_options files}
+ \subsubsection{Hand-written adjoint code}
+ %------------------------------------------------------------------
  \subsection{The cost function (dependent variable)
  \label{section_cost}}
  The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}.
  It is a function of the input variables $ \vec{u} $ via the composition
  $ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $.
- The input is referred to as the
+ The input are referred to as the
  {\sf independent variables} or {\sf control variables}.
  All aspects relevant to the treatment of the cost function $ {\cal J} $
  (parameter setting, initialization, accumulation,
  final evaluation), are controlled by the package {\it pkg/cost}.
+ The aspects relevant to the treatment of the independent variables
+ are controlled by the package {\it pkg/ctrl} and will be treated
+ in the next section.
  \input{part5/doc_cost_flow}
- \subsubsection{genmake and CPP options}
+ \subsubsection{Enabling the package}
- %
- \begin{itemize}
- %
- \item
  \fbox{
  \begin{minipage}{12cm}
- {\it genmake}, {\it CPP\_OPTIONS.h}, {\it ECCO\_CPPOPTIONS.h}
+ {\it packages.conf}, {\it ECCO\_CPPOPTIONS.h}
  \end{minipage}
  }
- \end{itemize}
+ \begin{itemize}
- %
- The directory {\it pkg/cost} can be included to the
- compile list in 3 different ways (cf. Section \ref{???}):
  %
- \begin{enumerate}
+ \item
+ The package is enabled by adding {\it cost} to your file {\it packages.conf}
+ (see Section ???)
  %
- \item {\it genmake}: \\
+ \item
- Change the default settings in the file {\it genmake} by adding
- {\bf cost} to the {\bf enable} list (not recommended).
- %
+ \end{itemize}
- \item {\it .genmakerc}: \\
- Customize the settings of {\bf enable}, {\bf disable} which are
- appropriate for your experiment in the file {\it .genmakerc}
- and add the file to your compile directory.
- %
- \item genmake-options: \\
- Call {\it genmake} with the option
- {\tt genmake -enable=cost}.
  %
- \end{enumerate}
+ N.B.: In general the following packages ought to be enabled
+ simultaneously: {\it autodiff, cost, ctrl}.
  The basic CPP option to enable the cost function is {\bf ALLOW\_COST}.
  Each specific cost function contribution has its own option.
  For the present example the option is {\bf ALLOW\_COST\_TRACER}.
  All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h}
  Since the cost function is usually used in conjunction with
  automatic differentiation, the CPP option
- {\bf ALLOW\_ADJOINT\_RUN} should be defined
+ {\bf ALLOW\_ADJOINT\_RUN} (file {\it CPP\_OPTIONS.h}) and
- (file {\it CPP\_OPTIONS.h}).
+ {\bf ALLOW\_AUTODIFF\_TAMC} (file {\it ECCO\_CPPOPTIONS.h})
+ should be defined.
  \subsubsection{Initialization}
  %
  The initialization of the {\it cost} package is readily enabled
- as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined.
+ as soon as the CPP option {\bf ALLOW\_COST} is defined.
  %
  \begin{itemize}
  %
-Line 831 
 from each contribution and sums over all
+Line 1106 
 from each contribution and sums over all
  \begin{equation}
  {\cal J} \, = \,
  {\rm fc} \, = \,
- {\rm mult\_tracer} \sum_{bi,\,bj}^{nSx,\,nSy}
+ {\rm mult\_tracer} \sum_{\text{global sum}} \sum_{bi,\,bj}^{nSx,\,nSy}
  {\rm objf\_tracer}(bi,bj) \, + \, ...
  \end{equation}
  %
-Line 879 
 are controlled by the package {\it pkg/c
+Line 1154 
 are controlled by the package {\it pkg/c
  %
  To enable the directory to be included to the compile list,
  {\bf ctrl} has to be added to the {\bf enable} list in
- {\it .genmakerc} (or {\it genmake} itself).
+ {\it .genmakerc} or in {\it genmake} itself (analogous to {\it cost}
+ package, cf. previous section).
  Each control variable is enabled via its own CPP option
  in {\it ECCO\_CPPOPTIONS.h}.
-Line 1023 
 in the code takes on the form
+Line 1299 
 in the code takes on the form
  %
  Note, that reading an active variable corresponds
  to a variable assignment. Its derivative corresponds
- to a write statement of the adjoint variable.
+ to a write statement of the adjoint variable, followed by
+ a reset.
  The 'active file' routines have been designed
  to support active read and corresponding adjoint active write
  operations (and vice versa).
-Line 1140 
 at intermediate times can be written usi
+Line 1417 
 at intermediate times can be written usi
  {\it addummy\_in\_stepping}.
  This routine is part of the adjoint support package
  {\it pkg/autodiff} (cf.f. below).
+ The procedure is enabled using via the CPP-option
+ {\bf ALLOW\_AUTODIFF\_MONITOR} (file {\it ECCO\_CPPOPTIONS.h}).
  To be part of the adjoint code, the corresponding S/R
  {\it dummy\_in\_stepping} has to be called in the forward
  model (S/R {\it the\_main\_loop}) at the appropriate place.
+ The adjoint common blocks are extracted from the adjoint code
+ via the header file {\it adcommon.h}.
  {\it dummy\_in\_stepping} is essentially empty,
  the corresponding adjoint routine is hand-written rather
-Line 1169 
 the common blocks
+Line 1450 
 the common blocks
  {\bf /adtr1\_r/}, {\bf /adffields/},
  which have been extracted from the adjoint code to enable
  access to the adjoint variables.
+ {\bf WARNING:} If the structure of the common blocks
+ {\bf /dynvars\_r/}, {\bf /dynvars\_cd/}, etc., changes
+ similar changes will occur in the adjoint common blocks.
+ Therefore, consistency between the TAMC-generated common blocks
+ and those in {\it adcommon.h} have to be checked.
  %
  \end{itemize}

 Legend:



Removed from v.1.14
 


changed lines


 
Added in v.1.16
 Legend:



Removed from v.1.14
 


changed lines


 
Added in v.1.16
-Removed from v.1.14
+Added in v.1.16

	ViewVC Help
Powered by ViewVC 1.1.22