/[MITgcm]/manual/s_autodiff/text/doc_ad_2.tex

Diff of /manual/s_autodiff/text/doc_ad_2.tex

Parent Directory | Revision Log | View Revision Graph Revision Graph | View Patch Patch

-revision 1.15 by heimbach,
Wed Apr 24 11:01:46 2002 UTC
+revision 1.21 by heimbach,
Thu Jan 17 22:32:06 2008 UTC
 Line 1
  % $Header$
  % $Name$
+ Author: Patrick Heimbach
  {\sf Automatic differentiation} (AD), also referred to as algorithmic
  (or, more loosely, computational) differentiation, involves
- automatically deriving code to calculate
+ automatically deriving code to calculate partial derivatives from an
- partial derivatives from an existing fully non-linear prognostic code.
+ existing fully non-linear prognostic code.  (see \cite{gri:00}).  A
- (see \cite{gri:00}).
+ software tool is used that parses and transforms source files
- A software tool is used that parses and transforms source files
+ according to a set of linguistic and mathematical rules.  AD tools are
- according to a set of linguistic and mathematical rules.
+ like source-to-source translators in that they parse a program code as
- AD tools are like source-to-source translators in that
+ input and produce a new program code as output
- they parse a program code as input and produce a new program code
+ (we restrict our discussion to source-to-source tools, ignoring
- as output.
+ operator-overloading tools).  However, unlike a
- However, unlike a pure source-to-source translation, the output program
+ pure source-to-source translation, the output program represents a new
- represents a new algorithm, such as the evaluation of the
+ algorithm, such as the evaluation of the Jacobian, the Hessian, or
- Jacobian, the Hessian, or higher derivative operators.
+ higher derivative operators.  In principle, a variety of derived
- In principle, a variety of derived algorithms
+ algorithms can be generated automatically in this way.
- can be generated automatically in this way.
+ MITgcm has been adapted for use with the Tangent linear and Adjoint
- The MITGCM has been adapted for use with the
+ Model Compiler (TAMC) and its successor TAF (Transformation of
- Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF
+ Algorithms in Fortran), developed by Ralf Giering (\cite{gie-kam:98},
- (Transformation of Algorithms in Fortran), developed
+ \cite{gie:99,gie:00}).  The first application of the adjoint of MITgcm
- by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}).
+ for sensitivity studies has been published by \cite{maro-eta:99}.
- The first application of the adjoint of the MITGCM for sensitivity
+ \cite{stam-etal:97,stam-etal:02} use MITgcm and its adjoint for ocean
- studies has been published by \cite{maro-eta:99}.
+ state estimation studies.  In the following we shall refer to TAMC and
- \cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint
+ TAF synonymously, except were explicitly stated otherwise.
- for ocean state estimation studies.
- In the following we shall refer to TAMC and TAF synonymously,
+ As of mid-2007 we are also able to generate fairly efficient
- except were explicitly stated otherwise.
+ adjoint code of the MITgcm using a new, open-source AD tool,
+ called OpenAD (see \cite{naum-etal:06,utke-etal:08}.
- TAMC exploits the chain rule for computing the first
+ This enables us for the first time to compare adjoint models
- derivative of a function with
+ generated from different AD tools, providing an additional
- respect to a set of input variables.
+ accuracy check, complementary to finite-difference gradient checks.
- Treating a given forward code as a composition of operations --
+ OpenAD and its application to  MITgcm is described in detail
- each line representing a compositional element, the chain rule is
+ in section \ref{sec_ad_openad}.
- rigorously applied to the code, line by line. The resulting
- tangent linear or adjoint code,
+ The AD tool exploits the chain rule for computing the first derivative of a
- then, may be thought of as the composition in
+ function with respect to a set of input variables.  Treating a given
- forward or reverse order, respectively, of the
+ forward code as a composition of operations -- each line representing
- Jacobian matrices of the forward code's compositional elements.
+ a compositional element, the chain rule is rigorously applied to the
+ code, line by line. The resulting tangent linear or adjoint code,
+ then, may be thought of as the composition in forward or reverse
+ order, respectively, of the Jacobian matrices of the forward code's
+ compositional elements.
  %**********************************************************************
  \section{Some basic algebra}
  \label{sec_ad_algebra}
+ \begin{rawhtml}
+ <!-- CMIREDIR:sec_ad_algebra: -->
+ \end{rawhtml}
  %**********************************************************************
  Let $ \cal{M} $ be a general nonlinear, model, i.e. a
-Line 676 
 Schematic view of intermediate dump and
+Line 685 
 Schematic view of intermediate dump and
  %**********************************************************************
  \section{TLM and ADM generation in general}
  \label{sec_ad_setup_gen}
+ \begin{rawhtml}
+ <!-- CMIREDIR:sec_ad_setup_gen: -->
+ \end{rawhtml}
  %**********************************************************************
  In this section we describe in a general fashion
  the parts of the code that are relevant for automatic
- differentiation using the software tool TAMC.
+ differentiation using the software tool TAF.
+ Modifications to use OpenAD are described in \ref{sec_ad_openad}.
  \input{part5/doc_ad_the_model}
  The basic flow is depicted in \ref{fig:adthemodel}.
- If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine
+ If CPP option \texttt{ALLOW\_AUTODIFF\_TAMC} is defined,
+ the driver routine
  {\it the\_model\_main}, instead of calling {\it the\_main\_loop},
- invokes the adjoint of this routine, {\it adthe\_main\_loop},
+ invokes the adjoint of this routine, {\it adthe\_main\_loop}
- which is the toplevel routine in terms of reverse mode computation.
+ (case \texttt{\#define ALLOW\_ADJOINT\_RUN}), or
- The routine {\it adthe\_main\_loop} has been generated by TAMC.
+ the tangent linear of this routine {\it g\_the\_main\_loop}
- It contains both the forward integration of the full model,
+ (case \texttt{\#define ALLOW\_TANGENTLINEAR\_RUN}),
+ which are the toplevel routines in terms of automatic differentiation.
+ The routines {\it adthe\_main\_loop} or {\it g\_the\_main\_loop}
+ are generated by TAF.
+ It contains both the forward integration of the full model, the
+ cost function calculation,
  any additional storing that is required for efficient checkpointing,
  and the reverse integration of the adjoint model.
- The structure of {\it adthe\_main\_loop} has been strongly
- simplified for clarification; in particular, no checkpointing
+ [DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM]
+ In Fig. \ref{fig:adthemodel}
+ the structure of {\it adthe\_main\_loop} has been strongly
+ simplified to focus on the essentials; in particular, no checkpointing
  procedures are shown here.
  Prior to the call of {\it adthe\_main\_loop}, the routine
  {\it ctrl\_unpack} is invoked to unpack the control vector
-Line 709 
 the driver routine {\it grdchk\_main} is
+Line 732 
 the driver routine {\it grdchk\_main} is
  the gradient has been computed via the adjoint
  (cf. Section \ref{section_grdchk}).
+ %------------------------------------------------------------------
+ \subsection{General setup
+ \label{section_ad_setup}}
+ In order to configure AD-related setups the following packages need
+ to be enabled:
+ {\it
+ \begin{table}[h!]
+ \begin{tabular}{l}
+ autodiff \\
+ ctrl \\
+ cost \\
+ grdchk \\
+ \end{tabular}
+ \end{table}
+ }
+ The packages are enabled by adding them to your experiment-specific
+ configuration file
+ {\it packages.conf} (see Section ???).
+ The following AD-specific CPP option files need to be customized:
+ %
+ \begin{itemize}
+ %
+ \item {\it ECCO\_CPPOPTIONS.h} \\
+ This header file collects CPP options for the packages
+ {\it autodiff, cost, ctrl} as well as AD-unrelated options for
+ the external forcing package {\it exf}.
+ \footnote{NOTE: These options are not set in their package-specific
+ headers such as {\it COST\_CPPOPTIONS.h}, but are instead collected
+ in the single header file {\it ECCO\_CPPOPTIONS.h}.
+ The package-specific header files serve as simple
+ placeholders at this point.}
+ %
+ \item {\it tamc.h} \\
+ This header configures the splitting of the time stepping loop
+ w.r.t. the 3-level checkpointing (see section ???).
+ %
+ \end{itemize}
+ %------------------------------------------------------------------
+ \subsection{Building the AD code using TAF
+ \label{section_ad_build}}
+ The build process of an AD code is very similar to building
+ the forward model. However, depending on which AD code one wishes
+ to generate, and on which AD tool is available (TAF or TAMC),
+ the following {\tt make} targets are available:
+ \begin{table}[h!]
+ {\footnotesize
+ \begin{tabular}{|ccll|}
+ \hline
+ ~ & {\it AD-target} & {\it output} & {\it description} \\
+ \hline
+ \hline
+ (1) & {\tt <MODE><TOOL>only} & {\tt <MODE>\_<TOOL>\_output.f}  &
+ generates code for $<$MODE$>$ using $<$TOOL$>$ \\
+ ~ & ~ & ~ & no {\tt make} dependencies on {\tt .F .h} \\
+ ~ & ~ & ~ & useful for compiling on remote platforms \\
+ \hline
+ (2) & {\tt <MODE><TOOL>} & {\tt <MODE>\_<TOOL>\_output.f}  &
+ generates code for $<$MODE$>$ using $<$TOOL$>$ \\
+ ~ & ~ & ~ & includes {\tt make} dependencies on {\tt .F .h} \\
+ ~ & ~ & ~ & i.e. input for $<$TOOL$>$ may be re-generated \\
+ \hline
+ (3) & {\tt <MODE>all} & {\tt mitgcmuv\_<MODE>}  &
+ generates code for $<$MODE$>$ using $<$TOOL$>$ \\
+ ~ & ~ & ~ & and compiles all code \\
+ ~ & ~ & ~ & (use of TAF is set as default) \\
+ \hline
+ \end{tabular}
+ }
+ \end{table}
+ %
+ Here, the following placeholders are used
+ %
+ \begin{itemize}
+ %
+ \item $<$TOOL$>$
+ %
+ \begin{itemize}
+ %
+ \item {\tt TAF}
+ \item {\tt TAMC}
+ %
+ \end{itemize}
+ %
+ \item $<$MODE$>$
+ %
+ \begin{itemize}
+ %
+ \item {\tt ad} generates the adjoint model (ADM)
+ \item {\tt ftl} generates the tangent linear model (TLM)
+ \item {\tt svd} generates both ADM and TLM for \\
+ singular value decomposition (SVD) type calculations
+ %
+ \end{itemize}
+ %
+ \end{itemize}
+ For example, to generate the adjoint model using TAF after routines ({\tt .F})
+ or headers ({\tt .h}) have been modified, but without compilation,
+ type {\tt make adtaf};
+ or, to generate the tangent linear model using TAMC without
+ re-generating the input code, type {\tt make ftltamconly}.
+ A typical full build process to generate the ADM via TAF would
+ look like follows:
+ \begin{verbatim}
+ % mkdir build
+ % cd build
+ % ../../../tools/genmake2 -mods=../code_ad
+ % make depend
+ % make adall
+ \end{verbatim}
+ %------------------------------------------------------------------
+ \subsection{The AD build process in detail
+ \label{section_ad_build_detail}}
+ The {\tt make <MODE>all} target consists of the following procedures:
+ \begin{enumerate}
+ %
+ \item
+ A header file {\tt AD\_CONFIG.h} is generated which contains a CPP option
+ on which code ought to be generated. Depending on the {\tt make} target,
+ the contents is one of the following:
+ \begin{itemize}
+ \item
+ {\tt \#define ALLOW\_ADJOINT\_RUN}
+ \item
+ {\tt \#define ALLOW\_TANGENTLINEAR\_RUN}
+ \item
+ {\tt \#define ALLOW\_ECCO\_OPTIMIZATION}
+ \end{itemize}
+ %
+ \item
+ A single file {\tt <MODE>\_input\_code.f} is concatenated
+ consisting of all {\tt .f} files that are part of the list {\bf AD\_FILES}
+ and all {\tt .flow} files that are part of the list {\bf AD\_FLOW\_FILES}.
+ %
+ \item
+ The AD tool is invoked with the {\tt <MODE>\_<TOOL>\_FLAGS}.
+ The default AD tool flags in {\tt genmake2} can be overrwritten by
+ an {\tt adjoint\_options} file (similar to the platform-specific
+ {\tt build\_options}, see Section ???.
+ The AD tool writes the resulting AD code into the file
+ {\tt <MODE>\_input\_code\_ad.f}
+ %
+ \item
+ A short sed script {\tt adjoint\_sed} is applied to
+ {\tt <MODE>\_input\_code\_ad.f}
+ to reinstate {\bf myThid} into the CALL argument list of active file I/O.
+ The result is written to file {\tt <MODE>\_<TOOL>\_output.f}.
+ %
+ \item
+ All routines are compiled and an executable is generated
+ (see Table ???).
+ %
+ \end{enumerate}
+ \subsubsection{The list AD\_FILES and {\tt .list} files}
+ Not all routines are presented to the AD tool.
+ Routines typically hidden are diagnostics routines which
+ do not influence the cost function, but may create
+ artificial flow dependencies such as I/O of active variables.
+ {\tt genmake2} generates a list (or variable) {\bf AD\_FILES}
+ which contains all routines that are shown to the AD tool.
+ This list is put together from all files with suffix {\tt .list}
+ that {\tt genmake2} finds in its search directories.
+ The list file for the core MITgcm routines is in {\tt model/src/}
+ is called {\tt model\_ad\_diff.list}.
+ Note that no wrapper routine is shown to TAF. These are either
+ not visible at all to the AD code, or hand-written AD code
+ is available (see next section).
+ Each package directory contains its package-specific
+ list file {\tt <PKG>\_ad\_diff.list}. For example,
+ {\tt pkg/ptracers/} contains the file {\tt ptracers\_ad\_diff.list}.
+ Thus, enabling a package will automatically extend the
+ {\bf AD\_FILES} list of {\tt genmake2} to incorporate the
+ package-specific routines.
+ Note that you will need to regenerate the {\tt Makefile} if
+ you enable a package (e.g. by adding it to {\tt packages.conf})
+ and a {\tt Makefile} already exists.
+ \subsubsection{The list AD\_FLOW\_FILES and {\tt .flow} files}
+ TAMC and TAF can evaluate user-specified directives
+ that start with a specific syntax ({\tt CADJ}, {\tt C\$TAF}, {\tt !\$TAF}).
+ The main categories of directives are STORE directives and
+ FLOW directives. Here, we are concerned with flow directives,
+ store directives are treated elsewhere.
+ Flow directives enable the AD tool to evaluate how it should treat
+ routines that are 'hidden' by the user, i.e. routines which are
+ not contained in the {\bf AD\_FILES} list (see previous section),
+ but which are called in part of the code that the AD tool does see.
+ The flow directive tell the AD tool
+ %
+ \begin{itemize}
+ %
+ \item which subroutine arguments are input/output
+ \item which subroutine arguments are active
+ \item which subroutine arguments are required to compute the cost
+ \item which subroutine arguments are dependent
+ %
+ \end{itemize}
+ %
+ The syntax for the flow directives can be found in the
+ AD tool manuals.
+ {\tt genmake2} generates a list (or variable) {\bf AD\_FLOW\_FILES}
+ which contains all files with suffix{\tt .flow} that it finds
+ in its search directories.
+ The flow directives for the core MITgcm routines of
+ {\tt eesupp/src/} and {\tt model/src/}
+ reside in {\tt pkg/autodiff/}.
+ This directory also contains hand-written adjoint code
+ for the MITgcm WRAPPER (section \ref{chap:sarch}).
+ Flow directives for package-specific routines are contained in
+ the corresponding package directories in the file
+ {\tt <PKG>\_ad.flow}, e.g. ptracers-specific directives are in
+ {\tt ptracers\_ad.flow}.
+ \subsubsection{Store directives for 3-level checkpointing}
+ The storing that is required at each period of the
+-level checkpointing is controled by three
+ top-level headers.
+ \begin{verbatim}
+ do ilev_3 = 1, nchklev_3
+ #  include ``checkpoint_lev3.h''
+    do ilev_2 = 1, nchklev_2
+ #     include ``checkpoint_lev2.h''
+       do ilev_1 = 1, nchklev_1
+ #        include ``checkpoint_lev1.h''
+ ...
+       end do
+    end do
+ end do
+ \end{verbatim}
+ All files {\tt checkpoint\_lev?.h} are contained in directory
+ {\tt pkg/autodiff/}.
+ \subsubsection{Changing the default AD tool flags: ad\_options files}
+ \subsubsection{Hand-written adjoint code}
+ %------------------------------------------------------------------
  \subsection{The cost function (dependent variable)
  \label{section_cost}}
-Line 726 
 in the next section.
+Line 1016 
 in the next section.
  \input{part5/doc_cost_flow}
- \subsubsection{genmake and CPP options}
+ \subsubsection{Enabling the package}
- %
- \begin{itemize}
- %
- \item
  \fbox{
  \begin{minipage}{12cm}
- {\it genmake}, {\it CPP\_OPTIONS.h}, {\it ECCO\_CPPOPTIONS.h}
+ {\it packages.conf}, {\it ECCO\_CPPOPTIONS.h}
  \end{minipage}
  }
- \end{itemize}
+ \begin{itemize}
- %
- The directory {\it pkg/cost} can be included to the
- compile list in 3 different ways (cf. Section \ref{???}):
  %
- \begin{enumerate}
+ \item
+ The package is enabled by adding {\it cost} to your file {\it packages.conf}
+ (see Section ???)
  %
- \item {\it genmake}: \\
+ \item
- Change the default settings in the file {\it genmake} by adding
- {\bf cost} to the {\bf enable} list (not recommended).
- %
+ \end{itemize}
- \item {\it .genmakerc}: \\
- Customize the settings of {\bf enable}, {\bf disable} which are
- appropriate for your experiment in the file {\it .genmakerc}
- and add the file to your compile directory.
- %
- \item genmake-options: \\
- Call {\it genmake} with the option
- {\tt genmake -enable=cost}.
  %
- \end{enumerate}
  N.B.: In general the following packages ought to be enabled
  simultaneously: {\it autodiff, cost, ctrl}.
  The basic CPP option to enable the cost function is {\bf ALLOW\_COST}.
-Line 851 
 from each contribution and sums over all
+Line 1129 
 from each contribution and sums over all
  \end{equation}
  %
  The total cost function {\bf fc} will be the
- 'dependent' variable in the argument list for TAMC, i.e.
+ 'dependent' variable in the argument list for TAF, i.e.
  \begin{verbatim}
- tamc -output 'fc' ...
+ taf -output 'fc' ...
  \end{verbatim}
  %%%% \end{document}
-Line 936 
 and their gradients: {\it ctrl\_unpack}
+Line 1214 
 and their gradients: {\it ctrl\_unpack}
  \\
  %
  Two important issues related to the handling of the control
- variables in the MITGCM need to be addressed.
+ variables in MITgcm need to be addressed.
  First, in order to save memory, the control variable arrays
  are not kept in memory, but rather read from file and added
  to the initial fields during the model initialization phase.
-Line 968 
 and gradient are generated and initialis
+Line 1246 
 and gradient are generated and initialis
  %
  The dependency flow for differentiation w.r.t. the controls
  starts with adding a perturbation onto the input variable,
- thus defining the independent or control variables for TAMC.
+ thus defining the independent or control variables for TAF.
  Three types of controls may be considered:
  %
  \begin{itemize}
-Line 1001 
 u         & = \, u_{[0]} \, + \, \Delta
+Line 1279 
 u         & = \, u_{[0]} \, + \, \Delta
  holding the perturbation. In the case of a simple
  sensitivity study this array is identical to zero.
  However, it's specification is essential in the context
- of automatic differentiation since TAMC
+ of automatic differentiation since TAF
  treats the corresponding line in the code symbolically
  when determining the differentiation chain and its origin.
  Thus, the variable names are part of the argument list
- when calling TAMC:
+ when calling TAF:
  %
  \begin{verbatim}
- tamc -input 'xx_tr1 ...' ...
+ taf -input 'xx_tr1 ...' ...
  \end{verbatim}
  %
- Now, as mentioned above, the MITGCM avoids maintaining
+ Now, as mentioned above, MITgcm avoids maintaining
  an array for each control variable by reading the
  perturbation to a temporary array from file.
- To ensure the symbolic link to be recognized by TAMC, a scalar
+ To ensure the symbolic link to be recognized by TAF, a scalar
  dummy variable {\bf xx\_tr1\_dummy} is introduced
  and an 'active read' routine of the adjoint support
  package {\it pkg/autodiff} is invoked.
  The read-procedure is tagged with the variable
- {\bf xx\_tr1\_dummy} enabling TAMC to recognize the
+ {\bf xx\_tr1\_dummy} enabling TAF to recognize the
  initialization of the perturbation.
- The modified call of TAMC thus reads
+ The modified call of TAF thus reads
  %
  \begin{verbatim}
- tamc -input 'xx_tr1_dummy ...' ...
+ taf -input 'xx_tr1_dummy ...' ...
  \end{verbatim}
  %
  and the modified operation to (\ref{perturb})

 Legend:



Removed from v.1.15
 


changed lines


 
Added in v.1.21
 Legend:



Removed from v.1.15
 


changed lines


 
Added in v.1.21
-Removed from v.1.15
+Added in v.1.21

	ViewVC Help
Powered by ViewVC 1.1.22