| 1 |
% $Header$ |
% $Header$ |
| 2 |
% $Name$ |
% $Name$ |
| 3 |
|
|
| 4 |
|
Author: Patrick Heimbach |
| 5 |
|
|
| 6 |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
| 7 |
(or, more loosely, computational) differentiation, involves |
(or, more loosely, computational) differentiation, involves |
| 8 |
automatically deriving code to calculate |
automatically deriving code to calculate partial derivatives from an |
| 9 |
partial derivatives from an existing fully non-linear prognostic code. |
existing fully non-linear prognostic code. (see \cite{gri:00}). A |
| 10 |
(see \cite{gri:00}). |
software tool is used that parses and transforms source files |
| 11 |
A software tool is used that parses and transforms source files |
according to a set of linguistic and mathematical rules. AD tools are |
| 12 |
according to a set of linguistic and mathematical rules. |
like source-to-source translators in that they parse a program code as |
| 13 |
AD tools are like source-to-source translators in that |
input and produce a new program code as output. However, unlike a |
| 14 |
they parse a program code as input and produce a new program code |
pure source-to-source translation, the output program represents a new |
| 15 |
as output. |
algorithm, such as the evaluation of the Jacobian, the Hessian, or |
| 16 |
However, unlike a pure source-to-source translation, the output program |
higher derivative operators. In principle, a variety of derived |
| 17 |
represents a new algorithm, such as the evaluation of the |
algorithms can be generated automatically in this way. |
| 18 |
Jacobian, the Hessian, or higher derivative operators. |
|
| 19 |
In principle, a variety of derived algorithms |
MITgcm has been adapted for use with the Tangent linear and Adjoint |
| 20 |
can be generated automatically in this way. |
Model Compiler (TAMC) and its successor TAF (Transformation of |
| 21 |
|
Algorithms in Fortran), developed by Ralf Giering (\cite{gie-kam:98}, |
| 22 |
The MITGCM has been adapted for use with the |
\cite{gie:99,gie:00}). The first application of the adjoint of MITgcm |
| 23 |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
for sensitivity studies has been published by \cite{maro-eta:99}. |
| 24 |
(Transformation of Algorithms in Fortran), developed |
\cite{sta-eta:97,sta-eta:01} use MITgcm and its adjoint for ocean |
| 25 |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
state estimation studies. In the following we shall refer to TAMC and |
| 26 |
The first application of the adjoint of the MITGCM for sensitivity |
TAF synonymously, except were explicitly stated otherwise. |
| 27 |
studies has been published by \cite{maro-eta:99}. |
|
| 28 |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
TAMC exploits the chain rule for computing the first derivative of a |
| 29 |
for ocean state estimation studies. |
function with respect to a set of input variables. Treating a given |
| 30 |
In the following we shall refer to TAMC and TAF synonymously, |
forward code as a composition of operations -- each line representing |
| 31 |
except were explicitly stated otherwise. |
a compositional element, the chain rule is rigorously applied to the |
| 32 |
|
code, line by line. The resulting tangent linear or adjoint code, |
| 33 |
TAMC exploits the chain rule for computing the first |
then, may be thought of as the composition in forward or reverse |
| 34 |
derivative of a function with |
order, respectively, of the Jacobian matrices of the forward code's |
| 35 |
respect to a set of input variables. |
compositional elements. |
|
Treating a given forward code as a composition of operations -- |
|
|
each line representing a compositional element, the chain rule is |
|
|
rigorously applied to the code, line by line. The resulting |
|
|
tangent linear or adjoint code, |
|
|
then, may be thought of as the composition in |
|
|
forward or reverse order, respectively, of the |
|
|
Jacobian matrices of the forward code's compositional elements. |
|
| 36 |
|
|
| 37 |
%********************************************************************** |
%********************************************************************** |
| 38 |
\section{Some basic algebra} |
\section{Some basic algebra} |
| 39 |
\label{sec_ad_algebra} |
\label{sec_ad_algebra} |
| 40 |
|
\begin{rawhtml} |
| 41 |
|
<!-- CMIREDIR:sec_ad_algebra: --> |
| 42 |
|
\end{rawhtml} |
| 43 |
%********************************************************************** |
%********************************************************************** |
| 44 |
|
|
| 45 |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
| 555 |
(a derivative is defined w.r.t. a point along the trajectory), |
(a derivative is defined w.r.t. a point along the trajectory), |
| 556 |
the intermediate results of the model trajectory |
the intermediate results of the model trajectory |
| 557 |
$\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$ |
$\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$ |
| 558 |
are needed to evaluate the intermediate Jacobian |
may be required to evaluate the intermediate Jacobian |
| 559 |
$M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $. |
$M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $. |
| 560 |
|
This is the case e.g. for nonlinear expressions |
| 561 |
|
(momentum advection, nonlinear equation of state), state-dependent |
| 562 |
|
conditional statements (parameterization schemes). |
| 563 |
In the forward mode, the intermediate results are required |
In the forward mode, the intermediate results are required |
| 564 |
in the same order as computed by the full forward model ${\cal M}$, |
in the same order as computed by the full forward model ${\cal M}$, |
| 565 |
but in the reverse mode they are required in the reverse order. |
but in the reverse mode they are required in the reverse order. |
| 570 |
|
|
| 571 |
A method to balance the amount of recomputations vs. |
A method to balance the amount of recomputations vs. |
| 572 |
storage requirements is called {\sf checkpointing} |
storage requirements is called {\sf checkpointing} |
| 573 |
(e.g. \cite{res-eta:98}). |
(e.g. \cite{gri:92}, \cite{res-eta:98}). |
| 574 |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
| 575 |
[as an example, we give explicit numbers for a 3-day |
[as an example, we give explicit numbers for a 3-day |
| 576 |
integration with a 1-hourly timestep in square brackets]. |
integration with a 1-hourly timestep in square brackets]. |
| 581 |
$ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals], |
$ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals], |
| 582 |
with the label $lev3$ for this outermost loop. |
with the label $lev3$ for this outermost loop. |
| 583 |
The model is then integrated along the full trajectory, |
The model is then integrated along the full trajectory, |
| 584 |
and the model state stored only at every $ k_{i}^{lev3} $-th timestep |
and the model state stored to disk only at every $ k_{i}^{lev3} $-th timestep |
| 585 |
[i.e. 3 times, at |
[i.e. 3 times, at |
| 586 |
$ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $]. |
$ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $]. |
| 587 |
|
In addition, the cost function is computed, if needed. |
| 588 |
% |
% |
| 589 |
\item [$lev2$] |
\item [$lev2$] |
| 590 |
In a second step each subsection itself is divided into |
In a second step each subsection itself is divided into |
| 591 |
$ {n}^{lev2} $ sub-subsections |
$ {n}^{lev2} $ subsections |
| 592 |
[$ {n}^{lev2} $=4 6-hour intervals per subsection]. |
[$ {n}^{lev2} $=4 6-hour intervals per subsection]. |
| 593 |
The model picks up at the last outermost dumped state |
The model picks up at the last outermost dumped state |
| 594 |
$ v_{k_{n}^{lev3}} $ and is integrated forward in time along |
$ v_{k_{n}^{lev3}} $ and is integrated forward in time along |
| 595 |
the last subsection, with the label $lev2$ for this |
the last subsection, with the label $lev2$ for this |
| 596 |
intermediate loop. |
intermediate loop. |
| 597 |
The model state is now stored at every $ k_{i}^{lev2} $-th |
The model state is now stored to disk at every $ k_{i}^{lev2} $-th |
| 598 |
timestep |
timestep |
| 599 |
[i.e. 4 times, at |
[i.e. 4 times, at |
| 600 |
$ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $]. |
$ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $]. |
| 602 |
\item [$lev1$] |
\item [$lev1$] |
| 603 |
Finally, the model picks up at the last intermediate dump state |
Finally, the model picks up at the last intermediate dump state |
| 604 |
$ v_{k_{n}^{lev2}} $ and is integrated forward in time along |
$ v_{k_{n}^{lev2}} $ and is integrated forward in time along |
| 605 |
the last sub-subsection, with the label $lev1$ for this |
the last subsection, with the label $lev1$ for this |
| 606 |
intermediate loop. |
intermediate loop. |
| 607 |
Within this sub-subsection only, the model state is stored |
Within this sub-subsection only, parts of the model state is stored |
| 608 |
at every timestep |
to memory at every timestep |
| 609 |
[i.e. every hour $ i=0,...,5$ corresponding to |
[i.e. every hour $ i=0,...,5$ corresponding to |
| 610 |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
| 611 |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
The final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
| 612 |
and the model state of all proceeding timesteps along the last |
and the model state of all preceding timesteps along the last |
| 613 |
sub-subsections are available, enabling integration backwards |
innermost subsection are available, enabling integration backwards |
| 614 |
in time along the last sub-subsection. |
in time along the last subsection. |
| 615 |
Thus, the adjoint can be computed along this last |
The adjoint can thus be computed along this last |
| 616 |
sub-subsection $k_{n}^{lev2}$. |
subsection $k_{n}^{lev2}$. |
| 617 |
% |
% |
| 618 |
\end{itemize} |
\end{itemize} |
| 619 |
% |
% |
| 620 |
This procedure is repeated consecutively for each previous |
This procedure is repeated consecutively for each previous |
| 621 |
sub-subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $ |
subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $ |
| 622 |
carrying the adjoint computation to the initial time |
carrying the adjoint computation to the initial time |
| 623 |
of the subsection $k_{n}^{lev3}$. |
of the subsection $k_{n}^{lev3}$. |
| 624 |
Then, the procedure is repeated for the previous subsection |
Then, the procedure is repeated for the previous subsection |
| 629 |
For the full model trajectory of |
For the full model trajectory of |
| 630 |
$ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps |
$ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps |
| 631 |
the required storing of the model state was significantly reduced to |
the required storing of the model state was significantly reduced to |
| 632 |
$ n^{lev1} + n^{lev2} + n^{lev3} $ |
$ n^{lev2} + n^{lev3} $ to disk and roughly $ n^{lev1} $ to memory |
| 633 |
[i.e. for the 3-day integration with a total oof 72 timesteps |
[i.e. for the 3-day integration with a total oof 72 timesteps |
| 634 |
the model state was stored 13 times]. |
the model state was stored 7 times to disk and roughly 6 times |
| 635 |
|
to memory]. |
| 636 |
This saving in memory comes at a cost of a required |
This saving in memory comes at a cost of a required |
| 637 |
3 full forward integrations of the model (one for each |
3 full forward integrations of the model (one for each |
| 638 |
checkpointing level). |
checkpointing level). |
| 639 |
The balance of storage vs. recomputation certainly depends |
The optimal balance of storage vs. recomputation certainly depends |
| 640 |
on the computing resources available. |
on the computing resources available and may be adjusted by |
| 641 |
|
adjusting the partitioning among the |
| 642 |
|
$ n^{lev3}, \,\, n^{lev2}, \,\, n^{lev1} $. |
| 643 |
|
|
| 644 |
\begin{figure}[t!] |
\begin{figure}[t!] |
| 645 |
\begin{center} |
\begin{center} |
| 669 |
% \subsection{Error covariance estimate and Hessian matrix} |
% \subsection{Error covariance estimate and Hessian matrix} |
| 670 |
% \label{sec_hessian} |
% \label{sec_hessian} |
| 671 |
|
|
| 672 |
\newpage |
\newpage |
| 673 |
|
|
| 674 |
%********************************************************************** |
%********************************************************************** |
| 675 |
\section{AD-specific setup by example: sensitivity of carbon sequestration} |
\section{TLM and ADM generation in general} |
| 676 |
\label{sec_ad_setup_ex} |
\label{sec_ad_setup_gen} |
| 677 |
|
\begin{rawhtml} |
| 678 |
|
<!-- CMIREDIR:sec_ad_setup_gen: --> |
| 679 |
|
\end{rawhtml} |
| 680 |
%********************************************************************** |
%********************************************************************** |
| 681 |
|
|
| 682 |
The MITGCM has been adapted to enable AD using TAMC or TAF. |
In this section we describe in a general fashion |
| 683 |
The present description, therefore, is specific to the |
the parts of the code that are relevant for automatic |
| 684 |
use of TAMC or TAF as AD tool. |
differentiation using the software tool TAF. |
| 685 |
The following sections describe the steps which are necessary to |
|
| 686 |
generate a tangent linear or adjoint model of the MITGCM. |
\input{part5/doc_ad_the_model} |
| 687 |
We take as an example the sensitivity of carbon sequestration |
|
| 688 |
in the ocean. |
The basic flow is depicted in \ref{fig:adthemodel}. |
| 689 |
The AD-relevant hooks in the code are sketched in |
If CPP option \texttt{ALLOW\_AUTODIFF\_TAMC} is defined, |
| 690 |
\ref{fig:adthemodel}, \ref{fig:adthemain}. |
the driver routine |
| 691 |
|
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
| 692 |
\subsection{Overview of the experiment} |
invokes the adjoint of this routine, {\it adthe\_main\_loop} |
| 693 |
|
(case \texttt{\#define ALLOW\_ADJOINT\_RUN}), or |
| 694 |
We describe an adjoint sensitivity analysis of out-gassing from |
the tangent linear of this routine {\it g\_the\_main\_loop} |
| 695 |
the ocean into the atmosphere of a carbon-like tracer injected |
(case \texttt{\#define ALLOW\_TANGENTLINEAR\_RUN}), |
| 696 |
into the ocean interior (see \cite{hil-eta:01}). |
which are the toplevel routines in terms of automatic differentiation. |
| 697 |
|
The routines {\it adthe\_main\_loop} or {\it g\_the\_main\_loop} |
| 698 |
\subsubsection{Passive tracer equation} |
are generated by TAF. |
| 699 |
|
It contains both the forward integration of the full model, the |
| 700 |
For this work the MITGCM was augmented with a thermodynamically |
cost function calculation, |
| 701 |
inactive tracer, $C$. Tracer residing in the ocean |
any additional storing that is required for efficient checkpointing, |
| 702 |
model surface layer is out-gassed according to a relaxation time scale, |
and the reverse integration of the adjoint model. |
| 703 |
$\mu$. Within the ocean interior, the tracer is passively advected |
|
| 704 |
by the ocean model currents. The full equation for the time evolution |
[DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM] |
| 705 |
% |
|
| 706 |
\begin{equation} |
In Fig. \ref{fig:adthemodel} |
| 707 |
\label{carbon_ddt} |
the structure of {\it adthe\_main\_loop} has been strongly |
| 708 |
\frac{\partial C}{\partial t} \, = \, |
simplified to focus on the essentials; in particular, no checkpointing |
| 709 |
-U\cdot \nabla C \, - \, \mu C \, + \, \Gamma(C) \,+ \, S |
procedures are shown here. |
| 710 |
\end{equation} |
Prior to the call of {\it adthe\_main\_loop}, the routine |
| 711 |
% |
{\it ctrl\_unpack} is invoked to unpack the control vector |
| 712 |
also includes a source term $S$. This term |
or initialise the control variables. |
| 713 |
represents interior sources of $C$ such as would arise due to |
Following the call of {\it adthe\_main\_loop}, |
| 714 |
direct injection. |
the routine {\it ctrl\_pack} |
| 715 |
The velocity term, $U$, is the sum of the |
is invoked to pack the control vector |
| 716 |
model Eulerian circulation and an eddy-induced velocity, the latter |
(cf. Section \ref{section_ctrl}). |
| 717 |
parameterized according to Gent/McWilliams |
If gradient checks are to be performed, the option |
| 718 |
(\cite{gen-mcw:90, gen-eta:95}). |
{\tt ALLOW\_GRADIENT\_CHECK} is defined. In this case |
| 719 |
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
the driver routine {\it grdchk\_main} is called after |
| 720 |
fluid is locally statically unstable. |
the gradient has been computed via the adjoint |
| 721 |
|
(cf. Section \ref{section_grdchk}). |
|
The out-gassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
|
|
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
|
|
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
|
|
is valid as a prognostic equation for small perturbations in oceanic |
|
|
carbon concentrations. This configuration provides a |
|
|
powerful tool for examining the impact of large-scale ocean circulation |
|
|
on $ CO_2 $ out-gassing due to interior injections. |
|
|
As source we choose a constant in time injection of |
|
|
$ S = 1 \,\, {\rm mol / s}$. |
|
|
|
|
|
\subsubsection{Model configuration} |
|
|
|
|
|
The model configuration employed has a constant |
|
|
$4^\circ \times 4^\circ$ resolution horizontal grid and realistic |
|
|
geography and bathymetry. Twenty vertical layers are used with |
|
|
vertical spacing ranging |
|
|
from 50 m near the surface to 815 m at depth. |
|
|
Driven to steady-state by climatological wind-stress, heat and |
|
|
fresh-water forcing the model reproduces well known large-scale |
|
|
features of the ocean general circulation. |
|
|
|
|
|
\subsubsection{Out-gassing cost function} |
|
|
|
|
|
To quantify and understand out-gassing due to injections of $C$ |
|
|
in eqn. (\ref{carbon_ddt}), |
|
|
we define a cost function $ {\cal J} $ that measures the total amount of |
|
|
tracer out-gassed at each timestep: |
|
|
% |
|
|
\begin{equation} |
|
|
\label{cost_tracer} |
|
|
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
|
|
\end{equation} |
|
|
% |
|
|
Equation(\ref{cost_tracer}) integrates the out-gassing term, $\mu C$, |
|
|
from (\ref{carbon_ddt}) |
|
|
over the entire ocean surface area, $A$, and accumulates it |
|
|
up to time $T$. |
|
|
Physically, ${\cal J}$ can be thought of as representing the amount of |
|
|
$CO_2$ that our model predicts would be out-gassed following an |
|
|
injection at rate $S$. |
|
|
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
|
|
$\frac{\partial {\cal J}}{\partial S}$, |
|
|
can be used to identify regions from which circulation |
|
|
would cause $CO_2$ to rapidly out-gas following injection |
|
|
and regions in which $CO_2$ injections would remain effectively |
|
|
sequestered within the ocean. |
|
|
|
|
|
\subsection{Code configuration} |
|
|
|
|
|
The model configuration for this experiment resides under the |
|
|
directory {\it verification/carbon/}. |
|
|
The code customization routines are in {\it verification/carbon/code/}: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\it .genmakerc} |
|
|
% |
|
|
\item {\it COST\_CPPOPTIONS.h} |
|
|
% |
|
|
\item {\it CPP\_EEOPTIONS.h} |
|
|
% |
|
|
\item {\it CPP\_OPTIONS.h} |
|
|
% |
|
|
\item {\it CTRL\_OPTIONS.h} |
|
|
% |
|
|
\item {\it ECCO\_OPTIONS.h} |
|
|
% |
|
|
\item {\it SIZE.h} |
|
|
% |
|
|
\item {\it adcommon.h} |
|
|
% |
|
|
\item {\it tamc.h} |
|
|
% |
|
|
\end{itemize} |
|
|
% |
|
|
The runtime flag and parameters settings are contained in |
|
|
{\it verification/carbon/input/}, |
|
|
together with the forcing fields and and restart files: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\it data} |
|
|
% |
|
|
\item {\it data.cost} |
|
|
% |
|
|
\item {\it data.ctrl} |
|
|
% |
|
|
\item {\it data.gmredi} |
|
|
% |
|
|
\item {\it data.grdchk} |
|
|
% |
|
|
\item {\it data.optim} |
|
|
% |
|
|
\item {\it data.pkg} |
|
|
% |
|
|
\item {\it eedata} |
|
|
% |
|
|
\item {\it topog.bin} |
|
|
% |
|
|
\item {\it windx.bin, windy.bin} |
|
|
% |
|
|
\item {\it salt.bin, theta.bin} |
|
|
% |
|
|
\item {\it SSS.bin, SST.bin} |
|
|
% |
|
|
\item {\it pickup*} |
|
|
% |
|
|
\end{itemize} |
|
|
% |
|
|
Finally, the file to generate the adjoint code resides in |
|
|
$ adjoint/ $: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\it makefile} |
|
|
% |
|
|
\end{itemize} |
|
|
% |
|
| 722 |
|
|
| 723 |
Below we describe the customizations of this files which are |
%------------------------------------------------------------------ |
|
specific to this experiment. |
|
| 724 |
|
|
| 725 |
\subsubsection{File {\it .genmakerc}} |
\subsection{General setup |
| 726 |
This file overwrites default settings of {\it genmake}. |
\label{section_ad_setup}} |
|
In the present example it is used to switch on the following |
|
|
packages which are related to automatic differentiation |
|
|
and are disabled by default: \\ |
|
|
\hspace*{4ex} {\tt set ENABLE=( autodiff cost ctrl ecco gmredi grdchk kpp )} \\ |
|
|
Other packages which are not needed are switched off: \\ |
|
|
\hspace*{4ex} {\tt set DISABLE=( aim obcs zonal\_filt shap\_filt cal exf )} |
|
|
|
|
|
\subsubsection{File {\it COST\_CPPOPTIONS.h, CTRL\_OPTIONS.h}} |
|
|
|
|
|
These files used to contain package-specific CPP-options |
|
|
(see Section \ref{???}). |
|
|
For technical reasons those options have been grouped together |
|
|
in the file {\it ECCO\_OPTIONS.h}. |
|
|
To retain the modularity, the files have been kept and contain |
|
|
the standard include of the {\it CPP\_OPTIONS.h} file. |
|
|
|
|
|
\subsubsection{File {\it CPP\_EEOPTIONS.h}} |
|
|
|
|
|
This file contains 'wrapper'-specific CPP options. |
|
|
It only needs to be changed if the code is to be run |
|
|
in a parallel environment (see Section \ref{???}). |
|
|
|
|
|
\subsubsection{File {\it CPP\_OPTIONS.h}} |
|
|
|
|
|
This file contains model-specific CPP options |
|
|
(see Section \ref{???}). |
|
|
Most options are related to the forward model setup. |
|
|
They are identical to the global steady circulation setup of |
|
|
{\it verification/exp2/}. |
|
|
The three options specific to this experiment are \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_PASSIVE\_TRACER} \\ |
|
|
This flag enables the code to carry through the |
|
|
advection/diffusion of a passive tracer along the |
|
|
model integration. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
|
|
This flag enables the inclusion of some AD-related fields |
|
|
concerning initialization, link between control variables |
|
|
and forward model variables, and the call to the top-level |
|
|
forward/adjoint subroutine {\it adthe\_main\_loop} |
|
|
instead of {\it the\_main\_loop}. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_GRADIENT\_CHECK} \\ |
|
|
This flag enables the gradient check package. |
|
|
After computing the unperturbed cost function and its gradient, |
|
|
a series of computations are performed for which \\ |
|
|
$\bullet$ an element of the control vector is perturbed \\ |
|
|
$\bullet$ the cost function w.r.t. the perturbed element is |
|
|
computed \\ |
|
|
$\bullet$ the difference between the perturbed and unperturbed |
|
|
cost function is computed to compute the finite difference gradient \\ |
|
|
$\bullet$ the finite difference gradient is compared with the |
|
|
adjoint-generated gradient. |
|
|
The gradient check package is further described in Section ???. |
|
| 727 |
|
|
| 728 |
\subsubsection{File {\it ECCO\_OPTIONS.h}} |
In order to configure AD-related setups the following packages need |
| 729 |
|
to be enabled: |
| 730 |
|
{\it |
| 731 |
|
\begin{table}[h!] |
| 732 |
|
\begin{tabular}{l} |
| 733 |
|
autodiff \\ |
| 734 |
|
ctrl \\ |
| 735 |
|
cost \\ |
| 736 |
|
grdchk \\ |
| 737 |
|
\end{tabular} |
| 738 |
|
\end{table} |
| 739 |
|
} |
| 740 |
|
The packages are enabled by adding them to your experiment-specific |
| 741 |
|
configuration file |
| 742 |
|
{\it packages.conf} (see Section ???). |
| 743 |
|
|
| 744 |
The CPP options of several AD-related packages are grouped |
The following AD-specific CPP option files need to be customized: |
|
in this file: |
|
| 745 |
% |
% |
| 746 |
\begin{itemize} |
\begin{itemize} |
| 747 |
% |
% |
| 748 |
\item |
\item {\it ECCO\_CPPOPTIONS.h} \\ |
| 749 |
Adjoint support package: {\it pkg/autodiff/} \\ |
This header file collects CPP options for the packages |
| 750 |
This package contains hand-written adjoint code such as |
{\it autodiff, cost, ctrl} as well as AD-unrelated options for |
| 751 |
active file handling, flow directives for files which must not |
the external forcing package {\it exf}. |
| 752 |
be differentiated, and TAMC-specific header files. \\ |
\footnote{NOTE: These options are not set in their package-specific |
| 753 |
\hspace*{4ex} {\tt \#define ALLOW\_AUTODIFF\_TAMC} \\ |
headers such as {\it COST\_CPPOPTIONS.h}, but are instead collected |
| 754 |
defines TAMC-related features in the code. \\ |
in the single header file {\it ECCO\_CPPOPTIONS.h}. |
| 755 |
\hspace*{4ex} {\tt \#define ALLOW\_TAMC\_CHECKPOINTING} \\ |
The package-specific header files serve as simple |
| 756 |
enables the checkpointing feature of TAMC |
placeholders at this point.} |
| 757 |
(see Section \ref{???}). |
% |
| 758 |
In the present example a 3-level checkpointing is implemented. |
\item {\it tamc.h} \\ |
| 759 |
The code contains the relevant store directives, common block |
This header configures the splitting of the time stepping loop |
| 760 |
and tape initializations, storing key computation, |
w.r.t. the 3-level checkpointing (see section ???). |
| 761 |
and loop index handling. |
|
|
The checkpointing length at each level is defined in |
|
|
file {\it tamc.h}, cf. below. |
|
|
% |
|
|
\item Cost function package: {\it pkg/cost/} \\ |
|
|
This package contains all relevant routines for |
|
|
initializing, accumulating and finalizing the cost function |
|
|
(see Section \ref{???}). \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
|
|
enables all general aspects of the cost function handling, |
|
|
in particular the hooks in the forward code for |
|
|
initializing, accumulating and finalizing the cost function. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
|
|
includes the call to the cost function for this |
|
|
particular experiment, eqn. (\ref{cost_tracer}). |
|
|
% |
|
|
\item Control variable package: {\it pkg/ctrl/} \\ |
|
|
This package contains all relevant routines for |
|
|
the handling of the control vector. |
|
|
Each control variable can be enabled/disabled with its own flag: \\ |
|
|
\begin{tabular}{ll} |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_THETA0\_CONTROL} & |
|
|
initial temperature \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SALT0\_CONTROL} & |
|
|
initial salinity \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TR0\_CONTROL} & |
|
|
initial passive tracer concentration \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUU0\_CONTROL} & |
|
|
zonal wind stress \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUV0\_CONTROL} & |
|
|
meridional wind stress \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SFLUX0\_CONTROL} & |
|
|
freshwater flux \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_HFLUX0\_CONTROL} & |
|
|
heat flux \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_DIFFKR\_CONTROL} & |
|
|
diapycnal diffusivity \\ |
|
|
\hspace*{2ex} {\tt \#undef ALLOW\_KAPPAGM\_CONTROL} & |
|
|
isopycnal diffusivity \\ |
|
|
\end{tabular} |
|
| 762 |
% |
% |
| 763 |
\end{itemize} |
\end{itemize} |
| 764 |
|
|
| 765 |
\subsubsection{File {\it SIZE.h}} |
%------------------------------------------------------------------ |
|
|
|
|
The file contains the grid point dimensions of the forward |
|
|
model. It is identical to the {\it verification/exp2/}: \\ |
|
|
\hspace*{4ex} {\tt sNx = 90} \\ |
|
|
\hspace*{4ex} {\tt sNy = 40} \\ |
|
|
\hspace*{4ex} {\tt Nr = 20} \\ |
|
|
It corresponds to a single-tile/single-processor setup: |
|
|
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
|
|
with standard overlap dimensioning |
|
|
{\tt OLx = OLy = 3}. |
|
|
|
|
|
\subsubsection{File {\it adcommon.h}} |
|
|
|
|
|
This file contains common blocks of some adjoint variables |
|
|
that are generated by TAMC. |
|
|
The common blocks are used by the adjoint support routine |
|
|
{\it addummy\_in\_stepping} which needs to access those variables: |
|
|
|
|
|
\begin{tabular}{ll} |
|
|
\hspace*{4ex} {\tt common /addynvars\_r/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /addynvars\_cd/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /addynvars\_diffkr/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /addynvars\_kapgm/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /adtr1\_r/} & |
|
|
\hspace*{4ex} is related to {\it TR1.h} \\ |
|
|
\hspace*{4ex} {\tt common /adffields/} & |
|
|
\hspace*{4ex} is related to {\it FFIELDS.h}\\ |
|
|
\end{tabular} |
|
|
|
|
|
Note that if the structure of the common block changes in the |
|
|
above header files of the forward code, the structure |
|
|
of the adjoint common blocks will change accordingly. |
|
|
Thus, it has to be made sure that the structure of the |
|
|
adjoint common block in the hand-written file {\it adcommon.h} |
|
|
complies with the automatically generated adjoint common blocks |
|
|
in {\it adjoint\_model.F}. |
|
| 766 |
|
|
| 767 |
\subsubsection{File {\it tamc.h}} |
\subsection{Building the AD code |
| 768 |
|
\label{section_ad_build}} |
| 769 |
|
|
| 770 |
This routine contains the dimensions for TAMC checkpointing. |
The build process of an AD code is very similar to building |
| 771 |
|
the forward model. However, depending on which AD code one wishes |
| 772 |
|
to generate, and on which AD tool is available (TAF or TAMC), |
| 773 |
|
the following {\tt make} targets are available: |
| 774 |
|
|
| 775 |
|
\begin{table}[h!] |
| 776 |
|
{\footnotesize |
| 777 |
|
\begin{tabular}{ccll} |
| 778 |
|
~ & {\it AD-target} & {\it output} & {\it description} \\ |
| 779 |
|
\hline |
| 780 |
|
\hline |
| 781 |
|
(1) & {\tt <MODE><TOOL>only} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 782 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 783 |
|
~ & ~ & ~ & no {\tt make} dependencies on {\tt .F .h} \\ |
| 784 |
|
~ & ~ & ~ & useful for compiling on remote platforms \\ |
| 785 |
|
\hline |
| 786 |
|
(2) & {\tt <MODE><TOOL>} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 787 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 788 |
|
~ & ~ & ~ & includes {\tt make} dependencies on {\tt .F .h} \\ |
| 789 |
|
~ & ~ & ~ & i.e. input for $<$TOOL$>$ may be re-generated \\ |
| 790 |
|
\hline |
| 791 |
|
(3) & {\tt <MODE>all} & {\tt mitgcmuv\_<MODE>} & |
| 792 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 793 |
|
~ & ~ & ~ & and compiles all code \\ |
| 794 |
|
~ & ~ & ~ & (use of TAF is set as default) \\ |
| 795 |
|
\hline |
| 796 |
|
\hline |
| 797 |
|
\end{tabular} |
| 798 |
|
} |
| 799 |
|
\end{table} |
| 800 |
|
% |
| 801 |
|
Here, the following placeholders are used |
| 802 |
% |
% |
| 803 |
\begin{itemize} |
\begin{itemize} |
| 804 |
% |
% |
| 805 |
\item {\tt \#ifdef ALLOW\_TAMC\_CHECKPOINTING} \\ |
\item [$<$TOOL$>$] |
|
3-level checkpointing is enabled, i.e. the timestepping |
|
|
is divided into three different levels (see Section \ref{???}). |
|
|
The model state of the outermost ({\tt nchklev\_3}) and the |
|
|
intermediate ({\tt nchklev\_2}) timestepping loop are stored to file |
|
|
(handled in {\it the\_main\_loop}). |
|
|
The innermost loop ({\tt nchklev\_1}) |
|
|
avoids I/O by storing all required variables |
|
|
to common blocks. This storing may also be necessary if |
|
|
no checkpointing is chosen |
|
|
(nonlinear functions, if-statements, iterative loops, ...). |
|
|
In the present example the dimensions are chosen as follows: \\ |
|
|
\hspace*{4ex} {\tt nchklev\_1 = 36 } \\ |
|
|
\hspace*{4ex} {\tt nchklev\_2 = 30 } \\ |
|
|
\hspace*{4ex} {\tt nchklev\_3 = 60 } \\ |
|
|
To guarantee that the checkpointing intervals span the entire |
|
|
integration period the following relation must be satisfied: \\ |
|
|
\hspace*{4ex} {\tt nchklev\_1*nchklev\_2*nchklev\_3 $ \ge $ nTimeSteps} \\ |
|
|
where {\tt nTimeSteps} is either specified in {\it data} |
|
|
or computed via \\ |
|
|
\hspace*{4ex} {\tt nTimeSteps = (endTime-startTime)/deltaTClock }. |
|
|
% |
|
|
\item {\tt \#undef ALLOW\_TAMC\_CHECKPOINTING} \\ |
|
|
No checkpointing is enabled. |
|
|
In this case the relevant counter is {\tt nchklev\_0}. |
|
|
Similar to above, the following relation has to be satisfied \\ |
|
|
\hspace*{4ex} {\tt nchklev\_0 $ \ge $ nTimeSteps}. |
|
| 806 |
% |
% |
| 807 |
\end{itemize} |
\begin{itemize} |
|
|
|
|
The following parameters may be worth describing: \\ |
|
| 808 |
% |
% |
| 809 |
\hspace*{4ex} {\tt isbyte} \\ |
\item {\tt TAF} |
| 810 |
\hspace*{4ex} {\tt maxpass} \\ |
\item {\tt TAMC} |
|
~ |
|
|
|
|
|
\subsubsection{File {\it makefile}} |
|
|
|
|
|
This file contains all relevant parameter flags and |
|
|
lists to run TAMC or TAF. |
|
|
It is assumed that TAMC is available to you, either locally, |
|
|
being installed on your network, or remotely through the 'TAMC Utility'. |
|
|
TAMC is called with the command {\tt tamc} followed by a |
|
|
number of options. They are described in detail in the |
|
|
TAMC manual \cite{gie:99}. |
|
|
Here we briefly discuss the main flags used in the {\it makefile} |
|
| 811 |
% |
% |
|
\begin{itemize} |
|
|
\item [{\tt tamc}] {\tt |
|
|
-input <variable names> |
|
|
-output <variable name> -r4 ... \\ |
|
|
-toplevel <S/R name> -reverse <file names> |
|
|
} |
|
| 812 |
\end{itemize} |
\end{itemize} |
| 813 |
% |
% |
| 814 |
|
\item [$<$MODE$>$] |
| 815 |
|
% |
| 816 |
\begin{itemize} |
\begin{itemize} |
| 817 |
% |
% |
| 818 |
\item {\tt -toplevel <S/R name>} \\ |
\item {\tt ad} generates the adjoint model (ADM) |
| 819 |
Name of the toplevel routine, with respect to which the |
\item {\tt ftl} generates the tangent linear model (TLM) |
| 820 |
control flow analysis is performed. |
\item {\tt svd} generates both ADM and TLM for \\ |
| 821 |
% |
singular value decomposition (SVD) type calculations |
|
\item {\tt -input <variable names>} \\ |
|
|
List of independent variables $ u $ with respect to which the |
|
|
dependent variable $ J $ is differentiated. |
|
|
% |
|
|
\item {\tt -output <variable name>} \\ |
|
|
Dependent variable $ J $ which is to be differentiated. |
|
|
% |
|
|
\item {\tt -reverse <file names>} \\ |
|
|
Adjoint code is generated to compute the sensitivity of an |
|
|
independent variable w.r.t. many dependent variables. |
|
|
In the discussion of Section ??? |
|
|
the generated adjoint top-level routine computes the product |
|
|
of the transposed Jacobian matrix $ M^T $ times |
|
|
the gradient vector $ \nabla_v J $. |
|
|
\\ |
|
|
{\tt <file names>} refers to the list of files {\it .f} which are to be |
|
|
analyzed by TAMC. This list is generally smaller than the full list |
|
|
of code to be compiled. The files not contained are either |
|
|
above the top-level routine (some initializations), or are |
|
|
deliberately hidden from TAMC, either because hand-written |
|
|
adjoint routines exist, or the routines must not (or don't have to) |
|
|
be differentiated. For each routine which is part of the flow tree |
|
|
of the top-level routine, but deliberately hidden from TAMC |
|
|
(or for each package which contains such routines), |
|
|
a corresponding file {\it .flow} exists containing flow directives |
|
|
for TAMC. |
|
| 822 |
% |
% |
| 823 |
\item {\tt -r4} \\ |
\end{itemize} |
|
~ |
|
| 824 |
% |
% |
| 825 |
\end{itemize} |
\end{itemize} |
| 826 |
|
|
| 827 |
|
For example, to generate the adjoint model using TAF after routines ({\tt .F}) |
| 828 |
|
or headers ({\tt .h}) have been modified, but without compilation, |
| 829 |
|
type {\tt make adtaf}; |
| 830 |
|
or, to generate the tangent linear model using TAMC without |
| 831 |
|
re-generating the input code, type {\tt make ftltamconly}. |
| 832 |
|
|
|
\subsubsection{The input parameter files} |
|
|
|
|
|
\paragraph{File {\it data}} |
|
|
|
|
|
\paragraph{File {\it data.cost}} |
|
|
|
|
|
\paragraph{File {\it data.ctrl}} |
|
|
|
|
|
\paragraph{File {\it data.gmredi}} |
|
|
|
|
|
\paragraph{File {\it data.grdchk}} |
|
|
|
|
|
\paragraph{File {\it data.optim}} |
|
|
|
|
|
\paragraph{File {\it data.pkg}} |
|
|
|
|
|
\paragraph{File {\it eedata}} |
|
|
|
|
|
\paragraph{File {\it topog.bin}} |
|
|
|
|
|
\paragraph{File {\it windx.bin, windy.bin}} |
|
| 833 |
|
|
| 834 |
\paragraph{File {\it salt.bin, theta.bin}} |
A typical full build process to generate the ADM via TAF would |
| 835 |
|
look like follows: |
| 836 |
\paragraph{File {\it SSS.bin, SST.bin}} |
\begin{verbatim} |
| 837 |
|
% mkdir build |
| 838 |
\paragraph{File {\it pickup*}} |
% cd build |
| 839 |
|
% ../../../tools/genmake2 -mods=../code_ad |
| 840 |
\subsection{Compiling the model and its adjoint} |
% make depend |
| 841 |
|
% make adall |
| 842 |
|
\end{verbatim} |
| 843 |
|
|
| 844 |
The built process of the adjoint model is slightly more |
%------------------------------------------------------------------ |
|
complex than that of compiling the forward code. |
|
|
The main reason is that the adjoint code generation requires |
|
|
a specific list of routines that are to be differentiated |
|
|
(as opposed to the automatic generation of a list of |
|
|
files to be compiled by genmake). |
|
|
This list excludes routines that don't have to be or must not be |
|
|
differentiated. For some of the latter routines flow directives |
|
|
may be necessary, a list of which has to be given as well. |
|
|
For this reason, a separate {\it makefile} is currently |
|
|
maintained in the directory {\tt adjoint/}. This |
|
|
makefile is responsible for the adjoint code generation. |
|
| 845 |
|
|
| 846 |
In the following we describe the build process step by step, |
\subsection{The AD build process in detail |
| 847 |
assuming you are in the directory {\tt bin/}. |
\label{section_ad_build_detail}} |
|
A summary of steps to follow is given at the end. |
|
| 848 |
|
|
| 849 |
\paragraph{Adjoint code generation and compilation -- step by step} |
The {\tt make <MODE>all} target consists of the following procedures: |
| 850 |
|
|
| 851 |
\begin{enumerate} |
\begin{enumerate} |
| 852 |
% |
% |
| 853 |
\item |
\item |
| 854 |
{\tt ln -s ../verification/???/code/.genmakerc .} \\ |
A header file {\tt AD\_CONFIG.h} is generated which contains a CPP option |
| 855 |
{\tt ln -s ../verification/???/code/*.[Fh] .} \\ |
on which code ought to be generated. Depending on the {\tt make} target, |
| 856 |
Link your customized genmake options, header files, |
the contents is |
| 857 |
and modified code to the compile directory. |
\begin{itemize} |
|
% |
|
| 858 |
\item |
\item |
| 859 |
{\tt ../tools/genmake -makefile} \\ |
{\tt \#define ALLOW\_ADJOINT\_RUN} |
|
Generate your Makefile (cf. Section ???). |
|
|
% |
|
| 860 |
\item |
\item |
| 861 |
{\tt make depend} \\ |
{\tt \#define ALLOW\_TANGENTLINEAR\_RUN} |
|
Dependency analysis for the CPP pre-compiler (cf. Section ???). |
|
|
% |
|
| 862 |
\item |
\item |
| 863 |
{\tt make small\_f} \\ |
{\tt \#define ALLOW\_ECCO\_OPTIMIZATION} |
| 864 |
This is the first difference between forward code compilation |
\end{itemize} |
|
and adjoint code generation and compilation. |
|
|
Instead of going through the entire compilation process |
|
|
(CPP precompiling -- {\tt .f}, object code generation -- {\tt .o}, |
|
|
linking of object files and libraries to generate executable), |
|
|
only the CPP compiler is invoked at this stage to generate |
|
|
the {\tt .f} files. |
|
| 865 |
% |
% |
| 866 |
\item |
\item |
| 867 |
{\tt cd ../adjoint} \\ |
A single file {\tt <MODE>\_input\_code.f} is concatenated |
| 868 |
{\tt make adtaf} or {\tt make adtamc} \\ |
consisting of all {\tt .f} files that are part of the list {\bf AD\_FILES} |
| 869 |
Depending on whether you have TAF or TAMC at your disposal, |
and all {\tt .flow} files that are part of the list {\bf AD\_FLOW\_FILES}. |
|
you'll choose {\tt adtaf} or {\tt adtamc} as your |
|
|
make target for the {\it makefile} in the directory {\tt adjoint/}. |
|
|
Several things happen at this stage. |
|
|
% |
|
|
\begin{enumerate} |
|
| 870 |
% |
% |
| 871 |
\item |
\item |
| 872 |
The initial template file {\it adjoint\_model.F} which is part |
The AD tool is invoked with the {\bf <MODE>\_<TOOL>\_FLAGS}. |
| 873 |
of the compiling list created by {\it genmake} is restored. |
The default AD tool flags in {\tt genmake2} can be overrwritten by |
| 874 |
|
an {\tt adjoint\_options} file (similar to the platform-specific |
| 875 |
|
{\tt build\_options}, see Section ???. |
| 876 |
|
The AD tool writes the resulting AD code into the file |
| 877 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 878 |
% |
% |
| 879 |
\item |
\item |
| 880 |
All Fortran routines {\tt *.f} in {\tt bin/} are |
A short sed script {\tt adjoint\_sed} is applied to |
| 881 |
concatenated into a single file (it's current name is |
{\tt <MODE>\_input\_code\_ad.f} |
| 882 |
{\it tamc\_code.f}). |
to reinstate {\bf myThid} into the CALL argument list of active file I/O. |
| 883 |
|
The result is written to file {\tt <MODE>\_<TOOL>\_output.f}. |
| 884 |
% |
% |
| 885 |
\item |
\item |
| 886 |
Adjoint code is generated by TAMC or TAF. |
All routines are compiled and an executable is generated |
| 887 |
The adjoint code is written to the file {\it tamc\_code\_ad.f}. |
(see Table ???). |
|
It contains all adjoint routines of the forward routines |
|
|
concatenated in {\it tamc\_code.f}. |
|
|
For a given forward routines {\tt subroutine routinename} |
|
|
the adjoint routine is named {\tt adsubroutine routinename} |
|
|
by default (that default can be changed via the flag |
|
|
{\tt -admark <markname>}). |
|
|
Furthermore, it may contain modified code which |
|
|
incorporates the translation of adjoint store directives |
|
|
into specific Fortran code. |
|
|
For a given forward routines {\tt subroutine routinename} |
|
|
the modified routine is named {\tt mdsubroutine routinename}. |
|
|
TAMC or TAF info is written to file |
|
|
{\it tamc\_code.prot} or {\it taf.log}, respectively. |
|
| 888 |
% |
% |
| 889 |
\end{enumerate} |
\end{enumerate} |
| 890 |
|
|
| 891 |
|
\subsubsection{The list AD\_FILES and {\tt .list} files} |
| 892 |
|
|
| 893 |
|
Not all routines are presented to the AD tool. |
| 894 |
|
Routines typically hidden are diagnostics routines which |
| 895 |
|
do not influence the cost function, but may create |
| 896 |
|
artificial flow dependencies such as I/O of active variables. |
| 897 |
|
|
| 898 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FILES} |
| 899 |
|
which contains all routines that are shown to the AD tool. |
| 900 |
|
This list is put together from all files with suffix {\tt .list} |
| 901 |
|
that {\tt genmake2} finds in its search directories. |
| 902 |
|
The list file for the core MITgcm routines is in {\tt model/src/} |
| 903 |
|
is called {\tt model\_ad\_diff.list}. |
| 904 |
|
Note that no wrapper routine is shown to TAF. These are either |
| 905 |
|
not visible at all to the AD code, or hand-written AD code |
| 906 |
|
is available (see next section). |
| 907 |
|
|
| 908 |
|
Each package directory contains its package-specific |
| 909 |
|
list file {\tt <PKG>\_ad\_diff.list}. For example, |
| 910 |
|
{\tt pkg/ptracers/} contains the file {\tt ptracers\_ad\_diff.list}. |
| 911 |
|
Thus, enabling a package will automatically extend the |
| 912 |
|
{\bf AD\_FILES} list of {\tt genmake2} to incorporate the |
| 913 |
|
package-specific routines. |
| 914 |
|
Note that you will need to regenerate the {\tt Makefile} if |
| 915 |
|
you enable a package (e.g. by adding it to {\tt packages.conf}) |
| 916 |
|
and a {\tt Makefile} already exists. |
| 917 |
|
|
| 918 |
|
\subsubsection{The list AD\_FLOW\_FILES and {\tt .flow} files} |
| 919 |
|
|
| 920 |
|
TAMC and TAF can evaluate user-specified directives |
| 921 |
|
that start with a specific syntax ({\tt CADJ}, {\tt C\$TAF}, {\tt !\$TAF}). |
| 922 |
|
The main categories of directives are STORE directives and |
| 923 |
|
FLOW directives. Here, we are concerned with flow directives, |
| 924 |
|
store directives are treated elsewhere. |
| 925 |
|
|
| 926 |
|
Flow directives enable the AD tool to evaluate how it should treat |
| 927 |
|
routines that are 'hidden' by the user, i.e. routines which are |
| 928 |
|
not contained in the {\bf AD\_FILES} list (see previous section), |
| 929 |
|
but which are called in part of the code that the AD tool does see. |
| 930 |
|
The flow directive tell the AD tool |
| 931 |
% |
% |
| 932 |
\item |
\begin{itemize} |
|
{\tt make adchange} \\ |
|
|
The multi-threading capability of the MITGCM requires a slight |
|
|
change in the parameter list of some routines that are related to |
|
|
to active file handling. |
|
|
This post-processing invokes the sed script {\it adjoint\_ecco\_sed.com} |
|
|
to insert the threading counter {\bf myThId} into the parameter list |
|
|
of those subroutines. |
|
|
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
|
|
and appended to the file {\it adjoint\_model.F}. |
|
|
This concludes the adjoint code generation. |
|
| 933 |
% |
% |
| 934 |
\item |
\item which subroutine arguments are input/output |
| 935 |
{\tt cd ../bin} \\ |
\item which subroutine arguments are active |
| 936 |
{\tt make} \\ |
\item which subroutine arguments are required to compute the cost |
| 937 |
The file {\it adjoint\_model.F} now contains the full adjoint code. |
\item which subroutine arguments are dependent |
|
All routines are now compiled. |
|
| 938 |
% |
% |
| 939 |
\end{enumerate} |
\end{itemize} |
| 940 |
|
% |
| 941 |
|
The syntax for the flow directives can be found in the |
| 942 |
|
AD tool manuals. |
| 943 |
|
|
| 944 |
\paragraph{Adjoint code generation and compilation -- summary} |
{\tt genmake2} generates a list (or variable) {\bf AD\_FLOW\_FILES} |
| 945 |
~ \\ |
which contains all files with suffix{\tt .flow} that it finds |
| 946 |
|
in its search directories. |
| 947 |
|
The flow directives for the core MITgcm routines of |
| 948 |
|
{\tt eesupp/src/} and {\tt model/src/} |
| 949 |
|
reside in {\tt pkg/autodiff/}. |
| 950 |
|
This directory also contains hand-written adjoint code |
| 951 |
|
for the MITgcm WRAPPER (section \ref{chap:sarch}). |
| 952 |
|
|
| 953 |
|
Flow directives for package-specific routines are contained in |
| 954 |
|
the corresponding package directories in the file |
| 955 |
|
{\tt <PKG>\_ad.flow}, e.g. ptracers-specific directives are in |
| 956 |
|
{\tt ptracers\_ad.flow}. |
| 957 |
|
|
| 958 |
|
\subsubsection{Store directives for 3-level checkpointing} |
| 959 |
|
|
| 960 |
|
The storing that is required at each period of the |
| 961 |
|
3-level checkpointing is controled by three |
| 962 |
|
top-level headers. |
| 963 |
|
|
| 964 |
\[ |
\begin{verbatim} |
| 965 |
\boxed{ |
do ilev_3 = 1, nchklev_3 |
| 966 |
\begin{split} |
# include ``checkpoint_lev3.h'' |
| 967 |
~ & \mbox{\tt cd bin} \\ |
do ilev_2 = 1, nchklev_2 |
| 968 |
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/.genmakerc .} \\ |
# include ``checkpoint_lev2.h'' |
| 969 |
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/*.[Fh] .} \\ |
do ilev_1 = 1, nchklev_1 |
| 970 |
~ & \mbox{\tt ../tools/genmake -makefile} \\ |
# include ``checkpoint_lev1.h'' |
| 971 |
~ & \mbox{\tt make depend} \\ |
|
| 972 |
~ & \mbox{\tt make small\_f} \\ |
... |
| 973 |
~ & \mbox{\tt cd ../adjoint} \\ |
|
| 974 |
~ & \mbox{\tt make adtaf <OR: make adtamc>} \\ |
end do |
| 975 |
~ & \mbox{\tt make adchange} \\ |
end do |
| 976 |
~ & \mbox{\tt cd ../bin} \\ |
end do |
| 977 |
~ & \mbox{\tt make} \\ |
\end{verbatim} |
|
\end{split} |
|
|
} |
|
|
\] |
|
| 978 |
|
|
| 979 |
\newpage |
All files {\tt checkpoint\_lev?.h} are contained in directory |
| 980 |
|
{\tt pkg/autodiff/}. |
| 981 |
|
|
|
%********************************************************************** |
|
|
\section{TLM and ADM generation in general} |
|
|
\label{sec_ad_setup_gen} |
|
|
%********************************************************************** |
|
| 982 |
|
|
| 983 |
In this section we describe in a general fashion |
\subsubsection{Changing the default AD tool flags: ad\_options files} |
|
the parts of the code that are relevant for automatic |
|
|
differentiation using the software tool TAMC. |
|
| 984 |
|
|
|
\begin{figure}[b!] |
|
|
\input{part5/doc_ad_the_model} |
|
|
\caption{~} |
|
|
\label{fig:adthemodel} |
|
|
\end{figure} |
|
| 985 |
|
|
| 986 |
The basic flow is depicted in \ref{fig:adthemodel}. |
\subsubsection{Hand-written adjoint code} |
| 987 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
|
| 988 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
%------------------------------------------------------------------ |
|
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
|
|
which is the toplevel routine in terms of reverse mode computation. |
|
|
The routine {\it adthe\_main\_loop} has been generated using TAMC. |
|
|
It contains both the forward integration of the full model, |
|
|
any additional storing that is required for efficient checkpointing, |
|
|
and the reverse integration of the adjoint model. |
|
|
The structure of {\it adthe\_main\_loop} has been strongly |
|
|
simplified for clarification; in particular, no checkpointing |
|
|
procedures are shown here. |
|
|
Prior to the call of {\it adthe\_main\_loop}, the routine |
|
|
{\it ctrl\_unpack} is invoked to unpack the control vector, |
|
|
and following that call, the routine {\it ctrl\_pack} |
|
|
is invoked to pack the control vector |
|
|
(cf. Section \ref{section_ctrl}). |
|
|
If gradient checks are to be performed, the option |
|
|
{\tt ALLOW\_GRADIENT\_CHECK} is defined. In this case |
|
|
the driver routine {\it grdchk\_main} is called after |
|
|
the gradient has been computed via the adjoint |
|
|
(cf. Section \ref{section_grdchk}). |
|
| 989 |
|
|
| 990 |
\subsection{The cost function (dependent variable) |
\subsection{The cost function (dependent variable) |
| 991 |
\label{section_cost}} |
\label{section_cost}} |
| 993 |
The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}. |
The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}. |
| 994 |
It is a function of the input variables $ \vec{u} $ via the composition |
It is a function of the input variables $ \vec{u} $ via the composition |
| 995 |
$ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $. |
$ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $. |
| 996 |
The input is referred to as the |
The input are referred to as the |
| 997 |
{\sf independent variables} or {\sf control variables}. |
{\sf independent variables} or {\sf control variables}. |
| 998 |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
| 999 |
(parameter setting, initialization, accumulation, |
(parameter setting, initialization, accumulation, |
| 1000 |
final evaluation), are controlled by the package {\it pkg/cost}. |
final evaluation), are controlled by the package {\it pkg/cost}. |
| 1001 |
|
The aspects relevant to the treatment of the independent variables |
| 1002 |
|
are controlled by the package {\it pkg/ctrl} and will be treated |
| 1003 |
|
in the next section. |
| 1004 |
|
|
|
\begin{figure}[h!] |
|
| 1005 |
\input{part5/doc_cost_flow} |
\input{part5/doc_cost_flow} |
|
\caption{~} |
|
|
\label{fig:costflow} |
|
|
\end{figure} |
|
| 1006 |
|
|
| 1007 |
\subsubsection{genmake and CPP options} |
\subsubsection{Enabling the package} |
| 1008 |
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item |
|
| 1009 |
\fbox{ |
\fbox{ |
| 1010 |
\begin{minipage}{12cm} |
\begin{minipage}{12cm} |
| 1011 |
{\it genmake}, {\it CPP\_OPTIONS.h}, {\it ECCO\_CPPOPTIONS.h} |
{\it packages.conf}, {\it ECCO\_CPPOPTIONS.h} |
| 1012 |
\end{minipage} |
\end{minipage} |
| 1013 |
} |
} |
| 1014 |
\end{itemize} |
\begin{itemize} |
|
% |
|
|
The directory {\it pkg/cost} can be included to the |
|
|
compile list in 3 different ways (cf. Section \ref{???}): |
|
| 1015 |
% |
% |
| 1016 |
\begin{enumerate} |
\item |
| 1017 |
|
The package is enabled by adding {\it cost} to your file {\it packages.conf} |
| 1018 |
|
(see Section ???) |
| 1019 |
% |
% |
| 1020 |
\item {\it genmake}: \\ |
\item |
| 1021 |
Change the default settings in the file {\it genmake} by adding |
|
| 1022 |
{\bf cost} to the {\bf enable} list (not recommended). |
|
| 1023 |
% |
\end{itemize} |
|
\item {\it .genmakerc}: \\ |
|
|
Customize the settings of {\bf enable}, {\bf disable} which are |
|
|
appropriate for your experiment in the file {\it .genmakerc} |
|
|
and add the file to your compile directory. |
|
|
% |
|
|
\item genmake-options: \\ |
|
|
Call {\it genmake} with the option |
|
|
{\tt genmake -enable=cost}. |
|
| 1024 |
% |
% |
| 1025 |
\end{enumerate} |
|
| 1026 |
|
N.B.: In general the following packages ought to be enabled |
| 1027 |
|
simultaneously: {\it autodiff, cost, ctrl}. |
| 1028 |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
| 1029 |
Each specific cost function contribution has its own option. |
Each specific cost function contribution has its own option. |
| 1030 |
For the present example the option is {\bf ALLOW\_COST\_TRACER}. |
For the present example the option is {\bf ALLOW\_COST\_TRACER}. |
| 1031 |
All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h} |
All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h} |
| 1032 |
Since the cost function is usually used in conjunction with |
Since the cost function is usually used in conjunction with |
| 1033 |
automatic differentiation, the CPP option |
automatic differentiation, the CPP option |
| 1034 |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
{\bf ALLOW\_ADJOINT\_RUN} (file {\it CPP\_OPTIONS.h}) and |
| 1035 |
(file {\it CPP\_OPTIONS.h}). |
{\bf ALLOW\_AUTODIFF\_TAMC} (file {\it ECCO\_CPPOPTIONS.h}) |
| 1036 |
|
should be defined. |
| 1037 |
|
|
| 1038 |
\subsubsection{Initialization} |
\subsubsection{Initialization} |
| 1039 |
% |
% |
| 1040 |
The initialization of the {\it cost} package is readily enabled |
The initialization of the {\it cost} package is readily enabled |
| 1041 |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
as soon as the CPP option {\bf ALLOW\_COST} is defined. |
| 1042 |
% |
% |
| 1043 |
\begin{itemize} |
\begin{itemize} |
| 1044 |
% |
% |
| 1112 |
\begin{equation} |
\begin{equation} |
| 1113 |
{\cal J} \, = \, |
{\cal J} \, = \, |
| 1114 |
{\rm fc} \, = \, |
{\rm fc} \, = \, |
| 1115 |
{\rm mult\_tracer} \sum_{bi,\,bj}^{nSx,\,nSy} |
{\rm mult\_tracer} \sum_{\text{global sum}} \sum_{bi,\,bj}^{nSx,\,nSy} |
| 1116 |
{\rm objf\_tracer}(bi,bj) \, + \, ... |
{\rm objf\_tracer}(bi,bj) \, + \, ... |
| 1117 |
\end{equation} |
\end{equation} |
| 1118 |
% |
% |
| 1124 |
|
|
| 1125 |
%%%% \end{document} |
%%%% \end{document} |
| 1126 |
|
|
|
\begin{figure} |
|
| 1127 |
\input{part5/doc_ad_the_main} |
\input{part5/doc_ad_the_main} |
|
\caption{~} |
|
|
\label{fig:adthemain} |
|
|
\end{figure} |
|
| 1128 |
|
|
| 1129 |
\subsection{The control variables (independent variables) |
\subsection{The control variables (independent variables) |
| 1130 |
\label{section_ctrl}} |
\label{section_ctrl}} |
| 1144 |
(parameter setting, initialization, perturbation) |
(parameter setting, initialization, perturbation) |
| 1145 |
are controlled by the package {\it pkg/ctrl}. |
are controlled by the package {\it pkg/ctrl}. |
| 1146 |
|
|
|
\begin{figure}[h!] |
|
| 1147 |
\input{part5/doc_ctrl_flow} |
\input{part5/doc_ctrl_flow} |
|
\caption{~} |
|
|
\label{fig:ctrlflow} |
|
|
\end{figure} |
|
| 1148 |
|
|
| 1149 |
\subsubsection{genmake and CPP options} |
\subsubsection{genmake and CPP options} |
| 1150 |
% |
% |
| 1160 |
% |
% |
| 1161 |
To enable the directory to be included to the compile list, |
To enable the directory to be included to the compile list, |
| 1162 |
{\bf ctrl} has to be added to the {\bf enable} list in |
{\bf ctrl} has to be added to the {\bf enable} list in |
| 1163 |
{\it .genmakerc} (or {\it genmake} itself). |
{\it .genmakerc} or in {\it genmake} itself (analogous to {\it cost} |
| 1164 |
|
package, cf. previous section). |
| 1165 |
Each control variable is enabled via its own CPP option |
Each control variable is enabled via its own CPP option |
| 1166 |
in {\it ECCO\_CPPOPTIONS.h}. |
in {\it ECCO\_CPPOPTIONS.h}. |
| 1167 |
|
|
| 1202 |
\\ |
\\ |
| 1203 |
% |
% |
| 1204 |
Two important issues related to the handling of the control |
Two important issues related to the handling of the control |
| 1205 |
variables in the MITGCM need to be addressed. |
variables in MITgcm need to be addressed. |
| 1206 |
First, in order to save memory, the control variable arrays |
First, in order to save memory, the control variable arrays |
| 1207 |
are not kept in memory, but rather read from file and added |
are not kept in memory, but rather read from file and added |
| 1208 |
to the initial fields during the model initialization phase. |
to the initial fields during the model initialization phase. |
| 1277 |
tamc -input 'xx_tr1 ...' ... |
tamc -input 'xx_tr1 ...' ... |
| 1278 |
\end{verbatim} |
\end{verbatim} |
| 1279 |
% |
% |
| 1280 |
Now, as mentioned above, the MITGCM avoids maintaining |
Now, as mentioned above, MITgcm avoids maintaining |
| 1281 |
an array for each control variable by reading the |
an array for each control variable by reading the |
| 1282 |
perturbation to a temporary array from file. |
perturbation to a temporary array from file. |
| 1283 |
To ensure the symbolic link to be recognized by TAMC, a scalar |
To ensure the symbolic link to be recognized by TAMC, a scalar |
| 1305 |
% |
% |
| 1306 |
Note, that reading an active variable corresponds |
Note, that reading an active variable corresponds |
| 1307 |
to a variable assignment. Its derivative corresponds |
to a variable assignment. Its derivative corresponds |
| 1308 |
to a write statement of the adjoint variable. |
to a write statement of the adjoint variable, followed by |
| 1309 |
|
a reset. |
| 1310 |
The 'active file' routines have been designed |
The 'active file' routines have been designed |
| 1311 |
to support active read and corresponding adjoint active write |
to support active read and corresponding adjoint active write |
| 1312 |
operations (and vice versa). |
operations (and vice versa). |
| 1423 |
{\it addummy\_in\_stepping}. |
{\it addummy\_in\_stepping}. |
| 1424 |
This routine is part of the adjoint support package |
This routine is part of the adjoint support package |
| 1425 |
{\it pkg/autodiff} (cf.f. below). |
{\it pkg/autodiff} (cf.f. below). |
| 1426 |
|
The procedure is enabled using via the CPP-option |
| 1427 |
|
{\bf ALLOW\_AUTODIFF\_MONITOR} (file {\it ECCO\_CPPOPTIONS.h}). |
| 1428 |
To be part of the adjoint code, the corresponding S/R |
To be part of the adjoint code, the corresponding S/R |
| 1429 |
{\it dummy\_in\_stepping} has to be called in the forward |
{\it dummy\_in\_stepping} has to be called in the forward |
| 1430 |
model (S/R {\it the\_main\_loop}) at the appropriate place. |
model (S/R {\it the\_main\_loop}) at the appropriate place. |
| 1431 |
|
The adjoint common blocks are extracted from the adjoint code |
| 1432 |
|
via the header file {\it adcommon.h}. |
| 1433 |
|
|
| 1434 |
{\it dummy\_in\_stepping} is essentially empty, |
{\it dummy\_in\_stepping} is essentially empty, |
| 1435 |
the corresponding adjoint routine is hand-written rather |
the corresponding adjoint routine is hand-written rather |
| 1456 |
{\bf /adtr1\_r/}, {\bf /adffields/}, |
{\bf /adtr1\_r/}, {\bf /adffields/}, |
| 1457 |
which have been extracted from the adjoint code to enable |
which have been extracted from the adjoint code to enable |
| 1458 |
access to the adjoint variables. |
access to the adjoint variables. |
| 1459 |
|
|
| 1460 |
|
{\bf WARNING:} If the structure of the common blocks |
| 1461 |
|
{\bf /dynvars\_r/}, {\bf /dynvars\_cd/}, etc., changes |
| 1462 |
|
similar changes will occur in the adjoint common blocks. |
| 1463 |
|
Therefore, consistency between the TAMC-generated common blocks |
| 1464 |
|
and those in {\it adcommon.h} have to be checked. |
| 1465 |
% |
% |
| 1466 |
\end{itemize} |
\end{itemize} |
| 1467 |
|
|
| 1476 |
with the value of the cost function itself $ {\cal J}(u_{[k]}) $ |
with the value of the cost function itself $ {\cal J}(u_{[k]}) $ |
| 1477 |
at iteration step $ k $ serve |
at iteration step $ k $ serve |
| 1478 |
as input to a minimization routine (e.g. quasi-Newton method, |
as input to a minimization routine (e.g. quasi-Newton method, |
| 1479 |
conjugate gradient, ... \cite{gil_lem:89}) |
conjugate gradient, ... \cite{gil-lem:89}) |
| 1480 |
to compute an update in the |
to compute an update in the |
| 1481 |
control variable for iteration step $k+1$ |
control variable for iteration step $k+1$ |
| 1482 |
\[ |
\[ |
| 1607 |
Finally, {\it ctrl\_pack} collects all adjoint files |
Finally, {\it ctrl\_pack} collects all adjoint files |
| 1608 |
and writes them to the compressed vector file |
and writes them to the compressed vector file |
| 1609 |
{\bf vector\_grad\_$<$k$>$}. |
{\bf vector\_grad\_$<$k$>$}. |
|
|
|
|
\subsection{TLM and ADM generation via TAMC} |
|
|
|
|
|
|
|
|
|
|
|
\subsection{Flow directives and adjoint support routines \label{section_flowdir}} |
|
|
|
|
|
\subsection{Store directives and checkpointing \label{section_checkpointing}} |
|
|
|
|
|
\subsection{Gradient checks \label{section_grdchk}} |
|
|
|
|
|
\subsection{Second derivative generation via TAMC} |
|
|
|
|
|
\section{Example of adjoint code} |
|
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch