| 21 |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
| 22 |
(Transformation of Algorithms in Fortran), developed |
(Transformation of Algorithms in Fortran), developed |
| 23 |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
| 24 |
The first application of the adjoint of the MITGCM for senistivity |
The first application of the adjoint of the MITGCM for sensitivity |
| 25 |
studies has been published by \cite{maro-eta:99}. |
studies has been published by \cite{maro-eta:99}. |
| 26 |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
| 27 |
for ocean state estimation studies. |
for ocean state estimation studies. |
| 52 |
such as forcing functions) to the $n$-dimensional space |
such as forcing functions) to the $n$-dimensional space |
| 53 |
$V \subset I\!\!R^n$ of |
$V \subset I\!\!R^n$ of |
| 54 |
model output variable $\vec{v}=(v_1,\ldots,v_n)$ |
model output variable $\vec{v}=(v_1,\ldots,v_n)$ |
| 55 |
(model state, model diagnostcs, objective function, ...) |
(model state, model diagnostics, objective function, ...) |
| 56 |
under consideration, |
under consideration, |
| 57 |
% |
% |
| 58 |
\begin{equation} |
\begin{equation} |
| 220 |
starting at step 0 and moving up to step $\Lambda$, with intermediate |
starting at step 0 and moving up to step $\Lambda$, with intermediate |
| 221 |
${\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}$ and final |
${\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}$ and final |
| 222 |
${\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}$. |
${\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}$. |
| 223 |
Let ${\cal J}$ be a cost funciton which explicitly depends on the |
Let ${\cal J}$ be a cost function which explicitly depends on the |
| 224 |
final state $\vec{v}$ only |
final state $\vec{v}$ only |
| 225 |
(this restriction is for clarity reasons only). |
(this restriction is for clarity reasons only). |
| 226 |
% |
% |
| 301 |
are the Lagrange multipliers of the model equations which determine |
are the Lagrange multipliers of the model equations which determine |
| 302 |
$ \vec{v}^{(\lambda)}$. |
$ \vec{v}^{(\lambda)}$. |
| 303 |
|
|
| 304 |
In coponents, eq. (\ref{adjoint}) reads as follows. |
In components, eq. (\ref{adjoint}) reads as follows. |
| 305 |
Let |
Let |
| 306 |
\[ |
\[ |
| 307 |
\begin{array}{rclcrcl} |
\begin{array}{rclcrcl} |
| 322 |
\end{array} |
\end{array} |
| 323 |
\] |
\] |
| 324 |
denote the perturbations in $\vec{u}$ and $\vec{v}$, respectively, |
denote the perturbations in $\vec{u}$ and $\vec{v}$, respectively, |
| 325 |
and their adjoint varaiables; |
and their adjoint variables; |
| 326 |
further |
further |
| 327 |
\[ |
\[ |
| 328 |
M \, = \, \left( |
M \, = \, \left( |
| 468 |
{\it all} intermediate states $ \vec{v}^{(\lambda)} $) are sought. |
{\it all} intermediate states $ \vec{v}^{(\lambda)} $) are sought. |
| 469 |
In order to be able to solve for each component of the gradient |
In order to be able to solve for each component of the gradient |
| 470 |
$ \partial {\cal J} / \partial u_{i} $ in (\ref{forward}) |
$ \partial {\cal J} / \partial u_{i} $ in (\ref{forward}) |
| 471 |
a forward calulation has to be performed for each component seperately, |
a forward calculation has to be performed for each component separately, |
| 472 |
i.e. $ \delta \vec{u} = \delta u_{i} {\vec{e}_{i}} $ |
i.e. $ \delta \vec{u} = \delta u_{i} {\vec{e}_{i}} $ |
| 473 |
for the $i$-th forward calculation. |
for the $i$-th forward calculation. |
| 474 |
Then, (\ref{forward}) represents the |
Then, (\ref{forward}) represents the |
| 487 |
\nabla_u {\cal J}^T \cdot \delta \vec{J} |
\nabla_u {\cal J}^T \cdot \delta \vec{J} |
| 488 |
\] |
\] |
| 489 |
where now $ \delta \vec{J} \in I\!\!R^l $ is a vector of |
where now $ \delta \vec{J} \in I\!\!R^l $ is a vector of |
| 490 |
dimenison $ l $. |
dimension $ l $. |
| 491 |
In this case $ l $ reverse simulations have to be performed |
In this case $ l $ reverse simulations have to be performed |
| 492 |
for each $ \delta J_{k}, \,\, k = 1, \ldots, l $. |
for each $ \delta J_{k}, \,\, k = 1, \ldots, l $. |
| 493 |
Then, the reverse mode is more efficient as long as |
Then, the reverse mode is more efficient as long as |
| 494 |
$ l < n $, otherwise the forward mode is preferable. |
$ l < n $, otherwise the forward mode is preferable. |
| 495 |
Stricly, the reverse mode is called adjoint mode only for |
Strictly, the reverse mode is called adjoint mode only for |
| 496 |
$ l = 1 $. |
$ l = 1 $. |
| 497 |
|
|
| 498 |
A detailed analysis of the underlying numerical operations |
A detailed analysis of the underlying numerical operations |
| 570 |
A method to balance the amount of recomputations vs. |
A method to balance the amount of recomputations vs. |
| 571 |
storage requirements is called {\sf checkpointing} |
storage requirements is called {\sf checkpointing} |
| 572 |
(e.g. \cite{res-eta:98}). |
(e.g. \cite{res-eta:98}). |
| 573 |
It is depicted in \reffig{3levelcheck} for a 3-level checkpointing |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
| 574 |
[as an example, we give explicit numbers for a 3-day |
[as an example, we give explicit numbers for a 3-day |
| 575 |
integration with a 1-hourly timestep in square brackets]. |
integration with a 1-hourly timestep in square brackets]. |
| 576 |
\begin{itemize} |
\begin{itemize} |
| 607 |
[i.e. every hour $ i=0,...,5$ corresponding to |
[i.e. every hour $ i=0,...,5$ corresponding to |
| 608 |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
| 609 |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
| 610 |
and the model state of all peceeding timesteps along the last |
and the model state of all proceeding timesteps along the last |
| 611 |
sub-subsections are available, enabling integration backwards |
sub-subsections are available, enabling integration backwards |
| 612 |
in time along the last sub-subsection. |
in time along the last sub-subsection. |
| 613 |
Thus, the adjoint can be computed along this last |
Thus, the adjoint can be computed along this last |
| 637 |
on the computing resources available. |
on the computing resources available. |
| 638 |
|
|
| 639 |
\begin{figure}[t!] |
\begin{figure}[t!] |
| 640 |
\centering |
\begin{center} |
| 641 |
%\psdraft |
%\psdraft |
| 642 |
\psfrag{v_k1^lev3}{\mathinfigure{v_{k_{1}^{lev3}}}} |
%\psfrag{v_k1^lev3}{\mathinfigure{v_{k_{1}^{lev3}}}} |
| 643 |
\psfrag{v_kn-1^lev3}{\mathinfigure{v_{k_{n-1}^{lev3}}}} |
%\psfrag{v_kn-1^lev3}{\mathinfigure{v_{k_{n-1}^{lev3}}}} |
| 644 |
\psfrag{v_kn^lev3}{\mathinfigure{v_{k_{n}^{lev3}}}} |
%\psfrag{v_kn^lev3}{\mathinfigure{v_{k_{n}^{lev3}}}} |
| 645 |
\psfrag{v_k1^lev2}{\mathinfigure{v_{k_{1}^{lev2}}}} |
%\psfrag{v_k1^lev2}{\mathinfigure{v_{k_{1}^{lev2}}}} |
| 646 |
\psfrag{v_kn-1^lev2}{\mathinfigure{v_{k_{n-1}^{lev2}}}} |
%\psfrag{v_kn-1^lev2}{\mathinfigure{v_{k_{n-1}^{lev2}}}} |
| 647 |
\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
%\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
| 648 |
\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
%\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
| 649 |
\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
%\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
| 650 |
\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
%\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
| 651 |
|
\resizebox{5.5in}{!}{\includegraphics{part5/checkpointing.eps}} |
| 652 |
%\psfull |
%\psfull |
| 653 |
\caption |
\end{center} |
| 654 |
{Schematic view of intermediate dump and restart for |
\caption{ |
| 655 |
|
Schematic view of intermediate dump and restart for |
| 656 |
3-level checkpointing.} |
3-level checkpointing.} |
| 657 |
\label{fig:3levelcheck} |
\label{fig:3levelcheck} |
| 658 |
\end{figure} |
\end{figure} |
| 664 |
% \subsection{Error covariance estimate and Hessian matrix} |
% \subsection{Error covariance estimate and Hessian matrix} |
| 665 |
% \label{sec_hessian} |
% \label{sec_hessian} |
| 666 |
|
|
|
\newpage |
|
|
|
|
|
%********************************************************************** |
|
|
\section{AD-specific setup by example: sensitivity of carbon sequestration} |
|
|
\label{sec_ad_setup_ex} |
|
|
%********************************************************************** |
|
|
|
|
|
The MITGCM has been adapted to enable AD using TAMC or TAF. |
|
|
The present description, therefore, is specific to the |
|
|
use of TAMC or TAF as AD tool. |
|
|
The following sections describe the steps which are necessary to |
|
|
generate a tangent linear or adjoint model of the MITGCM. |
|
|
We take as an example the sensitivity of carbon sequestration |
|
|
in the ocean. |
|
|
The AD-relevant hooks in the code are sketched in |
|
|
\reffig{adthemodel}, \reffig{adthemain}. |
|
|
|
|
|
\subsection{Overview of the experiment} |
|
|
|
|
|
We describe an adjoint sensitivity analysis of outgassing from |
|
|
the ocean into the atmosphere of a carbon-like tracer injected |
|
|
into the ocean interior (see \cite{hil-eta:01}). |
|
|
|
|
|
\subsubsection{Passive tracer equation} |
|
|
|
|
|
For this work the MITGCM was augmented with a thermodynamically |
|
|
inactive tracer, $C$. Tracer residing in the ocean |
|
|
model surface layer is outgassed according to a relaxation time scale, |
|
|
$\mu$. Within the ocean interior, the tracer is passively advected |
|
|
by the ocean model currents. The full equation for the time evolution |
|
|
% |
|
|
\begin{equation} |
|
|
\label{carbon_ddt} |
|
|
\frac{\partial C}{\partial t} \, = \, |
|
|
-U\cdot \nabla C \, - \, \mu C \, + \, \Gamma(C) \,+ \, S |
|
|
\end{equation} |
|
|
% |
|
|
also includes a source term $S$. This term |
|
|
represents interior sources of $C$ such as would arise due to |
|
|
direct injection. |
|
|
The velocity term, $U$, is the sum of the |
|
|
model Eulerian circulation and an eddy-induced velocity, the latter |
|
|
parameterized according to Gent/McWilliams |
|
|
(\cite{gen-mcw:90, gen-eta:95}). |
|
|
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
|
|
fluid is locally statically unstable. |
|
|
|
|
|
The outgassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
|
|
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
|
|
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
|
|
is valid as a prognostic equation for small perturbations in oceanic |
|
|
carbon concentrations. This configuration provides a |
|
|
powerful tool for examining the impact of large-scale ocean circulation |
|
|
on $ CO_2 $ outgassing due to interior injections. |
|
|
As source we choose a constant in time injection of |
|
|
$ S = 1 \,\, {\rm mol / s}$. |
|
|
|
|
|
\subsubsection{Model configuration} |
|
|
|
|
|
The model configuration employed has a constant |
|
|
$4^\circ \times 4^\circ$ resolution horizontal grid and realistic |
|
|
geography and bathymetry. Twenty vertical layers are used with |
|
|
vertical spacing ranging |
|
|
from 50 m near the surface to 815 m at depth. |
|
|
Driven to steady-state by climatalogical wind-stress, heat and |
|
|
fresh-water forcing the model reproduces well known large-scale |
|
|
features of the ocean general circulation. |
|
|
|
|
|
\subsubsection{Outgassing cost function} |
|
|
|
|
|
To quantify and understand outgassing due to injections of $C$ |
|
|
in eqn. (\ref{carbon_ddt}), |
|
|
we define a cost function $ {\cal J} $ that measures the total amount of |
|
|
tracer outgassed at each timestep: |
|
|
% |
|
|
\begin{equation} |
|
|
\label{cost_tracer} |
|
|
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
|
|
\end{equation} |
|
|
% |
|
|
Equation(\ref{cost_tracer}) integrates the outgassing term, $\mu C$, |
|
|
from (\ref{carbon_ddt}) |
|
|
over the entire ocean surface area, $A$, and accumulates it |
|
|
up to time $T$. |
|
|
Physically, ${\cal J}$ can be thought of as representing the amount of |
|
|
$CO_2$ that our model predicts would be outgassed following an |
|
|
injection at rate $S$. |
|
|
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
|
|
$\frac{\partial {\cal J}}{\partial S}$, |
|
|
can be used to identify regions from which circulation |
|
|
would cause $CO_2$ to rapidly outgas following injection |
|
|
and regions in which $CO_2$ injections would remain effectively |
|
|
sequesterd within the ocean. |
|
|
|
|
|
\subsection{Code configuration} |
|
|
|
|
|
The model configuration for this experiment resides under the |
|
|
directory {\it verification/carbon/}. |
|
|
The code customisation routines are in {\it verification/carbon/code/}: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\it .genmakerc} |
|
|
% |
|
|
\item {\it COST\_CPPOPTIONS.h} |
|
|
% |
|
|
\item {\it CPP\_EEOPTIONS.h} |
|
|
% |
|
|
\item {\it CPP\_OPTIONS.h} |
|
|
% |
|
|
\item {\it CTRL\_OPTIONS.h} |
|
|
% |
|
|
\item {\it ECCO\_OPTIONS.h} |
|
|
% |
|
|
\item {\it SIZE.h} |
|
|
% |
|
|
\item {\it adcommon.h} |
|
|
% |
|
|
\item {\it tamc.h} |
|
|
% |
|
|
\end{itemize} |
|
|
% |
|
|
The runtime flag and parameters settings are contained in |
|
|
{\it verification/carbon/input/}, |
|
|
together with the forcing fields and and restart files: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\it data} |
|
|
% |
|
|
\item {\it data.cost} |
|
|
% |
|
|
\item {\it data.ctrl} |
|
|
% |
|
|
\item {\it data.gmredi} |
|
|
% |
|
|
\item {\it data.grdchk} |
|
|
% |
|
|
\item {\it data.optim} |
|
|
% |
|
|
\item {\it data.pkg} |
|
|
% |
|
|
\item {\it eedata} |
|
|
% |
|
|
\item {\it topog.bin} |
|
|
% |
|
|
\item {\it windx.bin, windy.bin} |
|
|
% |
|
|
\item {\it salt.bin, theta.bin} |
|
|
% |
|
|
\item {\it SSS.bin, SST.bin} |
|
|
% |
|
|
\item {\it pickup*} |
|
|
% |
|
|
\end{itemize} |
|
|
% |
|
|
Finally, the file to generate the adjoint code resides in |
|
|
$ adjoint/ $: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\it makefile} |
|
|
% |
|
|
\end{itemize} |
|
|
% |
|
|
|
|
|
Below we describe the customisations of this files which are |
|
|
specific to this experiment. |
|
|
|
|
|
\subsubsection{File {\it .genmakerc}} |
|
|
This file overwrites default settings of {\it genmake}. |
|
|
In the present example it is used to switch on the following |
|
|
packages which are related to automatic differentiation |
|
|
and are disabled by default: \\ |
|
|
\hspace*{4ex} {\tt set ENABLE=( autodiff cost ctrl ecco gmredi grdchk kpp )} \\ |
|
|
Other packages which are not needed are switched off: \\ |
|
|
\hspace*{4ex} {\tt set DISABLE=( aim obcs zonal\_filt shap\_filt cal exf )} |
|
|
|
|
|
\subsubsection{File {\it COST\_CPPOPTIONS.h, CTRL\_OPTIONS.h}} |
|
|
|
|
|
These files used to contain package-specific CPP-options |
|
|
(see Section \ref{???}). |
|
|
For technical reasons those options have been grouped together |
|
|
in the file {\it ECCO\_OPTIONS.h}. |
|
|
To retain the modularity, the files have been kept and contain |
|
|
the standard include of the {\it CPP\_OPTIONS.h} file. |
|
|
|
|
|
\subsubsection{File {\it CPP\_EEOPTIONS.h}} |
|
|
|
|
|
This file contains 'wrapper'-specific CPP options. |
|
|
It only needs to be changed if the code is to be run |
|
|
in a parallel environment (see Section \ref{???}). |
|
|
|
|
|
\subsubsection{File {\it CPP\_OPTIONS.h}} |
|
|
|
|
|
This file contains model-specific CPP options |
|
|
(see Section \ref{???}). |
|
|
Most options are related to the forward model setup. |
|
|
They are identical to the global steady circulation setup of |
|
|
{\it verification/exp2/}. |
|
|
The three options specific to this experiment are \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_PASSIVE\_TRACER} \\ |
|
|
This flag enables the code to carry through the |
|
|
advection/diffusion of a passive tracer along the |
|
|
model integration. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
|
|
This flag enables the inclusion of some AD-related fields |
|
|
concerning initialisation, link between control variables |
|
|
and forward model variables, and the call to the top-level |
|
|
forward/adjoint subroutine {\it adthe\_main\_loop} |
|
|
instead of {\it the\_main\_loop}. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_GRADIENT\_CHECK} \\ |
|
|
This flag enables the gradient check package. |
|
|
After computing the unperturbed cost function and its gradient, |
|
|
a series of computations are performed for which \\ |
|
|
$\bullet$ an element of the control vector is perturbed \\ |
|
|
$\bullet$ the cost function w.r.t. the perturbed element is |
|
|
computed \\ |
|
|
$\bullet$ the difference between the perturbed and unperturbed |
|
|
cost function is computed to compute the finite difference gradient \\ |
|
|
$\bullet$ the finite difference gradient is compared with the |
|
|
adjoint-generated gradient. |
|
|
The gradient check package is further described in Section ???. |
|
|
|
|
|
\subsubsection{File {\it ECCO\_OPTIONS.h}} |
|
|
|
|
|
The CPP options of several AD-related packages are grouped |
|
|
in this file: |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item |
|
|
Adjoint support package: {\it pkg/autodiff/} \\ |
|
|
This package contains hand-written adjoint code such as |
|
|
active file handling, flow directives for files which must not |
|
|
be differentiated, and TAMC-specific header files. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_AUTODIFF\_TAMC} \\ |
|
|
defines TAMC-related features in the code. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_TAMC\_CHECKPOINTING} \\ |
|
|
enables the checkpointing feature of TAMC |
|
|
(see Section \ref{???}). |
|
|
In the present example a 3-level checkpointing is implemented. |
|
|
The code contains the relevant store directives, common block |
|
|
and tape initialisations, storing key computation, |
|
|
and loop index handling. |
|
|
The checkpointing length at each level is defined in |
|
|
file {\it tamc.h}, cf. below. |
|
|
% |
|
|
\item Cost function package: {\it pkg/cost/} \\ |
|
|
This package contains all relevant routines for |
|
|
initialising, accumulating and finalizing the cost function |
|
|
(see Section \ref{???}). \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
|
|
enables all general aspects of the cost function handling, |
|
|
in particular the hooks in the foorward code for |
|
|
initialising, accumulating and finalizing the cost function. \\ |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
|
|
includes the call to the cost function for this |
|
|
particular experiment, eqn. (\ref{cost_tracer}). |
|
|
% |
|
|
\item Control variable package: {\it pkg/ctrl/} \\ |
|
|
This package contains all relevant routines for |
|
|
the handling of the control vector. |
|
|
Each control variable can be enabled/disabled with its own flag: \\ |
|
|
\begin{tabular}{ll} |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_THETA0\_CONTROL} & |
|
|
initial temperature \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SALT0\_CONTROL} & |
|
|
initial salinity \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TR0\_CONTROL} & |
|
|
initial passive tracer concentration \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUU0\_CONTROL} & |
|
|
zonal wind stress \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUV0\_CONTROL} & |
|
|
meridional wind stress \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SFLUX0\_CONTROL} & |
|
|
freshwater flux \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_HFLUX0\_CONTROL} & |
|
|
heat flux \\ |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_DIFFKR\_CONTROL} & |
|
|
diapycnal diffusivity \\ |
|
|
\hspace*{2ex} {\tt \#undef ALLOW\_KAPPAGM\_CONTROL} & |
|
|
isopycnal diffusivity \\ |
|
|
\end{tabular} |
|
|
% |
|
|
\end{itemize} |
|
|
|
|
|
\subsubsection{File {\it SIZE.h}} |
|
|
|
|
|
The file contains the grid point dimensions of the forward |
|
|
model. It is identical to the {\it verification/exp2/}: \\ |
|
|
\hspace*{4ex} {\tt sNx = 90} \\ |
|
|
\hspace*{4ex} {\tt sNy = 40} \\ |
|
|
\hspace*{4ex} {\tt Nr = 20} \\ |
|
|
It correpsponds to a single-tile/single-processor setup: |
|
|
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
|
|
with standard overlap dimensioning |
|
|
{\tt OLx = OLy = 3}. |
|
|
|
|
|
\subsubsection{File {\it adcommon.h}} |
|
|
|
|
|
This file contains common blocks of some adjoint variables |
|
|
that are generated by TAMC. |
|
|
The common blocks are used by the adjoint support routine |
|
|
{\it addummy\_in\_stepping} which needs to access those variables: |
|
|
|
|
|
\begin{tabular}{ll} |
|
|
\hspace*{4ex} {\tt common /addynvars\_r/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /addynvars\_cd/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /addynvars\_diffkr/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /addynvars\_kapgm/} & |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
|
|
\hspace*{4ex} {\tt common /adtr1\_r/} & |
|
|
\hspace*{4ex} is related to {\it TR1.h} \\ |
|
|
\hspace*{4ex} {\tt common /adffields/} & |
|
|
\hspace*{4ex} is related to {\it FFIELDS.h}\\ |
|
|
\end{tabular} |
|
|
|
|
|
Note that if the structure of the common block changes in the |
|
|
above header files of the forward code, the structure |
|
|
of the adjoint common blocks will change accordingly. |
|
|
Thus, it has to be made sure that the structure of the |
|
|
adjoint common block in the hand-written file {\it adcommon.h} |
|
|
complies with the automatically generated adjoint common blocks |
|
|
in {\it adjoint\_model.F}. |
|
|
|
|
|
\subsubsection{File {\it tamc.h}} |
|
|
|
|
|
This routine contains the dimensions for TAMC checkpointing. |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\tt \#ifdef ALLOW\_TAMC\_CHECKPOINTING} \\ |
|
|
3-level checkpointing is enabled, i.e. the timestepping |
|
|
is divided into three different levels (see Section \ref{???}). |
|
|
The model state of the outermost ({\tt nchklev\_3}) and the |
|
|
intermediate ({\tt nchklev\_2}) timestepping loop are stored to file |
|
|
(handled in {\it the\_main\_loop}). |
|
|
The innermost loop ({\tt nchklev\_1}) |
|
|
avoids I/O by storing all required variables |
|
|
to common blocks. This storing may also be necessary if |
|
|
no checkpointing is chosen |
|
|
(nonlinear functions, if-statements, iterative loops, ...). |
|
|
In the present example the dimensions are chosen as follows: \\ |
|
|
\hspace*{4ex} {\tt nchklev\_1 = 36 } \\ |
|
|
\hspace*{4ex} {\tt nchklev\_2 = 30 } \\ |
|
|
\hspace*{4ex} {\tt nchklev\_3 = 60 } \\ |
|
|
To guarantee that the checkpointing intervals span the entire |
|
|
integration period the following relation must be satisfied: \\ |
|
|
\hspace*{4ex} {\tt nchklev\_1*nchklev\_2*nchklev\_3 $ \ge $ nTimeSteps} \\ |
|
|
where {\tt nTimeSteps} is either specified in {\it data} |
|
|
or computed via \\ |
|
|
\hspace*{4ex} {\tt nTimeSteps = (endTime-startTime)/deltaTClock }. |
|
|
% |
|
|
\item {\tt \#undef ALLOW\_TAMC\_CHECKPOINTING} \\ |
|
|
No checkpointing is enabled. |
|
|
In this case the relevant counter is {\tt nchklev\_0}. |
|
|
Similar to above, the following relation has to be satisfied \\ |
|
|
\hspace*{4ex} {\tt nchklev\_0 $ \ge $ nTimeSteps}. |
|
|
% |
|
|
\end{itemize} |
|
|
|
|
|
The following parameters may be worth describing: \\ |
|
|
% |
|
|
\hspace*{4ex} {\tt isbyte} \\ |
|
|
\hspace*{4ex} {\tt maxpass} \\ |
|
|
~ |
|
|
|
|
|
\subsubsection{File {\it makefile}} |
|
|
|
|
|
This file contains all relevant paramter flags and |
|
|
lists to run TAMC or TAF. |
|
|
It is assumed that TAMC is available to you, either locally, |
|
|
being installed on your network, or remotely through the 'TAMC Utility'. |
|
|
TAMC is called with the command {\tt tamc} followed by a |
|
|
number of options. They are described in detail in the |
|
|
TAMC manual \cite{gie:99}. |
|
|
Here we briefly discuss the main flags used in the {\it makefile} |
|
|
% |
|
|
\begin{itemize} |
|
|
\item [{\tt tamc}] {\tt |
|
|
-input <variable names> |
|
|
-output <variable name> -r4 ... \\ |
|
|
-toplevel <S/R name> -reverse <file names> |
|
|
} |
|
|
\end{itemize} |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item {\tt -toplevel <S/R name>} \\ |
|
|
Name of the toplevel routine, with respect to which the |
|
|
control flow analysis is performed. |
|
|
% |
|
|
\item {\tt -input <variable names>} \\ |
|
|
List of independent variables $ u $ with respect to which the |
|
|
dependent variable $ J $ is differentiated. |
|
|
% |
|
|
\item {\tt -output <variable name>} \\ |
|
|
Dependent variable $ J $ which is to be differentiated. |
|
|
% |
|
|
\item {\tt -reverse <file names>} \\ |
|
|
Adjoint code is generated to compute the sensitivity of an |
|
|
independent variable w.r.t. many dependent variables. |
|
|
In the discussion of Section ??? |
|
|
the generated adjoint top-level routine computes the product |
|
|
of the transposed Jacobian matrix $ M^T $ times |
|
|
the gradient vector $ \nabla_v J $. |
|
|
\\ |
|
|
{\tt <file names>} refers to the list of files {\it .f} which are to be |
|
|
analyzed by TAMC. This list is generally smaller than the full list |
|
|
of code to be compiled. The files not contained are either |
|
|
above the top-level routine (some initialisations), or are |
|
|
deliberately hidden from TAMC, either because hand-written |
|
|
adjoint routines exist, or the routines must not (or don't have to) |
|
|
be differentiated. For each routine which is part of the flow tree |
|
|
of the top-level routine, but deliberately hidden from TAMC |
|
|
(or for each package which contains such routines), |
|
|
a corresponding file {\it .flow} exists containing flow directives |
|
|
for TAMC. |
|
|
% |
|
|
\item {\tt -r4} \\ |
|
|
~ |
|
|
% |
|
|
\end{itemize} |
|
|
|
|
|
|
|
|
\subsubsection{The input parameter files} |
|
|
|
|
|
\paragraph{File {\it data}} |
|
|
|
|
|
\paragraph{File {\it data.cost}} |
|
|
|
|
|
\paragraph{File {\it data.ctrl}} |
|
|
|
|
|
\paragraph{File {\it data.gmredi}} |
|
|
|
|
|
\paragraph{File {\it data.grdchk}} |
|
|
|
|
|
\paragraph{File {\it data.optim}} |
|
|
|
|
|
\paragraph{File {\it data.pkg}} |
|
|
|
|
|
\paragraph{File {\it eedata}} |
|
|
|
|
|
\paragraph{File {\it topog.bin}} |
|
|
|
|
|
\paragraph{File {\it windx.bin, windy.bin}} |
|
|
|
|
|
\paragraph{File {\it salt.bin, theta.bin}} |
|
|
|
|
|
\paragraph{File {\it SSS.bin, SST.bin}} |
|
|
|
|
|
\paragraph{File {\it pickup*}} |
|
|
|
|
|
\subsection{Compiling the model and its adjoint} |
|
|
|
|
|
The built process of the adjoint model is slightly more |
|
|
complex than that of compiling the forward code. |
|
|
The main reason is that the adjoint code generation requires |
|
|
a specific list of routines that are to be differentiated |
|
|
(as opposed to the automatic generation of a list of |
|
|
files to be compiled by genmake). |
|
|
This list excludes routines that don't have to be or must not be |
|
|
differentiated. For some of the latter routines flow directives |
|
|
may be necessary, a list of which has to be given as well. |
|
|
For this reason, a separate {\it makefile} is currently |
|
|
maintained in the directory {\tt adjoint/}. This |
|
|
makefile is responsible for the adjoint code generation. |
|
|
|
|
|
In the following we describe the build process step by step, |
|
|
assuming you are in the directory {\tt bin/}. |
|
|
A summary of steps to follow is given at the end. |
|
|
|
|
|
\paragraph{Adjoint code generation and compilation -- step by step} |
|
|
|
|
|
\begin{enumerate} |
|
|
% |
|
|
\item |
|
|
{\tt ln -s ../verification/???/code/.genmakerc .} \\ |
|
|
{\tt ln -s ../verification/???/code/*.[Fh] .} \\ |
|
|
Link your customized genmake options, header files, |
|
|
and modified code to the compile directory. |
|
|
% |
|
|
\item |
|
|
{\tt ../tools/genmake -makefile} \\ |
|
|
Generate your Makefile (cf. Section ???). |
|
|
% |
|
|
\item |
|
|
{\tt make depend} \\ |
|
|
Dependency analysis for the CPP pre-compiler (cf. Section ???). |
|
|
% |
|
|
\item |
|
|
{\tt make small\_f} \\ |
|
|
This is the first difference between forward code compilation |
|
|
and adjoint code generation and compilation. |
|
|
Instead of going through the entire compilation process |
|
|
(CPP precompiling -- {\tt .f}, object code generation -- {\tt .o}, |
|
|
linking of object files and libraries to generate executable), |
|
|
only the CPP compiler is invoked at this stage to generate |
|
|
the {\tt .f} files. |
|
|
% |
|
|
\item |
|
|
{\tt cd ../adjoint} \\ |
|
|
{\tt make adtaf} or {\tt make adtamc} \\ |
|
|
Depending on whether you have TAF or TAMC at your disposal, |
|
|
you'll choose {\tt adtaf} or {\tt adtamc} as your |
|
|
make target for the {\it makefile} in the directory {\tt adjoint/}. |
|
|
Several things happen at this stage. |
|
|
% |
|
|
\begin{enumerate} |
|
|
% |
|
|
\item |
|
|
The initial template file {\it adjoint\_model.F} which is part |
|
|
of the compiling list created by {\it genmake} is restored. |
|
|
% |
|
|
\item |
|
|
All Fortran routines {\tt *.f} in {\tt bin/} are |
|
|
concatenated into a single file (it's current name is |
|
|
{\it tamc\_code.f}). |
|
|
% |
|
|
\item |
|
|
Adjoint code is generated by TAMC or TAF. |
|
|
The adjoint code is written to the file {\it tamc\_code\_ad.f}. |
|
|
It contains all adjoint routines of the forward routines |
|
|
concatenated in {\it tamc\_code.f}. |
|
|
For a given forward routines {\tt subroutine routinename} |
|
|
the adjoint routine is named {\tt adsubroutine routinename} |
|
|
by default (that default can be changed via the flag |
|
|
{\tt -admark <markname>}). |
|
|
Furthermore, it may contain modified code which |
|
|
incorporates the translation of adjoint store directives |
|
|
into specific Fortran code. |
|
|
For a given forward routines {\tt subroutine routinename} |
|
|
the modified routine is named {\tt mdsubroutine routinename}. |
|
|
TAMC or TAF info is written to file |
|
|
{\it tamc\_code.prot} or {\it taf.log}, respectively. |
|
|
% |
|
|
\end{enumerate} |
|
|
% |
|
|
\item |
|
|
{\tt make adchange} \\ |
|
|
The multi-threading capability of the MITGCM requires a slight |
|
|
change in the parameter list of some routines that are related to |
|
|
to active file handling. |
|
|
This postprocessing invokes the sed script {\it adjoint\_ecco\_sed.com} |
|
|
to insert the threading counter {\bf myThId} into the parameter list |
|
|
of those subroutines. |
|
|
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
|
|
and appended to the file {\it adjoint\_model.F}. |
|
|
This concludes the adjoint codel generation. |
|
|
% |
|
|
\item |
|
|
{\tt cd ../bin} \\ |
|
|
{\tt make} \\ |
|
|
The file {\it adjoint\_model.F} now contains the full adjoint code. |
|
|
All routines are now compiled. |
|
|
% |
|
|
\end{enumerate} |
|
|
|
|
|
\paragraph{Adjoint code generation and compilation -- summary} |
|
|
~ \\ |
|
|
|
|
|
\[ |
|
|
\boxed{ |
|
|
\begin{split} |
|
|
~ & \mbox{\tt cd bin} \\ |
|
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/.genmakerc .} \\ |
|
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/*.[Fh] .} \\ |
|
|
~ & \mbox{\tt ../tools/genmake -makefile} \\ |
|
|
~ & \mbox{\tt make depend} \\ |
|
|
~ & \mbox{\tt make small\_f} \\ |
|
|
~ & \mbox{\tt cd ../adjoint} \\ |
|
|
~ & \mbox{\tt make adtaf <OR: make adtamc>} \\ |
|
|
~ & \mbox{\tt make adchange} \\ |
|
|
~ & \mbox{\tt cd ../bin} \\ |
|
|
~ & \mbox{\tt make} \\ |
|
|
\end{split} |
|
|
} |
|
|
\] |
|
|
|
|
| 667 |
\newpage |
\newpage |
| 668 |
|
|
| 669 |
%********************************************************************** |
%********************************************************************** |
| 675 |
the parts of the code that are relevant for automatic |
the parts of the code that are relevant for automatic |
| 676 |
differentiation using the software tool TAMC. |
differentiation using the software tool TAMC. |
| 677 |
|
|
|
\begin{figure}[b!] |
|
| 678 |
\input{part5/doc_ad_the_model} |
\input{part5/doc_ad_the_model} |
|
\caption{~} |
|
|
\label{fig:adthemodel} |
|
|
\end{figure} |
|
| 679 |
|
|
| 680 |
The basic flow is depicted in \reffig{adthemodel}. |
The basic flow is depicted in \ref{fig:adthemodel}. |
| 681 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
| 682 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
| 683 |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
| 709 |
The input is referred to as the |
The input is referred to as the |
| 710 |
{\sf independent variables} or {\sf control variables}. |
{\sf independent variables} or {\sf control variables}. |
| 711 |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
| 712 |
(parameter setting, initialisation, accumulation, |
(parameter setting, initialization, accumulation, |
| 713 |
final evaluation), are controlled by the package {\it pkg/cost}. |
final evaluation), are controlled by the package {\it pkg/cost}. |
| 714 |
|
|
|
\begin{figure}[h!] |
|
| 715 |
\input{part5/doc_cost_flow} |
\input{part5/doc_cost_flow} |
|
\caption{~} |
|
|
\label{fig:costflow} |
|
|
\end{figure} |
|
| 716 |
|
|
| 717 |
\subsubsection{genmake and CPP options} |
\subsubsection{genmake and CPP options} |
| 718 |
% |
% |
| 754 |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
| 755 |
(file {\it CPP\_OPTIONS.h}). |
(file {\it CPP\_OPTIONS.h}). |
| 756 |
|
|
| 757 |
\subsubsection{Initialisation} |
\subsubsection{Initialization} |
| 758 |
% |
% |
| 759 |
The initialisation of the {\it cost} package is readily enabled |
The initialization of the {\it cost} package is readily enabled |
| 760 |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
| 761 |
% |
% |
| 762 |
\begin{itemize} |
\begin{itemize} |
| 787 |
} |
} |
| 788 |
\\ |
\\ |
| 789 |
This S/R |
This S/R |
| 790 |
initialises the different cost function contributions. |
initializes the different cost function contributions. |
| 791 |
The contribtion for the present example is {\bf objf\_tracer} |
The contribution for the present example is {\bf objf\_tracer} |
| 792 |
which is defined on each tile (bi,bj). |
which is defined on each tile (bi,bj). |
| 793 |
% |
% |
| 794 |
\end{itemize} |
\end{itemize} |
| 843 |
|
|
| 844 |
%%%% \end{document} |
%%%% \end{document} |
| 845 |
|
|
|
\begin{figure} |
|
| 846 |
\input{part5/doc_ad_the_main} |
\input{part5/doc_ad_the_main} |
|
\caption{~} |
|
|
\label{fig:adthemain} |
|
|
\end{figure} |
|
| 847 |
|
|
| 848 |
\subsection{The control variables (independent variables) |
\subsection{The control variables (independent variables) |
| 849 |
\label{section_ctrl}} |
\label{section_ctrl}} |
| 860 |
active variables are written and from which active variables |
active variables are written and from which active variables |
| 861 |
are read are called {\sf active files}. |
are read are called {\sf active files}. |
| 862 |
All aspects relevant to the treatment of the control variables |
All aspects relevant to the treatment of the control variables |
| 863 |
(parameter setting, initialisation, perturbation) |
(parameter setting, initialization, perturbation) |
| 864 |
are controled by the package {\it pkg/ctrl}. |
are controlled by the package {\it pkg/ctrl}. |
| 865 |
|
|
|
\begin{figure}[h!] |
|
| 866 |
\input{part5/doc_ctrl_flow} |
\input{part5/doc_ctrl_flow} |
|
\caption{~} |
|
|
\label{fig:ctrlflow} |
|
|
\end{figure} |
|
| 867 |
|
|
| 868 |
\subsubsection{genmake and CPP options} |
\subsubsection{genmake and CPP options} |
| 869 |
% |
% |
| 883 |
Each control variable is enabled via its own CPP option |
Each control variable is enabled via its own CPP option |
| 884 |
in {\it ECCO\_CPPOPTIONS.h}. |
in {\it ECCO\_CPPOPTIONS.h}. |
| 885 |
|
|
| 886 |
\subsubsection{Initialisation} |
\subsubsection{Initialization} |
| 887 |
% |
% |
| 888 |
\begin{itemize} |
\begin{itemize} |
| 889 |
% |
% |
| 923 |
variables in the MITGCM need to be addressed. |
variables in the MITGCM need to be addressed. |
| 924 |
First, in order to save memory, the control variable arrays |
First, in order to save memory, the control variable arrays |
| 925 |
are not kept in memory, but rather read from file and added |
are not kept in memory, but rather read from file and added |
| 926 |
to the initial fields during the model initialisation phase. |
to the initial fields during the model initialization phase. |
| 927 |
Similarly, the corresponding adjoint fields which represent |
Similarly, the corresponding adjoint fields which represent |
| 928 |
the gradient of the cost function w.r.t. the control variables |
the gradient of the cost function w.r.t. the control variables |
| 929 |
are written to file at the end of the adjoint integration. |
are written to file at the end of the adjoint integration. |
| 1003 |
and an 'active read' routine of the adjoint support |
and an 'active read' routine of the adjoint support |
| 1004 |
package {\it pkg/autodiff} is invoked. |
package {\it pkg/autodiff} is invoked. |
| 1005 |
The read-procedure is tagged with the variable |
The read-procedure is tagged with the variable |
| 1006 |
{\bf xx\_tr1\_dummy} enabbling TAMC to recognize the |
{\bf xx\_tr1\_dummy} enabling TAMC to recognize the |
| 1007 |
initialisation of the perturbation. |
initialization of the perturbation. |
| 1008 |
The modified call of TAMC thus reads |
The modified call of TAMC thus reads |
| 1009 |
% |
% |
| 1010 |
\begin{verbatim} |
\begin{verbatim} |
| 1115 |
\begin{itemize} |
\begin{itemize} |
| 1116 |
% |
% |
| 1117 |
\item {\bf vector\_ctrl}: the control vector \\ |
\item {\bf vector\_ctrl}: the control vector \\ |
| 1118 |
At the very beginning of the model initialisation, |
At the very beginning of the model initialization, |
| 1119 |
the updated compressed control vector is read (or initialised) |
the updated compressed control vector is read (or initialised) |
| 1120 |
and distributed to 2-dim. and 3-dim. control variable fields. |
and distributed to 2-dim. and 3-dim. control variable fields. |
| 1121 |
% |
% |
| 1183 |
with the value of the cost function itself $ {\cal J}(u_{[k]}) $ |
with the value of the cost function itself $ {\cal J}(u_{[k]}) $ |
| 1184 |
at iteration step $ k $ serve |
at iteration step $ k $ serve |
| 1185 |
as input to a minimization routine (e.g. quasi-Newton method, |
as input to a minimization routine (e.g. quasi-Newton method, |
| 1186 |
conjugate gradient, ... \cite{gil_lem:89}) |
conjugate gradient, ... \cite{gil-lem:89}) |
| 1187 |
to compute an update in the |
to compute an update in the |
| 1188 |
control variable for iteration step $k+1$ |
control variable for iteration step $k+1$ |
| 1189 |
\[ |
\[ |
| 1314 |
Finally, {\it ctrl\_pack} collects all adjoint files |
Finally, {\it ctrl\_pack} collects all adjoint files |
| 1315 |
and writes them to the compressed vector file |
and writes them to the compressed vector file |
| 1316 |
{\bf vector\_grad\_$<$k$>$}. |
{\bf vector\_grad\_$<$k$>$}. |
|
|
|
|
\subsection{TLM and ADM generation via TAMC} |
|
|
|
|
|
|
|
|
|
|
|
\subsection{Flow directives and adjoint support routines \label{section_flowdir}} |
|
|
|
|
|
\subsection{Store directives and checkpointing \label{section_checkpointing}} |
|
|
|
|
|
\subsection{Gradient checks \label{section_grdchk}} |
|
|
|
|
|
\subsection{Second derivative generation via TAMC} |
|
|
|
|
|
\section{Example of adjoint code} |
|
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch