21 |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
22 |
(Transformation of Algorithms in Fortran), developed |
(Transformation of Algorithms in Fortran), developed |
23 |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
24 |
The first application of the adjoint of the MITGCM for senistivity |
The first application of the adjoint of the MITGCM for sensitivity |
25 |
studies has been published by \cite{maro-eta:99}. |
studies has been published by \cite{maro-eta:99}. |
26 |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
27 |
for ocean state estimation studies. |
for ocean state estimation studies. |
52 |
such as forcing functions) to the $n$-dimensional space |
such as forcing functions) to the $n$-dimensional space |
53 |
$V \subset I\!\!R^n$ of |
$V \subset I\!\!R^n$ of |
54 |
model output variable $\vec{v}=(v_1,\ldots,v_n)$ |
model output variable $\vec{v}=(v_1,\ldots,v_n)$ |
55 |
(model state, model diagnostcs, objective function, ...) |
(model state, model diagnostics, objective function, ...) |
56 |
under consideration, |
under consideration, |
57 |
% |
% |
58 |
\begin{equation} |
\begin{equation} |
220 |
starting at step 0 and moving up to step $\Lambda$, with intermediate |
starting at step 0 and moving up to step $\Lambda$, with intermediate |
221 |
${\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}$ and final |
${\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}$ and final |
222 |
${\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}$. |
${\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}$. |
223 |
Let ${\cal J}$ be a cost funciton which explicitly depends on the |
Let ${\cal J}$ be a cost function which explicitly depends on the |
224 |
final state $\vec{v}$ only |
final state $\vec{v}$ only |
225 |
(this restriction is for clarity reasons only). |
(this restriction is for clarity reasons only). |
226 |
% |
% |
301 |
are the Lagrange multipliers of the model equations which determine |
are the Lagrange multipliers of the model equations which determine |
302 |
$ \vec{v}^{(\lambda)}$. |
$ \vec{v}^{(\lambda)}$. |
303 |
|
|
304 |
In coponents, eq. (\ref{adjoint}) reads as follows. |
In components, eq. (\ref{adjoint}) reads as follows. |
305 |
Let |
Let |
306 |
\[ |
\[ |
307 |
\begin{array}{rclcrcl} |
\begin{array}{rclcrcl} |
322 |
\end{array} |
\end{array} |
323 |
\] |
\] |
324 |
denote the perturbations in $\vec{u}$ and $\vec{v}$, respectively, |
denote the perturbations in $\vec{u}$ and $\vec{v}$, respectively, |
325 |
and their adjoint varaiables; |
and their adjoint variables; |
326 |
further |
further |
327 |
\[ |
\[ |
328 |
M \, = \, \left( |
M \, = \, \left( |
468 |
{\it all} intermediate states $ \vec{v}^{(\lambda)} $) are sought. |
{\it all} intermediate states $ \vec{v}^{(\lambda)} $) are sought. |
469 |
In order to be able to solve for each component of the gradient |
In order to be able to solve for each component of the gradient |
470 |
$ \partial {\cal J} / \partial u_{i} $ in (\ref{forward}) |
$ \partial {\cal J} / \partial u_{i} $ in (\ref{forward}) |
471 |
a forward calulation has to be performed for each component seperately, |
a forward calculation has to be performed for each component separately, |
472 |
i.e. $ \delta \vec{u} = \delta u_{i} {\vec{e}_{i}} $ |
i.e. $ \delta \vec{u} = \delta u_{i} {\vec{e}_{i}} $ |
473 |
for the $i$-th forward calculation. |
for the $i$-th forward calculation. |
474 |
Then, (\ref{forward}) represents the |
Then, (\ref{forward}) represents the |
487 |
\nabla_u {\cal J}^T \cdot \delta \vec{J} |
\nabla_u {\cal J}^T \cdot \delta \vec{J} |
488 |
\] |
\] |
489 |
where now $ \delta \vec{J} \in I\!\!R^l $ is a vector of |
where now $ \delta \vec{J} \in I\!\!R^l $ is a vector of |
490 |
dimenison $ l $. |
dimension $ l $. |
491 |
In this case $ l $ reverse simulations have to be performed |
In this case $ l $ reverse simulations have to be performed |
492 |
for each $ \delta J_{k}, \,\, k = 1, \ldots, l $. |
for each $ \delta J_{k}, \,\, k = 1, \ldots, l $. |
493 |
Then, the reverse mode is more efficient as long as |
Then, the reverse mode is more efficient as long as |
494 |
$ l < n $, otherwise the forward mode is preferable. |
$ l < n $, otherwise the forward mode is preferable. |
495 |
Stricly, the reverse mode is called adjoint mode only for |
Strictly, the reverse mode is called adjoint mode only for |
496 |
$ l = 1 $. |
$ l = 1 $. |
497 |
|
|
498 |
A detailed analysis of the underlying numerical operations |
A detailed analysis of the underlying numerical operations |
607 |
[i.e. every hour $ i=0,...,5$ corresponding to |
[i.e. every hour $ i=0,...,5$ corresponding to |
608 |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
609 |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
610 |
and the model state of all peceeding timesteps along the last |
and the model state of all proceeding timesteps along the last |
611 |
sub-subsections are available, enabling integration backwards |
sub-subsections are available, enabling integration backwards |
612 |
in time along the last sub-subsection. |
in time along the last sub-subsection. |
613 |
Thus, the adjoint can be computed along this last |
Thus, the adjoint can be computed along this last |
683 |
|
|
684 |
\subsection{Overview of the experiment} |
\subsection{Overview of the experiment} |
685 |
|
|
686 |
We describe an adjoint sensitivity analysis of outgassing from |
We describe an adjoint sensitivity analysis of out-gassing from |
687 |
the ocean into the atmosphere of a carbon-like tracer injected |
the ocean into the atmosphere of a carbon-like tracer injected |
688 |
into the ocean interior (see \cite{hil-eta:01}). |
into the ocean interior (see \cite{hil-eta:01}). |
689 |
|
|
691 |
|
|
692 |
For this work the MITGCM was augmented with a thermodynamically |
For this work the MITGCM was augmented with a thermodynamically |
693 |
inactive tracer, $C$. Tracer residing in the ocean |
inactive tracer, $C$. Tracer residing in the ocean |
694 |
model surface layer is outgassed according to a relaxation time scale, |
model surface layer is out-gassed according to a relaxation time scale, |
695 |
$\mu$. Within the ocean interior, the tracer is passively advected |
$\mu$. Within the ocean interior, the tracer is passively advected |
696 |
by the ocean model currents. The full equation for the time evolution |
by the ocean model currents. The full equation for the time evolution |
697 |
% |
% |
711 |
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
712 |
fluid is locally statically unstable. |
fluid is locally statically unstable. |
713 |
|
|
714 |
The outgassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
The out-gassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
715 |
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
716 |
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
717 |
is valid as a prognostic equation for small perturbations in oceanic |
is valid as a prognostic equation for small perturbations in oceanic |
718 |
carbon concentrations. This configuration provides a |
carbon concentrations. This configuration provides a |
719 |
powerful tool for examining the impact of large-scale ocean circulation |
powerful tool for examining the impact of large-scale ocean circulation |
720 |
on $ CO_2 $ outgassing due to interior injections. |
on $ CO_2 $ out-gassing due to interior injections. |
721 |
As source we choose a constant in time injection of |
As source we choose a constant in time injection of |
722 |
$ S = 1 \,\, {\rm mol / s}$. |
$ S = 1 \,\, {\rm mol / s}$. |
723 |
|
|
728 |
geography and bathymetry. Twenty vertical layers are used with |
geography and bathymetry. Twenty vertical layers are used with |
729 |
vertical spacing ranging |
vertical spacing ranging |
730 |
from 50 m near the surface to 815 m at depth. |
from 50 m near the surface to 815 m at depth. |
731 |
Driven to steady-state by climatalogical wind-stress, heat and |
Driven to steady-state by climatological wind-stress, heat and |
732 |
fresh-water forcing the model reproduces well known large-scale |
fresh-water forcing the model reproduces well known large-scale |
733 |
features of the ocean general circulation. |
features of the ocean general circulation. |
734 |
|
|
735 |
\subsubsection{Outgassing cost function} |
\subsubsection{Out-gassing cost function} |
736 |
|
|
737 |
To quantify and understand outgassing due to injections of $C$ |
To quantify and understand out-gassing due to injections of $C$ |
738 |
in eqn. (\ref{carbon_ddt}), |
in eqn. (\ref{carbon_ddt}), |
739 |
we define a cost function $ {\cal J} $ that measures the total amount of |
we define a cost function $ {\cal J} $ that measures the total amount of |
740 |
tracer outgassed at each timestep: |
tracer out-gassed at each timestep: |
741 |
% |
% |
742 |
\begin{equation} |
\begin{equation} |
743 |
\label{cost_tracer} |
\label{cost_tracer} |
744 |
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
745 |
\end{equation} |
\end{equation} |
746 |
% |
% |
747 |
Equation(\ref{cost_tracer}) integrates the outgassing term, $\mu C$, |
Equation(\ref{cost_tracer}) integrates the out-gassing term, $\mu C$, |
748 |
from (\ref{carbon_ddt}) |
from (\ref{carbon_ddt}) |
749 |
over the entire ocean surface area, $A$, and accumulates it |
over the entire ocean surface area, $A$, and accumulates it |
750 |
up to time $T$. |
up to time $T$. |
751 |
Physically, ${\cal J}$ can be thought of as representing the amount of |
Physically, ${\cal J}$ can be thought of as representing the amount of |
752 |
$CO_2$ that our model predicts would be outgassed following an |
$CO_2$ that our model predicts would be out-gassed following an |
753 |
injection at rate $S$. |
injection at rate $S$. |
754 |
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
755 |
$\frac{\partial {\cal J}}{\partial S}$, |
$\frac{\partial {\cal J}}{\partial S}$, |
756 |
can be used to identify regions from which circulation |
can be used to identify regions from which circulation |
757 |
would cause $CO_2$ to rapidly outgas following injection |
would cause $CO_2$ to rapidly out-gas following injection |
758 |
and regions in which $CO_2$ injections would remain effectively |
and regions in which $CO_2$ injections would remain effectively |
759 |
sequesterd within the ocean. |
sequestered within the ocean. |
760 |
|
|
761 |
\subsection{Code configuration} |
\subsection{Code configuration} |
762 |
|
|
763 |
The model configuration for this experiment resides under the |
The model configuration for this experiment resides under the |
764 |
directory {\it verification/carbon/}. |
directory {\it verification/carbon/}. |
765 |
The code customisation routines are in {\it verification/carbon/code/}: |
The code customization routines are in {\it verification/carbon/code/}: |
766 |
% |
% |
767 |
\begin{itemize} |
\begin{itemize} |
768 |
% |
% |
830 |
\end{itemize} |
\end{itemize} |
831 |
% |
% |
832 |
|
|
833 |
Below we describe the customisations of this files which are |
Below we describe the customizations of this files which are |
834 |
specific to this experiment. |
specific to this experiment. |
835 |
|
|
836 |
\subsubsection{File {\it .genmakerc}} |
\subsubsection{File {\it .genmakerc}} |
871 |
model integration. \\ |
model integration. \\ |
872 |
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
873 |
This flag enables the inclusion of some AD-related fields |
This flag enables the inclusion of some AD-related fields |
874 |
concerning initialisation, link between control variables |
concerning initialization, link between control variables |
875 |
and forward model variables, and the call to the top-level |
and forward model variables, and the call to the top-level |
876 |
forward/adjoint subroutine {\it adthe\_main\_loop} |
forward/adjoint subroutine {\it adthe\_main\_loop} |
877 |
instead of {\it the\_main\_loop}. \\ |
instead of {\it the\_main\_loop}. \\ |
907 |
(see Section \ref{???}). |
(see Section \ref{???}). |
908 |
In the present example a 3-level checkpointing is implemented. |
In the present example a 3-level checkpointing is implemented. |
909 |
The code contains the relevant store directives, common block |
The code contains the relevant store directives, common block |
910 |
and tape initialisations, storing key computation, |
and tape initializations, storing key computation, |
911 |
and loop index handling. |
and loop index handling. |
912 |
The checkpointing length at each level is defined in |
The checkpointing length at each level is defined in |
913 |
file {\it tamc.h}, cf. below. |
file {\it tamc.h}, cf. below. |
914 |
% |
% |
915 |
\item Cost function package: {\it pkg/cost/} \\ |
\item Cost function package: {\it pkg/cost/} \\ |
916 |
This package contains all relevant routines for |
This package contains all relevant routines for |
917 |
initialising, accumulating and finalizing the cost function |
initializing, accumulating and finalizing the cost function |
918 |
(see Section \ref{???}). \\ |
(see Section \ref{???}). \\ |
919 |
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
920 |
enables all general aspects of the cost function handling, |
enables all general aspects of the cost function handling, |
921 |
in particular the hooks in the foorward code for |
in particular the hooks in the forward code for |
922 |
initialising, accumulating and finalizing the cost function. \\ |
initializing, accumulating and finalizing the cost function. \\ |
923 |
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
924 |
includes the call to the cost function for this |
includes the call to the cost function for this |
925 |
particular experiment, eqn. (\ref{cost_tracer}). |
particular experiment, eqn. (\ref{cost_tracer}). |
958 |
\hspace*{4ex} {\tt sNx = 90} \\ |
\hspace*{4ex} {\tt sNx = 90} \\ |
959 |
\hspace*{4ex} {\tt sNy = 40} \\ |
\hspace*{4ex} {\tt sNy = 40} \\ |
960 |
\hspace*{4ex} {\tt Nr = 20} \\ |
\hspace*{4ex} {\tt Nr = 20} \\ |
961 |
It correpsponds to a single-tile/single-processor setup: |
It corresponds to a single-tile/single-processor setup: |
962 |
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
963 |
with standard overlap dimensioning |
with standard overlap dimensioning |
964 |
{\tt OLx = OLy = 3}. |
{\tt OLx = OLy = 3}. |
1037 |
|
|
1038 |
\subsubsection{File {\it makefile}} |
\subsubsection{File {\it makefile}} |
1039 |
|
|
1040 |
This file contains all relevant paramter flags and |
This file contains all relevant parameter flags and |
1041 |
lists to run TAMC or TAF. |
lists to run TAMC or TAF. |
1042 |
It is assumed that TAMC is available to you, either locally, |
It is assumed that TAMC is available to you, either locally, |
1043 |
being installed on your network, or remotely through the 'TAMC Utility'. |
being installed on your network, or remotely through the 'TAMC Utility'. |
1078 |
{\tt <file names>} refers to the list of files {\it .f} which are to be |
{\tt <file names>} refers to the list of files {\it .f} which are to be |
1079 |
analyzed by TAMC. This list is generally smaller than the full list |
analyzed by TAMC. This list is generally smaller than the full list |
1080 |
of code to be compiled. The files not contained are either |
of code to be compiled. The files not contained are either |
1081 |
above the top-level routine (some initialisations), or are |
above the top-level routine (some initializations), or are |
1082 |
deliberately hidden from TAMC, either because hand-written |
deliberately hidden from TAMC, either because hand-written |
1083 |
adjoint routines exist, or the routines must not (or don't have to) |
adjoint routines exist, or the routines must not (or don't have to) |
1084 |
be differentiated. For each routine which is part of the flow tree |
be differentiated. For each routine which is part of the flow tree |
1211 |
The multi-threading capability of the MITGCM requires a slight |
The multi-threading capability of the MITGCM requires a slight |
1212 |
change in the parameter list of some routines that are related to |
change in the parameter list of some routines that are related to |
1213 |
to active file handling. |
to active file handling. |
1214 |
This postprocessing invokes the sed script {\it adjoint\_ecco\_sed.com} |
This post-processing invokes the sed script {\it adjoint\_ecco\_sed.com} |
1215 |
to insert the threading counter {\bf myThId} into the parameter list |
to insert the threading counter {\bf myThId} into the parameter list |
1216 |
of those subroutines. |
of those subroutines. |
1217 |
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
1218 |
and appended to the file {\it adjoint\_model.F}. |
and appended to the file {\it adjoint\_model.F}. |
1219 |
This concludes the adjoint codel generation. |
This concludes the adjoint code generation. |
1220 |
% |
% |
1221 |
\item |
\item |
1222 |
{\tt cd ../bin} \\ |
{\tt cd ../bin} \\ |
1296 |
The input is referred to as the |
The input is referred to as the |
1297 |
{\sf independent variables} or {\sf control variables}. |
{\sf independent variables} or {\sf control variables}. |
1298 |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
1299 |
(parameter setting, initialisation, accumulation, |
(parameter setting, initialization, accumulation, |
1300 |
final evaluation), are controlled by the package {\it pkg/cost}. |
final evaluation), are controlled by the package {\it pkg/cost}. |
1301 |
|
|
1302 |
\begin{figure}[h!] |
\begin{figure}[h!] |
1345 |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
1346 |
(file {\it CPP\_OPTIONS.h}). |
(file {\it CPP\_OPTIONS.h}). |
1347 |
|
|
1348 |
\subsubsection{Initialisation} |
\subsubsection{Initialization} |
1349 |
% |
% |
1350 |
The initialisation of the {\it cost} package is readily enabled |
The initialization of the {\it cost} package is readily enabled |
1351 |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
1352 |
% |
% |
1353 |
\begin{itemize} |
\begin{itemize} |
1378 |
} |
} |
1379 |
\\ |
\\ |
1380 |
This S/R |
This S/R |
1381 |
initialises the different cost function contributions. |
initializes the different cost function contributions. |
1382 |
The contribtion for the present example is {\bf objf\_tracer} |
The contribution for the present example is {\bf objf\_tracer} |
1383 |
which is defined on each tile (bi,bj). |
which is defined on each tile (bi,bj). |
1384 |
% |
% |
1385 |
\end{itemize} |
\end{itemize} |
1455 |
active variables are written and from which active variables |
active variables are written and from which active variables |
1456 |
are read are called {\sf active files}. |
are read are called {\sf active files}. |
1457 |
All aspects relevant to the treatment of the control variables |
All aspects relevant to the treatment of the control variables |
1458 |
(parameter setting, initialisation, perturbation) |
(parameter setting, initialization, perturbation) |
1459 |
are controled by the package {\it pkg/ctrl}. |
are controlled by the package {\it pkg/ctrl}. |
1460 |
|
|
1461 |
\begin{figure}[h!] |
\begin{figure}[h!] |
1462 |
\input{part5/doc_ctrl_flow} |
\input{part5/doc_ctrl_flow} |
1482 |
Each control variable is enabled via its own CPP option |
Each control variable is enabled via its own CPP option |
1483 |
in {\it ECCO\_CPPOPTIONS.h}. |
in {\it ECCO\_CPPOPTIONS.h}. |
1484 |
|
|
1485 |
\subsubsection{Initialisation} |
\subsubsection{Initialization} |
1486 |
% |
% |
1487 |
\begin{itemize} |
\begin{itemize} |
1488 |
% |
% |
1522 |
variables in the MITGCM need to be addressed. |
variables in the MITGCM need to be addressed. |
1523 |
First, in order to save memory, the control variable arrays |
First, in order to save memory, the control variable arrays |
1524 |
are not kept in memory, but rather read from file and added |
are not kept in memory, but rather read from file and added |
1525 |
to the initial fields during the model initialisation phase. |
to the initial fields during the model initialization phase. |
1526 |
Similarly, the corresponding adjoint fields which represent |
Similarly, the corresponding adjoint fields which represent |
1527 |
the gradient of the cost function w.r.t. the control variables |
the gradient of the cost function w.r.t. the control variables |
1528 |
are written to file at the end of the adjoint integration. |
are written to file at the end of the adjoint integration. |
1602 |
and an 'active read' routine of the adjoint support |
and an 'active read' routine of the adjoint support |
1603 |
package {\it pkg/autodiff} is invoked. |
package {\it pkg/autodiff} is invoked. |
1604 |
The read-procedure is tagged with the variable |
The read-procedure is tagged with the variable |
1605 |
{\bf xx\_tr1\_dummy} enabbling TAMC to recognize the |
{\bf xx\_tr1\_dummy} enabling TAMC to recognize the |
1606 |
initialisation of the perturbation. |
initialization of the perturbation. |
1607 |
The modified call of TAMC thus reads |
The modified call of TAMC thus reads |
1608 |
% |
% |
1609 |
\begin{verbatim} |
\begin{verbatim} |
1714 |
\begin{itemize} |
\begin{itemize} |
1715 |
% |
% |
1716 |
\item {\bf vector\_ctrl}: the control vector \\ |
\item {\bf vector\_ctrl}: the control vector \\ |
1717 |
At the very beginning of the model initialisation, |
At the very beginning of the model initialization, |
1718 |
the updated compressed control vector is read (or initialised) |
the updated compressed control vector is read (or initialised) |
1719 |
and distributed to 2-dim. and 3-dim. control variable fields. |
and distributed to 2-dim. and 3-dim. control variable fields. |
1720 |
% |
% |