21 |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
22 |
(Transformation of Algorithms in Fortran), developed |
(Transformation of Algorithms in Fortran), developed |
23 |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
24 |
The first application of the adjoint of the MITGCM for senistivity |
The first application of the adjoint of the MITGCM for sensitivity |
25 |
studies has been published by \cite{maro-eta:99}. |
studies has been published by \cite{maro-eta:99}. |
26 |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
27 |
for ocean state estimation studies. |
for ocean state estimation studies. |
52 |
such as forcing functions) to the $n$-dimensional space |
such as forcing functions) to the $n$-dimensional space |
53 |
$V \subset I\!\!R^n$ of |
$V \subset I\!\!R^n$ of |
54 |
model output variable $\vec{v}=(v_1,\ldots,v_n)$ |
model output variable $\vec{v}=(v_1,\ldots,v_n)$ |
55 |
(model state, model diagnostcs, objective function, ...) |
(model state, model diagnostics, objective function, ...) |
56 |
under consideration, |
under consideration, |
57 |
% |
% |
58 |
\begin{equation} |
\begin{equation} |
220 |
starting at step 0 and moving up to step $\Lambda$, with intermediate |
starting at step 0 and moving up to step $\Lambda$, with intermediate |
221 |
${\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}$ and final |
${\cal M}_{\lambda} (\vec{u}) = \vec{v}^{(\lambda+1)}$ and final |
222 |
${\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}$. |
${\cal M}_{\Lambda} (\vec{u}) = \vec{v}^{(\Lambda+1)} = \vec{v}$. |
223 |
Let ${\cal J}$ be a cost funciton which explicitly depends on the |
Let ${\cal J}$ be a cost function which explicitly depends on the |
224 |
final state $\vec{v}$ only |
final state $\vec{v}$ only |
225 |
(this restriction is for clarity reasons only). |
(this restriction is for clarity reasons only). |
226 |
% |
% |
301 |
are the Lagrange multipliers of the model equations which determine |
are the Lagrange multipliers of the model equations which determine |
302 |
$ \vec{v}^{(\lambda)}$. |
$ \vec{v}^{(\lambda)}$. |
303 |
|
|
304 |
In coponents, eq. (\ref{adjoint}) reads as follows. |
In components, eq. (\ref{adjoint}) reads as follows. |
305 |
Let |
Let |
306 |
\[ |
\[ |
307 |
\begin{array}{rclcrcl} |
\begin{array}{rclcrcl} |
322 |
\end{array} |
\end{array} |
323 |
\] |
\] |
324 |
denote the perturbations in $\vec{u}$ and $\vec{v}$, respectively, |
denote the perturbations in $\vec{u}$ and $\vec{v}$, respectively, |
325 |
and their adjoint varaiables; |
and their adjoint variables; |
326 |
further |
further |
327 |
\[ |
\[ |
328 |
M \, = \, \left( |
M \, = \, \left( |
468 |
{\it all} intermediate states $ \vec{v}^{(\lambda)} $) are sought. |
{\it all} intermediate states $ \vec{v}^{(\lambda)} $) are sought. |
469 |
In order to be able to solve for each component of the gradient |
In order to be able to solve for each component of the gradient |
470 |
$ \partial {\cal J} / \partial u_{i} $ in (\ref{forward}) |
$ \partial {\cal J} / \partial u_{i} $ in (\ref{forward}) |
471 |
a forward calulation has to be performed for each component seperately, |
a forward calculation has to be performed for each component separately, |
472 |
i.e. $ \delta \vec{u} = \delta u_{i} {\vec{e}_{i}} $ |
i.e. $ \delta \vec{u} = \delta u_{i} {\vec{e}_{i}} $ |
473 |
for the $i$-th forward calculation. |
for the $i$-th forward calculation. |
474 |
Then, (\ref{forward}) represents the |
Then, (\ref{forward}) represents the |
487 |
\nabla_u {\cal J}^T \cdot \delta \vec{J} |
\nabla_u {\cal J}^T \cdot \delta \vec{J} |
488 |
\] |
\] |
489 |
where now $ \delta \vec{J} \in I\!\!R^l $ is a vector of |
where now $ \delta \vec{J} \in I\!\!R^l $ is a vector of |
490 |
dimenison $ l $. |
dimension $ l $. |
491 |
In this case $ l $ reverse simulations have to be performed |
In this case $ l $ reverse simulations have to be performed |
492 |
for each $ \delta J_{k}, \,\, k = 1, \ldots, l $. |
for each $ \delta J_{k}, \,\, k = 1, \ldots, l $. |
493 |
Then, the reverse mode is more efficient as long as |
Then, the reverse mode is more efficient as long as |
494 |
$ l < n $, otherwise the forward mode is preferable. |
$ l < n $, otherwise the forward mode is preferable. |
495 |
Stricly, the reverse mode is called adjoint mode only for |
Strictly, the reverse mode is called adjoint mode only for |
496 |
$ l = 1 $. |
$ l = 1 $. |
497 |
|
|
498 |
A detailed analysis of the underlying numerical operations |
A detailed analysis of the underlying numerical operations |
570 |
A method to balance the amount of recomputations vs. |
A method to balance the amount of recomputations vs. |
571 |
storage requirements is called {\sf checkpointing} |
storage requirements is called {\sf checkpointing} |
572 |
(e.g. \cite{res-eta:98}). |
(e.g. \cite{res-eta:98}). |
573 |
It is depicted in \reffig{3levelcheck} for a 3-level checkpointing |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
574 |
[as an example, we give explicit numbers for a 3-day |
[as an example, we give explicit numbers for a 3-day |
575 |
integration with a 1-hourly timestep in square brackets]. |
integration with a 1-hourly timestep in square brackets]. |
576 |
\begin{itemize} |
\begin{itemize} |
607 |
[i.e. every hour $ i=0,...,5$ corresponding to |
[i.e. every hour $ i=0,...,5$ corresponding to |
608 |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
609 |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
610 |
and the model state of all peceeding timesteps along the last |
and the model state of all proceeding timesteps along the last |
611 |
sub-subsections are available, enabling integration backwards |
sub-subsections are available, enabling integration backwards |
612 |
in time along the last sub-subsection. |
in time along the last sub-subsection. |
613 |
Thus, the adjoint can be computed along this last |
Thus, the adjoint can be computed along this last |
637 |
on the computing resources available. |
on the computing resources available. |
638 |
|
|
639 |
\begin{figure}[t!] |
\begin{figure}[t!] |
640 |
\centering |
\begin{center} |
641 |
%\psdraft |
%\psdraft |
642 |
\psfrag{v_k1^lev3}{\mathinfigure{v_{k_{1}^{lev3}}}} |
%\psfrag{v_k1^lev3}{\mathinfigure{v_{k_{1}^{lev3}}}} |
643 |
\psfrag{v_kn-1^lev3}{\mathinfigure{v_{k_{n-1}^{lev3}}}} |
%\psfrag{v_kn-1^lev3}{\mathinfigure{v_{k_{n-1}^{lev3}}}} |
644 |
\psfrag{v_kn^lev3}{\mathinfigure{v_{k_{n}^{lev3}}}} |
%\psfrag{v_kn^lev3}{\mathinfigure{v_{k_{n}^{lev3}}}} |
645 |
\psfrag{v_k1^lev2}{\mathinfigure{v_{k_{1}^{lev2}}}} |
%\psfrag{v_k1^lev2}{\mathinfigure{v_{k_{1}^{lev2}}}} |
646 |
\psfrag{v_kn-1^lev2}{\mathinfigure{v_{k_{n-1}^{lev2}}}} |
%\psfrag{v_kn-1^lev2}{\mathinfigure{v_{k_{n-1}^{lev2}}}} |
647 |
\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
%\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
648 |
\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
%\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
649 |
\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
%\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
650 |
\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
%\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
651 |
|
\resizebox{5.5in}{!}{\includegraphics{part5/checkpointing.eps}} |
652 |
%\psfull |
%\psfull |
653 |
\caption |
\end{center} |
654 |
{Schematic view of intermediate dump and restart for |
\caption{ |
655 |
|
Schematic view of intermediate dump and restart for |
656 |
3-level checkpointing.} |
3-level checkpointing.} |
657 |
\label{fig:3levelcheck} |
\label{fig:3levelcheck} |
658 |
\end{figure} |
\end{figure} |
679 |
We take as an example the sensitivity of carbon sequestration |
We take as an example the sensitivity of carbon sequestration |
680 |
in the ocean. |
in the ocean. |
681 |
The AD-relevant hooks in the code are sketched in |
The AD-relevant hooks in the code are sketched in |
682 |
\reffig{adthemodel}, \reffig{adthemain}. |
\ref{fig:adthemodel}, \ref{fig:adthemain}. |
683 |
|
|
684 |
\subsection{Overview of the experiment} |
\subsection{Overview of the experiment} |
685 |
|
|
686 |
We describe an adjoint sensitivity analysis of outgassing from |
We describe an adjoint sensitivity analysis of out-gassing from |
687 |
the ocean into the atmosphere of a carbon-like tracer injected |
the ocean into the atmosphere of a carbon-like tracer injected |
688 |
into the ocean interior (see \cite{hil-eta:01}). |
into the ocean interior (see \cite{hil-eta:01}). |
689 |
|
|
691 |
|
|
692 |
For this work the MITGCM was augmented with a thermodynamically |
For this work the MITGCM was augmented with a thermodynamically |
693 |
inactive tracer, $C$. Tracer residing in the ocean |
inactive tracer, $C$. Tracer residing in the ocean |
694 |
model surface layer is outgassed according to a relaxation time scale, |
model surface layer is out-gassed according to a relaxation time scale, |
695 |
$\mu$. Within the ocean interior, the tracer is passively advected |
$\mu$. Within the ocean interior, the tracer is passively advected |
696 |
by the ocean model currents. The full equation for the time evolution |
by the ocean model currents. The full equation for the time evolution |
697 |
% |
% |
711 |
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
712 |
fluid is locally statically unstable. |
fluid is locally statically unstable. |
713 |
|
|
714 |
The outgassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
The out-gassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
715 |
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
716 |
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
717 |
is valid as a prognostic equation for small perturbations in oceanic |
is valid as a prognostic equation for small perturbations in oceanic |
718 |
carbon concentrations. This configuration provides a |
carbon concentrations. This configuration provides a |
719 |
powerful tool for examining the impact of large-scale ocean circulation |
powerful tool for examining the impact of large-scale ocean circulation |
720 |
on $ CO_2 $ outgassing due to interior injections. |
on $ CO_2 $ out-gassing due to interior injections. |
721 |
As source we choose a constant in time injection of |
As source we choose a constant in time injection of |
722 |
$ S = 1 \,\, {\rm mol / s}$. |
$ S = 1 \,\, {\rm mol / s}$. |
723 |
|
|
728 |
geography and bathymetry. Twenty vertical layers are used with |
geography and bathymetry. Twenty vertical layers are used with |
729 |
vertical spacing ranging |
vertical spacing ranging |
730 |
from 50 m near the surface to 815 m at depth. |
from 50 m near the surface to 815 m at depth. |
731 |
Driven to steady-state by climatalogical wind-stress, heat and |
Driven to steady-state by climatological wind-stress, heat and |
732 |
fresh-water forcing the model reproduces well known large-scale |
fresh-water forcing the model reproduces well known large-scale |
733 |
features of the ocean general circulation. |
features of the ocean general circulation. |
734 |
|
|
735 |
\subsubsection{Outgassing cost function} |
\subsubsection{Out-gassing cost function} |
736 |
|
|
737 |
To quantify and understand outgassing due to injections of $C$ |
To quantify and understand out-gassing due to injections of $C$ |
738 |
in eqn. (\ref{carbon_ddt}), |
in eqn. (\ref{carbon_ddt}), |
739 |
we define a cost function $ {\cal J} $ that measures the total amount of |
we define a cost function $ {\cal J} $ that measures the total amount of |
740 |
tracer outgassed at each timestep: |
tracer out-gassed at each timestep: |
741 |
% |
% |
742 |
\begin{equation} |
\begin{equation} |
743 |
\label{cost_tracer} |
\label{cost_tracer} |
744 |
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
745 |
\end{equation} |
\end{equation} |
746 |
% |
% |
747 |
Equation(\ref{cost_tracer}) integrates the outgassing term, $\mu C$, |
Equation(\ref{cost_tracer}) integrates the out-gassing term, $\mu C$, |
748 |
from (\ref{carbon_ddt}) |
from (\ref{carbon_ddt}) |
749 |
over the entire ocean surface area, $A$, and accumulates it |
over the entire ocean surface area, $A$, and accumulates it |
750 |
up to time $T$. |
up to time $T$. |
751 |
Physically, ${\cal J}$ can be thought of as representing the amount of |
Physically, ${\cal J}$ can be thought of as representing the amount of |
752 |
$CO_2$ that our model predicts would be outgassed following an |
$CO_2$ that our model predicts would be out-gassed following an |
753 |
injection at rate $S$. |
injection at rate $S$. |
754 |
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
755 |
$\frac{\partial {\cal J}}{\partial S}$, |
$\frac{\partial {\cal J}}{\partial S}$, |
756 |
can be used to identify regions from which circulation |
can be used to identify regions from which circulation |
757 |
would cause $CO_2$ to rapidly outgas following injection |
would cause $CO_2$ to rapidly out-gas following injection |
758 |
and regions in which $CO_2$ injections would remain effectively |
and regions in which $CO_2$ injections would remain effectively |
759 |
sequesterd within the ocean. |
sequestered within the ocean. |
760 |
|
|
761 |
\subsection{Code configuration} |
\subsection{Code configuration} |
762 |
|
|
763 |
The model configuration for this experiment resides under the |
The model configuration for this experiment resides under the |
764 |
directory {\it verification/carbon/}. |
directory {\it verification/carbon/}. |
765 |
The code customisation routines are in {\it verification/carbon/code/}: |
The code customization routines are in {\it verification/carbon/code/}: |
766 |
% |
% |
767 |
\begin{itemize} |
\begin{itemize} |
768 |
% |
% |
830 |
\end{itemize} |
\end{itemize} |
831 |
% |
% |
832 |
|
|
833 |
Below we describe the customisations of this files which are |
Below we describe the customizations of this files which are |
834 |
specific to this experiment. |
specific to this experiment. |
835 |
|
|
836 |
\subsubsection{File {\it .genmakerc}} |
\subsubsection{File {\it .genmakerc}} |
871 |
model integration. \\ |
model integration. \\ |
872 |
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
873 |
This flag enables the inclusion of some AD-related fields |
This flag enables the inclusion of some AD-related fields |
874 |
concerning initialisation, link between control variables |
concerning initialization, link between control variables |
875 |
and forward model variables, and the call to the top-level |
and forward model variables, and the call to the top-level |
876 |
forward/adjoint subroutine {\it adthe\_main\_loop} |
forward/adjoint subroutine {\it adthe\_main\_loop} |
877 |
instead of {\it the\_main\_loop}. \\ |
instead of {\it the\_main\_loop}. \\ |
907 |
(see Section \ref{???}). |
(see Section \ref{???}). |
908 |
In the present example a 3-level checkpointing is implemented. |
In the present example a 3-level checkpointing is implemented. |
909 |
The code contains the relevant store directives, common block |
The code contains the relevant store directives, common block |
910 |
and tape initialisations, storing key computation, |
and tape initializations, storing key computation, |
911 |
and loop index handling. |
and loop index handling. |
912 |
The checkpointing length at each level is defined in |
The checkpointing length at each level is defined in |
913 |
file {\it tamc.h}, cf. below. |
file {\it tamc.h}, cf. below. |
914 |
% |
% |
915 |
\item Cost function package: {\it pkg/cost/} \\ |
\item Cost function package: {\it pkg/cost/} \\ |
916 |
This package contains all relevant routines for |
This package contains all relevant routines for |
917 |
initialising, accumulating and finalizing the cost function |
initializing, accumulating and finalizing the cost function |
918 |
(see Section \ref{???}). \\ |
(see Section \ref{???}). \\ |
919 |
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
920 |
enables all general aspects of the cost function handling, |
enables all general aspects of the cost function handling, |
921 |
in particular the hooks in the foorward code for |
in particular the hooks in the forward code for |
922 |
initialising, accumulating and finalizing the cost function. \\ |
initializing, accumulating and finalizing the cost function. \\ |
923 |
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
924 |
includes the call to the cost function for this |
includes the call to the cost function for this |
925 |
particular experiment, eqn. (\ref{cost_tracer}). |
particular experiment, eqn. (\ref{cost_tracer}). |
958 |
\hspace*{4ex} {\tt sNx = 90} \\ |
\hspace*{4ex} {\tt sNx = 90} \\ |
959 |
\hspace*{4ex} {\tt sNy = 40} \\ |
\hspace*{4ex} {\tt sNy = 40} \\ |
960 |
\hspace*{4ex} {\tt Nr = 20} \\ |
\hspace*{4ex} {\tt Nr = 20} \\ |
961 |
It correpsponds to a single-tile/single-processor setup: |
It corresponds to a single-tile/single-processor setup: |
962 |
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
963 |
with standard overlap dimensioning |
with standard overlap dimensioning |
964 |
{\tt OLx = OLy = 3}. |
{\tt OLx = OLy = 3}. |
1037 |
|
|
1038 |
\subsubsection{File {\it makefile}} |
\subsubsection{File {\it makefile}} |
1039 |
|
|
1040 |
This file contains all relevant paramter flags and |
This file contains all relevant parameter flags and |
1041 |
lists to run TAMC or TAF. |
lists to run TAMC or TAF. |
1042 |
It is assumed that TAMC is available to you, either locally, |
It is assumed that TAMC is available to you, either locally, |
1043 |
being installed on your network, or remotely through the 'TAMC Utility'. |
being installed on your network, or remotely through the 'TAMC Utility'. |
1078 |
{\tt <file names>} refers to the list of files {\it .f} which are to be |
{\tt <file names>} refers to the list of files {\it .f} which are to be |
1079 |
analyzed by TAMC. This list is generally smaller than the full list |
analyzed by TAMC. This list is generally smaller than the full list |
1080 |
of code to be compiled. The files not contained are either |
of code to be compiled. The files not contained are either |
1081 |
above the top-level routine (some initialisations), or are |
above the top-level routine (some initializations), or are |
1082 |
deliberately hidden from TAMC, either because hand-written |
deliberately hidden from TAMC, either because hand-written |
1083 |
adjoint routines exist, or the routines must not (or don't have to) |
adjoint routines exist, or the routines must not (or don't have to) |
1084 |
be differentiated. For each routine which is part of the flow tree |
be differentiated. For each routine which is part of the flow tree |
1093 |
\end{itemize} |
\end{itemize} |
1094 |
|
|
1095 |
|
|
1096 |
\subsubsection{File {\it data}} |
\subsubsection{The input parameter files} |
1097 |
|
|
1098 |
\subsubsection{File {\it data.cost}} |
\paragraph{File {\it data}} |
1099 |
|
|
1100 |
\subsubsection{File {\it data.ctrl}} |
\paragraph{File {\it data.cost}} |
1101 |
|
|
1102 |
\subsubsection{File {\it data.gmredi}} |
\paragraph{File {\it data.ctrl}} |
1103 |
|
|
1104 |
\subsubsection{File {\it data.grdchk}} |
\paragraph{File {\it data.gmredi}} |
1105 |
|
|
1106 |
\subsubsection{File {\it data.optim}} |
\paragraph{File {\it data.grdchk}} |
1107 |
|
|
1108 |
\subsubsection{File {\it data.pkg}} |
\paragraph{File {\it data.optim}} |
1109 |
|
|
1110 |
\subsubsection{File {\it eedata}} |
\paragraph{File {\it data.pkg}} |
1111 |
|
|
1112 |
\subsubsection{File {\it topog.bin}} |
\paragraph{File {\it eedata}} |
1113 |
|
|
1114 |
\subsubsection{File {\it windx.bin, windy.bin}} |
\paragraph{File {\it topog.bin}} |
1115 |
|
|
1116 |
\subsubsection{File {\it salt.bin, theta.bin}} |
\paragraph{File {\it windx.bin, windy.bin}} |
1117 |
|
|
1118 |
\subsubsection{File {\it SSS.bin, SST.bin}} |
\paragraph{File {\it salt.bin, theta.bin}} |
1119 |
|
|
1120 |
\subsubsection{File {\it pickup*}} |
\paragraph{File {\it SSS.bin, SST.bin}} |
1121 |
|
|
1122 |
\subsection{Compiling the model and its adjoint} |
\paragraph{File {\it pickup*}} |
1123 |
|
|
1124 |
|
\subsection{Compiling the model and its adjoint} |
1125 |
|
|
1126 |
|
The built process of the adjoint model is slightly more |
1127 |
|
complex than that of compiling the forward code. |
1128 |
|
The main reason is that the adjoint code generation requires |
1129 |
|
a specific list of routines that are to be differentiated |
1130 |
|
(as opposed to the automatic generation of a list of |
1131 |
|
files to be compiled by genmake). |
1132 |
|
This list excludes routines that don't have to be or must not be |
1133 |
|
differentiated. For some of the latter routines flow directives |
1134 |
|
may be necessary, a list of which has to be given as well. |
1135 |
|
For this reason, a separate {\it makefile} is currently |
1136 |
|
maintained in the directory {\tt adjoint/}. This |
1137 |
|
makefile is responsible for the adjoint code generation. |
1138 |
|
|
1139 |
|
In the following we describe the build process step by step, |
1140 |
|
assuming you are in the directory {\tt bin/}. |
1141 |
|
A summary of steps to follow is given at the end. |
1142 |
|
|
1143 |
|
\paragraph{Adjoint code generation and compilation -- step by step} |
1144 |
|
|
1145 |
|
\begin{enumerate} |
1146 |
|
% |
1147 |
|
\item |
1148 |
|
{\tt ln -s ../verification/???/code/.genmakerc .} \\ |
1149 |
|
{\tt ln -s ../verification/???/code/*.[Fh] .} \\ |
1150 |
|
Link your customized genmake options, header files, |
1151 |
|
and modified code to the compile directory. |
1152 |
|
% |
1153 |
|
\item |
1154 |
|
{\tt ../tools/genmake -makefile} \\ |
1155 |
|
Generate your Makefile (cf. Section ???). |
1156 |
|
% |
1157 |
|
\item |
1158 |
|
{\tt make depend} \\ |
1159 |
|
Dependency analysis for the CPP pre-compiler (cf. Section ???). |
1160 |
|
% |
1161 |
|
\item |
1162 |
|
{\tt make small\_f} \\ |
1163 |
|
This is the first difference between forward code compilation |
1164 |
|
and adjoint code generation and compilation. |
1165 |
|
Instead of going through the entire compilation process |
1166 |
|
(CPP precompiling -- {\tt .f}, object code generation -- {\tt .o}, |
1167 |
|
linking of object files and libraries to generate executable), |
1168 |
|
only the CPP compiler is invoked at this stage to generate |
1169 |
|
the {\tt .f} files. |
1170 |
|
% |
1171 |
|
\item |
1172 |
|
{\tt cd ../adjoint} \\ |
1173 |
|
{\tt make adtaf} or {\tt make adtamc} \\ |
1174 |
|
Depending on whether you have TAF or TAMC at your disposal, |
1175 |
|
you'll choose {\tt adtaf} or {\tt adtamc} as your |
1176 |
|
make target for the {\it makefile} in the directory {\tt adjoint/}. |
1177 |
|
Several things happen at this stage. |
1178 |
|
% |
1179 |
|
\begin{enumerate} |
1180 |
|
% |
1181 |
|
\item |
1182 |
|
The initial template file {\it adjoint\_model.F} which is part |
1183 |
|
of the compiling list created by {\it genmake} is restored. |
1184 |
|
% |
1185 |
|
\item |
1186 |
|
All Fortran routines {\tt *.f} in {\tt bin/} are |
1187 |
|
concatenated into a single file (it's current name is |
1188 |
|
{\it tamc\_code.f}). |
1189 |
|
% |
1190 |
|
\item |
1191 |
|
Adjoint code is generated by TAMC or TAF. |
1192 |
|
The adjoint code is written to the file {\it tamc\_code\_ad.f}. |
1193 |
|
It contains all adjoint routines of the forward routines |
1194 |
|
concatenated in {\it tamc\_code.f}. |
1195 |
|
For a given forward routines {\tt subroutine routinename} |
1196 |
|
the adjoint routine is named {\tt adsubroutine routinename} |
1197 |
|
by default (that default can be changed via the flag |
1198 |
|
{\tt -admark <markname>}). |
1199 |
|
Furthermore, it may contain modified code which |
1200 |
|
incorporates the translation of adjoint store directives |
1201 |
|
into specific Fortran code. |
1202 |
|
For a given forward routines {\tt subroutine routinename} |
1203 |
|
the modified routine is named {\tt mdsubroutine routinename}. |
1204 |
|
TAMC or TAF info is written to file |
1205 |
|
{\it tamc\_code.prot} or {\it taf.log}, respectively. |
1206 |
|
% |
1207 |
|
\end{enumerate} |
1208 |
|
% |
1209 |
|
\item |
1210 |
|
{\tt make adchange} \\ |
1211 |
|
The multi-threading capability of the MITGCM requires a slight |
1212 |
|
change in the parameter list of some routines that are related to |
1213 |
|
to active file handling. |
1214 |
|
This post-processing invokes the sed script {\it adjoint\_ecco\_sed.com} |
1215 |
|
to insert the threading counter {\bf myThId} into the parameter list |
1216 |
|
of those subroutines. |
1217 |
|
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
1218 |
|
and appended to the file {\it adjoint\_model.F}. |
1219 |
|
This concludes the adjoint code generation. |
1220 |
|
% |
1221 |
|
\item |
1222 |
|
{\tt cd ../bin} \\ |
1223 |
|
{\tt make} \\ |
1224 |
|
The file {\it adjoint\_model.F} now contains the full adjoint code. |
1225 |
|
All routines are now compiled. |
1226 |
|
% |
1227 |
|
\end{enumerate} |
1228 |
|
|
1229 |
|
\paragraph{Adjoint code generation and compilation -- summary} |
1230 |
|
~ \\ |
1231 |
|
|
1232 |
|
\[ |
1233 |
|
\boxed{ |
1234 |
|
\begin{split} |
1235 |
|
~ & \mbox{\tt cd bin} \\ |
1236 |
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/.genmakerc .} \\ |
1237 |
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/*.[Fh] .} \\ |
1238 |
|
~ & \mbox{\tt ../tools/genmake -makefile} \\ |
1239 |
|
~ & \mbox{\tt make depend} \\ |
1240 |
|
~ & \mbox{\tt make small\_f} \\ |
1241 |
|
~ & \mbox{\tt cd ../adjoint} \\ |
1242 |
|
~ & \mbox{\tt make adtaf <OR: make adtamc>} \\ |
1243 |
|
~ & \mbox{\tt make adchange} \\ |
1244 |
|
~ & \mbox{\tt cd ../bin} \\ |
1245 |
|
~ & \mbox{\tt make} \\ |
1246 |
|
\end{split} |
1247 |
|
} |
1248 |
|
\] |
1249 |
|
|
1250 |
\newpage |
\newpage |
1251 |
|
|
1264 |
\label{fig:adthemodel} |
\label{fig:adthemodel} |
1265 |
\end{figure} |
\end{figure} |
1266 |
|
|
1267 |
The basic flow is depicted in \reffig{adthemodel}. |
The basic flow is depicted in \ref{fig:adthemodel}. |
1268 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
1269 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
1270 |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
1296 |
The input is referred to as the |
The input is referred to as the |
1297 |
{\sf independent variables} or {\sf control variables}. |
{\sf independent variables} or {\sf control variables}. |
1298 |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
1299 |
(parameter setting, initialisation, accumulation, |
(parameter setting, initialization, accumulation, |
1300 |
final evaluation), are controlled by the package {\it pkg/cost}. |
final evaluation), are controlled by the package {\it pkg/cost}. |
1301 |
|
|
1302 |
\begin{figure}[h!] |
\begin{figure}[h!] |
1345 |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
1346 |
(file {\it CPP\_OPTIONS.h}). |
(file {\it CPP\_OPTIONS.h}). |
1347 |
|
|
1348 |
\subsubsection{Initialisation} |
\subsubsection{Initialization} |
1349 |
% |
% |
1350 |
The initialisation of the {\it cost} package is readily enabled |
The initialization of the {\it cost} package is readily enabled |
1351 |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
1352 |
% |
% |
1353 |
\begin{itemize} |
\begin{itemize} |
1378 |
} |
} |
1379 |
\\ |
\\ |
1380 |
This S/R |
This S/R |
1381 |
initialises the different cost function contributions. |
initializes the different cost function contributions. |
1382 |
The contribtion for the present example is {\bf objf\_tracer} |
The contribution for the present example is {\bf objf\_tracer} |
1383 |
which is defined on each tile (bi,bj). |
which is defined on each tile (bi,bj). |
1384 |
% |
% |
1385 |
\end{itemize} |
\end{itemize} |
1455 |
active variables are written and from which active variables |
active variables are written and from which active variables |
1456 |
are read are called {\sf active files}. |
are read are called {\sf active files}. |
1457 |
All aspects relevant to the treatment of the control variables |
All aspects relevant to the treatment of the control variables |
1458 |
(parameter setting, initialisation, perturbation) |
(parameter setting, initialization, perturbation) |
1459 |
are controled by the package {\it pkg/ctrl}. |
are controlled by the package {\it pkg/ctrl}. |
1460 |
|
|
1461 |
\begin{figure}[h!] |
\begin{figure}[h!] |
1462 |
\input{part5/doc_ctrl_flow} |
\input{part5/doc_ctrl_flow} |
1482 |
Each control variable is enabled via its own CPP option |
Each control variable is enabled via its own CPP option |
1483 |
in {\it ECCO\_CPPOPTIONS.h}. |
in {\it ECCO\_CPPOPTIONS.h}. |
1484 |
|
|
1485 |
\subsubsection{Initialisation} |
\subsubsection{Initialization} |
1486 |
% |
% |
1487 |
\begin{itemize} |
\begin{itemize} |
1488 |
% |
% |
1522 |
variables in the MITGCM need to be addressed. |
variables in the MITGCM need to be addressed. |
1523 |
First, in order to save memory, the control variable arrays |
First, in order to save memory, the control variable arrays |
1524 |
are not kept in memory, but rather read from file and added |
are not kept in memory, but rather read from file and added |
1525 |
to the initial fields during the model initialisation phase. |
to the initial fields during the model initialization phase. |
1526 |
Similarly, the corresponding adjoint fields which represent |
Similarly, the corresponding adjoint fields which represent |
1527 |
the gradient of the cost function w.r.t. the control variables |
the gradient of the cost function w.r.t. the control variables |
1528 |
are written to file at the end of the adjoint integration. |
are written to file at the end of the adjoint integration. |
1602 |
and an 'active read' routine of the adjoint support |
and an 'active read' routine of the adjoint support |
1603 |
package {\it pkg/autodiff} is invoked. |
package {\it pkg/autodiff} is invoked. |
1604 |
The read-procedure is tagged with the variable |
The read-procedure is tagged with the variable |
1605 |
{\bf xx\_tr1\_dummy} enabbling TAMC to recognize the |
{\bf xx\_tr1\_dummy} enabling TAMC to recognize the |
1606 |
initialisation of the perturbation. |
initialization of the perturbation. |
1607 |
The modified call of TAMC thus reads |
The modified call of TAMC thus reads |
1608 |
% |
% |
1609 |
\begin{verbatim} |
\begin{verbatim} |
1714 |
\begin{itemize} |
\begin{itemize} |
1715 |
% |
% |
1716 |
\item {\bf vector\_ctrl}: the control vector \\ |
\item {\bf vector\_ctrl}: the control vector \\ |
1717 |
At the very beginning of the model initialisation, |
At the very beginning of the model initialization, |
1718 |
the updated compressed control vector is read (or initialised) |
the updated compressed control vector is read (or initialised) |
1719 |
and distributed to 2-dim. and 3-dim. control variable fields. |
and distributed to 2-dim. and 3-dim. control variable fields. |
1720 |
% |
% |