1 |
adcroft |
1.5 |
% $Header: /u/gcmpack/mitgcmdoc/part5/doc_ad_examples.tex,v 1.4 2002/04/24 19:23:58 adcroft Exp $ |
2 |
cnh |
1.1 |
% $Name: $ |
3 |
|
|
|
4 |
|
|
%********************************************************************** |
5 |
|
|
\section{Sensitivity of Air-Sea Exchange to Tracer Injection Site } |
6 |
adcroft |
1.5 |
\label{www:tutorials} |
7 |
cnh |
1.1 |
\label{sect:eg-simple-tracer-adjoint} |
8 |
|
|
\label{sec_ad_setup_ex} |
9 |
adcroft |
1.4 |
\label{sect:tutorialIII} |
10 |
cnh |
1.1 |
%********************************************************************** |
11 |
|
|
|
12 |
|
|
The MITGCM has been adapted to enable AD using TAMC or TAF. |
13 |
|
|
The present description, therefore, is specific to the |
14 |
|
|
use of TAMC or TAF as AD tool. |
15 |
|
|
The following sections describe the steps which are necessary to |
16 |
|
|
generate a tangent linear or adjoint model of the MITGCM. |
17 |
|
|
We take as an example the sensitivity of carbon sequestration |
18 |
|
|
in the ocean. |
19 |
|
|
The AD-relevant hooks in the code are sketched in |
20 |
|
|
\ref{fig:adthemodel}, \ref{fig:adthemain}. |
21 |
|
|
|
22 |
|
|
\subsection{Overview of the experiment} |
23 |
adcroft |
1.5 |
\label{www:tutorials} |
24 |
cnh |
1.1 |
|
25 |
|
|
We describe an adjoint sensitivity analysis of out-gassing from |
26 |
|
|
the ocean into the atmosphere of a carbon-like tracer injected |
27 |
|
|
into the ocean interior (see \cite{hil-eta:01}). |
28 |
|
|
|
29 |
|
|
\subsubsection{Passive tracer equation} |
30 |
adcroft |
1.5 |
\label{www:tutorials} |
31 |
cnh |
1.1 |
|
32 |
|
|
For this work the MITGCM was augmented with a thermodynamically |
33 |
|
|
inactive tracer, $C$. Tracer residing in the ocean |
34 |
|
|
model surface layer is out-gassed according to a relaxation time scale, |
35 |
|
|
$\mu$. Within the ocean interior, the tracer is passively advected |
36 |
|
|
by the ocean model currents. The full equation for the time evolution |
37 |
|
|
% |
38 |
|
|
\begin{equation} |
39 |
|
|
\label{carbon_ddt} |
40 |
|
|
\frac{\partial C}{\partial t} \, = \, |
41 |
|
|
-U\cdot \nabla C \, - \, \mu C \, + \, \Gamma(C) \,+ \, S |
42 |
|
|
\end{equation} |
43 |
|
|
% |
44 |
|
|
also includes a source term $S$. This term |
45 |
|
|
represents interior sources of $C$ such as would arise due to |
46 |
|
|
direct injection. |
47 |
|
|
The velocity term, $U$, is the sum of the |
48 |
|
|
model Eulerian circulation and an eddy-induced velocity, the latter |
49 |
|
|
parameterized according to Gent/McWilliams |
50 |
|
|
(\cite{gen-mcw:90, gen-eta:95}). |
51 |
|
|
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
52 |
|
|
fluid is locally statically unstable. |
53 |
|
|
|
54 |
|
|
The out-gassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
55 |
|
|
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
56 |
|
|
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
57 |
|
|
is valid as a prognostic equation for small perturbations in oceanic |
58 |
|
|
carbon concentrations. This configuration provides a |
59 |
|
|
powerful tool for examining the impact of large-scale ocean circulation |
60 |
|
|
on $ CO_2 $ out-gassing due to interior injections. |
61 |
|
|
As source we choose a constant in time injection of |
62 |
|
|
$ S = 1 \,\, {\rm mol / s}$. |
63 |
|
|
|
64 |
|
|
\subsubsection{Model configuration} |
65 |
adcroft |
1.5 |
\label{www:tutorials} |
66 |
cnh |
1.1 |
|
67 |
|
|
The model configuration employed has a constant |
68 |
|
|
$4^\circ \times 4^\circ$ resolution horizontal grid and realistic |
69 |
|
|
geography and bathymetry. Twenty vertical layers are used with |
70 |
|
|
vertical spacing ranging |
71 |
|
|
from 50 m near the surface to 815 m at depth. |
72 |
|
|
Driven to steady-state by climatological wind-stress, heat and |
73 |
|
|
fresh-water forcing the model reproduces well known large-scale |
74 |
|
|
features of the ocean general circulation. |
75 |
|
|
|
76 |
|
|
\subsubsection{Out-gassing cost function} |
77 |
adcroft |
1.5 |
\label{www:tutorials} |
78 |
cnh |
1.1 |
|
79 |
|
|
To quantify and understand out-gassing due to injections of $C$ |
80 |
|
|
in eqn. (\ref{carbon_ddt}), |
81 |
|
|
we define a cost function $ {\cal J} $ that measures the total amount of |
82 |
|
|
tracer out-gassed at each timestep: |
83 |
|
|
% |
84 |
|
|
\begin{equation} |
85 |
|
|
\label{cost_tracer} |
86 |
|
|
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
87 |
|
|
\end{equation} |
88 |
|
|
% |
89 |
|
|
Equation(\ref{cost_tracer}) integrates the out-gassing term, $\mu C$, |
90 |
|
|
from (\ref{carbon_ddt}) |
91 |
|
|
over the entire ocean surface area, $A$, and accumulates it |
92 |
|
|
up to time $T$. |
93 |
|
|
Physically, ${\cal J}$ can be thought of as representing the amount of |
94 |
|
|
$CO_2$ that our model predicts would be out-gassed following an |
95 |
|
|
injection at rate $S$. |
96 |
|
|
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
97 |
|
|
$\frac{\partial {\cal J}}{\partial S}$, |
98 |
|
|
can be used to identify regions from which circulation |
99 |
|
|
would cause $CO_2$ to rapidly out-gas following injection |
100 |
|
|
and regions in which $CO_2$ injections would remain effectively |
101 |
|
|
sequestered within the ocean. |
102 |
|
|
|
103 |
|
|
\subsection{Code configuration} |
104 |
adcroft |
1.5 |
\label{www:tutorials} |
105 |
cnh |
1.1 |
|
106 |
|
|
The model configuration for this experiment resides under the |
107 |
|
|
directory {\it verification/carbon/}. |
108 |
|
|
The code customization routines are in {\it verification/carbon/code/}: |
109 |
|
|
% |
110 |
|
|
\begin{itemize} |
111 |
|
|
% |
112 |
|
|
\item {\it .genmakerc} |
113 |
|
|
% |
114 |
|
|
\item {\it COST\_CPPOPTIONS.h} |
115 |
|
|
% |
116 |
|
|
\item {\it CPP\_EEOPTIONS.h} |
117 |
|
|
% |
118 |
|
|
\item {\it CPP\_OPTIONS.h} |
119 |
|
|
% |
120 |
|
|
\item {\it CTRL\_OPTIONS.h} |
121 |
|
|
% |
122 |
|
|
\item {\it ECCO\_OPTIONS.h} |
123 |
|
|
% |
124 |
|
|
\item {\it SIZE.h} |
125 |
|
|
% |
126 |
|
|
\item {\it adcommon.h} |
127 |
|
|
% |
128 |
|
|
\item {\it tamc.h} |
129 |
|
|
% |
130 |
|
|
\end{itemize} |
131 |
|
|
% |
132 |
|
|
The runtime flag and parameters settings are contained in |
133 |
|
|
{\it verification/carbon/input/}, |
134 |
|
|
together with the forcing fields and and restart files: |
135 |
|
|
% |
136 |
|
|
\begin{itemize} |
137 |
|
|
% |
138 |
|
|
\item {\it data} |
139 |
|
|
% |
140 |
|
|
\item {\it data.cost} |
141 |
|
|
% |
142 |
|
|
\item {\it data.ctrl} |
143 |
|
|
% |
144 |
|
|
\item {\it data.gmredi} |
145 |
|
|
% |
146 |
|
|
\item {\it data.grdchk} |
147 |
|
|
% |
148 |
|
|
\item {\it data.optim} |
149 |
|
|
% |
150 |
|
|
\item {\it data.pkg} |
151 |
|
|
% |
152 |
|
|
\item {\it eedata} |
153 |
|
|
% |
154 |
|
|
\item {\it topog.bin} |
155 |
|
|
% |
156 |
|
|
\item {\it windx.bin, windy.bin} |
157 |
|
|
% |
158 |
|
|
\item {\it salt.bin, theta.bin} |
159 |
|
|
% |
160 |
|
|
\item {\it SSS.bin, SST.bin} |
161 |
|
|
% |
162 |
|
|
\item {\it pickup*} |
163 |
|
|
% |
164 |
|
|
\end{itemize} |
165 |
|
|
% |
166 |
|
|
Finally, the file to generate the adjoint code resides in |
167 |
|
|
$ adjoint/ $: |
168 |
|
|
% |
169 |
|
|
\begin{itemize} |
170 |
|
|
% |
171 |
|
|
\item {\it makefile} |
172 |
|
|
% |
173 |
|
|
\end{itemize} |
174 |
|
|
% |
175 |
|
|
|
176 |
|
|
Below we describe the customizations of this files which are |
177 |
|
|
specific to this experiment. |
178 |
|
|
|
179 |
|
|
\subsubsection{File {\it .genmakerc}} |
180 |
adcroft |
1.5 |
\label{www:tutorials} |
181 |
cnh |
1.1 |
This file overwrites default settings of {\it genmake}. |
182 |
|
|
In the present example it is used to switch on the following |
183 |
|
|
packages which are related to automatic differentiation |
184 |
|
|
and are disabled by default: \\ |
185 |
|
|
\hspace*{4ex} {\tt set ENABLE=( autodiff cost ctrl ecco gmredi grdchk kpp )} \\ |
186 |
|
|
Other packages which are not needed are switched off: \\ |
187 |
|
|
\hspace*{4ex} {\tt set DISABLE=( aim obcs zonal\_filt shap\_filt cal exf )} |
188 |
|
|
|
189 |
|
|
\subsubsection{File {\it COST\_CPPOPTIONS.h, CTRL\_OPTIONS.h}} |
190 |
adcroft |
1.5 |
\label{www:tutorials} |
191 |
cnh |
1.1 |
|
192 |
|
|
These files used to contain package-specific CPP-options |
193 |
|
|
(see Section \ref{???}). |
194 |
|
|
For technical reasons those options have been grouped together |
195 |
|
|
in the file {\it ECCO\_OPTIONS.h}. |
196 |
|
|
To retain the modularity, the files have been kept and contain |
197 |
|
|
the standard include of the {\it CPP\_OPTIONS.h} file. |
198 |
|
|
|
199 |
|
|
\subsubsection{File {\it CPP\_EEOPTIONS.h}} |
200 |
adcroft |
1.5 |
\label{www:tutorials} |
201 |
cnh |
1.1 |
|
202 |
|
|
This file contains 'wrapper'-specific CPP options. |
203 |
|
|
It only needs to be changed if the code is to be run |
204 |
|
|
in a parallel environment (see Section \ref{???}). |
205 |
|
|
|
206 |
|
|
\subsubsection{File {\it CPP\_OPTIONS.h}} |
207 |
adcroft |
1.5 |
\label{www:tutorials} |
208 |
cnh |
1.1 |
|
209 |
|
|
This file contains model-specific CPP options |
210 |
|
|
(see Section \ref{???}). |
211 |
|
|
Most options are related to the forward model setup. |
212 |
|
|
They are identical to the global steady circulation setup of |
213 |
heimbach |
1.2 |
{\it verification/global\_ocean.90x40x15/}. |
214 |
cnh |
1.1 |
The three options specific to this experiment are \\ |
215 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_PASSIVE\_TRACER} \\ |
216 |
|
|
This flag enables the code to carry through the |
217 |
|
|
advection/diffusion of a passive tracer along the |
218 |
|
|
model integration. \\ |
219 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
220 |
|
|
This flag enables the inclusion of some AD-related fields |
221 |
|
|
concerning initialization, link between control variables |
222 |
|
|
and forward model variables, and the call to the top-level |
223 |
|
|
forward/adjoint subroutine {\it adthe\_main\_loop} |
224 |
|
|
instead of {\it the\_main\_loop}. \\ |
225 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_GRADIENT\_CHECK} \\ |
226 |
|
|
This flag enables the gradient check package. |
227 |
|
|
After computing the unperturbed cost function and its gradient, |
228 |
|
|
a series of computations are performed for which \\ |
229 |
|
|
$\bullet$ an element of the control vector is perturbed \\ |
230 |
|
|
$\bullet$ the cost function w.r.t. the perturbed element is |
231 |
|
|
computed \\ |
232 |
|
|
$\bullet$ the difference between the perturbed and unperturbed |
233 |
|
|
cost function is computed to compute the finite difference gradient \\ |
234 |
|
|
$\bullet$ the finite difference gradient is compared with the |
235 |
|
|
adjoint-generated gradient. |
236 |
|
|
The gradient check package is further described in Section ???. |
237 |
|
|
|
238 |
|
|
\subsubsection{File {\it ECCO\_OPTIONS.h}} |
239 |
adcroft |
1.5 |
\label{www:tutorials} |
240 |
cnh |
1.1 |
|
241 |
|
|
The CPP options of several AD-related packages are grouped |
242 |
|
|
in this file: |
243 |
|
|
% |
244 |
|
|
\begin{itemize} |
245 |
|
|
% |
246 |
|
|
\item |
247 |
heimbach |
1.2 |
Overall ECCO-related execution modus: \\ |
248 |
|
|
These determine whether a pure forward run, |
249 |
|
|
a sensitivity run or an iteration of optimization is |
250 |
|
|
performed. These options are not needed in the present context. |
251 |
|
|
% |
252 |
|
|
\item |
253 |
cnh |
1.1 |
Adjoint support package: {\it pkg/autodiff/} \\ |
254 |
|
|
This package contains hand-written adjoint code such as |
255 |
|
|
active file handling, flow directives for files which must not |
256 |
|
|
be differentiated, and TAMC-specific header files. \\ |
257 |
heimbach |
1.2 |
% |
258 |
cnh |
1.1 |
\hspace*{4ex} {\tt \#define ALLOW\_AUTODIFF\_TAMC} \\ |
259 |
|
|
defines TAMC-related features in the code. \\ |
260 |
heimbach |
1.2 |
% |
261 |
cnh |
1.1 |
\hspace*{4ex} {\tt \#define ALLOW\_TAMC\_CHECKPOINTING} \\ |
262 |
|
|
enables the checkpointing feature of TAMC |
263 |
|
|
(see Section \ref{???}). |
264 |
|
|
In the present example a 3-level checkpointing is implemented. |
265 |
|
|
The code contains the relevant store directives, common block |
266 |
|
|
and tape initializations, storing key computation, |
267 |
|
|
and loop index handling. |
268 |
|
|
The checkpointing length at each level is defined in |
269 |
|
|
file {\it tamc.h}, cf. below. |
270 |
heimbach |
1.2 |
The out and intermediate loop directivs are contained |
271 |
|
|
in the files {\it checkpoint\_lev3\_directives.h}, |
272 |
|
|
{\it checkpoint\_lev2\_directives.h} (package {\it pkg/autodiff}). \\ |
273 |
|
|
% |
274 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_AUTODIFF\_MONITOOR} \\ |
275 |
|
|
enables the monitoring of intermediate adjoint variables |
276 |
|
|
(see Section \ref{???}). \\ |
277 |
|
|
% |
278 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_DIVIDED\_ADJOINT} \\ |
279 |
|
|
enables adjoint dump and restart |
280 |
|
|
(see Section \ref{???}). |
281 |
cnh |
1.1 |
% |
282 |
|
|
\item Cost function package: {\it pkg/cost/} \\ |
283 |
|
|
This package contains all relevant routines for |
284 |
|
|
initializing, accumulating and finalizing the cost function |
285 |
|
|
(see Section \ref{???}). \\ |
286 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
287 |
|
|
enables all general aspects of the cost function handling, |
288 |
|
|
in particular the hooks in the forward code for |
289 |
|
|
initializing, accumulating and finalizing the cost function. \\ |
290 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
291 |
|
|
includes the call to the cost function for this |
292 |
|
|
particular experiment, eqn. (\ref{cost_tracer}). |
293 |
|
|
% |
294 |
|
|
\item Control variable package: {\it pkg/ctrl/} \\ |
295 |
|
|
This package contains all relevant routines for |
296 |
|
|
the handling of the control vector. |
297 |
|
|
Each control variable can be enabled/disabled with its own flag: \\ |
298 |
|
|
\begin{tabular}{ll} |
299 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_THETA0\_CONTROL} & |
300 |
|
|
initial temperature \\ |
301 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SALT0\_CONTROL} & |
302 |
|
|
initial salinity \\ |
303 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TR0\_CONTROL} & |
304 |
|
|
initial passive tracer concentration \\ |
305 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUU0\_CONTROL} & |
306 |
|
|
zonal wind stress \\ |
307 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUV0\_CONTROL} & |
308 |
|
|
meridional wind stress \\ |
309 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SFLUX0\_CONTROL} & |
310 |
|
|
freshwater flux \\ |
311 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_HFLUX0\_CONTROL} & |
312 |
|
|
heat flux \\ |
313 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_DIFFKR\_CONTROL} & |
314 |
|
|
diapycnal diffusivity \\ |
315 |
|
|
\hspace*{2ex} {\tt \#undef ALLOW\_KAPPAGM\_CONTROL} & |
316 |
|
|
isopycnal diffusivity \\ |
317 |
|
|
\end{tabular} |
318 |
|
|
% |
319 |
|
|
\end{itemize} |
320 |
|
|
|
321 |
|
|
\subsubsection{File {\it SIZE.h}} |
322 |
adcroft |
1.5 |
\label{www:tutorials} |
323 |
cnh |
1.1 |
|
324 |
|
|
The file contains the grid point dimensions of the forward |
325 |
|
|
model. It is identical to the {\it verification/exp2/}: \\ |
326 |
|
|
\hspace*{4ex} {\tt sNx = 90} \\ |
327 |
|
|
\hspace*{4ex} {\tt sNy = 40} \\ |
328 |
|
|
\hspace*{4ex} {\tt Nr = 20} \\ |
329 |
|
|
It corresponds to a single-tile/single-processor setup: |
330 |
|
|
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
331 |
|
|
with standard overlap dimensioning |
332 |
|
|
{\tt OLx = OLy = 3}. |
333 |
|
|
|
334 |
|
|
\subsubsection{File {\it adcommon.h}} |
335 |
adcroft |
1.5 |
\label{www:tutorials} |
336 |
cnh |
1.1 |
|
337 |
|
|
This file contains common blocks of some adjoint variables |
338 |
|
|
that are generated by TAMC. |
339 |
|
|
The common blocks are used by the adjoint support routine |
340 |
|
|
{\it addummy\_in\_stepping} which needs to access those variables: |
341 |
|
|
|
342 |
|
|
\begin{tabular}{ll} |
343 |
|
|
\hspace*{4ex} {\tt common /addynvars\_r/} & |
344 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
345 |
|
|
\hspace*{4ex} {\tt common /addynvars\_cd/} & |
346 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
347 |
|
|
\hspace*{4ex} {\tt common /addynvars\_diffkr/} & |
348 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
349 |
|
|
\hspace*{4ex} {\tt common /addynvars\_kapgm/} & |
350 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
351 |
|
|
\hspace*{4ex} {\tt common /adtr1\_r/} & |
352 |
|
|
\hspace*{4ex} is related to {\it TR1.h} \\ |
353 |
|
|
\hspace*{4ex} {\tt common /adffields/} & |
354 |
|
|
\hspace*{4ex} is related to {\it FFIELDS.h}\\ |
355 |
|
|
\end{tabular} |
356 |
|
|
|
357 |
|
|
Note that if the structure of the common block changes in the |
358 |
|
|
above header files of the forward code, the structure |
359 |
|
|
of the adjoint common blocks will change accordingly. |
360 |
|
|
Thus, it has to be made sure that the structure of the |
361 |
|
|
adjoint common block in the hand-written file {\it adcommon.h} |
362 |
|
|
complies with the automatically generated adjoint common blocks |
363 |
|
|
in {\it adjoint\_model.F}. |
364 |
heimbach |
1.2 |
The header file is enabled via the CPP-option |
365 |
|
|
{\bf ALLOW\_AUTODIFF\_MONITOR}. |
366 |
cnh |
1.1 |
|
367 |
|
|
\subsubsection{File {\it tamc.h}} |
368 |
adcroft |
1.5 |
\label{www:tutorials} |
369 |
cnh |
1.1 |
|
370 |
heimbach |
1.2 |
This routine contains the dimensions for TAMC checkpointing |
371 |
|
|
and some indices relevant for storing ky computations. |
372 |
cnh |
1.1 |
% |
373 |
|
|
\begin{itemize} |
374 |
|
|
% |
375 |
|
|
\item {\tt \#ifdef ALLOW\_TAMC\_CHECKPOINTING} \\ |
376 |
|
|
3-level checkpointing is enabled, i.e. the timestepping |
377 |
|
|
is divided into three different levels (see Section \ref{???}). |
378 |
|
|
The model state of the outermost ({\tt nchklev\_3}) and the |
379 |
|
|
intermediate ({\tt nchklev\_2}) timestepping loop are stored to file |
380 |
|
|
(handled in {\it the\_main\_loop}). |
381 |
|
|
The innermost loop ({\tt nchklev\_1}) |
382 |
|
|
avoids I/O by storing all required variables |
383 |
|
|
to common blocks. This storing may also be necessary if |
384 |
|
|
no checkpointing is chosen |
385 |
|
|
(nonlinear functions, if-statements, iterative loops, ...). |
386 |
|
|
In the present example the dimensions are chosen as follows: \\ |
387 |
|
|
\hspace*{4ex} {\tt nchklev\_1 = 36 } \\ |
388 |
|
|
\hspace*{4ex} {\tt nchklev\_2 = 30 } \\ |
389 |
|
|
\hspace*{4ex} {\tt nchklev\_3 = 60 } \\ |
390 |
|
|
To guarantee that the checkpointing intervals span the entire |
391 |
|
|
integration period the following relation must be satisfied: \\ |
392 |
|
|
\hspace*{4ex} {\tt nchklev\_1*nchklev\_2*nchklev\_3 $ \ge $ nTimeSteps} \\ |
393 |
|
|
where {\tt nTimeSteps} is either specified in {\it data} |
394 |
|
|
or computed via \\ |
395 |
|
|
\hspace*{4ex} {\tt nTimeSteps = (endTime-startTime)/deltaTClock }. |
396 |
|
|
% |
397 |
|
|
\item {\tt \#undef ALLOW\_TAMC\_CHECKPOINTING} \\ |
398 |
|
|
No checkpointing is enabled. |
399 |
|
|
In this case the relevant counter is {\tt nchklev\_0}. |
400 |
|
|
Similar to above, the following relation has to be satisfied \\ |
401 |
|
|
\hspace*{4ex} {\tt nchklev\_0 $ \ge $ nTimeSteps}. |
402 |
|
|
% |
403 |
|
|
\end{itemize} |
404 |
|
|
|
405 |
|
|
The following parameters may be worth describing: \\ |
406 |
|
|
% |
407 |
|
|
\hspace*{4ex} {\tt isbyte} \\ |
408 |
|
|
\hspace*{4ex} {\tt maxpass} \\ |
409 |
|
|
~ |
410 |
|
|
|
411 |
|
|
\subsubsection{File {\it makefile}} |
412 |
adcroft |
1.5 |
\label{www:tutorials} |
413 |
cnh |
1.1 |
|
414 |
|
|
This file contains all relevant parameter flags and |
415 |
|
|
lists to run TAMC or TAF. |
416 |
|
|
It is assumed that TAMC is available to you, either locally, |
417 |
|
|
being installed on your network, or remotely through the 'TAMC Utility'. |
418 |
|
|
TAMC is called with the command {\tt tamc} followed by a |
419 |
|
|
number of options. They are described in detail in the |
420 |
|
|
TAMC manual \cite{gie:99}. |
421 |
heimbach |
1.3 |
Here we briefly discuss the main flags used in the {\it makefile}. |
422 |
|
|
The standard output for TAF is written to file |
423 |
|
|
{\it taf.log}. |
424 |
cnh |
1.1 |
% |
425 |
|
|
\begin{itemize} |
426 |
|
|
\item [{\tt tamc}] {\tt |
427 |
|
|
-input <variable names> |
428 |
heimbach |
1.2 |
-output <variable name> -i4 -r4 ... \\ |
429 |
cnh |
1.1 |
-toplevel <S/R name> -reverse <file names> |
430 |
|
|
} |
431 |
heimbach |
1.2 |
\item [{\tt taf}] {\tt |
432 |
|
|
-input <variable names> |
433 |
|
|
-output <variable name> -i4 -r4 ... \\ |
434 |
|
|
-toplevel <S/R name> -reverse <file names> \\ |
435 |
heimbach |
1.3 |
-flow taf\_flow.log -nonew\_arg |
436 |
heimbach |
1.2 |
} |
437 |
cnh |
1.1 |
\end{itemize} |
438 |
|
|
% |
439 |
|
|
\begin{itemize} |
440 |
|
|
% |
441 |
|
|
\item {\tt -toplevel <S/R name>} \\ |
442 |
|
|
Name of the toplevel routine, with respect to which the |
443 |
|
|
control flow analysis is performed. |
444 |
|
|
% |
445 |
|
|
\item {\tt -input <variable names>} \\ |
446 |
|
|
List of independent variables $ u $ with respect to which the |
447 |
|
|
dependent variable $ J $ is differentiated. |
448 |
|
|
% |
449 |
|
|
\item {\tt -output <variable name>} \\ |
450 |
|
|
Dependent variable $ J $ which is to be differentiated. |
451 |
|
|
% |
452 |
|
|
\item {\tt -reverse <file names>} \\ |
453 |
|
|
Adjoint code is generated to compute the sensitivity of an |
454 |
|
|
independent variable w.r.t. many dependent variables. |
455 |
|
|
In the discussion of Section ??? |
456 |
|
|
the generated adjoint top-level routine computes the product |
457 |
|
|
of the transposed Jacobian matrix $ M^T $ times |
458 |
|
|
the gradient vector $ \nabla_v J $. |
459 |
|
|
\\ |
460 |
|
|
{\tt <file names>} refers to the list of files {\it .f} which are to be |
461 |
|
|
analyzed by TAMC. This list is generally smaller than the full list |
462 |
|
|
of code to be compiled. The files not contained are either |
463 |
|
|
above the top-level routine (some initializations), or are |
464 |
|
|
deliberately hidden from TAMC, either because hand-written |
465 |
|
|
adjoint routines exist, or the routines must not (or don't have to) |
466 |
|
|
be differentiated. For each routine which is part of the flow tree |
467 |
|
|
of the top-level routine, but deliberately hidden from TAMC |
468 |
|
|
(or for each package which contains such routines), |
469 |
|
|
a corresponding file {\it .flow} exists containing flow directives |
470 |
|
|
for TAMC. |
471 |
|
|
% |
472 |
heimbach |
1.2 |
\item {\tt -i4 -r4} \\ |
473 |
cnh |
1.1 |
~ |
474 |
heimbach |
1.2 |
% |
475 |
heimbach |
1.3 |
\item {\tt -flow taf\_flow.log} \\ |
476 |
|
|
Will cause TAF to produce a flow listing file |
477 |
|
|
named {\it taf\_flow.log} in which |
478 |
heimbach |
1.2 |
the set of active and passive variables are identified |
479 |
|
|
for each subroutine. |
480 |
|
|
% |
481 |
heimbach |
1.3 |
\item {\tt -nonew\_arg} \\ |
482 |
heimbach |
1.2 |
The default in the order of the parameter list of |
483 |
|
|
adjoint routines has changed. |
484 |
|
|
Before TAF 1.3 the default was compatible with the |
485 |
|
|
TAMC-generated list. As of TAF 1.3 the order of adjoint |
486 |
|
|
routine parameter lists is no longer copatible with TAMC. |
487 |
|
|
To restore compatibility when using TAF 1.3 and higher, |
488 |
|
|
this argument is needed. |
489 |
|
|
It is currently crucial to use since all hand-written |
490 |
|
|
adjoint routines refer to the TAMC default. |
491 |
cnh |
1.1 |
% |
492 |
|
|
\end{itemize} |
493 |
|
|
|
494 |
|
|
|
495 |
|
|
\subsubsection{The input parameter files} |
496 |
adcroft |
1.5 |
\label{www:tutorials} |
497 |
cnh |
1.1 |
|
498 |
|
|
\paragraph{File {\it data}} |
499 |
|
|
|
500 |
|
|
\paragraph{File {\it data.cost}} |
501 |
|
|
|
502 |
|
|
\paragraph{File {\it data.ctrl}} |
503 |
|
|
|
504 |
|
|
\paragraph{File {\it data.gmredi}} |
505 |
|
|
|
506 |
|
|
\paragraph{File {\it data.grdchk}} |
507 |
|
|
|
508 |
|
|
\paragraph{File {\it data.optim}} |
509 |
|
|
|
510 |
|
|
\paragraph{File {\it data.pkg}} |
511 |
|
|
|
512 |
|
|
\paragraph{File {\it eedata}} |
513 |
|
|
|
514 |
heimbach |
1.3 |
\paragraph{File {\it topog.bin}} ~ \\ |
515 |
|
|
% |
516 |
|
|
Contains two-dimendional bathymetry information |
517 |
cnh |
1.1 |
|
518 |
heimbach |
1.3 |
\paragraph{File {\it windx.bin, windy.bin, salt.bin, theta.bin, |
519 |
|
|
SSS.bin, SST.bin}} ~ \\ |
520 |
|
|
% |
521 |
|
|
These contain the initial values |
522 |
|
|
(salinity, temperature, {\it salt.bin, theta.bin}), |
523 |
|
|
surface boundary values (surface wind stresses, |
524 |
|
|
({\it windx.bin, windy.bin}), and surface restoring fields |
525 |
|
|
({\it SSS.bin, SST.bin}). |
526 |
cnh |
1.1 |
|
527 |
heimbach |
1.3 |
\paragraph{File {\it pickup*}} ~ \\ |
528 |
|
|
% |
529 |
|
|
Contains model state after model spinup. |
530 |
cnh |
1.1 |
|
531 |
|
|
\subsection{Compiling the model and its adjoint} |
532 |
adcroft |
1.5 |
\label{www:tutorials} |
533 |
cnh |
1.1 |
|
534 |
|
|
The built process of the adjoint model is slightly more |
535 |
|
|
complex than that of compiling the forward code. |
536 |
|
|
The main reason is that the adjoint code generation requires |
537 |
|
|
a specific list of routines that are to be differentiated |
538 |
|
|
(as opposed to the automatic generation of a list of |
539 |
|
|
files to be compiled by genmake). |
540 |
|
|
This list excludes routines that don't have to be or must not be |
541 |
|
|
differentiated. For some of the latter routines flow directives |
542 |
|
|
may be necessary, a list of which has to be given as well. |
543 |
|
|
For this reason, a separate {\it makefile} is currently |
544 |
|
|
maintained in the directory {\tt adjoint/}. This |
545 |
|
|
makefile is responsible for the adjoint code generation. |
546 |
|
|
|
547 |
|
|
In the following we describe the build process step by step, |
548 |
|
|
assuming you are in the directory {\tt bin/}. |
549 |
|
|
A summary of steps to follow is given at the end. |
550 |
|
|
|
551 |
|
|
\paragraph{Adjoint code generation and compilation -- step by step} |
552 |
|
|
|
553 |
|
|
\begin{enumerate} |
554 |
|
|
% |
555 |
|
|
\item |
556 |
|
|
{\tt ln -s ../verification/???/code/.genmakerc .} \\ |
557 |
|
|
{\tt ln -s ../verification/???/code/*.[Fh] .} \\ |
558 |
|
|
Link your customized genmake options, header files, |
559 |
|
|
and modified code to the compile directory. |
560 |
|
|
% |
561 |
|
|
\item |
562 |
|
|
{\tt ../tools/genmake -makefile} \\ |
563 |
|
|
Generate your Makefile (cf. Section ???). |
564 |
|
|
% |
565 |
|
|
\item |
566 |
|
|
{\tt make depend} \\ |
567 |
|
|
Dependency analysis for the CPP pre-compiler (cf. Section ???). |
568 |
|
|
% |
569 |
|
|
\item |
570 |
|
|
{\tt cd ../adjoint} \\ |
571 |
|
|
{\tt make adtaf} or {\tt make adtamc} \\ |
572 |
|
|
Depending on whether you have TAF or TAMC at your disposal, |
573 |
|
|
you'll choose {\tt adtaf} or {\tt adtamc} as your |
574 |
|
|
make target for the {\it makefile} in the directory {\tt adjoint/}. |
575 |
|
|
Several things happen at this stage. |
576 |
|
|
% |
577 |
|
|
\begin{enumerate} |
578 |
|
|
% |
579 |
|
|
\item |
580 |
heimbach |
1.3 |
{\tt make adrestore, make ftlrestore} \\ |
581 |
|
|
The initial template files {\it adjoint\_model.F} |
582 |
|
|
and {\it tangentlinear\_model.F} in {\it pkg/autodiff} |
583 |
|
|
which are part |
584 |
|
|
of the compiling list created by {\it genmake} are restored. |
585 |
|
|
% |
586 |
|
|
\item {\tt make depend, make small\_f} \\ |
587 |
|
|
The {\tt bin/} directory is brought up to date, |
588 |
|
|
i.e. for recent changes in header or source code |
589 |
|
|
{\it .[Fh]}, corresponding {\it .f} routines are generated |
590 |
|
|
or re-generated. |
591 |
|
|
Note that here, only CPP precompiling is performed; |
592 |
|
|
no object code {\it .o} is generated as yet. |
593 |
|
|
Precompiling is necessary for TAMC to see the full code. |
594 |
cnh |
1.1 |
% |
595 |
|
|
\item |
596 |
heimbach |
1.3 |
{\tt make allcode} \\ |
597 |
cnh |
1.1 |
All Fortran routines {\tt *.f} in {\tt bin/} are |
598 |
heimbach |
1.3 |
concatenated into a single file called |
599 |
|
|
{\it tamc\_code.f}. |
600 |
cnh |
1.1 |
% |
601 |
|
|
\item |
602 |
heimbach |
1.3 |
{\tt make admodeltaf/admodeltamc} \\ |
603 |
cnh |
1.1 |
Adjoint code is generated by TAMC or TAF. |
604 |
|
|
The adjoint code is written to the file {\it tamc\_code\_ad.f}. |
605 |
|
|
It contains all adjoint routines of the forward routines |
606 |
|
|
concatenated in {\it tamc\_code.f}. |
607 |
|
|
For a given forward routines {\tt subroutine routinename} |
608 |
|
|
the adjoint routine is named {\tt adsubroutine routinename} |
609 |
|
|
by default (that default can be changed via the flag |
610 |
|
|
{\tt -admark <markname>}). |
611 |
|
|
Furthermore, it may contain modified code which |
612 |
|
|
incorporates the translation of adjoint store directives |
613 |
|
|
into specific Fortran code. |
614 |
|
|
For a given forward routines {\tt subroutine routinename} |
615 |
|
|
the modified routine is named {\tt mdsubroutine routinename}. |
616 |
|
|
TAMC or TAF info is written to file |
617 |
|
|
{\it tamc\_code.prot} or {\it taf.log}, respectively. |
618 |
|
|
% |
619 |
|
|
\item |
620 |
|
|
{\tt make adchange} \\ |
621 |
|
|
The multi-threading capability of the MITGCM requires a slight |
622 |
|
|
change in the parameter list of some routines that are related to |
623 |
|
|
to active file handling. |
624 |
|
|
This post-processing invokes the sed script {\it adjoint\_ecco\_sed.com} |
625 |
|
|
to insert the threading counter {\bf myThId} into the parameter list |
626 |
|
|
of those subroutines. |
627 |
|
|
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
628 |
|
|
and appended to the file {\it adjoint\_model.F}. |
629 |
|
|
This concludes the adjoint code generation. |
630 |
|
|
% |
631 |
heimbach |
1.3 |
\end{enumerate} |
632 |
|
|
% |
633 |
cnh |
1.1 |
\item |
634 |
|
|
{\tt cd ../bin} \\ |
635 |
|
|
{\tt make} \\ |
636 |
|
|
The file {\it adjoint\_model.F} now contains the full adjoint code. |
637 |
|
|
All routines are now compiled. |
638 |
|
|
% |
639 |
|
|
\end{enumerate} |
640 |
|
|
|
641 |
heimbach |
1.3 |
N.B.: The targets {\tt make adtaf/adtamc} now comprise a |
642 |
|
|
series of targets that in previous versions had to be |
643 |
|
|
invoked separarely. This was probably preferable at a more |
644 |
|
|
experimental stage, but has now been dropped in favour of |
645 |
|
|
a more straightforward build process. |
646 |
|
|
|
647 |
|
|
|
648 |
cnh |
1.1 |
\paragraph{Adjoint code generation and compilation -- summary} |
649 |
|
|
~ \\ |
650 |
|
|
|
651 |
heimbach |
1.3 |
{\small |
652 |
cnh |
1.1 |
\[ |
653 |
|
|
\boxed{ |
654 |
|
|
\begin{split} |
655 |
|
|
~ & \mbox{\tt cd bin} \\ |
656 |
|
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/.genmakerc .} \\ |
657 |
|
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/*.[Fh] .} \\ |
658 |
|
|
~ & \mbox{\tt ../tools/genmake -makefile} \\ |
659 |
|
|
~ & \mbox{\tt make depend} \\ |
660 |
|
|
~ & \mbox{\tt cd ../adjoint} \\ |
661 |
|
|
~ & \mbox{\tt make adtaf <OR: make adtamc>} \\ |
662 |
heimbach |
1.3 |
~ & \hspace*{6ex} \mbox{\tt contains the targets:} \\ |
663 |
|
|
~ & \hspace*{6ex} \mbox{\tt adrestore small\_f allcode admodeltaf/admodeltamc adchange} \\ |
664 |
cnh |
1.1 |
~ & \mbox{\tt cd ../bin} \\ |
665 |
|
|
~ & \mbox{\tt make} \\ |
666 |
|
|
\end{split} |
667 |
|
|
} |
668 |
|
|
\] |
669 |
heimbach |
1.3 |
} |