1 |
jmc |
1.4 |
% $Header: /u/gcmpack/manual/s_examples/tracer_adjsens/doc_ad_examples.tex,v 1.3 2010/08/30 23:09:20 jmc Exp $ |
2 |
molod |
1.1 |
% $Name: $ |
3 |
|
|
|
4 |
|
|
%********************************************************************** |
5 |
|
|
\section{Sensitivity of Air-Sea Exchange to Tracer Injection Site } |
6 |
jmc |
1.3 |
%\label{www:tutorials} |
7 |
|
|
\label{sec:eg-simple-tracer-adjoint} |
8 |
molod |
1.1 |
\label{sec_ad_setup_ex} |
9 |
jmc |
1.3 |
\label{sec:tutorialIII} |
10 |
molod |
1.1 |
\begin{rawhtml} |
11 |
|
|
<!-- CMIREDIR:sec_ad_setup_ex: --> |
12 |
|
|
\end{rawhtml} |
13 |
jmc |
1.2 |
\begin{center} |
14 |
|
|
(in directory: {\it verification/tutorial\_tracer\_adjsens/}) |
15 |
|
|
\end{center} |
16 |
molod |
1.1 |
%********************************************************************** |
17 |
|
|
|
18 |
|
|
MITgcm has been adapted to enable AD using TAMC or TAF. |
19 |
|
|
The present description, therefore, is specific to the |
20 |
|
|
use of TAMC or TAF as AD tool. |
21 |
|
|
The following sections describe the steps which are necessary to |
22 |
|
|
generate a tangent linear or adjoint model of MITgcm. |
23 |
|
|
We take as an example the sensitivity of carbon sequestration |
24 |
|
|
in the ocean. |
25 |
|
|
The AD-relevant hooks in the code are sketched in |
26 |
|
|
\ref{fig:adthemodel}, \ref{fig:adthemain}. |
27 |
|
|
|
28 |
|
|
\subsection{Overview of the experiment} |
29 |
jmc |
1.3 |
%\label{www:tutorials} |
30 |
molod |
1.1 |
|
31 |
|
|
We describe an adjoint sensitivity analysis of out-gassing from |
32 |
|
|
the ocean into the atmosphere of a carbon-like tracer injected |
33 |
|
|
into the ocean interior (see \cite{hil-eta:01}). |
34 |
|
|
|
35 |
|
|
\subsubsection{Passive tracer equation} |
36 |
jmc |
1.3 |
%\label{www:tutorials} |
37 |
molod |
1.1 |
|
38 |
|
|
For this work MITgcm was augmented with a thermodynamically |
39 |
|
|
inactive tracer, $C$. Tracer residing in the ocean |
40 |
|
|
model surface layer is out-gassed according to a relaxation time scale, |
41 |
|
|
$\mu$. Within the ocean interior, the tracer is passively advected |
42 |
|
|
by the ocean model currents. The full equation for the time evolution |
43 |
|
|
% |
44 |
|
|
\begin{equation} |
45 |
|
|
\label{carbon_ddt} |
46 |
|
|
\frac{\partial C}{\partial t} \, = \, |
47 |
|
|
-U\cdot \nabla C \, - \, \mu C \, + \, \Gamma(C) \,+ \, S |
48 |
|
|
\end{equation} |
49 |
|
|
% |
50 |
|
|
also includes a source term $S$. This term |
51 |
|
|
represents interior sources of $C$ such as would arise due to |
52 |
|
|
direct injection. |
53 |
|
|
The velocity term, $U$, is the sum of the |
54 |
|
|
model Eulerian circulation and an eddy-induced velocity, the latter |
55 |
|
|
parameterized according to Gent/McWilliams |
56 |
|
|
(\cite{gen-mcw:90, gen-eta:95}). |
57 |
|
|
The convection function, $\Gamma$, mixes $C$ vertically wherever the |
58 |
|
|
fluid is locally statically unstable. |
59 |
|
|
|
60 |
|
|
The out-gassing time scale, $\mu$, in eqn. (\ref{carbon_ddt}) |
61 |
|
|
is set so that \( 1/\mu \sim 1 \ \mathrm{year} \) for the surface |
62 |
|
|
ocean and $\mu=0$ elsewhere. With this value, eqn. (\ref{carbon_ddt}) |
63 |
|
|
is valid as a prognostic equation for small perturbations in oceanic |
64 |
|
|
carbon concentrations. This configuration provides a |
65 |
|
|
powerful tool for examining the impact of large-scale ocean circulation |
66 |
|
|
on $ CO_2 $ out-gassing due to interior injections. |
67 |
|
|
As source we choose a constant in time injection of |
68 |
|
|
$ S = 1 \,\, {\rm mol / s}$. |
69 |
|
|
|
70 |
|
|
\subsubsection{Model configuration} |
71 |
jmc |
1.3 |
%\label{www:tutorials} |
72 |
molod |
1.1 |
|
73 |
|
|
The model configuration employed has a constant |
74 |
|
|
$4^\circ \times 4^\circ$ resolution horizontal grid and realistic |
75 |
|
|
geography and bathymetry. Twenty vertical layers are used with |
76 |
|
|
vertical spacing ranging |
77 |
|
|
from 50 m near the surface to 815 m at depth. |
78 |
|
|
Driven to steady-state by climatological wind-stress, heat and |
79 |
|
|
fresh-water forcing the model reproduces well known large-scale |
80 |
|
|
features of the ocean general circulation. |
81 |
|
|
|
82 |
|
|
\subsubsection{Out-gassing cost function} |
83 |
jmc |
1.3 |
%\label{www:tutorials} |
84 |
molod |
1.1 |
|
85 |
|
|
To quantify and understand out-gassing due to injections of $C$ |
86 |
|
|
in eqn. (\ref{carbon_ddt}), |
87 |
|
|
we define a cost function $ {\cal J} $ that measures the total amount of |
88 |
|
|
tracer out-gassed at each timestep: |
89 |
|
|
% |
90 |
|
|
\begin{equation} |
91 |
|
|
\label{cost_tracer} |
92 |
|
|
{\cal J}(t=T)=\int_{t=0}^{t=T}\int_{A} \mu C \, dA \, dt |
93 |
|
|
\end{equation} |
94 |
|
|
% |
95 |
|
|
Equation(\ref{cost_tracer}) integrates the out-gassing term, $\mu C$, |
96 |
|
|
from (\ref{carbon_ddt}) |
97 |
|
|
over the entire ocean surface area, $A$, and accumulates it |
98 |
|
|
up to time $T$. |
99 |
|
|
Physically, ${\cal J}$ can be thought of as representing the amount of |
100 |
|
|
$CO_2$ that our model predicts would be out-gassed following an |
101 |
|
|
injection at rate $S$. |
102 |
|
|
The sensitivity of ${\cal J}$ to the spatial location of $S$, |
103 |
|
|
$\frac{\partial {\cal J}}{\partial S}$, |
104 |
|
|
can be used to identify regions from which circulation |
105 |
|
|
would cause $CO_2$ to rapidly out-gas following injection |
106 |
|
|
and regions in which $CO_2$ injections would remain effectively |
107 |
|
|
sequestered within the ocean. |
108 |
|
|
|
109 |
|
|
\subsection{Code configuration} |
110 |
jmc |
1.3 |
%\label{www:tutorials} |
111 |
molod |
1.1 |
|
112 |
|
|
The model configuration for this experiment resides under the |
113 |
|
|
directory {\it verification/carbon/}. |
114 |
|
|
The code customization routines are in {\it verification/carbon/code/}: |
115 |
|
|
% |
116 |
|
|
\begin{itemize} |
117 |
|
|
% |
118 |
|
|
\item {\it .genmakerc} |
119 |
|
|
% |
120 |
|
|
\item {\it COST\_CPPOPTIONS.h} |
121 |
|
|
% |
122 |
|
|
\item {\it CPP\_EEOPTIONS.h} |
123 |
|
|
% |
124 |
|
|
\item {\it CPP\_OPTIONS.h} |
125 |
|
|
% |
126 |
|
|
\item {\it CTRL\_OPTIONS.h} |
127 |
|
|
% |
128 |
|
|
\item {\it ECCO\_OPTIONS.h} |
129 |
|
|
% |
130 |
|
|
\item {\it SIZE.h} |
131 |
|
|
% |
132 |
|
|
\item {\it adcommon.h} |
133 |
|
|
% |
134 |
|
|
\item {\it tamc.h} |
135 |
|
|
% |
136 |
|
|
\end{itemize} |
137 |
|
|
% |
138 |
|
|
The runtime flag and parameters settings are contained in |
139 |
|
|
{\it verification/carbon/input/}, |
140 |
|
|
together with the forcing fields and and restart files: |
141 |
|
|
% |
142 |
|
|
\begin{itemize} |
143 |
|
|
% |
144 |
|
|
\item {\it data} |
145 |
|
|
% |
146 |
|
|
\item {\it data.cost} |
147 |
|
|
% |
148 |
|
|
\item {\it data.ctrl} |
149 |
|
|
% |
150 |
|
|
\item {\it data.gmredi} |
151 |
|
|
% |
152 |
|
|
\item {\it data.grdchk} |
153 |
|
|
% |
154 |
|
|
\item {\it data.optim} |
155 |
|
|
% |
156 |
|
|
\item {\it data.pkg} |
157 |
|
|
% |
158 |
|
|
\item {\it eedata} |
159 |
|
|
% |
160 |
|
|
\item {\it topog.bin} |
161 |
|
|
% |
162 |
|
|
\item {\it windx.bin, windy.bin} |
163 |
|
|
% |
164 |
|
|
\item {\it salt.bin, theta.bin} |
165 |
|
|
% |
166 |
|
|
\item {\it SSS.bin, SST.bin} |
167 |
|
|
% |
168 |
|
|
\item {\it pickup*} |
169 |
|
|
% |
170 |
|
|
\end{itemize} |
171 |
|
|
% |
172 |
|
|
Finally, the file to generate the adjoint code resides in |
173 |
|
|
$ adjoint/ $: |
174 |
|
|
% |
175 |
|
|
\begin{itemize} |
176 |
|
|
% |
177 |
|
|
\item {\it makefile} |
178 |
|
|
% |
179 |
|
|
\end{itemize} |
180 |
|
|
% |
181 |
|
|
|
182 |
|
|
Below we describe the customizations of this files which are |
183 |
|
|
specific to this experiment. |
184 |
|
|
|
185 |
|
|
\subsubsection{File {\it .genmakerc}} |
186 |
jmc |
1.3 |
%\label{www:tutorials} |
187 |
molod |
1.1 |
This file overwrites default settings of {\it genmake}. |
188 |
|
|
In the present example it is used to switch on the following |
189 |
|
|
packages which are related to automatic differentiation |
190 |
|
|
and are disabled by default: \\ |
191 |
|
|
\hspace*{4ex} {\tt set ENABLE=( autodiff cost ctrl ecco gmredi grdchk kpp )} \\ |
192 |
|
|
Other packages which are not needed are switched off: \\ |
193 |
|
|
\hspace*{4ex} {\tt set DISABLE=( aim obcs zonal\_filt shap\_filt cal exf )} |
194 |
|
|
|
195 |
|
|
\subsubsection{File {\it COST\_CPPOPTIONS.h, CTRL\_OPTIONS.h}} |
196 |
jmc |
1.3 |
%\label{www:tutorials} |
197 |
molod |
1.1 |
|
198 |
|
|
These files used to contain package-specific CPP-options |
199 |
jmc |
1.4 |
(see Section ref:ask-the-author). |
200 |
molod |
1.1 |
For technical reasons those options have been grouped together |
201 |
|
|
in the file {\it ECCO\_OPTIONS.h}. |
202 |
|
|
To retain the modularity, the files have been kept and contain |
203 |
|
|
the standard include of the {\it CPP\_OPTIONS.h} file. |
204 |
|
|
|
205 |
|
|
\subsubsection{File {\it CPP\_EEOPTIONS.h}} |
206 |
jmc |
1.3 |
%\label{www:tutorials} |
207 |
molod |
1.1 |
|
208 |
|
|
This file contains 'wrapper'-specific CPP options. |
209 |
|
|
It only needs to be changed if the code is to be run |
210 |
jmc |
1.4 |
in a parallel environment (see Section ref:ask-the-author). |
211 |
molod |
1.1 |
|
212 |
|
|
\subsubsection{File {\it CPP\_OPTIONS.h}} |
213 |
jmc |
1.3 |
%\label{www:tutorials} |
214 |
molod |
1.1 |
|
215 |
|
|
This file contains model-specific CPP options |
216 |
jmc |
1.4 |
(see Section ref:ask-the-author). |
217 |
molod |
1.1 |
Most options are related to the forward model setup. |
218 |
|
|
They are identical to the global steady circulation setup of |
219 |
|
|
{\it verification/global\_ocean.90x40x15/}. |
220 |
|
|
The three options specific to this experiment are \\ |
221 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_PASSIVE\_TRACER} \\ |
222 |
|
|
This flag enables the code to carry through the |
223 |
|
|
advection/diffusion of a passive tracer along the |
224 |
|
|
model integration. \\ |
225 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_MIT\_ADJOINT\_RUN} \\ |
226 |
|
|
This flag enables the inclusion of some AD-related fields |
227 |
|
|
concerning initialization, link between control variables |
228 |
|
|
and forward model variables, and the call to the top-level |
229 |
|
|
forward/adjoint subroutine {\it adthe\_main\_loop} |
230 |
|
|
instead of {\it the\_main\_loop}. \\ |
231 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_GRADIENT\_CHECK} \\ |
232 |
|
|
This flag enables the gradient check package. |
233 |
|
|
After computing the unperturbed cost function and its gradient, |
234 |
|
|
a series of computations are performed for which \\ |
235 |
|
|
$\bullet$ an element of the control vector is perturbed \\ |
236 |
|
|
$\bullet$ the cost function w.r.t. the perturbed element is |
237 |
|
|
computed \\ |
238 |
|
|
$\bullet$ the difference between the perturbed and unperturbed |
239 |
|
|
cost function is computed to compute the finite difference gradient \\ |
240 |
|
|
$\bullet$ the finite difference gradient is compared with the |
241 |
|
|
adjoint-generated gradient. |
242 |
|
|
The gradient check package is further described in Section ???. |
243 |
|
|
|
244 |
|
|
\subsubsection{File {\it ECCO\_OPTIONS.h}} |
245 |
jmc |
1.3 |
%\label{www:tutorials} |
246 |
molod |
1.1 |
|
247 |
|
|
The CPP options of several AD-related packages are grouped |
248 |
|
|
in this file: |
249 |
|
|
% |
250 |
|
|
\begin{itemize} |
251 |
|
|
% |
252 |
|
|
\item |
253 |
|
|
Overall ECCO-related execution modus: \\ |
254 |
|
|
These determine whether a pure forward run, |
255 |
|
|
a sensitivity run or an iteration of optimization is |
256 |
|
|
performed. These options are not needed in the present context. |
257 |
|
|
% |
258 |
|
|
\item |
259 |
|
|
Adjoint support package: {\it pkg/autodiff/} \\ |
260 |
|
|
This package contains hand-written adjoint code such as |
261 |
|
|
active file handling, flow directives for files which must not |
262 |
|
|
be differentiated, and TAMC-specific header files. \\ |
263 |
|
|
% |
264 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_AUTODIFF\_TAMC} \\ |
265 |
|
|
defines TAMC-related features in the code. \\ |
266 |
|
|
% |
267 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_TAMC\_CHECKPOINTING} \\ |
268 |
|
|
enables the checkpointing feature of TAMC |
269 |
jmc |
1.4 |
(see Section ref:ask-the-author). |
270 |
molod |
1.1 |
In the present example a 3-level checkpointing is implemented. |
271 |
|
|
The code contains the relevant store directives, common block |
272 |
|
|
and tape initializations, storing key computation, |
273 |
|
|
and loop index handling. |
274 |
|
|
The checkpointing length at each level is defined in |
275 |
|
|
file {\it tamc.h}, cf. below. |
276 |
|
|
The out and intermediate loop directivs are contained |
277 |
|
|
in the files {\it checkpoint\_lev3\_directives.h}, |
278 |
|
|
{\it checkpoint\_lev2\_directives.h} (package {\it pkg/autodiff}). \\ |
279 |
|
|
% |
280 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_AUTODIFF\_MONITOOR} \\ |
281 |
|
|
enables the monitoring of intermediate adjoint variables |
282 |
jmc |
1.4 |
(see Section ref:ask-the-author). \\ |
283 |
molod |
1.1 |
% |
284 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_DIVIDED\_ADJOINT} \\ |
285 |
|
|
enables adjoint dump and restart |
286 |
jmc |
1.4 |
(see Section ref:ask-the-author). |
287 |
molod |
1.1 |
% |
288 |
|
|
\item Cost function package: {\it pkg/cost/} \\ |
289 |
|
|
This package contains all relevant routines for |
290 |
|
|
initializing, accumulating and finalizing the cost function |
291 |
jmc |
1.4 |
(see Section ref:ask-the-author). \\ |
292 |
molod |
1.1 |
\hspace*{4ex} {\tt \#define ALLOW\_COST} \\ |
293 |
|
|
enables all general aspects of the cost function handling, |
294 |
|
|
in particular the hooks in the forward code for |
295 |
|
|
initializing, accumulating and finalizing the cost function. \\ |
296 |
|
|
\hspace*{4ex} {\tt \#define ALLOW\_COST\_TRACER} \\ |
297 |
|
|
includes the call to the cost function for this |
298 |
|
|
particular experiment, eqn. (\ref{cost_tracer}). |
299 |
|
|
% |
300 |
|
|
\item Control variable package: {\it pkg/ctrl/} \\ |
301 |
|
|
This package contains all relevant routines for |
302 |
|
|
the handling of the control vector. |
303 |
|
|
Each control variable can be enabled/disabled with its own flag: \\ |
304 |
|
|
\begin{tabular}{ll} |
305 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_THETA0\_CONTROL} & |
306 |
|
|
initial temperature \\ |
307 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SALT0\_CONTROL} & |
308 |
|
|
initial salinity \\ |
309 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TR0\_CONTROL} & |
310 |
|
|
initial passive tracer concentration \\ |
311 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUU0\_CONTROL} & |
312 |
|
|
zonal wind stress \\ |
313 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_TAUV0\_CONTROL} & |
314 |
|
|
meridional wind stress \\ |
315 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_SFLUX0\_CONTROL} & |
316 |
|
|
freshwater flux \\ |
317 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_HFLUX0\_CONTROL} & |
318 |
|
|
heat flux \\ |
319 |
|
|
\hspace*{2ex} {\tt \#define ALLOW\_DIFFKR\_CONTROL} & |
320 |
|
|
diapycnal diffusivity \\ |
321 |
|
|
\hspace*{2ex} {\tt \#undef ALLOW\_KAPPAGM\_CONTROL} & |
322 |
|
|
isopycnal diffusivity \\ |
323 |
|
|
\end{tabular} |
324 |
|
|
% |
325 |
|
|
\end{itemize} |
326 |
|
|
|
327 |
|
|
\subsubsection{File {\it SIZE.h}} |
328 |
jmc |
1.3 |
%\label{www:tutorials} |
329 |
molod |
1.1 |
|
330 |
|
|
The file contains the grid point dimensions of the forward |
331 |
|
|
model. It is identical to the {\it verification/exp2/}: \\ |
332 |
|
|
\hspace*{4ex} {\tt sNx = 90} \\ |
333 |
|
|
\hspace*{4ex} {\tt sNy = 40} \\ |
334 |
|
|
\hspace*{4ex} {\tt Nr = 20} \\ |
335 |
|
|
It corresponds to a single-tile/single-processor setup: |
336 |
|
|
{\tt nSx = nSy = 1, nPx = nPy = 1}, |
337 |
|
|
with standard overlap dimensioning |
338 |
|
|
{\tt OLx = OLy = 3}. |
339 |
|
|
|
340 |
|
|
\subsubsection{File {\it adcommon.h}} |
341 |
jmc |
1.3 |
%\label{www:tutorials} |
342 |
molod |
1.1 |
|
343 |
|
|
This file contains common blocks of some adjoint variables |
344 |
|
|
that are generated by TAMC. |
345 |
|
|
The common blocks are used by the adjoint support routine |
346 |
|
|
{\it addummy\_in\_stepping} which needs to access those variables: |
347 |
|
|
|
348 |
|
|
\begin{tabular}{ll} |
349 |
|
|
\hspace*{4ex} {\tt common /addynvars\_r/} & |
350 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
351 |
|
|
\hspace*{4ex} {\tt common /addynvars\_cd/} & |
352 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
353 |
|
|
\hspace*{4ex} {\tt common /addynvars\_diffkr/} & |
354 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
355 |
|
|
\hspace*{4ex} {\tt common /addynvars\_kapgm/} & |
356 |
|
|
\hspace*{4ex} is related to {\it DYNVARS.h} \\ |
357 |
|
|
\hspace*{4ex} {\tt common /adtr1\_r/} & |
358 |
|
|
\hspace*{4ex} is related to {\it TR1.h} \\ |
359 |
|
|
\hspace*{4ex} {\tt common /adffields/} & |
360 |
|
|
\hspace*{4ex} is related to {\it FFIELDS.h}\\ |
361 |
|
|
\end{tabular} |
362 |
|
|
|
363 |
|
|
Note that if the structure of the common block changes in the |
364 |
|
|
above header files of the forward code, the structure |
365 |
|
|
of the adjoint common blocks will change accordingly. |
366 |
|
|
Thus, it has to be made sure that the structure of the |
367 |
|
|
adjoint common block in the hand-written file {\it adcommon.h} |
368 |
|
|
complies with the automatically generated adjoint common blocks |
369 |
|
|
in {\it adjoint\_model.F}. |
370 |
|
|
The header file is enabled via the CPP-option |
371 |
|
|
{\bf ALLOW\_AUTODIFF\_MONITOR}. |
372 |
|
|
|
373 |
|
|
\subsubsection{File {\it tamc.h}} |
374 |
jmc |
1.3 |
%\label{www:tutorials} |
375 |
molod |
1.1 |
|
376 |
|
|
This routine contains the dimensions for TAMC checkpointing |
377 |
|
|
and some indices relevant for storing ky computations. |
378 |
|
|
% |
379 |
|
|
\begin{itemize} |
380 |
|
|
% |
381 |
|
|
\item {\tt \#ifdef ALLOW\_TAMC\_CHECKPOINTING} \\ |
382 |
|
|
3-level checkpointing is enabled, i.e. the timestepping |
383 |
jmc |
1.4 |
is divided into three different levels (see Section ref:ask-the-author). |
384 |
molod |
1.1 |
The model state of the outermost ({\tt nchklev\_3}) and the |
385 |
|
|
intermediate ({\tt nchklev\_2}) timestepping loop are stored to file |
386 |
|
|
(handled in {\it the\_main\_loop}). |
387 |
|
|
The innermost loop ({\tt nchklev\_1}) |
388 |
|
|
avoids I/O by storing all required variables |
389 |
|
|
to common blocks. This storing may also be necessary if |
390 |
|
|
no checkpointing is chosen |
391 |
|
|
(nonlinear functions, if-statements, iterative loops, ...). |
392 |
|
|
In the present example the dimensions are chosen as follows: \\ |
393 |
|
|
\hspace*{4ex} {\tt nchklev\_1 = 36 } \\ |
394 |
|
|
\hspace*{4ex} {\tt nchklev\_2 = 30 } \\ |
395 |
|
|
\hspace*{4ex} {\tt nchklev\_3 = 60 } \\ |
396 |
|
|
To guarantee that the checkpointing intervals span the entire |
397 |
|
|
integration period the following relation must be satisfied: \\ |
398 |
|
|
\hspace*{4ex} {\tt nchklev\_1*nchklev\_2*nchklev\_3 $ \ge $ nTimeSteps} \\ |
399 |
|
|
where {\tt nTimeSteps} is either specified in {\it data} |
400 |
|
|
or computed via \\ |
401 |
|
|
\hspace*{4ex} {\tt nTimeSteps = (endTime-startTime)/deltaTClock }. |
402 |
|
|
% |
403 |
|
|
\item {\tt \#undef ALLOW\_TAMC\_CHECKPOINTING} \\ |
404 |
|
|
No checkpointing is enabled. |
405 |
|
|
In this case the relevant counter is {\tt nchklev\_0}. |
406 |
|
|
Similar to above, the following relation has to be satisfied \\ |
407 |
|
|
\hspace*{4ex} {\tt nchklev\_0 $ \ge $ nTimeSteps}. |
408 |
|
|
% |
409 |
|
|
\end{itemize} |
410 |
|
|
|
411 |
|
|
The following parameters may be worth describing: \\ |
412 |
|
|
% |
413 |
|
|
\hspace*{4ex} {\tt isbyte} \\ |
414 |
|
|
\hspace*{4ex} {\tt maxpass} \\ |
415 |
|
|
~ |
416 |
|
|
|
417 |
|
|
\subsubsection{File {\it makefile}} |
418 |
jmc |
1.3 |
%\label{www:tutorials} |
419 |
molod |
1.1 |
|
420 |
|
|
This file contains all relevant parameter flags and |
421 |
|
|
lists to run TAMC or TAF. |
422 |
|
|
It is assumed that TAMC is available to you, either locally, |
423 |
|
|
being installed on your network, or remotely through the 'TAMC Utility'. |
424 |
|
|
TAMC is called with the command {\tt tamc} followed by a |
425 |
|
|
number of options. They are described in detail in the |
426 |
|
|
TAMC manual \cite{gie:99}. |
427 |
|
|
Here we briefly discuss the main flags used in the {\it makefile}. |
428 |
|
|
The standard output for TAF is written to file |
429 |
|
|
{\it taf.log}. |
430 |
|
|
% |
431 |
|
|
\begin{itemize} |
432 |
|
|
\item [{\tt tamc}] {\tt |
433 |
|
|
-input <variable names> |
434 |
|
|
-output <variable name> -i4 -r4 ... \\ |
435 |
|
|
-toplevel <S/R name> -reverse <file names> |
436 |
|
|
} |
437 |
|
|
\item [{\tt taf}] {\tt |
438 |
|
|
-input <variable names> |
439 |
|
|
-output <variable name> -i4 -r4 ... \\ |
440 |
|
|
-toplevel <S/R name> -reverse <file names> \\ |
441 |
|
|
-flow taf\_flow.log -nonew\_arg |
442 |
|
|
} |
443 |
|
|
\end{itemize} |
444 |
|
|
% |
445 |
|
|
\begin{itemize} |
446 |
|
|
% |
447 |
|
|
\item {\tt -toplevel <S/R name>} \\ |
448 |
|
|
Name of the toplevel routine, with respect to which the |
449 |
|
|
control flow analysis is performed. |
450 |
|
|
% |
451 |
|
|
\item {\tt -input <variable names>} \\ |
452 |
|
|
List of independent variables $ u $ with respect to which the |
453 |
|
|
dependent variable $ J $ is differentiated. |
454 |
|
|
% |
455 |
|
|
\item {\tt -output <variable name>} \\ |
456 |
|
|
Dependent variable $ J $ which is to be differentiated. |
457 |
|
|
% |
458 |
|
|
\item {\tt -reverse <file names>} \\ |
459 |
|
|
Adjoint code is generated to compute the sensitivity of an |
460 |
|
|
independent variable w.r.t. many dependent variables. |
461 |
|
|
In the discussion of Section ??? |
462 |
|
|
the generated adjoint top-level routine computes the product |
463 |
|
|
of the transposed Jacobian matrix $ M^T $ times |
464 |
|
|
the gradient vector $ \nabla_v J $. |
465 |
|
|
\\ |
466 |
|
|
{\tt <file names>} refers to the list of files {\it .f} which are to be |
467 |
|
|
analyzed by TAMC. This list is generally smaller than the full list |
468 |
|
|
of code to be compiled. The files not contained are either |
469 |
|
|
above the top-level routine (some initializations), or are |
470 |
|
|
deliberately hidden from TAMC, either because hand-written |
471 |
|
|
adjoint routines exist, or the routines must not (or don't have to) |
472 |
|
|
be differentiated. For each routine which is part of the flow tree |
473 |
|
|
of the top-level routine, but deliberately hidden from TAMC |
474 |
|
|
(or for each package which contains such routines), |
475 |
|
|
a corresponding file {\it .flow} exists containing flow directives |
476 |
|
|
for TAMC. |
477 |
|
|
% |
478 |
|
|
\item {\tt -i4 -r4} \\ |
479 |
|
|
~ |
480 |
|
|
% |
481 |
|
|
\item {\tt -flow taf\_flow.log} \\ |
482 |
|
|
Will cause TAF to produce a flow listing file |
483 |
|
|
named {\it taf\_flow.log} in which |
484 |
|
|
the set of active and passive variables are identified |
485 |
|
|
for each subroutine. |
486 |
|
|
% |
487 |
|
|
\item {\tt -nonew\_arg} \\ |
488 |
|
|
The default in the order of the parameter list of |
489 |
|
|
adjoint routines has changed. |
490 |
|
|
Before TAF 1.3 the default was compatible with the |
491 |
|
|
TAMC-generated list. As of TAF 1.3 the order of adjoint |
492 |
|
|
routine parameter lists is no longer copatible with TAMC. |
493 |
|
|
To restore compatibility when using TAF 1.3 and higher, |
494 |
|
|
this argument is needed. |
495 |
|
|
It is currently crucial to use since all hand-written |
496 |
|
|
adjoint routines refer to the TAMC default. |
497 |
|
|
% |
498 |
|
|
\end{itemize} |
499 |
|
|
|
500 |
|
|
|
501 |
|
|
\subsubsection{The input parameter files} |
502 |
jmc |
1.3 |
%\label{www:tutorials} |
503 |
molod |
1.1 |
|
504 |
|
|
\paragraph{File {\it data}} |
505 |
|
|
|
506 |
|
|
\paragraph{File {\it data.cost}} |
507 |
|
|
|
508 |
|
|
\paragraph{File {\it data.ctrl}} |
509 |
|
|
|
510 |
|
|
\paragraph{File {\it data.gmredi}} |
511 |
|
|
|
512 |
|
|
\paragraph{File {\it data.grdchk}} |
513 |
|
|
|
514 |
|
|
\paragraph{File {\it data.optim}} |
515 |
|
|
|
516 |
|
|
\paragraph{File {\it data.pkg}} |
517 |
|
|
|
518 |
|
|
\paragraph{File {\it eedata}} |
519 |
|
|
|
520 |
|
|
\paragraph{File {\it topog.bin}} ~ \\ |
521 |
|
|
% |
522 |
|
|
Contains two-dimendional bathymetry information |
523 |
|
|
|
524 |
|
|
\paragraph{File {\it windx.bin, windy.bin, salt.bin, theta.bin, |
525 |
|
|
SSS.bin, SST.bin}} ~ \\ |
526 |
|
|
% |
527 |
|
|
These contain the initial values |
528 |
|
|
(salinity, temperature, {\it salt.bin, theta.bin}), |
529 |
|
|
surface boundary values (surface wind stresses, |
530 |
|
|
({\it windx.bin, windy.bin}), and surface restoring fields |
531 |
|
|
({\it SSS.bin, SST.bin}). |
532 |
|
|
|
533 |
|
|
\paragraph{File {\it pickup*}} ~ \\ |
534 |
|
|
% |
535 |
|
|
Contains model state after model spinup. |
536 |
|
|
|
537 |
|
|
\subsection{Compiling the model and its adjoint} |
538 |
jmc |
1.3 |
%\label{www:tutorials} |
539 |
molod |
1.1 |
|
540 |
|
|
The built process of the adjoint model is slightly more |
541 |
|
|
complex than that of compiling the forward code. |
542 |
|
|
The main reason is that the adjoint code generation requires |
543 |
|
|
a specific list of routines that are to be differentiated |
544 |
|
|
(as opposed to the automatic generation of a list of |
545 |
|
|
files to be compiled by genmake). |
546 |
|
|
This list excludes routines that don't have to be or must not be |
547 |
|
|
differentiated. For some of the latter routines flow directives |
548 |
|
|
may be necessary, a list of which has to be given as well. |
549 |
|
|
For this reason, a separate {\it makefile} is currently |
550 |
|
|
maintained in the directory {\tt adjoint/}. This |
551 |
|
|
makefile is responsible for the adjoint code generation. |
552 |
|
|
|
553 |
|
|
In the following we describe the build process step by step, |
554 |
|
|
assuming you are in the directory {\tt bin/}. |
555 |
|
|
A summary of steps to follow is given at the end. |
556 |
|
|
|
557 |
|
|
\paragraph{Adjoint code generation and compilation -- step by step} |
558 |
|
|
|
559 |
|
|
\begin{enumerate} |
560 |
|
|
% |
561 |
|
|
\item |
562 |
|
|
{\tt ln -s ../verification/???/code/.genmakerc .} \\ |
563 |
|
|
{\tt ln -s ../verification/???/code/*.[Fh] .} \\ |
564 |
|
|
Link your customized genmake options, header files, |
565 |
|
|
and modified code to the compile directory. |
566 |
|
|
% |
567 |
|
|
\item |
568 |
|
|
{\tt ../tools/genmake -makefile} \\ |
569 |
|
|
Generate your Makefile (cf. Section ???). |
570 |
|
|
% |
571 |
|
|
\item |
572 |
|
|
{\tt make depend} \\ |
573 |
|
|
Dependency analysis for the CPP pre-compiler (cf. Section ???). |
574 |
|
|
% |
575 |
|
|
\item |
576 |
|
|
{\tt cd ../adjoint} \\ |
577 |
|
|
{\tt make adtaf} or {\tt make adtamc} \\ |
578 |
|
|
Depending on whether you have TAF or TAMC at your disposal, |
579 |
|
|
you'll choose {\tt adtaf} or {\tt adtamc} as your |
580 |
|
|
make target for the {\it makefile} in the directory {\tt adjoint/}. |
581 |
|
|
Several things happen at this stage. |
582 |
|
|
% |
583 |
|
|
\begin{enumerate} |
584 |
|
|
% |
585 |
|
|
\item |
586 |
|
|
{\tt make adrestore, make ftlrestore} \\ |
587 |
|
|
The initial template files {\it adjoint\_model.F} |
588 |
|
|
and {\it tangentlinear\_model.F} in {\it pkg/autodiff} |
589 |
|
|
which are part |
590 |
|
|
of the compiling list created by {\it genmake} are restored. |
591 |
|
|
% |
592 |
|
|
\item {\tt make depend, make small\_f} \\ |
593 |
|
|
The {\tt bin/} directory is brought up to date, |
594 |
|
|
i.e. for recent changes in header or source code |
595 |
|
|
{\it .[Fh]}, corresponding {\it .f} routines are generated |
596 |
|
|
or re-generated. |
597 |
|
|
Note that here, only CPP precompiling is performed; |
598 |
|
|
no object code {\it .o} is generated as yet. |
599 |
|
|
Precompiling is necessary for TAMC to see the full code. |
600 |
|
|
% |
601 |
|
|
\item |
602 |
|
|
{\tt make allcode} \\ |
603 |
|
|
All Fortran routines {\tt *.f} in {\tt bin/} are |
604 |
|
|
concatenated into a single file called |
605 |
|
|
{\it tamc\_code.f}. |
606 |
|
|
% |
607 |
|
|
\item |
608 |
|
|
{\tt make admodeltaf/admodeltamc} \\ |
609 |
|
|
Adjoint code is generated by TAMC or TAF. |
610 |
|
|
The adjoint code is written to the file {\it tamc\_code\_ad.f}. |
611 |
|
|
It contains all adjoint routines of the forward routines |
612 |
|
|
concatenated in {\it tamc\_code.f}. |
613 |
|
|
For a given forward routines {\tt subroutine routinename} |
614 |
|
|
the adjoint routine is named {\tt adsubroutine routinename} |
615 |
|
|
by default (that default can be changed via the flag |
616 |
|
|
{\tt -admark <markname>}). |
617 |
|
|
Furthermore, it may contain modified code which |
618 |
|
|
incorporates the translation of adjoint store directives |
619 |
|
|
into specific Fortran code. |
620 |
|
|
For a given forward routines {\tt subroutine routinename} |
621 |
|
|
the modified routine is named {\tt mdsubroutine routinename}. |
622 |
|
|
TAMC or TAF info is written to file |
623 |
|
|
{\it tamc\_code.prot} or {\it taf.log}, respectively. |
624 |
|
|
% |
625 |
|
|
\item |
626 |
|
|
{\tt make adchange} \\ |
627 |
|
|
The multi-threading capability of MITgcm requires a slight |
628 |
|
|
change in the parameter list of some routines that are related to |
629 |
|
|
to active file handling. |
630 |
|
|
This post-processing invokes the sed script {\it adjoint\_ecco\_sed.com} |
631 |
|
|
to insert the threading counter {\bf myThId} into the parameter list |
632 |
|
|
of those subroutines. |
633 |
|
|
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
634 |
|
|
and appended to the file {\it adjoint\_model.F}. |
635 |
|
|
This concludes the adjoint code generation. |
636 |
|
|
% |
637 |
|
|
\end{enumerate} |
638 |
|
|
% |
639 |
|
|
\item |
640 |
|
|
{\tt cd ../bin} \\ |
641 |
|
|
{\tt make} \\ |
642 |
|
|
The file {\it adjoint\_model.F} now contains the full adjoint code. |
643 |
|
|
All routines are now compiled. |
644 |
|
|
% |
645 |
|
|
\end{enumerate} |
646 |
|
|
|
647 |
|
|
N.B.: The targets {\tt make adtaf/adtamc} now comprise a |
648 |
|
|
series of targets that in previous versions had to be |
649 |
|
|
invoked separarely. This was probably preferable at a more |
650 |
|
|
experimental stage, but has now been dropped in favour of |
651 |
|
|
a more straightforward build process. |
652 |
|
|
|
653 |
|
|
|
654 |
|
|
\paragraph{Adjoint code generation and compilation -- summary} |
655 |
|
|
~ \\ |
656 |
|
|
|
657 |
|
|
{\small |
658 |
|
|
\[ |
659 |
|
|
\boxed{ |
660 |
jmc |
1.3 |
\begin{aligned} |
661 |
molod |
1.1 |
~ & \mbox{\tt cd bin} \\ |
662 |
|
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/.genmakerc .} \\ |
663 |
|
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/*.[Fh] .} \\ |
664 |
|
|
~ & \mbox{\tt ../tools/genmake -makefile} \\ |
665 |
|
|
~ & \mbox{\tt make depend} \\ |
666 |
|
|
~ & \mbox{\tt cd ../adjoint} \\ |
667 |
|
|
~ & \mbox{\tt make adtaf <OR: make adtamc>} \\ |
668 |
|
|
~ & \hspace*{6ex} \mbox{\tt contains the targets:} \\ |
669 |
|
|
~ & \hspace*{6ex} \mbox{\tt adrestore small\_f allcode admodeltaf/admodeltamc adchange} \\ |
670 |
|
|
~ & \mbox{\tt cd ../bin} \\ |
671 |
|
|
~ & \mbox{\tt make} \\ |
672 |
jmc |
1.3 |
\end{aligned} |
673 |
molod |
1.1 |
} |
674 |
|
|
\] |
675 |
|
|
} |