| 570 |
A method to balance the amount of recomputations vs. |
A method to balance the amount of recomputations vs. |
| 571 |
storage requirements is called {\sf checkpointing} |
storage requirements is called {\sf checkpointing} |
| 572 |
(e.g. \cite{res-eta:98}). |
(e.g. \cite{res-eta:98}). |
| 573 |
It is depicted in \reffig{3levelcheck} for a 3-level checkpointing |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
| 574 |
[as an example, we give explicit numbers for a 3-day |
[as an example, we give explicit numbers for a 3-day |
| 575 |
integration with a 1-hourly timestep in square brackets]. |
integration with a 1-hourly timestep in square brackets]. |
| 576 |
\begin{itemize} |
\begin{itemize} |
| 637 |
on the computing resources available. |
on the computing resources available. |
| 638 |
|
|
| 639 |
\begin{figure}[t!] |
\begin{figure}[t!] |
| 640 |
\centering |
\begin{center} |
| 641 |
%\psdraft |
%\psdraft |
| 642 |
\psfrag{v_k1^lev3}{\mathinfigure{v_{k_{1}^{lev3}}}} |
%\psfrag{v_k1^lev3}{\mathinfigure{v_{k_{1}^{lev3}}}} |
| 643 |
\psfrag{v_kn-1^lev3}{\mathinfigure{v_{k_{n-1}^{lev3}}}} |
%\psfrag{v_kn-1^lev3}{\mathinfigure{v_{k_{n-1}^{lev3}}}} |
| 644 |
\psfrag{v_kn^lev3}{\mathinfigure{v_{k_{n}^{lev3}}}} |
%\psfrag{v_kn^lev3}{\mathinfigure{v_{k_{n}^{lev3}}}} |
| 645 |
\psfrag{v_k1^lev2}{\mathinfigure{v_{k_{1}^{lev2}}}} |
%\psfrag{v_k1^lev2}{\mathinfigure{v_{k_{1}^{lev2}}}} |
| 646 |
\psfrag{v_kn-1^lev2}{\mathinfigure{v_{k_{n-1}^{lev2}}}} |
%\psfrag{v_kn-1^lev2}{\mathinfigure{v_{k_{n-1}^{lev2}}}} |
| 647 |
\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
%\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
| 648 |
\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
%\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
| 649 |
\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
%\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
| 650 |
\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
%\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
| 651 |
|
\resizebox{5.5in}{!}{\includegraphics{part5/checkpointing.eps}} |
| 652 |
%\psfull |
%\psfull |
| 653 |
\caption |
\end{center} |
| 654 |
{Schematic view of intermediate dump and restart for |
\caption{ |
| 655 |
|
Schematic view of intermediate dump and restart for |
| 656 |
3-level checkpointing.} |
3-level checkpointing.} |
| 657 |
\label{fig:3levelcheck} |
\label{fig:3levelcheck} |
| 658 |
\end{figure} |
\end{figure} |
| 679 |
We take as an example the sensitivity of carbon sequestration |
We take as an example the sensitivity of carbon sequestration |
| 680 |
in the ocean. |
in the ocean. |
| 681 |
The AD-relevant hooks in the code are sketched in |
The AD-relevant hooks in the code are sketched in |
| 682 |
\reffig{adthemodel}, \reffig{adthemain}. |
\ref{fig:adthemodel}, \ref{fig:adthemain}. |
| 683 |
|
|
| 684 |
\subsection{Overview of the experiment} |
\subsection{Overview of the experiment} |
| 685 |
|
|
| 1093 |
\end{itemize} |
\end{itemize} |
| 1094 |
|
|
| 1095 |
|
|
| 1096 |
\subsubsection{File {\it data}} |
\subsubsection{The input parameter files} |
| 1097 |
|
|
| 1098 |
\subsubsection{File {\it data.cost}} |
\paragraph{File {\it data}} |
| 1099 |
|
|
| 1100 |
\subsubsection{File {\it data.ctrl}} |
\paragraph{File {\it data.cost}} |
| 1101 |
|
|
| 1102 |
\subsubsection{File {\it data.gmredi}} |
\paragraph{File {\it data.ctrl}} |
| 1103 |
|
|
| 1104 |
\subsubsection{File {\it data.grdchk}} |
\paragraph{File {\it data.gmredi}} |
| 1105 |
|
|
| 1106 |
\subsubsection{File {\it data.optim}} |
\paragraph{File {\it data.grdchk}} |
| 1107 |
|
|
| 1108 |
\subsubsection{File {\it data.pkg}} |
\paragraph{File {\it data.optim}} |
| 1109 |
|
|
| 1110 |
\subsubsection{File {\it eedata}} |
\paragraph{File {\it data.pkg}} |
| 1111 |
|
|
| 1112 |
\subsubsection{File {\it topog.bin}} |
\paragraph{File {\it eedata}} |
| 1113 |
|
|
| 1114 |
\subsubsection{File {\it windx.bin, windy.bin}} |
\paragraph{File {\it topog.bin}} |
| 1115 |
|
|
| 1116 |
\subsubsection{File {\it salt.bin, theta.bin}} |
\paragraph{File {\it windx.bin, windy.bin}} |
| 1117 |
|
|
| 1118 |
\subsubsection{File {\it SSS.bin, SST.bin}} |
\paragraph{File {\it salt.bin, theta.bin}} |
| 1119 |
|
|
| 1120 |
\subsubsection{File {\it pickup*}} |
\paragraph{File {\it SSS.bin, SST.bin}} |
| 1121 |
|
|
| 1122 |
\subsection{Compiling the model and its adjoint} |
\paragraph{File {\it pickup*}} |
| 1123 |
|
|
| 1124 |
|
\subsection{Compiling the model and its adjoint} |
| 1125 |
|
|
| 1126 |
|
The built process of the adjoint model is slightly more |
| 1127 |
|
complex than that of compiling the forward code. |
| 1128 |
|
The main reason is that the adjoint code generation requires |
| 1129 |
|
a specific list of routines that are to be differentiated |
| 1130 |
|
(as opposed to the automatic generation of a list of |
| 1131 |
|
files to be compiled by genmake). |
| 1132 |
|
This list excludes routines that don't have to be or must not be |
| 1133 |
|
differentiated. For some of the latter routines flow directives |
| 1134 |
|
may be necessary, a list of which has to be given as well. |
| 1135 |
|
For this reason, a separate {\it makefile} is currently |
| 1136 |
|
maintained in the directory {\tt adjoint/}. This |
| 1137 |
|
makefile is responsible for the adjoint code generation. |
| 1138 |
|
|
| 1139 |
|
In the following we describe the build process step by step, |
| 1140 |
|
assuming you are in the directory {\tt bin/}. |
| 1141 |
|
A summary of steps to follow is given at the end. |
| 1142 |
|
|
| 1143 |
|
\paragraph{Adjoint code generation and compilation -- step by step} |
| 1144 |
|
|
| 1145 |
|
\begin{enumerate} |
| 1146 |
|
% |
| 1147 |
|
\item |
| 1148 |
|
{\tt ln -s ../verification/???/code/.genmakerc .} \\ |
| 1149 |
|
{\tt ln -s ../verification/???/code/*.[Fh] .} \\ |
| 1150 |
|
Link your customized genmake options, header files, |
| 1151 |
|
and modified code to the compile directory. |
| 1152 |
|
% |
| 1153 |
|
\item |
| 1154 |
|
{\tt ../tools/genmake -makefile} \\ |
| 1155 |
|
Generate your Makefile (cf. Section ???). |
| 1156 |
|
% |
| 1157 |
|
\item |
| 1158 |
|
{\tt make depend} \\ |
| 1159 |
|
Dependency analysis for the CPP pre-compiler (cf. Section ???). |
| 1160 |
|
% |
| 1161 |
|
\item |
| 1162 |
|
{\tt make small\_f} \\ |
| 1163 |
|
This is the first difference between forward code compilation |
| 1164 |
|
and adjoint code generation and compilation. |
| 1165 |
|
Instead of going through the entire compilation process |
| 1166 |
|
(CPP precompiling -- {\tt .f}, object code generation -- {\tt .o}, |
| 1167 |
|
linking of object files and libraries to generate executable), |
| 1168 |
|
only the CPP compiler is invoked at this stage to generate |
| 1169 |
|
the {\tt .f} files. |
| 1170 |
|
% |
| 1171 |
|
\item |
| 1172 |
|
{\tt cd ../adjoint} \\ |
| 1173 |
|
{\tt make adtaf} or {\tt make adtamc} \\ |
| 1174 |
|
Depending on whether you have TAF or TAMC at your disposal, |
| 1175 |
|
you'll choose {\tt adtaf} or {\tt adtamc} as your |
| 1176 |
|
make target for the {\it makefile} in the directory {\tt adjoint/}. |
| 1177 |
|
Several things happen at this stage. |
| 1178 |
|
% |
| 1179 |
|
\begin{enumerate} |
| 1180 |
|
% |
| 1181 |
|
\item |
| 1182 |
|
The initial template file {\it adjoint\_model.F} which is part |
| 1183 |
|
of the compiling list created by {\it genmake} is restored. |
| 1184 |
|
% |
| 1185 |
|
\item |
| 1186 |
|
All Fortran routines {\tt *.f} in {\tt bin/} are |
| 1187 |
|
concatenated into a single file (it's current name is |
| 1188 |
|
{\it tamc\_code.f}). |
| 1189 |
|
% |
| 1190 |
|
\item |
| 1191 |
|
Adjoint code is generated by TAMC or TAF. |
| 1192 |
|
The adjoint code is written to the file {\it tamc\_code\_ad.f}. |
| 1193 |
|
It contains all adjoint routines of the forward routines |
| 1194 |
|
concatenated in {\it tamc\_code.f}. |
| 1195 |
|
For a given forward routines {\tt subroutine routinename} |
| 1196 |
|
the adjoint routine is named {\tt adsubroutine routinename} |
| 1197 |
|
by default (that default can be changed via the flag |
| 1198 |
|
{\tt -admark <markname>}). |
| 1199 |
|
Furthermore, it may contain modified code which |
| 1200 |
|
incorporates the translation of adjoint store directives |
| 1201 |
|
into specific Fortran code. |
| 1202 |
|
For a given forward routines {\tt subroutine routinename} |
| 1203 |
|
the modified routine is named {\tt mdsubroutine routinename}. |
| 1204 |
|
TAMC or TAF info is written to file |
| 1205 |
|
{\it tamc\_code.prot} or {\it taf.log}, respectively. |
| 1206 |
|
% |
| 1207 |
|
\end{enumerate} |
| 1208 |
|
% |
| 1209 |
|
\item |
| 1210 |
|
{\tt make adchange} \\ |
| 1211 |
|
The multi-threading capability of the MITGCM requires a slight |
| 1212 |
|
change in the parameter list of some routines that are related to |
| 1213 |
|
to active file handling. |
| 1214 |
|
This postprocessing invokes the sed script {\it adjoint\_ecco\_sed.com} |
| 1215 |
|
to insert the threading counter {\bf myThId} into the parameter list |
| 1216 |
|
of those subroutines. |
| 1217 |
|
The resulting code is written to file {\it tamc\_code\_sed\_ad.f} |
| 1218 |
|
and appended to the file {\it adjoint\_model.F}. |
| 1219 |
|
This concludes the adjoint codel generation. |
| 1220 |
|
% |
| 1221 |
|
\item |
| 1222 |
|
{\tt cd ../bin} \\ |
| 1223 |
|
{\tt make} \\ |
| 1224 |
|
The file {\it adjoint\_model.F} now contains the full adjoint code. |
| 1225 |
|
All routines are now compiled. |
| 1226 |
|
% |
| 1227 |
|
\end{enumerate} |
| 1228 |
|
|
| 1229 |
|
\paragraph{Adjoint code generation and compilation -- summary} |
| 1230 |
|
~ \\ |
| 1231 |
|
|
| 1232 |
|
\[ |
| 1233 |
|
\boxed{ |
| 1234 |
|
\begin{split} |
| 1235 |
|
~ & \mbox{\tt cd bin} \\ |
| 1236 |
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/.genmakerc .} \\ |
| 1237 |
|
~ & \mbox{\tt ln -s ../verification/my\_experiment/code/*.[Fh] .} \\ |
| 1238 |
|
~ & \mbox{\tt ../tools/genmake -makefile} \\ |
| 1239 |
|
~ & \mbox{\tt make depend} \\ |
| 1240 |
|
~ & \mbox{\tt make small\_f} \\ |
| 1241 |
|
~ & \mbox{\tt cd ../adjoint} \\ |
| 1242 |
|
~ & \mbox{\tt make adtaf <OR: make adtamc>} \\ |
| 1243 |
|
~ & \mbox{\tt make adchange} \\ |
| 1244 |
|
~ & \mbox{\tt cd ../bin} \\ |
| 1245 |
|
~ & \mbox{\tt make} \\ |
| 1246 |
|
\end{split} |
| 1247 |
|
} |
| 1248 |
|
\] |
| 1249 |
|
|
| 1250 |
\newpage |
\newpage |
| 1251 |
|
|
| 1264 |
\label{fig:adthemodel} |
\label{fig:adthemodel} |
| 1265 |
\end{figure} |
\end{figure} |
| 1266 |
|
|
| 1267 |
The basic flow is depicted in \reffig{adthemodel}. |
The basic flow is depicted in \ref{fig:adthemodel}. |
| 1268 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
| 1269 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
| 1270 |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch