| 1 |
% $Header$ |
% $Header$ |
| 2 |
% $Name$ |
% $Name$ |
| 3 |
|
|
| 4 |
|
Author: Patrick Heimbach |
| 5 |
|
|
| 6 |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
| 7 |
(or, more loosely, computational) differentiation, involves |
(or, more loosely, computational) differentiation, involves |
| 8 |
automatically deriving code to calculate |
automatically deriving code to calculate partial derivatives from an |
| 9 |
partial derivatives from an existing fully non-linear prognostic code. |
existing fully non-linear prognostic code. (see \cite{gri:00}). A |
| 10 |
(see \cite{gri:00}). |
software tool is used that parses and transforms source files |
| 11 |
A software tool is used that parses and transforms source files |
according to a set of linguistic and mathematical rules. AD tools are |
| 12 |
according to a set of linguistic and mathematical rules. |
like source-to-source translators in that they parse a program code as |
| 13 |
AD tools are like source-to-source translators in that |
input and produce a new program code as output |
| 14 |
they parse a program code as input and produce a new program code |
(we restrict our discussion to source-to-source tools, ignoring |
| 15 |
as output. |
operator-overloading tools). However, unlike a |
| 16 |
However, unlike a pure source-to-source translation, the output program |
pure source-to-source translation, the output program represents a new |
| 17 |
represents a new algorithm, such as the evaluation of the |
algorithm, such as the evaluation of the Jacobian, the Hessian, or |
| 18 |
Jacobian, the Hessian, or higher derivative operators. |
higher derivative operators. In principle, a variety of derived |
| 19 |
In principle, a variety of derived algorithms |
algorithms can be generated automatically in this way. |
| 20 |
can be generated automatically in this way. |
|
| 21 |
|
MITgcm has been adapted for use with the Tangent linear and Adjoint |
| 22 |
The MITGCM has been adapted for use with the |
Model Compiler (TAMC) and its successor TAF (Transformation of |
| 23 |
Tangent linear and Adjoint Model Compiler (TAMC) and its successor TAF |
Algorithms in Fortran), developed by Ralf Giering (\cite{gie-kam:98}, |
| 24 |
(Transformation of Algorithms in Fortran), developed |
\cite{gie:99,gie:00}). The first application of the adjoint of MITgcm |
| 25 |
by Ralf Giering (\cite{gie-kam:98}, \cite{gie:99,gie:00}). |
for sensitivity studies has been published by \cite{maro-eta:99}. |
| 26 |
The first application of the adjoint of the MITGCM for sensitivity |
\cite{stam-etal:97,stam-etal:02} use MITgcm and its adjoint for ocean |
| 27 |
studies has been published by \cite{maro-eta:99}. |
state estimation studies. In the following we shall refer to TAMC and |
| 28 |
\cite{sta-eta:97,sta-eta:01} use the MITGCM and its adjoint |
TAF synonymously, except were explicitly stated otherwise. |
| 29 |
for ocean state estimation studies. |
|
| 30 |
In the following we shall refer to TAMC and TAF synonymously, |
As of mid-2007 we are also able to generate fairly efficient |
| 31 |
except were explicitly stated otherwise. |
adjoint code of the MITgcm using a new, open-source AD tool, |
| 32 |
|
called OpenAD (see \cite{naum-etal:06,utke-etal:08}. |
| 33 |
TAMC exploits the chain rule for computing the first |
This enables us for the first time to compare adjoint models |
| 34 |
derivative of a function with |
generated from different AD tools, providing an additional |
| 35 |
respect to a set of input variables. |
accuracy check, complementary to finite-difference gradient checks. |
| 36 |
Treating a given forward code as a composition of operations -- |
OpenAD and its application to MITgcm is described in detail |
| 37 |
each line representing a compositional element, the chain rule is |
in section \ref{sec_ad_openad}. |
| 38 |
rigorously applied to the code, line by line. The resulting |
|
| 39 |
tangent linear or adjoint code, |
The AD tool exploits the chain rule for computing the first derivative of a |
| 40 |
then, may be thought of as the composition in |
function with respect to a set of input variables. Treating a given |
| 41 |
forward or reverse order, respectively, of the |
forward code as a composition of operations -- each line representing |
| 42 |
Jacobian matrices of the forward code's compositional elements. |
a compositional element, the chain rule is rigorously applied to the |
| 43 |
|
code, line by line. The resulting tangent linear or adjoint code, |
| 44 |
|
then, may be thought of as the composition in forward or reverse |
| 45 |
|
order, respectively, of the Jacobian matrices of the forward code's |
| 46 |
|
compositional elements. |
| 47 |
|
|
| 48 |
%********************************************************************** |
%********************************************************************** |
| 49 |
\section{Some basic algebra} |
\section{Some basic algebra} |
| 50 |
\label{sec_ad_algebra} |
\label{sec_ad_algebra} |
| 51 |
|
\begin{rawhtml} |
| 52 |
|
<!-- CMIREDIR:sec_ad_algebra: --> |
| 53 |
|
\end{rawhtml} |
| 54 |
%********************************************************************** |
%********************************************************************** |
| 55 |
|
|
| 56 |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
| 566 |
(a derivative is defined w.r.t. a point along the trajectory), |
(a derivative is defined w.r.t. a point along the trajectory), |
| 567 |
the intermediate results of the model trajectory |
the intermediate results of the model trajectory |
| 568 |
$\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$ |
$\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$ |
| 569 |
are needed to evaluate the intermediate Jacobian |
may be required to evaluate the intermediate Jacobian |
| 570 |
$M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $. |
$M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $. |
| 571 |
|
This is the case e.g. for nonlinear expressions |
| 572 |
|
(momentum advection, nonlinear equation of state), state-dependent |
| 573 |
|
conditional statements (parameterization schemes). |
| 574 |
In the forward mode, the intermediate results are required |
In the forward mode, the intermediate results are required |
| 575 |
in the same order as computed by the full forward model ${\cal M}$, |
in the same order as computed by the full forward model ${\cal M}$, |
| 576 |
but in the reverse mode they are required in the reverse order. |
but in the reverse mode they are required in the reverse order. |
| 581 |
|
|
| 582 |
A method to balance the amount of recomputations vs. |
A method to balance the amount of recomputations vs. |
| 583 |
storage requirements is called {\sf checkpointing} |
storage requirements is called {\sf checkpointing} |
| 584 |
(e.g. \cite{res-eta:98}). |
(e.g. \cite{gri:92}, \cite{res-eta:98}). |
| 585 |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
| 586 |
[as an example, we give explicit numbers for a 3-day |
[as an example, we give explicit numbers for a 3-day |
| 587 |
integration with a 1-hourly timestep in square brackets]. |
integration with a 1-hourly timestep in square brackets]. |
| 592 |
$ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals], |
$ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals], |
| 593 |
with the label $lev3$ for this outermost loop. |
with the label $lev3$ for this outermost loop. |
| 594 |
The model is then integrated along the full trajectory, |
The model is then integrated along the full trajectory, |
| 595 |
and the model state stored only at every $ k_{i}^{lev3} $-th timestep |
and the model state stored to disk only at every $ k_{i}^{lev3} $-th timestep |
| 596 |
[i.e. 3 times, at |
[i.e. 3 times, at |
| 597 |
$ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $]. |
$ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $]. |
| 598 |
|
In addition, the cost function is computed, if needed. |
| 599 |
% |
% |
| 600 |
\item [$lev2$] |
\item [$lev2$] |
| 601 |
In a second step each subsection itself is divided into |
In a second step each subsection itself is divided into |
| 602 |
$ {n}^{lev2} $ sub-subsections |
$ {n}^{lev2} $ subsections |
| 603 |
[$ {n}^{lev2} $=4 6-hour intervals per subsection]. |
[$ {n}^{lev2} $=4 6-hour intervals per subsection]. |
| 604 |
The model picks up at the last outermost dumped state |
The model picks up at the last outermost dumped state |
| 605 |
$ v_{k_{n}^{lev3}} $ and is integrated forward in time along |
$ v_{k_{n}^{lev3}} $ and is integrated forward in time along |
| 606 |
the last subsection, with the label $lev2$ for this |
the last subsection, with the label $lev2$ for this |
| 607 |
intermediate loop. |
intermediate loop. |
| 608 |
The model state is now stored at every $ k_{i}^{lev2} $-th |
The model state is now stored to disk at every $ k_{i}^{lev2} $-th |
| 609 |
timestep |
timestep |
| 610 |
[i.e. 4 times, at |
[i.e. 4 times, at |
| 611 |
$ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $]. |
$ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $]. |
| 613 |
\item [$lev1$] |
\item [$lev1$] |
| 614 |
Finally, the model picks up at the last intermediate dump state |
Finally, the model picks up at the last intermediate dump state |
| 615 |
$ v_{k_{n}^{lev2}} $ and is integrated forward in time along |
$ v_{k_{n}^{lev2}} $ and is integrated forward in time along |
| 616 |
the last sub-subsection, with the label $lev1$ for this |
the last subsection, with the label $lev1$ for this |
| 617 |
intermediate loop. |
intermediate loop. |
| 618 |
Within this sub-subsection only, the model state is stored |
Within this sub-subsection only, parts of the model state is stored |
| 619 |
at every timestep |
to memory at every timestep |
| 620 |
[i.e. every hour $ i=0,...,5$ corresponding to |
[i.e. every hour $ i=0,...,5$ corresponding to |
| 621 |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
| 622 |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
The final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
| 623 |
and the model state of all proceeding timesteps along the last |
and the model state of all preceding timesteps along the last |
| 624 |
sub-subsections are available, enabling integration backwards |
innermost subsection are available, enabling integration backwards |
| 625 |
in time along the last sub-subsection. |
in time along the last subsection. |
| 626 |
Thus, the adjoint can be computed along this last |
The adjoint can thus be computed along this last |
| 627 |
sub-subsection $k_{n}^{lev2}$. |
subsection $k_{n}^{lev2}$. |
| 628 |
% |
% |
| 629 |
\end{itemize} |
\end{itemize} |
| 630 |
% |
% |
| 631 |
This procedure is repeated consecutively for each previous |
This procedure is repeated consecutively for each previous |
| 632 |
sub-subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $ |
subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $ |
| 633 |
carrying the adjoint computation to the initial time |
carrying the adjoint computation to the initial time |
| 634 |
of the subsection $k_{n}^{lev3}$. |
of the subsection $k_{n}^{lev3}$. |
| 635 |
Then, the procedure is repeated for the previous subsection |
Then, the procedure is repeated for the previous subsection |
| 640 |
For the full model trajectory of |
For the full model trajectory of |
| 641 |
$ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps |
$ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps |
| 642 |
the required storing of the model state was significantly reduced to |
the required storing of the model state was significantly reduced to |
| 643 |
$ n^{lev1} + n^{lev2} + n^{lev3} $ |
$ n^{lev2} + n^{lev3} $ to disk and roughly $ n^{lev1} $ to memory |
| 644 |
[i.e. for the 3-day integration with a total oof 72 timesteps |
[i.e. for the 3-day integration with a total oof 72 timesteps |
| 645 |
the model state was stored 13 times]. |
the model state was stored 7 times to disk and roughly 6 times |
| 646 |
|
to memory]. |
| 647 |
This saving in memory comes at a cost of a required |
This saving in memory comes at a cost of a required |
| 648 |
3 full forward integrations of the model (one for each |
3 full forward integrations of the model (one for each |
| 649 |
checkpointing level). |
checkpointing level). |
| 650 |
The balance of storage vs. recomputation certainly depends |
The optimal balance of storage vs. recomputation certainly depends |
| 651 |
on the computing resources available. |
on the computing resources available and may be adjusted by |
| 652 |
|
adjusting the partitioning among the |
| 653 |
|
$ n^{lev3}, \,\, n^{lev2}, \,\, n^{lev1} $. |
| 654 |
|
|
| 655 |
\begin{figure}[t!] |
\begin{figure}[t!] |
| 656 |
\begin{center} |
\begin{center} |
| 663 |
%\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
%\psfrag{v_kn^lev2}{\mathinfigure{v_{k_{n}^{lev2}}}} |
| 664 |
%\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
%\psfrag{v_k1^lev1}{\mathinfigure{v_{k_{1}^{lev1}}}} |
| 665 |
%\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
%\psfrag{v_kn^lev1}{\mathinfigure{v_{k_{n}^{lev1}}}} |
| 666 |
%\mbox{\epsfig{file=part5/checkpointing.eps, width=0.8\textwidth}} |
%\mbox{\epsfig{file=s_autodiff/figs/checkpointing.eps, width=0.8\textwidth}} |
| 667 |
\resizebox{5.5in}{!}{\includegraphics{part5/checkpointing.eps}} |
\resizebox{5.5in}{!}{\includegraphics{s_autodiff/figs/checkpointing.eps}} |
| 668 |
%\psfull |
%\psfull |
| 669 |
\end{center} |
\end{center} |
| 670 |
\caption{ |
\caption{ |
| 685 |
%********************************************************************** |
%********************************************************************** |
| 686 |
\section{TLM and ADM generation in general} |
\section{TLM and ADM generation in general} |
| 687 |
\label{sec_ad_setup_gen} |
\label{sec_ad_setup_gen} |
| 688 |
|
\begin{rawhtml} |
| 689 |
|
<!-- CMIREDIR:sec_ad_setup_gen: --> |
| 690 |
|
\end{rawhtml} |
| 691 |
%********************************************************************** |
%********************************************************************** |
| 692 |
|
|
| 693 |
In this section we describe in a general fashion |
In this section we describe in a general fashion |
| 694 |
the parts of the code that are relevant for automatic |
the parts of the code that are relevant for automatic |
| 695 |
differentiation using the software tool TAMC. |
differentiation using the software tool TAF. |
| 696 |
|
Modifications to use OpenAD are described in \ref{sec_ad_openad}. |
| 697 |
|
|
| 698 |
\input{part5/doc_ad_the_model} |
\input{s_autodiff/text/doc_ad_the_model} |
| 699 |
|
|
| 700 |
The basic flow is depicted in \ref{fig:adthemodel}. |
The basic flow is depicted in \ref{fig:adthemodel}. |
| 701 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
If CPP option \texttt{ALLOW\_AUTODIFF\_TAMC} is defined, |
| 702 |
|
the driver routine |
| 703 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
| 704 |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
invokes the adjoint of this routine, {\it adthe\_main\_loop} |
| 705 |
which is the toplevel routine in terms of reverse mode computation. |
(case \texttt{\#define ALLOW\_ADJOINT\_RUN}), or |
| 706 |
The routine {\it adthe\_main\_loop} has been generated using TAMC. |
the tangent linear of this routine {\it g\_the\_main\_loop} |
| 707 |
It contains both the forward integration of the full model, |
(case \texttt{\#define ALLOW\_TANGENTLINEAR\_RUN}), |
| 708 |
|
which are the toplevel routines in terms of automatic differentiation. |
| 709 |
|
The routines {\it adthe\_main\_loop} or {\it g\_the\_main\_loop} |
| 710 |
|
are generated by TAF. |
| 711 |
|
It contains both the forward integration of the full model, the |
| 712 |
|
cost function calculation, |
| 713 |
any additional storing that is required for efficient checkpointing, |
any additional storing that is required for efficient checkpointing, |
| 714 |
and the reverse integration of the adjoint model. |
and the reverse integration of the adjoint model. |
| 715 |
The structure of {\it adthe\_main\_loop} has been strongly |
|
| 716 |
simplified for clarification; in particular, no checkpointing |
[DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM] |
| 717 |
|
|
| 718 |
|
In Fig. \ref{fig:adthemodel} |
| 719 |
|
the structure of {\it adthe\_main\_loop} has been strongly |
| 720 |
|
simplified to focus on the essentials; in particular, no checkpointing |
| 721 |
procedures are shown here. |
procedures are shown here. |
| 722 |
Prior to the call of {\it adthe\_main\_loop}, the routine |
Prior to the call of {\it adthe\_main\_loop}, the routine |
| 723 |
{\it ctrl\_unpack} is invoked to unpack the control vector, |
{\it ctrl\_unpack} is invoked to unpack the control vector |
| 724 |
and following that call, the routine {\it ctrl\_pack} |
or initialise the control variables. |
| 725 |
|
Following the call of {\it adthe\_main\_loop}, |
| 726 |
|
the routine {\it ctrl\_pack} |
| 727 |
is invoked to pack the control vector |
is invoked to pack the control vector |
| 728 |
(cf. Section \ref{section_ctrl}). |
(cf. Section \ref{section_ctrl}). |
| 729 |
If gradient checks are to be performed, the option |
If gradient checks are to be performed, the option |
| 732 |
the gradient has been computed via the adjoint |
the gradient has been computed via the adjoint |
| 733 |
(cf. Section \ref{section_grdchk}). |
(cf. Section \ref{section_grdchk}). |
| 734 |
|
|
| 735 |
|
%------------------------------------------------------------------ |
| 736 |
|
|
| 737 |
|
\subsection{General setup |
| 738 |
|
\label{section_ad_setup}} |
| 739 |
|
|
| 740 |
|
In order to configure AD-related setups the following packages need |
| 741 |
|
to be enabled: |
| 742 |
|
{\it |
| 743 |
|
\begin{table}[h!] |
| 744 |
|
\begin{tabular}{l} |
| 745 |
|
autodiff \\ |
| 746 |
|
ctrl \\ |
| 747 |
|
cost \\ |
| 748 |
|
grdchk \\ |
| 749 |
|
\end{tabular} |
| 750 |
|
\end{table} |
| 751 |
|
} |
| 752 |
|
The packages are enabled by adding them to your experiment-specific |
| 753 |
|
configuration file |
| 754 |
|
{\it packages.conf} (see Section ???). |
| 755 |
|
|
| 756 |
|
The following AD-specific CPP option files need to be customized: |
| 757 |
|
% |
| 758 |
|
\begin{itemize} |
| 759 |
|
% |
| 760 |
|
\item {\it ECCO\_CPPOPTIONS.h} \\ |
| 761 |
|
This header file collects CPP options for the packages |
| 762 |
|
{\it autodiff, cost, ctrl} as well as AD-unrelated options for |
| 763 |
|
the external forcing package {\it exf}. |
| 764 |
|
\footnote{NOTE: These options are not set in their package-specific |
| 765 |
|
headers such as {\it COST\_CPPOPTIONS.h}, but are instead collected |
| 766 |
|
in the single header file {\it ECCO\_CPPOPTIONS.h}. |
| 767 |
|
The package-specific header files serve as simple |
| 768 |
|
placeholders at this point.} |
| 769 |
|
% |
| 770 |
|
\item {\it tamc.h} \\ |
| 771 |
|
This header configures the splitting of the time stepping loop |
| 772 |
|
w.r.t. the 3-level checkpointing (see section ???). |
| 773 |
|
|
| 774 |
|
% |
| 775 |
|
\end{itemize} |
| 776 |
|
|
| 777 |
|
%------------------------------------------------------------------ |
| 778 |
|
|
| 779 |
|
\subsection{Building the AD code using TAF |
| 780 |
|
\label{section_ad_build}} |
| 781 |
|
|
| 782 |
|
The build process of an AD code is very similar to building |
| 783 |
|
the forward model. However, depending on which AD code one wishes |
| 784 |
|
to generate, and on which AD tool is available (TAF or TAMC), |
| 785 |
|
the following {\tt make} targets are available: |
| 786 |
|
|
| 787 |
|
\begin{table}[h!] |
| 788 |
|
{\footnotesize |
| 789 |
|
\begin{tabular}{|ccll|} |
| 790 |
|
\hline |
| 791 |
|
~ & {\it AD-target} & {\it output} & {\it description} \\ |
| 792 |
|
\hline |
| 793 |
|
\hline |
| 794 |
|
(1) & {\tt <MODE><TOOL>only} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 795 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 796 |
|
~ & ~ & ~ & no {\tt make} dependencies on {\tt .F .h} \\ |
| 797 |
|
~ & ~ & ~ & useful for compiling on remote platforms \\ |
| 798 |
|
\hline |
| 799 |
|
(2) & {\tt <MODE><TOOL>} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 800 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 801 |
|
~ & ~ & ~ & includes {\tt make} dependencies on {\tt .F .h} \\ |
| 802 |
|
~ & ~ & ~ & i.e. input for $<$TOOL$>$ may be re-generated \\ |
| 803 |
|
\hline |
| 804 |
|
(3) & {\tt <MODE>all} & {\tt mitgcmuv\_<MODE>} & |
| 805 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 806 |
|
~ & ~ & ~ & and compiles all code \\ |
| 807 |
|
~ & ~ & ~ & (use of TAF is set as default) \\ |
| 808 |
|
\hline |
| 809 |
|
\end{tabular} |
| 810 |
|
} |
| 811 |
|
\end{table} |
| 812 |
|
% |
| 813 |
|
Here, the following placeholders are used |
| 814 |
|
% |
| 815 |
|
\begin{itemize} |
| 816 |
|
% |
| 817 |
|
\item $<$TOOL$>$ |
| 818 |
|
% |
| 819 |
|
\begin{itemize} |
| 820 |
|
% |
| 821 |
|
\item {\tt TAF} |
| 822 |
|
\item {\tt TAMC} |
| 823 |
|
% |
| 824 |
|
\end{itemize} |
| 825 |
|
% |
| 826 |
|
\item $<$MODE$>$ |
| 827 |
|
% |
| 828 |
|
\begin{itemize} |
| 829 |
|
% |
| 830 |
|
\item {\tt ad} generates the adjoint model (ADM) |
| 831 |
|
\item {\tt ftl} generates the tangent linear model (TLM) |
| 832 |
|
\item {\tt svd} generates both ADM and TLM for \\ |
| 833 |
|
singular value decomposition (SVD) type calculations |
| 834 |
|
% |
| 835 |
|
\end{itemize} |
| 836 |
|
% |
| 837 |
|
\end{itemize} |
| 838 |
|
|
| 839 |
|
For example, to generate the adjoint model using TAF after routines ({\tt .F}) |
| 840 |
|
or headers ({\tt .h}) have been modified, but without compilation, |
| 841 |
|
type {\tt make adtaf}; |
| 842 |
|
or, to generate the tangent linear model using TAMC without |
| 843 |
|
re-generating the input code, type {\tt make ftltamconly}. |
| 844 |
|
|
| 845 |
|
|
| 846 |
|
A typical full build process to generate the ADM via TAF would |
| 847 |
|
look like follows: |
| 848 |
|
\begin{verbatim} |
| 849 |
|
% mkdir build |
| 850 |
|
% cd build |
| 851 |
|
% ../../../tools/genmake2 -mods=../code_ad |
| 852 |
|
% make depend |
| 853 |
|
% make adall |
| 854 |
|
\end{verbatim} |
| 855 |
|
|
| 856 |
|
%------------------------------------------------------------------ |
| 857 |
|
|
| 858 |
|
\subsection{The AD build process in detail |
| 859 |
|
\label{section_ad_build_detail}} |
| 860 |
|
|
| 861 |
|
The {\tt make <MODE>all} target consists of the following procedures: |
| 862 |
|
|
| 863 |
|
\begin{enumerate} |
| 864 |
|
% |
| 865 |
|
\item |
| 866 |
|
A header file {\tt AD\_CONFIG.h} is generated which contains a CPP option |
| 867 |
|
on which code ought to be generated. Depending on the {\tt make} target, |
| 868 |
|
the contents is one of the following: |
| 869 |
|
\begin{itemize} |
| 870 |
|
\item |
| 871 |
|
{\tt \#define ALLOW\_ADJOINT\_RUN} |
| 872 |
|
\item |
| 873 |
|
{\tt \#define ALLOW\_TANGENTLINEAR\_RUN} |
| 874 |
|
\item |
| 875 |
|
{\tt \#define ALLOW\_ECCO\_OPTIMIZATION} |
| 876 |
|
\end{itemize} |
| 877 |
|
% |
| 878 |
|
\item |
| 879 |
|
A single file {\tt <MODE>\_input\_code.f} is concatenated |
| 880 |
|
consisting of all {\tt .f} files that are part of the list {\bf AD\_FILES} |
| 881 |
|
and all {\tt .flow} files that are part of the list {\bf AD\_FLOW\_FILES}. |
| 882 |
|
% |
| 883 |
|
\item |
| 884 |
|
The AD tool is invoked with the {\tt <MODE>\_<TOOL>\_FLAGS}. |
| 885 |
|
The default AD tool flags in {\tt genmake2} can be overrwritten by |
| 886 |
|
an {\tt adjoint\_options} file (similar to the platform-specific |
| 887 |
|
{\tt build\_options}, see Section ???. |
| 888 |
|
The AD tool writes the resulting AD code into the file |
| 889 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 890 |
|
% |
| 891 |
|
\item |
| 892 |
|
A short sed script {\tt adjoint\_sed} is applied to |
| 893 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 894 |
|
to reinstate {\bf myThid} into the CALL argument list of active file I/O. |
| 895 |
|
The result is written to file {\tt <MODE>\_<TOOL>\_output.f}. |
| 896 |
|
% |
| 897 |
|
\item |
| 898 |
|
All routines are compiled and an executable is generated |
| 899 |
|
(see Table ???). |
| 900 |
|
% |
| 901 |
|
\end{enumerate} |
| 902 |
|
|
| 903 |
|
\subsubsection{The list AD\_FILES and {\tt .list} files} |
| 904 |
|
|
| 905 |
|
Not all routines are presented to the AD tool. |
| 906 |
|
Routines typically hidden are diagnostics routines which |
| 907 |
|
do not influence the cost function, but may create |
| 908 |
|
artificial flow dependencies such as I/O of active variables. |
| 909 |
|
|
| 910 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FILES} |
| 911 |
|
which contains all routines that are shown to the AD tool. |
| 912 |
|
This list is put together from all files with suffix {\tt .list} |
| 913 |
|
that {\tt genmake2} finds in its search directories. |
| 914 |
|
The list file for the core MITgcm routines is in {\tt model/src/} |
| 915 |
|
is called {\tt model\_ad\_diff.list}. |
| 916 |
|
Note that no wrapper routine is shown to TAF. These are either |
| 917 |
|
not visible at all to the AD code, or hand-written AD code |
| 918 |
|
is available (see next section). |
| 919 |
|
|
| 920 |
|
Each package directory contains its package-specific |
| 921 |
|
list file {\tt <PKG>\_ad\_diff.list}. For example, |
| 922 |
|
{\tt pkg/ptracers/} contains the file {\tt ptracers\_ad\_diff.list}. |
| 923 |
|
Thus, enabling a package will automatically extend the |
| 924 |
|
{\bf AD\_FILES} list of {\tt genmake2} to incorporate the |
| 925 |
|
package-specific routines. |
| 926 |
|
Note that you will need to regenerate the {\tt Makefile} if |
| 927 |
|
you enable a package (e.g. by adding it to {\tt packages.conf}) |
| 928 |
|
and a {\tt Makefile} already exists. |
| 929 |
|
|
| 930 |
|
\subsubsection{The list AD\_FLOW\_FILES and {\tt .flow} files} |
| 931 |
|
|
| 932 |
|
TAMC and TAF can evaluate user-specified directives |
| 933 |
|
that start with a specific syntax ({\tt CADJ}, {\tt C\$TAF}, {\tt !\$TAF}). |
| 934 |
|
The main categories of directives are STORE directives and |
| 935 |
|
FLOW directives. Here, we are concerned with flow directives, |
| 936 |
|
store directives are treated elsewhere. |
| 937 |
|
|
| 938 |
|
Flow directives enable the AD tool to evaluate how it should treat |
| 939 |
|
routines that are 'hidden' by the user, i.e. routines which are |
| 940 |
|
not contained in the {\bf AD\_FILES} list (see previous section), |
| 941 |
|
but which are called in part of the code that the AD tool does see. |
| 942 |
|
The flow directive tell the AD tool |
| 943 |
|
% |
| 944 |
|
\begin{itemize} |
| 945 |
|
% |
| 946 |
|
\item which subroutine arguments are input/output |
| 947 |
|
\item which subroutine arguments are active |
| 948 |
|
\item which subroutine arguments are required to compute the cost |
| 949 |
|
\item which subroutine arguments are dependent |
| 950 |
|
% |
| 951 |
|
\end{itemize} |
| 952 |
|
% |
| 953 |
|
The syntax for the flow directives can be found in the |
| 954 |
|
AD tool manuals. |
| 955 |
|
|
| 956 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FLOW\_FILES} |
| 957 |
|
which contains all files with suffix{\tt .flow} that it finds |
| 958 |
|
in its search directories. |
| 959 |
|
The flow directives for the core MITgcm routines of |
| 960 |
|
{\tt eesupp/src/} and {\tt model/src/} |
| 961 |
|
reside in {\tt pkg/autodiff/}. |
| 962 |
|
This directory also contains hand-written adjoint code |
| 963 |
|
for the MITgcm WRAPPER (section \ref{chap:sarch}). |
| 964 |
|
|
| 965 |
|
Flow directives for package-specific routines are contained in |
| 966 |
|
the corresponding package directories in the file |
| 967 |
|
{\tt <PKG>\_ad.flow}, e.g. ptracers-specific directives are in |
| 968 |
|
{\tt ptracers\_ad.flow}. |
| 969 |
|
|
| 970 |
|
\subsubsection{Store directives for 3-level checkpointing} |
| 971 |
|
|
| 972 |
|
The storing that is required at each period of the |
| 973 |
|
3-level checkpointing is controled by three |
| 974 |
|
top-level headers. |
| 975 |
|
|
| 976 |
|
\begin{verbatim} |
| 977 |
|
do ilev_3 = 1, nchklev_3 |
| 978 |
|
# include ``checkpoint_lev3.h'' |
| 979 |
|
do ilev_2 = 1, nchklev_2 |
| 980 |
|
# include ``checkpoint_lev2.h'' |
| 981 |
|
do ilev_1 = 1, nchklev_1 |
| 982 |
|
# include ``checkpoint_lev1.h'' |
| 983 |
|
|
| 984 |
|
... |
| 985 |
|
|
| 986 |
|
end do |
| 987 |
|
end do |
| 988 |
|
end do |
| 989 |
|
\end{verbatim} |
| 990 |
|
|
| 991 |
|
All files {\tt checkpoint\_lev?.h} are contained in directory |
| 992 |
|
{\tt pkg/autodiff/}. |
| 993 |
|
|
| 994 |
|
|
| 995 |
|
\subsubsection{Changing the default AD tool flags: ad\_options files} |
| 996 |
|
|
| 997 |
|
|
| 998 |
|
\subsubsection{Hand-written adjoint code} |
| 999 |
|
|
| 1000 |
|
%------------------------------------------------------------------ |
| 1001 |
|
|
| 1002 |
\subsection{The cost function (dependent variable) |
\subsection{The cost function (dependent variable) |
| 1003 |
\label{section_cost}} |
\label{section_cost}} |
| 1004 |
|
|
| 1005 |
The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}. |
The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}. |
| 1006 |
It is a function of the input variables $ \vec{u} $ via the composition |
It is a function of the input variables $ \vec{u} $ via the composition |
| 1007 |
$ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $. |
$ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $. |
| 1008 |
The input is referred to as the |
The input are referred to as the |
| 1009 |
{\sf independent variables} or {\sf control variables}. |
{\sf independent variables} or {\sf control variables}. |
| 1010 |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
| 1011 |
(parameter setting, initialization, accumulation, |
(parameter setting, initialization, accumulation, |
| 1012 |
final evaluation), are controlled by the package {\it pkg/cost}. |
final evaluation), are controlled by the package {\it pkg/cost}. |
| 1013 |
|
The aspects relevant to the treatment of the independent variables |
| 1014 |
|
are controlled by the package {\it pkg/ctrl} and will be treated |
| 1015 |
|
in the next section. |
| 1016 |
|
|
| 1017 |
\input{part5/doc_cost_flow} |
\input{s_autodiff/text/doc_cost_flow} |
| 1018 |
|
|
| 1019 |
|
\subsubsection{Enabling the package} |
| 1020 |
|
|
|
\subsubsection{genmake and CPP options} |
|
|
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item |
|
| 1021 |
\fbox{ |
\fbox{ |
| 1022 |
\begin{minipage}{12cm} |
\begin{minipage}{12cm} |
| 1023 |
{\it genmake}, {\it CPP\_OPTIONS.h}, {\it ECCO\_CPPOPTIONS.h} |
{\it packages.conf}, {\it ECCO\_CPPOPTIONS.h} |
| 1024 |
\end{minipage} |
\end{minipage} |
| 1025 |
} |
} |
| 1026 |
\end{itemize} |
\begin{itemize} |
| 1027 |
% |
% |
| 1028 |
The directory {\it pkg/cost} can be included to the |
\item |
| 1029 |
compile list in 3 different ways (cf. Section \ref{???}): |
The package is enabled by adding {\it cost} to your file {\it packages.conf} |
| 1030 |
|
(see Section ???) |
| 1031 |
% |
% |
| 1032 |
\begin{enumerate} |
\item |
| 1033 |
% |
|
| 1034 |
\item {\it genmake}: \\ |
|
| 1035 |
Change the default settings in the file {\it genmake} by adding |
\end{itemize} |
|
{\bf cost} to the {\bf enable} list (not recommended). |
|
|
% |
|
|
\item {\it .genmakerc}: \\ |
|
|
Customize the settings of {\bf enable}, {\bf disable} which are |
|
|
appropriate for your experiment in the file {\it .genmakerc} |
|
|
and add the file to your compile directory. |
|
|
% |
|
|
\item genmake-options: \\ |
|
|
Call {\it genmake} with the option |
|
|
{\tt genmake -enable=cost}. |
|
| 1036 |
% |
% |
| 1037 |
\end{enumerate} |
|
| 1038 |
|
N.B.: In general the following packages ought to be enabled |
| 1039 |
|
simultaneously: {\it autodiff, cost, ctrl}. |
| 1040 |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
| 1041 |
Each specific cost function contribution has its own option. |
Each specific cost function contribution has its own option. |
| 1042 |
For the present example the option is {\bf ALLOW\_COST\_TRACER}. |
For the present example the option is {\bf ALLOW\_COST\_TRACER}. |
| 1043 |
All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h} |
All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h} |
| 1044 |
Since the cost function is usually used in conjunction with |
Since the cost function is usually used in conjunction with |
| 1045 |
automatic differentiation, the CPP option |
automatic differentiation, the CPP option |
| 1046 |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
{\bf ALLOW\_ADJOINT\_RUN} (file {\it CPP\_OPTIONS.h}) and |
| 1047 |
(file {\it CPP\_OPTIONS.h}). |
{\bf ALLOW\_AUTODIFF\_TAMC} (file {\it ECCO\_CPPOPTIONS.h}) |
| 1048 |
|
should be defined. |
| 1049 |
|
|
| 1050 |
\subsubsection{Initialization} |
\subsubsection{Initialization} |
| 1051 |
% |
% |
| 1052 |
The initialization of the {\it cost} package is readily enabled |
The initialization of the {\it cost} package is readily enabled |
| 1053 |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
as soon as the CPP option {\bf ALLOW\_COST} is defined. |
| 1054 |
% |
% |
| 1055 |
\begin{itemize} |
\begin{itemize} |
| 1056 |
% |
% |
| 1124 |
\begin{equation} |
\begin{equation} |
| 1125 |
{\cal J} \, = \, |
{\cal J} \, = \, |
| 1126 |
{\rm fc} \, = \, |
{\rm fc} \, = \, |
| 1127 |
{\rm mult\_tracer} \sum_{bi,\,bj}^{nSx,\,nSy} |
{\rm mult\_tracer} \sum_{\text{global sum}} \sum_{bi,\,bj}^{nSx,\,nSy} |
| 1128 |
{\rm objf\_tracer}(bi,bj) \, + \, ... |
{\rm objf\_tracer}(bi,bj) \, + \, ... |
| 1129 |
\end{equation} |
\end{equation} |
| 1130 |
% |
% |
| 1131 |
The total cost function {\bf fc} will be the |
The total cost function {\bf fc} will be the |
| 1132 |
'dependent' variable in the argument list for TAMC, i.e. |
'dependent' variable in the argument list for TAF, i.e. |
| 1133 |
\begin{verbatim} |
\begin{verbatim} |
| 1134 |
tamc -output 'fc' ... |
taf -output 'fc' ... |
| 1135 |
\end{verbatim} |
\end{verbatim} |
| 1136 |
|
|
| 1137 |
%%%% \end{document} |
%%%% \end{document} |
| 1138 |
|
|
| 1139 |
\input{part5/doc_ad_the_main} |
\input{s_autodiff/text/doc_ad_the_main} |
| 1140 |
|
|
| 1141 |
\subsection{The control variables (independent variables) |
\subsection{The control variables (independent variables) |
| 1142 |
\label{section_ctrl}} |
\label{section_ctrl}} |
| 1156 |
(parameter setting, initialization, perturbation) |
(parameter setting, initialization, perturbation) |
| 1157 |
are controlled by the package {\it pkg/ctrl}. |
are controlled by the package {\it pkg/ctrl}. |
| 1158 |
|
|
| 1159 |
\input{part5/doc_ctrl_flow} |
\input{s_autodiff/text/doc_ctrl_flow} |
| 1160 |
|
|
| 1161 |
\subsubsection{genmake and CPP options} |
\subsubsection{genmake and CPP options} |
| 1162 |
% |
% |
| 1172 |
% |
% |
| 1173 |
To enable the directory to be included to the compile list, |
To enable the directory to be included to the compile list, |
| 1174 |
{\bf ctrl} has to be added to the {\bf enable} list in |
{\bf ctrl} has to be added to the {\bf enable} list in |
| 1175 |
{\it .genmakerc} (or {\it genmake} itself). |
{\it .genmakerc} or in {\it genmake} itself (analogous to {\it cost} |
| 1176 |
|
package, cf. previous section). |
| 1177 |
Each control variable is enabled via its own CPP option |
Each control variable is enabled via its own CPP option |
| 1178 |
in {\it ECCO\_CPPOPTIONS.h}. |
in {\it ECCO\_CPPOPTIONS.h}. |
| 1179 |
|
|
| 1214 |
\\ |
\\ |
| 1215 |
% |
% |
| 1216 |
Two important issues related to the handling of the control |
Two important issues related to the handling of the control |
| 1217 |
variables in the MITGCM need to be addressed. |
variables in MITgcm need to be addressed. |
| 1218 |
First, in order to save memory, the control variable arrays |
First, in order to save memory, the control variable arrays |
| 1219 |
are not kept in memory, but rather read from file and added |
are not kept in memory, but rather read from file and added |
| 1220 |
to the initial fields during the model initialization phase. |
to the initial fields during the model initialization phase. |
| 1246 |
% |
% |
| 1247 |
The dependency flow for differentiation w.r.t. the controls |
The dependency flow for differentiation w.r.t. the controls |
| 1248 |
starts with adding a perturbation onto the input variable, |
starts with adding a perturbation onto the input variable, |
| 1249 |
thus defining the independent or control variables for TAMC. |
thus defining the independent or control variables for TAF. |
| 1250 |
Three types of controls may be considered: |
Three types of controls may be considered: |
| 1251 |
% |
% |
| 1252 |
\begin{itemize} |
\begin{itemize} |
| 1279 |
holding the perturbation. In the case of a simple |
holding the perturbation. In the case of a simple |
| 1280 |
sensitivity study this array is identical to zero. |
sensitivity study this array is identical to zero. |
| 1281 |
However, it's specification is essential in the context |
However, it's specification is essential in the context |
| 1282 |
of automatic differentiation since TAMC |
of automatic differentiation since TAF |
| 1283 |
treats the corresponding line in the code symbolically |
treats the corresponding line in the code symbolically |
| 1284 |
when determining the differentiation chain and its origin. |
when determining the differentiation chain and its origin. |
| 1285 |
Thus, the variable names are part of the argument list |
Thus, the variable names are part of the argument list |
| 1286 |
when calling TAMC: |
when calling TAF: |
| 1287 |
% |
% |
| 1288 |
\begin{verbatim} |
\begin{verbatim} |
| 1289 |
tamc -input 'xx_tr1 ...' ... |
taf -input 'xx_tr1 ...' ... |
| 1290 |
\end{verbatim} |
\end{verbatim} |
| 1291 |
% |
% |
| 1292 |
Now, as mentioned above, the MITGCM avoids maintaining |
Now, as mentioned above, MITgcm avoids maintaining |
| 1293 |
an array for each control variable by reading the |
an array for each control variable by reading the |
| 1294 |
perturbation to a temporary array from file. |
perturbation to a temporary array from file. |
| 1295 |
To ensure the symbolic link to be recognized by TAMC, a scalar |
To ensure the symbolic link to be recognized by TAF, a scalar |
| 1296 |
dummy variable {\bf xx\_tr1\_dummy} is introduced |
dummy variable {\bf xx\_tr1\_dummy} is introduced |
| 1297 |
and an 'active read' routine of the adjoint support |
and an 'active read' routine of the adjoint support |
| 1298 |
package {\it pkg/autodiff} is invoked. |
package {\it pkg/autodiff} is invoked. |
| 1299 |
The read-procedure is tagged with the variable |
The read-procedure is tagged with the variable |
| 1300 |
{\bf xx\_tr1\_dummy} enabling TAMC to recognize the |
{\bf xx\_tr1\_dummy} enabling TAF to recognize the |
| 1301 |
initialization of the perturbation. |
initialization of the perturbation. |
| 1302 |
The modified call of TAMC thus reads |
The modified call of TAF thus reads |
| 1303 |
% |
% |
| 1304 |
\begin{verbatim} |
\begin{verbatim} |
| 1305 |
tamc -input 'xx_tr1_dummy ...' ... |
taf -input 'xx_tr1_dummy ...' ... |
| 1306 |
\end{verbatim} |
\end{verbatim} |
| 1307 |
% |
% |
| 1308 |
and the modified operation to (\ref{perturb}) |
and the modified operation to (\ref{perturb}) |
| 1317 |
% |
% |
| 1318 |
Note, that reading an active variable corresponds |
Note, that reading an active variable corresponds |
| 1319 |
to a variable assignment. Its derivative corresponds |
to a variable assignment. Its derivative corresponds |
| 1320 |
to a write statement of the adjoint variable. |
to a write statement of the adjoint variable, followed by |
| 1321 |
|
a reset. |
| 1322 |
The 'active file' routines have been designed |
The 'active file' routines have been designed |
| 1323 |
to support active read and corresponding adjoint active write |
to support active read and corresponding adjoint active write |
| 1324 |
operations (and vice versa). |
operations (and vice versa). |
| 1435 |
{\it addummy\_in\_stepping}. |
{\it addummy\_in\_stepping}. |
| 1436 |
This routine is part of the adjoint support package |
This routine is part of the adjoint support package |
| 1437 |
{\it pkg/autodiff} (cf.f. below). |
{\it pkg/autodiff} (cf.f. below). |
| 1438 |
|
The procedure is enabled using via the CPP-option |
| 1439 |
|
{\bf ALLOW\_AUTODIFF\_MONITOR} (file {\it ECCO\_CPPOPTIONS.h}). |
| 1440 |
To be part of the adjoint code, the corresponding S/R |
To be part of the adjoint code, the corresponding S/R |
| 1441 |
{\it dummy\_in\_stepping} has to be called in the forward |
{\it dummy\_in\_stepping} has to be called in the forward |
| 1442 |
model (S/R {\it the\_main\_loop}) at the appropriate place. |
model (S/R {\it the\_main\_loop}) at the appropriate place. |
| 1443 |
|
The adjoint common blocks are extracted from the adjoint code |
| 1444 |
|
via the header file {\it adcommon.h}. |
| 1445 |
|
|
| 1446 |
{\it dummy\_in\_stepping} is essentially empty, |
{\it dummy\_in\_stepping} is essentially empty, |
| 1447 |
the corresponding adjoint routine is hand-written rather |
the corresponding adjoint routine is hand-written rather |
| 1468 |
{\bf /adtr1\_r/}, {\bf /adffields/}, |
{\bf /adtr1\_r/}, {\bf /adffields/}, |
| 1469 |
which have been extracted from the adjoint code to enable |
which have been extracted from the adjoint code to enable |
| 1470 |
access to the adjoint variables. |
access to the adjoint variables. |
| 1471 |
|
|
| 1472 |
|
{\bf WARNING:} If the structure of the common blocks |
| 1473 |
|
{\bf /dynvars\_r/}, {\bf /dynvars\_cd/}, etc., changes |
| 1474 |
|
similar changes will occur in the adjoint common blocks. |
| 1475 |
|
Therefore, consistency between the TAMC-generated common blocks |
| 1476 |
|
and those in {\it adcommon.h} have to be checked. |
| 1477 |
% |
% |
| 1478 |
\end{itemize} |
\end{itemize} |
| 1479 |
|
|
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch