| 1 |
% $Header$ |
% $Header$ |
| 2 |
% $Name$ |
% $Name$ |
| 3 |
|
|
| 4 |
|
Author: Patrick Heimbach |
| 5 |
|
|
| 6 |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
| 7 |
(or, more loosely, computational) differentiation, involves |
(or, more loosely, computational) differentiation, involves |
| 8 |
automatically deriving code to calculate |
automatically deriving code to calculate |
| 44 |
%********************************************************************** |
%********************************************************************** |
| 45 |
\section{Some basic algebra} |
\section{Some basic algebra} |
| 46 |
\label{sec_ad_algebra} |
\label{sec_ad_algebra} |
| 47 |
|
\begin{rawhtml} |
| 48 |
|
<!-- CMIREDIR:sec_ad_algebra: --> |
| 49 |
|
\end{rawhtml} |
| 50 |
%********************************************************************** |
%********************************************************************** |
| 51 |
|
|
| 52 |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
| 562 |
(a derivative is defined w.r.t. a point along the trajectory), |
(a derivative is defined w.r.t. a point along the trajectory), |
| 563 |
the intermediate results of the model trajectory |
the intermediate results of the model trajectory |
| 564 |
$\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$ |
$\vec{v}^{(\lambda+1)}={\cal M}_{\lambda}(v^{(\lambda)})$ |
| 565 |
are needed to evaluate the intermediate Jacobian |
may be required to evaluate the intermediate Jacobian |
| 566 |
$M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $. |
$M_{\lambda}|_{\vec{v}^{(\lambda)}} \, \delta \vec{v}^{(\lambda)} $. |
| 567 |
|
This is the case e.g. for nonlinear expressions |
| 568 |
|
(momentum advection, nonlinear equation of state), state-dependent |
| 569 |
|
conditional statements (parameterization schemes). |
| 570 |
In the forward mode, the intermediate results are required |
In the forward mode, the intermediate results are required |
| 571 |
in the same order as computed by the full forward model ${\cal M}$, |
in the same order as computed by the full forward model ${\cal M}$, |
| 572 |
but in the reverse mode they are required in the reverse order. |
but in the reverse mode they are required in the reverse order. |
| 577 |
|
|
| 578 |
A method to balance the amount of recomputations vs. |
A method to balance the amount of recomputations vs. |
| 579 |
storage requirements is called {\sf checkpointing} |
storage requirements is called {\sf checkpointing} |
| 580 |
(e.g. \cite{res-eta:98}). |
(e.g. \cite{gri:92}, \cite{res-eta:98}). |
| 581 |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
It is depicted in \ref{fig:3levelcheck} for a 3-level checkpointing |
| 582 |
[as an example, we give explicit numbers for a 3-day |
[as an example, we give explicit numbers for a 3-day |
| 583 |
integration with a 1-hourly timestep in square brackets]. |
integration with a 1-hourly timestep in square brackets]. |
| 588 |
$ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals], |
$ {n}^{lev3} $ subsections [$ {n}^{lev3} $=3 1-day intervals], |
| 589 |
with the label $lev3$ for this outermost loop. |
with the label $lev3$ for this outermost loop. |
| 590 |
The model is then integrated along the full trajectory, |
The model is then integrated along the full trajectory, |
| 591 |
and the model state stored only at every $ k_{i}^{lev3} $-th timestep |
and the model state stored to disk only at every $ k_{i}^{lev3} $-th timestep |
| 592 |
[i.e. 3 times, at |
[i.e. 3 times, at |
| 593 |
$ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $]. |
$ i = 0,1,2 $ corresponding to $ k_{i}^{lev3} = 0, 24, 48 $]. |
| 594 |
|
In addition, the cost function is computed, if needed. |
| 595 |
% |
% |
| 596 |
\item [$lev2$] |
\item [$lev2$] |
| 597 |
In a second step each subsection itself is divided into |
In a second step each subsection itself is divided into |
| 598 |
$ {n}^{lev2} $ sub-subsections |
$ {n}^{lev2} $ subsections |
| 599 |
[$ {n}^{lev2} $=4 6-hour intervals per subsection]. |
[$ {n}^{lev2} $=4 6-hour intervals per subsection]. |
| 600 |
The model picks up at the last outermost dumped state |
The model picks up at the last outermost dumped state |
| 601 |
$ v_{k_{n}^{lev3}} $ and is integrated forward in time along |
$ v_{k_{n}^{lev3}} $ and is integrated forward in time along |
| 602 |
the last subsection, with the label $lev2$ for this |
the last subsection, with the label $lev2$ for this |
| 603 |
intermediate loop. |
intermediate loop. |
| 604 |
The model state is now stored at every $ k_{i}^{lev2} $-th |
The model state is now stored to disk at every $ k_{i}^{lev2} $-th |
| 605 |
timestep |
timestep |
| 606 |
[i.e. 4 times, at |
[i.e. 4 times, at |
| 607 |
$ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $]. |
$ i = 0,1,2,3 $ corresponding to $ k_{i}^{lev2} = 48, 54, 60, 66 $]. |
| 609 |
\item [$lev1$] |
\item [$lev1$] |
| 610 |
Finally, the model picks up at the last intermediate dump state |
Finally, the model picks up at the last intermediate dump state |
| 611 |
$ v_{k_{n}^{lev2}} $ and is integrated forward in time along |
$ v_{k_{n}^{lev2}} $ and is integrated forward in time along |
| 612 |
the last sub-subsection, with the label $lev1$ for this |
the last subsection, with the label $lev1$ for this |
| 613 |
intermediate loop. |
intermediate loop. |
| 614 |
Within this sub-subsection only, the model state is stored |
Within this sub-subsection only, parts of the model state is stored |
| 615 |
at every timestep |
to memory at every timestep |
| 616 |
[i.e. every hour $ i=0,...,5$ corresponding to |
[i.e. every hour $ i=0,...,5$ corresponding to |
| 617 |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
$ k_{i}^{lev1} = 66, 67, \ldots, 71 $]. |
| 618 |
Thus, the final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
The final state $ v_n = v_{k_{n}^{lev1}} $ is reached |
| 619 |
and the model state of all proceeding timesteps along the last |
and the model state of all preceding timesteps along the last |
| 620 |
sub-subsections are available, enabling integration backwards |
innermost subsection are available, enabling integration backwards |
| 621 |
in time along the last sub-subsection. |
in time along the last subsection. |
| 622 |
Thus, the adjoint can be computed along this last |
The adjoint can thus be computed along this last |
| 623 |
sub-subsection $k_{n}^{lev2}$. |
subsection $k_{n}^{lev2}$. |
| 624 |
% |
% |
| 625 |
\end{itemize} |
\end{itemize} |
| 626 |
% |
% |
| 627 |
This procedure is repeated consecutively for each previous |
This procedure is repeated consecutively for each previous |
| 628 |
sub-subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $ |
subsection $k_{n-1}^{lev2}, \ldots, k_{1}^{lev2} $ |
| 629 |
carrying the adjoint computation to the initial time |
carrying the adjoint computation to the initial time |
| 630 |
of the subsection $k_{n}^{lev3}$. |
of the subsection $k_{n}^{lev3}$. |
| 631 |
Then, the procedure is repeated for the previous subsection |
Then, the procedure is repeated for the previous subsection |
| 636 |
For the full model trajectory of |
For the full model trajectory of |
| 637 |
$ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps |
$ n^{lev3} \cdot n^{lev2} \cdot n^{lev1} $ timesteps |
| 638 |
the required storing of the model state was significantly reduced to |
the required storing of the model state was significantly reduced to |
| 639 |
$ n^{lev1} + n^{lev2} + n^{lev3} $ |
$ n^{lev2} + n^{lev3} $ to disk and roughly $ n^{lev1} $ to memory |
| 640 |
[i.e. for the 3-day integration with a total oof 72 timesteps |
[i.e. for the 3-day integration with a total oof 72 timesteps |
| 641 |
the model state was stored 13 times]. |
the model state was stored 7 times to disk and roughly 6 times |
| 642 |
|
to memory]. |
| 643 |
This saving in memory comes at a cost of a required |
This saving in memory comes at a cost of a required |
| 644 |
3 full forward integrations of the model (one for each |
3 full forward integrations of the model (one for each |
| 645 |
checkpointing level). |
checkpointing level). |
| 646 |
The balance of storage vs. recomputation certainly depends |
The optimal balance of storage vs. recomputation certainly depends |
| 647 |
on the computing resources available. |
on the computing resources available and may be adjusted by |
| 648 |
|
adjusting the partitioning among the |
| 649 |
|
$ n^{lev3}, \,\, n^{lev2}, \,\, n^{lev1} $. |
| 650 |
|
|
| 651 |
\begin{figure}[t!] |
\begin{figure}[t!] |
| 652 |
\begin{center} |
\begin{center} |
| 681 |
%********************************************************************** |
%********************************************************************** |
| 682 |
\section{TLM and ADM generation in general} |
\section{TLM and ADM generation in general} |
| 683 |
\label{sec_ad_setup_gen} |
\label{sec_ad_setup_gen} |
| 684 |
|
\begin{rawhtml} |
| 685 |
|
<!-- CMIREDIR:sec_ad_setup_gen: --> |
| 686 |
|
\end{rawhtml} |
| 687 |
%********************************************************************** |
%********************************************************************** |
| 688 |
|
|
| 689 |
In this section we describe in a general fashion |
In this section we describe in a general fashion |
| 690 |
the parts of the code that are relevant for automatic |
the parts of the code that are relevant for automatic |
| 691 |
differentiation using the software tool TAMC. |
differentiation using the software tool TAF. |
| 692 |
|
|
| 693 |
\input{part5/doc_ad_the_model} |
\input{part5/doc_ad_the_model} |
| 694 |
|
|
| 695 |
The basic flow is depicted in \ref{fig:adthemodel}. |
The basic flow is depicted in \ref{fig:adthemodel}. |
| 696 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
If CPP option \texttt{ALLOW\_AUTODIFF\_TAMC} is defined, |
| 697 |
|
the driver routine |
| 698 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
| 699 |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
invokes the adjoint of this routine, {\it adthe\_main\_loop} |
| 700 |
which is the toplevel routine in terms of reverse mode computation. |
(case \texttt{\#define ALLOW\_ADJOINT\_RUN}), or |
| 701 |
The routine {\it adthe\_main\_loop} has been generated using TAMC. |
the tangent linear of this routine {\it g\_the\_main\_loop} |
| 702 |
It contains both the forward integration of the full model, |
(case \texttt{\#define ALLOW\_TANGENTLINEAR\_RUN}), |
| 703 |
|
which are the toplevel routines in terms of automatic differentiation. |
| 704 |
|
The routines {\it adthe\_main\_loop} or {\it g\_the\_main\_loop} |
| 705 |
|
are generated by TAF. |
| 706 |
|
It contains both the forward integration of the full model, the |
| 707 |
|
cost function calculation, |
| 708 |
any additional storing that is required for efficient checkpointing, |
any additional storing that is required for efficient checkpointing, |
| 709 |
and the reverse integration of the adjoint model. |
and the reverse integration of the adjoint model. |
| 710 |
The structure of {\it adthe\_main\_loop} has been strongly |
|
| 711 |
simplified for clarification; in particular, no checkpointing |
[DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM] |
| 712 |
|
|
| 713 |
|
In Fig. \ref{fig:adthemodel} |
| 714 |
|
the structure of {\it adthe\_main\_loop} has been strongly |
| 715 |
|
simplified to focus on the essentials; in particular, no checkpointing |
| 716 |
procedures are shown here. |
procedures are shown here. |
| 717 |
Prior to the call of {\it adthe\_main\_loop}, the routine |
Prior to the call of {\it adthe\_main\_loop}, the routine |
| 718 |
{\it ctrl\_unpack} is invoked to unpack the control vector, |
{\it ctrl\_unpack} is invoked to unpack the control vector |
| 719 |
and following that call, the routine {\it ctrl\_pack} |
or initialise the control variables. |
| 720 |
|
Following the call of {\it adthe\_main\_loop}, |
| 721 |
|
the routine {\it ctrl\_pack} |
| 722 |
is invoked to pack the control vector |
is invoked to pack the control vector |
| 723 |
(cf. Section \ref{section_ctrl}). |
(cf. Section \ref{section_ctrl}). |
| 724 |
If gradient checks are to be performed, the option |
If gradient checks are to be performed, the option |
| 727 |
the gradient has been computed via the adjoint |
the gradient has been computed via the adjoint |
| 728 |
(cf. Section \ref{section_grdchk}). |
(cf. Section \ref{section_grdchk}). |
| 729 |
|
|
| 730 |
|
%------------------------------------------------------------------ |
| 731 |
|
|
| 732 |
|
\subsection{General setup |
| 733 |
|
\label{section_ad_setup}} |
| 734 |
|
|
| 735 |
|
In order to configure AD-related setups the following packages need |
| 736 |
|
to be enabled: |
| 737 |
|
{\it |
| 738 |
|
\begin{table}[h!] |
| 739 |
|
\begin{tabular}{l} |
| 740 |
|
autodiff \\ |
| 741 |
|
ctrl \\ |
| 742 |
|
cost \\ |
| 743 |
|
grdchk \\ |
| 744 |
|
\end{tabular} |
| 745 |
|
\end{table} |
| 746 |
|
} |
| 747 |
|
The packages are enabled by adding them to your experiment-specific |
| 748 |
|
configuration file |
| 749 |
|
{\it packages.conf} (see Section ???). |
| 750 |
|
|
| 751 |
|
The following AD-specific CPP option files need to be customized: |
| 752 |
|
% |
| 753 |
|
\begin{itemize} |
| 754 |
|
% |
| 755 |
|
\item {\it ECCO\_CPPOPTIONS.h} \\ |
| 756 |
|
This header file collects CPP options for the packages |
| 757 |
|
{\it autodiff, cost, ctrl} as well as AD-unrelated options for |
| 758 |
|
the external forcing package {\it exf}. |
| 759 |
|
\footnote{NOTE: These options are not set in their package-specific |
| 760 |
|
headers such as {\it COST\_CPPOPTIONS.h}, but are instead collected |
| 761 |
|
in the single header file {\it ECCO\_CPPOPTIONS.h}. |
| 762 |
|
The package-specific header files serve as simple |
| 763 |
|
placeholders at this point.} |
| 764 |
|
% |
| 765 |
|
\item {\it tamc.h} \\ |
| 766 |
|
This header configures the splitting of the time stepping loop |
| 767 |
|
w.r.t. the 3-level checkpointing (see section ???). |
| 768 |
|
|
| 769 |
|
% |
| 770 |
|
\end{itemize} |
| 771 |
|
|
| 772 |
|
%------------------------------------------------------------------ |
| 773 |
|
|
| 774 |
|
\subsection{Building the AD code |
| 775 |
|
\label{section_ad_build}} |
| 776 |
|
|
| 777 |
|
The build process of an AD code is very similar to building |
| 778 |
|
the forward model. However, depending on which AD code one wishes |
| 779 |
|
to generate, and on which AD tool is available (TAF or TAMC), |
| 780 |
|
the following {\tt make} targets are available: |
| 781 |
|
|
| 782 |
|
\begin{table}[h!] |
| 783 |
|
{\footnotesize |
| 784 |
|
\begin{tabular}{ccll} |
| 785 |
|
~ & {\it AD-target} & {\it output} & {\it description} \\ |
| 786 |
|
\hline |
| 787 |
|
\hline |
| 788 |
|
(1) & {\tt <MODE><TOOL>only} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 789 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 790 |
|
~ & ~ & ~ & no {\tt make} dependencies on {\tt .F .h} \\ |
| 791 |
|
~ & ~ & ~ & useful for compiling on remote platforms \\ |
| 792 |
|
\hline |
| 793 |
|
(2) & {\tt <MODE><TOOL>} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 794 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 795 |
|
~ & ~ & ~ & includes {\tt make} dependencies on {\tt .F .h} \\ |
| 796 |
|
~ & ~ & ~ & i.e. input for $<$TOOL$>$ may be re-generated \\ |
| 797 |
|
\hline |
| 798 |
|
(3) & {\tt <MODE>all} & {\tt mitgcmuv\_<MODE>} & |
| 799 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 800 |
|
~ & ~ & ~ & and compiles all code \\ |
| 801 |
|
~ & ~ & ~ & (use of TAF is set as default) \\ |
| 802 |
|
\hline |
| 803 |
|
\hline |
| 804 |
|
\end{tabular} |
| 805 |
|
} |
| 806 |
|
\end{table} |
| 807 |
|
% |
| 808 |
|
Here, the following placeholders are used |
| 809 |
|
% |
| 810 |
|
\begin{itemize} |
| 811 |
|
% |
| 812 |
|
\item [$<$TOOL$>$] |
| 813 |
|
% |
| 814 |
|
\begin{itemize} |
| 815 |
|
% |
| 816 |
|
\item {\tt TAF} |
| 817 |
|
\item {\tt TAMC} |
| 818 |
|
% |
| 819 |
|
\end{itemize} |
| 820 |
|
% |
| 821 |
|
\item [$<$MODE$>$] |
| 822 |
|
% |
| 823 |
|
\begin{itemize} |
| 824 |
|
% |
| 825 |
|
\item {\tt ad} generates the adjoint model (ADM) |
| 826 |
|
\item {\tt ftl} generates the tangent linear model (TLM) |
| 827 |
|
\item {\tt svd} generates both ADM and TLM for \\ |
| 828 |
|
singular value decomposition (SVD) type calculations |
| 829 |
|
% |
| 830 |
|
\end{itemize} |
| 831 |
|
% |
| 832 |
|
\end{itemize} |
| 833 |
|
|
| 834 |
|
For example, to generate the adjoint model using TAF after routines ({\tt .F}) |
| 835 |
|
or headers ({\tt .h}) have been modified, but without compilation, |
| 836 |
|
type {\tt make adtaf}; |
| 837 |
|
or, to generate the tangent linear model using TAMC without |
| 838 |
|
re-generating the input code, type {\tt make ftltamconly}. |
| 839 |
|
|
| 840 |
|
|
| 841 |
|
A typical full build process to generate the ADM via TAF would |
| 842 |
|
look like follows: |
| 843 |
|
\begin{verbatim} |
| 844 |
|
% mkdir build |
| 845 |
|
% cd build |
| 846 |
|
% ../../../tools/genmake2 -mods=../code_ad |
| 847 |
|
% make depend |
| 848 |
|
% make adall |
| 849 |
|
\end{verbatim} |
| 850 |
|
|
| 851 |
|
%------------------------------------------------------------------ |
| 852 |
|
|
| 853 |
|
\subsection{The AD build process in detail |
| 854 |
|
\label{section_ad_build_detail}} |
| 855 |
|
|
| 856 |
|
The {\tt make <MODE>all} target consists of the following procedures: |
| 857 |
|
|
| 858 |
|
\begin{enumerate} |
| 859 |
|
% |
| 860 |
|
\item |
| 861 |
|
A header file {\tt AD\_CONFIG.h} is generated which contains a CPP option |
| 862 |
|
on which code ought to be generated. Depending on the {\tt make} target, |
| 863 |
|
the contents is |
| 864 |
|
\begin{itemize} |
| 865 |
|
\item |
| 866 |
|
{\tt \#define ALLOW\_ADJOINT\_RUN} |
| 867 |
|
\item |
| 868 |
|
{\tt \#define ALLOW\_TANGENTLINEAR\_RUN} |
| 869 |
|
\item |
| 870 |
|
{\tt \#define ALLOW\_ECCO\_OPTIMIZATION} |
| 871 |
|
\end{itemize} |
| 872 |
|
% |
| 873 |
|
\item |
| 874 |
|
A single file {\tt <MODE>\_input\_code.f} is concatenated |
| 875 |
|
consisting of all {\tt .f} files that are part of the list {\bf AD\_FILES} |
| 876 |
|
and all {\tt .flow} files that are part of the list {\bf AD\_FLOW\_FILES}. |
| 877 |
|
% |
| 878 |
|
\item |
| 879 |
|
The AD tool is invoked with the {\bf <MODE>\_<TOOL>\_FLAGS}. |
| 880 |
|
The default AD tool flags in {\tt genmake2} can be overrwritten by |
| 881 |
|
an {\tt adjoint\_options} file (similar to the platform-specific |
| 882 |
|
{\tt build\_options}, see Section ???. |
| 883 |
|
The AD tool writes the resulting AD code into the file |
| 884 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 885 |
|
% |
| 886 |
|
\item |
| 887 |
|
A short sed script {\tt adjoint\_sed} is applied to |
| 888 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 889 |
|
to reinstate {\bf myThid} into the CALL argument list of active file I/O. |
| 890 |
|
The result is written to file {\tt <MODE>\_<TOOL>\_output.f}. |
| 891 |
|
% |
| 892 |
|
\item |
| 893 |
|
All routines are compiled and an executable is generated |
| 894 |
|
(see Table ???). |
| 895 |
|
% |
| 896 |
|
\end{enumerate} |
| 897 |
|
|
| 898 |
|
\subsubsection{The list AD\_FILES and {\tt .list} files} |
| 899 |
|
|
| 900 |
|
Not all routines are presented to the AD tool. |
| 901 |
|
Routines typically hidden are diagnostics routines which |
| 902 |
|
do not influence the cost function, but may create |
| 903 |
|
artificial flow dependencies such as I/O of active variables. |
| 904 |
|
|
| 905 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FILES} |
| 906 |
|
which contains all routines that are shown to the AD tool. |
| 907 |
|
This list is put together from all files with suffix {\tt .list} |
| 908 |
|
that {\tt genmake2} finds in its search directories. |
| 909 |
|
The list file for the core MITgcm routines is in {\tt model/src/} |
| 910 |
|
is called {\tt model\_ad\_diff.list}. |
| 911 |
|
Note that no wrapper routine is shown to TAF. These are either |
| 912 |
|
not visible at all to the AD code, or hand-written AD code |
| 913 |
|
is available (see next section). |
| 914 |
|
|
| 915 |
|
Each package directory contains its package-specific |
| 916 |
|
list file {\tt <PKG>\_ad\_diff.list}. For example, |
| 917 |
|
{\tt pkg/ptracers/} contains the file {\tt ptracers\_ad\_diff.list}. |
| 918 |
|
Thus, enabling a package will automatically extend the |
| 919 |
|
{\bf AD\_FILES} list of {\tt genmake2} to incorporate the |
| 920 |
|
package-specific routines. |
| 921 |
|
Note that you will need to regenerate the {\tt Makefile} if |
| 922 |
|
you enable a package (e.g. by adding it to {\tt packages.conf}) |
| 923 |
|
and a {\tt Makefile} already exists. |
| 924 |
|
|
| 925 |
|
\subsubsection{The list AD\_FLOW\_FILES and {\tt .flow} files} |
| 926 |
|
|
| 927 |
|
TAMC and TAF can evaluate user-specified directives |
| 928 |
|
that start with a specific syntax ({\tt CADJ}, {\tt C\$TAF}, {\tt !\$TAF}). |
| 929 |
|
The main categories of directives are STORE directives and |
| 930 |
|
FLOW directives. Here, we are concerned with flow directives, |
| 931 |
|
store directives are treated elsewhere. |
| 932 |
|
|
| 933 |
|
Flow directives enable the AD tool to evaluate how it should treat |
| 934 |
|
routines that are 'hidden' by the user, i.e. routines which are |
| 935 |
|
not contained in the {\bf AD\_FILES} list (see previous section), |
| 936 |
|
but which are called in part of the code that the AD tool does see. |
| 937 |
|
The flow directive tell the AD tool |
| 938 |
|
% |
| 939 |
|
\begin{itemize} |
| 940 |
|
% |
| 941 |
|
\item which subroutine arguments are input/output |
| 942 |
|
\item which subroutine arguments are active |
| 943 |
|
\item which subroutine arguments are required to compute the cost |
| 944 |
|
\item which subroutine arguments are dependent |
| 945 |
|
% |
| 946 |
|
\end{itemize} |
| 947 |
|
% |
| 948 |
|
The syntax for the flow directives can be found in the |
| 949 |
|
AD tool manuals. |
| 950 |
|
|
| 951 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FLOW\_FILES} |
| 952 |
|
which contains all files with suffix{\tt .flow} that it finds |
| 953 |
|
in its search directories. |
| 954 |
|
The flow directives for the core MITgcm routines of |
| 955 |
|
{\tt eesupp/src/} and {\tt model/src/} |
| 956 |
|
reside in {\tt pkg/autodiff/}. |
| 957 |
|
This directory also contains hand-written adjoint code |
| 958 |
|
for the MITgcm WRAPPER (see Section ???). |
| 959 |
|
|
| 960 |
|
Flow directives for package-specific routines are contained in |
| 961 |
|
the corresponding package directories in the file |
| 962 |
|
{\tt <PKG>\_ad.flow}, e.g. ptracers-specific directives are in |
| 963 |
|
{\tt ptracers\_ad.flow}. |
| 964 |
|
|
| 965 |
|
\subsubsection{Store directives for 3-level checkpointing} |
| 966 |
|
|
| 967 |
|
The storing that is required at each period of the |
| 968 |
|
3-level checkpointing is controled by three |
| 969 |
|
top-level headers. |
| 970 |
|
|
| 971 |
|
\begin{verbatim} |
| 972 |
|
do ilev_3 = 1, nchklev_3 |
| 973 |
|
# include ``checkpoint_lev3.h'' |
| 974 |
|
do ilev_2 = 1, nchklev_2 |
| 975 |
|
# include ``checkpoint_lev2.h'' |
| 976 |
|
do ilev_1 = 1, nchklev_1 |
| 977 |
|
# include ``checkpoint_lev1.h'' |
| 978 |
|
|
| 979 |
|
... |
| 980 |
|
|
| 981 |
|
end do |
| 982 |
|
end do |
| 983 |
|
end do |
| 984 |
|
\end{verbatim} |
| 985 |
|
|
| 986 |
|
All files {\tt checkpoint\_lev?.h} are contained in directory |
| 987 |
|
{\tt pkg/autodiff/}. |
| 988 |
|
|
| 989 |
|
|
| 990 |
|
\subsubsection{Changing the default AD tool flags: ad\_options files} |
| 991 |
|
|
| 992 |
|
|
| 993 |
|
\subsubsection{Hand-written adjoint code} |
| 994 |
|
|
| 995 |
|
%------------------------------------------------------------------ |
| 996 |
|
|
| 997 |
\subsection{The cost function (dependent variable) |
\subsection{The cost function (dependent variable) |
| 998 |
\label{section_cost}} |
\label{section_cost}} |
| 999 |
|
|
| 1000 |
The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}. |
The cost function $ {\cal J} $ is referred to as the {\sf dependent variable}. |
| 1001 |
It is a function of the input variables $ \vec{u} $ via the composition |
It is a function of the input variables $ \vec{u} $ via the composition |
| 1002 |
$ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $. |
$ {\cal J}(\vec{u}) \, = \, {\cal J}(M(\vec{u})) $. |
| 1003 |
The input is referred to as the |
The input are referred to as the |
| 1004 |
{\sf independent variables} or {\sf control variables}. |
{\sf independent variables} or {\sf control variables}. |
| 1005 |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
All aspects relevant to the treatment of the cost function $ {\cal J} $ |
| 1006 |
(parameter setting, initialization, accumulation, |
(parameter setting, initialization, accumulation, |
| 1007 |
final evaluation), are controlled by the package {\it pkg/cost}. |
final evaluation), are controlled by the package {\it pkg/cost}. |
| 1008 |
|
The aspects relevant to the treatment of the independent variables |
| 1009 |
|
are controlled by the package {\it pkg/ctrl} and will be treated |
| 1010 |
|
in the next section. |
| 1011 |
|
|
| 1012 |
\input{part5/doc_cost_flow} |
\input{part5/doc_cost_flow} |
| 1013 |
|
|
| 1014 |
\subsubsection{genmake and CPP options} |
\subsubsection{Enabling the package} |
| 1015 |
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item |
|
| 1016 |
\fbox{ |
\fbox{ |
| 1017 |
\begin{minipage}{12cm} |
\begin{minipage}{12cm} |
| 1018 |
{\it genmake}, {\it CPP\_OPTIONS.h}, {\it ECCO\_CPPOPTIONS.h} |
{\it packages.conf}, {\it ECCO\_CPPOPTIONS.h} |
| 1019 |
\end{minipage} |
\end{minipage} |
| 1020 |
} |
} |
| 1021 |
\end{itemize} |
\begin{itemize} |
| 1022 |
% |
% |
| 1023 |
The directory {\it pkg/cost} can be included to the |
\item |
| 1024 |
compile list in 3 different ways (cf. Section \ref{???}): |
The package is enabled by adding {\it cost} to your file {\it packages.conf} |
| 1025 |
|
(see Section ???) |
| 1026 |
% |
% |
| 1027 |
\begin{enumerate} |
\item |
| 1028 |
% |
|
| 1029 |
\item {\it genmake}: \\ |
|
| 1030 |
Change the default settings in the file {\it genmake} by adding |
\end{itemize} |
|
{\bf cost} to the {\bf enable} list (not recommended). |
|
|
% |
|
|
\item {\it .genmakerc}: \\ |
|
|
Customize the settings of {\bf enable}, {\bf disable} which are |
|
|
appropriate for your experiment in the file {\it .genmakerc} |
|
|
and add the file to your compile directory. |
|
|
% |
|
|
\item genmake-options: \\ |
|
|
Call {\it genmake} with the option |
|
|
{\tt genmake -enable=cost}. |
|
| 1031 |
% |
% |
| 1032 |
\end{enumerate} |
|
| 1033 |
|
N.B.: In general the following packages ought to be enabled |
| 1034 |
|
simultaneously: {\it autodiff, cost, ctrl}. |
| 1035 |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
| 1036 |
Each specific cost function contribution has its own option. |
Each specific cost function contribution has its own option. |
| 1037 |
For the present example the option is {\bf ALLOW\_COST\_TRACER}. |
For the present example the option is {\bf ALLOW\_COST\_TRACER}. |
| 1038 |
All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h} |
All cost-specific options are set in {\it ECCO\_CPPOPTIONS.h} |
| 1039 |
Since the cost function is usually used in conjunction with |
Since the cost function is usually used in conjunction with |
| 1040 |
automatic differentiation, the CPP option |
automatic differentiation, the CPP option |
| 1041 |
{\bf ALLOW\_ADJOINT\_RUN} should be defined |
{\bf ALLOW\_ADJOINT\_RUN} (file {\it CPP\_OPTIONS.h}) and |
| 1042 |
(file {\it CPP\_OPTIONS.h}). |
{\bf ALLOW\_AUTODIFF\_TAMC} (file {\it ECCO\_CPPOPTIONS.h}) |
| 1043 |
|
should be defined. |
| 1044 |
|
|
| 1045 |
\subsubsection{Initialization} |
\subsubsection{Initialization} |
| 1046 |
% |
% |
| 1047 |
The initialization of the {\it cost} package is readily enabled |
The initialization of the {\it cost} package is readily enabled |
| 1048 |
as soon as the CPP option {\bf ALLOW\_ADJOINT\_RUN} is defined. |
as soon as the CPP option {\bf ALLOW\_COST} is defined. |
| 1049 |
% |
% |
| 1050 |
\begin{itemize} |
\begin{itemize} |
| 1051 |
% |
% |
| 1119 |
\begin{equation} |
\begin{equation} |
| 1120 |
{\cal J} \, = \, |
{\cal J} \, = \, |
| 1121 |
{\rm fc} \, = \, |
{\rm fc} \, = \, |
| 1122 |
{\rm mult\_tracer} \sum_{bi,\,bj}^{nSx,\,nSy} |
{\rm mult\_tracer} \sum_{\text{global sum}} \sum_{bi,\,bj}^{nSx,\,nSy} |
| 1123 |
{\rm objf\_tracer}(bi,bj) \, + \, ... |
{\rm objf\_tracer}(bi,bj) \, + \, ... |
| 1124 |
\end{equation} |
\end{equation} |
| 1125 |
% |
% |
| 1167 |
% |
% |
| 1168 |
To enable the directory to be included to the compile list, |
To enable the directory to be included to the compile list, |
| 1169 |
{\bf ctrl} has to be added to the {\bf enable} list in |
{\bf ctrl} has to be added to the {\bf enable} list in |
| 1170 |
{\it .genmakerc} (or {\it genmake} itself). |
{\it .genmakerc} or in {\it genmake} itself (analogous to {\it cost} |
| 1171 |
|
package, cf. previous section). |
| 1172 |
Each control variable is enabled via its own CPP option |
Each control variable is enabled via its own CPP option |
| 1173 |
in {\it ECCO\_CPPOPTIONS.h}. |
in {\it ECCO\_CPPOPTIONS.h}. |
| 1174 |
|
|
| 1312 |
% |
% |
| 1313 |
Note, that reading an active variable corresponds |
Note, that reading an active variable corresponds |
| 1314 |
to a variable assignment. Its derivative corresponds |
to a variable assignment. Its derivative corresponds |
| 1315 |
to a write statement of the adjoint variable. |
to a write statement of the adjoint variable, followed by |
| 1316 |
|
a reset. |
| 1317 |
The 'active file' routines have been designed |
The 'active file' routines have been designed |
| 1318 |
to support active read and corresponding adjoint active write |
to support active read and corresponding adjoint active write |
| 1319 |
operations (and vice versa). |
operations (and vice versa). |
| 1430 |
{\it addummy\_in\_stepping}. |
{\it addummy\_in\_stepping}. |
| 1431 |
This routine is part of the adjoint support package |
This routine is part of the adjoint support package |
| 1432 |
{\it pkg/autodiff} (cf.f. below). |
{\it pkg/autodiff} (cf.f. below). |
| 1433 |
|
The procedure is enabled using via the CPP-option |
| 1434 |
|
{\bf ALLOW\_AUTODIFF\_MONITOR} (file {\it ECCO\_CPPOPTIONS.h}). |
| 1435 |
To be part of the adjoint code, the corresponding S/R |
To be part of the adjoint code, the corresponding S/R |
| 1436 |
{\it dummy\_in\_stepping} has to be called in the forward |
{\it dummy\_in\_stepping} has to be called in the forward |
| 1437 |
model (S/R {\it the\_main\_loop}) at the appropriate place. |
model (S/R {\it the\_main\_loop}) at the appropriate place. |
| 1438 |
|
The adjoint common blocks are extracted from the adjoint code |
| 1439 |
|
via the header file {\it adcommon.h}. |
| 1440 |
|
|
| 1441 |
{\it dummy\_in\_stepping} is essentially empty, |
{\it dummy\_in\_stepping} is essentially empty, |
| 1442 |
the corresponding adjoint routine is hand-written rather |
the corresponding adjoint routine is hand-written rather |
| 1463 |
{\bf /adtr1\_r/}, {\bf /adffields/}, |
{\bf /adtr1\_r/}, {\bf /adffields/}, |
| 1464 |
which have been extracted from the adjoint code to enable |
which have been extracted from the adjoint code to enable |
| 1465 |
access to the adjoint variables. |
access to the adjoint variables. |
| 1466 |
|
|
| 1467 |
|
{\bf WARNING:} If the structure of the common blocks |
| 1468 |
|
{\bf /dynvars\_r/}, {\bf /dynvars\_cd/}, etc., changes |
| 1469 |
|
similar changes will occur in the adjoint common blocks. |
| 1470 |
|
Therefore, consistency between the TAMC-generated common blocks |
| 1471 |
|
and those in {\it adcommon.h} have to be checked. |
| 1472 |
% |
% |
| 1473 |
\end{itemize} |
\end{itemize} |
| 1474 |
|
|
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch