| 1 |
% $Header$ |
% $Header$ |
| 2 |
% $Name$ |
% $Name$ |
| 3 |
|
|
| 4 |
|
Author: Patrick Heimbach |
| 5 |
|
|
| 6 |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
{\sf Automatic differentiation} (AD), also referred to as algorithmic |
| 7 |
(or, more loosely, computational) differentiation, involves |
(or, more loosely, computational) differentiation, involves |
| 8 |
automatically deriving code to calculate |
automatically deriving code to calculate |
| 44 |
%********************************************************************** |
%********************************************************************** |
| 45 |
\section{Some basic algebra} |
\section{Some basic algebra} |
| 46 |
\label{sec_ad_algebra} |
\label{sec_ad_algebra} |
| 47 |
|
\begin{rawhtml} |
| 48 |
|
<!-- CMIREDIR:sec_ad_algebra: --> |
| 49 |
|
\end{rawhtml} |
| 50 |
%********************************************************************** |
%********************************************************************** |
| 51 |
|
|
| 52 |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
Let $ \cal{M} $ be a general nonlinear, model, i.e. a |
| 681 |
%********************************************************************** |
%********************************************************************** |
| 682 |
\section{TLM and ADM generation in general} |
\section{TLM and ADM generation in general} |
| 683 |
\label{sec_ad_setup_gen} |
\label{sec_ad_setup_gen} |
| 684 |
|
\begin{rawhtml} |
| 685 |
|
<!-- CMIREDIR:sec_ad_setup_gen: --> |
| 686 |
|
\end{rawhtml} |
| 687 |
%********************************************************************** |
%********************************************************************** |
| 688 |
|
|
| 689 |
In this section we describe in a general fashion |
In this section we describe in a general fashion |
| 690 |
the parts of the code that are relevant for automatic |
the parts of the code that are relevant for automatic |
| 691 |
differentiation using the software tool TAMC. |
differentiation using the software tool TAF. |
| 692 |
|
|
| 693 |
\input{part5/doc_ad_the_model} |
\input{part5/doc_ad_the_model} |
| 694 |
|
|
| 695 |
The basic flow is depicted in \ref{fig:adthemodel}. |
The basic flow is depicted in \ref{fig:adthemodel}. |
| 696 |
If the option {\tt ALLOW\_AUTODIFF\_TAMC} is defined, the driver routine |
If CPP option \texttt{ALLOW\_AUTODIFF\_TAMC} is defined, |
| 697 |
|
the driver routine |
| 698 |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
{\it the\_model\_main}, instead of calling {\it the\_main\_loop}, |
| 699 |
invokes the adjoint of this routine, {\it adthe\_main\_loop}, |
invokes the adjoint of this routine, {\it adthe\_main\_loop} |
| 700 |
which is the toplevel routine in terms of reverse mode computation. |
(case \texttt{\#define ALLOW\_ADJOINT\_RUN}), or |
| 701 |
The routine {\it adthe\_main\_loop} has been generated by TAMC. |
the tangent linear of this routine {\it g\_the\_main\_loop} |
| 702 |
It contains both the forward integration of the full model, |
(case \texttt{\#define ALLOW\_TANGENTLINEAR\_RUN}), |
| 703 |
|
which are the toplevel routines in terms of automatic differentiation. |
| 704 |
|
The routines {\it adthe\_main\_loop} or {\it g\_the\_main\_loop} |
| 705 |
|
are generated by TAF. |
| 706 |
|
It contains both the forward integration of the full model, the |
| 707 |
|
cost function calculation, |
| 708 |
any additional storing that is required for efficient checkpointing, |
any additional storing that is required for efficient checkpointing, |
| 709 |
and the reverse integration of the adjoint model. |
and the reverse integration of the adjoint model. |
| 710 |
The structure of {\it adthe\_main\_loop} has been strongly |
|
| 711 |
simplified for clarification; in particular, no checkpointing |
[DESCRIBE IN A SEPARATE SECTION THE WORKING OF THE TLM] |
| 712 |
|
|
| 713 |
|
In Fig. \ref{fig:adthemodel} |
| 714 |
|
the structure of {\it adthe\_main\_loop} has been strongly |
| 715 |
|
simplified to focus on the essentials; in particular, no checkpointing |
| 716 |
procedures are shown here. |
procedures are shown here. |
| 717 |
Prior to the call of {\it adthe\_main\_loop}, the routine |
Prior to the call of {\it adthe\_main\_loop}, the routine |
| 718 |
{\it ctrl\_unpack} is invoked to unpack the control vector |
{\it ctrl\_unpack} is invoked to unpack the control vector |
| 727 |
the gradient has been computed via the adjoint |
the gradient has been computed via the adjoint |
| 728 |
(cf. Section \ref{section_grdchk}). |
(cf. Section \ref{section_grdchk}). |
| 729 |
|
|
| 730 |
|
%------------------------------------------------------------------ |
| 731 |
|
|
| 732 |
|
\subsection{General setup |
| 733 |
|
\label{section_ad_setup}} |
| 734 |
|
|
| 735 |
|
In order to configure AD-related setups the following packages need |
| 736 |
|
to be enabled: |
| 737 |
|
{\it |
| 738 |
|
\begin{table}[h!] |
| 739 |
|
\begin{tabular}{l} |
| 740 |
|
autodiff \\ |
| 741 |
|
ctrl \\ |
| 742 |
|
cost \\ |
| 743 |
|
grdchk \\ |
| 744 |
|
\end{tabular} |
| 745 |
|
\end{table} |
| 746 |
|
} |
| 747 |
|
The packages are enabled by adding them to your experiment-specific |
| 748 |
|
configuration file |
| 749 |
|
{\it packages.conf} (see Section ???). |
| 750 |
|
|
| 751 |
|
The following AD-specific CPP option files need to be customized: |
| 752 |
|
% |
| 753 |
|
\begin{itemize} |
| 754 |
|
% |
| 755 |
|
\item {\it ECCO\_CPPOPTIONS.h} \\ |
| 756 |
|
This header file collects CPP options for the packages |
| 757 |
|
{\it autodiff, cost, ctrl} as well as AD-unrelated options for |
| 758 |
|
the external forcing package {\it exf}. |
| 759 |
|
\footnote{NOTE: These options are not set in their package-specific |
| 760 |
|
headers such as {\it COST\_CPPOPTIONS.h}, but are instead collected |
| 761 |
|
in the single header file {\it ECCO\_CPPOPTIONS.h}. |
| 762 |
|
The package-specific header files serve as simple |
| 763 |
|
placeholders at this point.} |
| 764 |
|
% |
| 765 |
|
\item {\it tamc.h} \\ |
| 766 |
|
This header configures the splitting of the time stepping loop |
| 767 |
|
w.r.t. the 3-level checkpointing (see section ???). |
| 768 |
|
|
| 769 |
|
% |
| 770 |
|
\end{itemize} |
| 771 |
|
|
| 772 |
|
%------------------------------------------------------------------ |
| 773 |
|
|
| 774 |
|
\subsection{Building the AD code |
| 775 |
|
\label{section_ad_build}} |
| 776 |
|
|
| 777 |
|
The build process of an AD code is very similar to building |
| 778 |
|
the forward model. However, depending on which AD code one wishes |
| 779 |
|
to generate, and on which AD tool is available (TAF or TAMC), |
| 780 |
|
the following {\tt make} targets are available: |
| 781 |
|
|
| 782 |
|
\begin{table}[h!] |
| 783 |
|
{\footnotesize |
| 784 |
|
\begin{tabular}{ccll} |
| 785 |
|
~ & {\it AD-target} & {\it output} & {\it description} \\ |
| 786 |
|
\hline |
| 787 |
|
\hline |
| 788 |
|
(1) & {\tt <MODE><TOOL>only} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 789 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 790 |
|
~ & ~ & ~ & no {\tt make} dependencies on {\tt .F .h} \\ |
| 791 |
|
~ & ~ & ~ & useful for compiling on remote platforms \\ |
| 792 |
|
\hline |
| 793 |
|
(2) & {\tt <MODE><TOOL>} & {\tt <MODE>\_<TOOL>\_output.f} & |
| 794 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 795 |
|
~ & ~ & ~ & includes {\tt make} dependencies on {\tt .F .h} \\ |
| 796 |
|
~ & ~ & ~ & i.e. input for $<$TOOL$>$ may be re-generated \\ |
| 797 |
|
\hline |
| 798 |
|
(3) & {\tt <MODE>all} & {\tt mitgcmuv\_<MODE>} & |
| 799 |
|
generates code for $<$MODE$>$ using $<$TOOL$>$ \\ |
| 800 |
|
~ & ~ & ~ & and compiles all code \\ |
| 801 |
|
~ & ~ & ~ & (use of TAF is set as default) \\ |
| 802 |
|
\hline |
| 803 |
|
\hline |
| 804 |
|
\end{tabular} |
| 805 |
|
} |
| 806 |
|
\end{table} |
| 807 |
|
% |
| 808 |
|
Here, the following placeholders are used |
| 809 |
|
% |
| 810 |
|
\begin{itemize} |
| 811 |
|
% |
| 812 |
|
\item [$<$TOOL$>$] |
| 813 |
|
% |
| 814 |
|
\begin{itemize} |
| 815 |
|
% |
| 816 |
|
\item {\tt TAF} |
| 817 |
|
\item {\tt TAMC} |
| 818 |
|
% |
| 819 |
|
\end{itemize} |
| 820 |
|
% |
| 821 |
|
\item [$<$MODE$>$] |
| 822 |
|
% |
| 823 |
|
\begin{itemize} |
| 824 |
|
% |
| 825 |
|
\item {\tt ad} generates the adjoint model (ADM) |
| 826 |
|
\item {\tt ftl} generates the tangent linear model (TLM) |
| 827 |
|
\item {\tt svd} generates both ADM and TLM for \\ |
| 828 |
|
singular value decomposition (SVD) type calculations |
| 829 |
|
% |
| 830 |
|
\end{itemize} |
| 831 |
|
% |
| 832 |
|
\end{itemize} |
| 833 |
|
|
| 834 |
|
For example, to generate the adjoint model using TAF after routines ({\tt .F}) |
| 835 |
|
or headers ({\tt .h}) have been modified, but without compilation, |
| 836 |
|
type {\tt make adtaf}; |
| 837 |
|
or, to generate the tangent linear model using TAMC without |
| 838 |
|
re-generating the input code, type {\tt make ftltamconly}. |
| 839 |
|
|
| 840 |
|
|
| 841 |
|
A typical full build process to generate the ADM via TAF would |
| 842 |
|
look like follows: |
| 843 |
|
\begin{verbatim} |
| 844 |
|
% mkdir build |
| 845 |
|
% cd build |
| 846 |
|
% ../../../tools/genmake2 -mods=../code_ad |
| 847 |
|
% make depend |
| 848 |
|
% make adall |
| 849 |
|
\end{verbatim} |
| 850 |
|
|
| 851 |
|
%------------------------------------------------------------------ |
| 852 |
|
|
| 853 |
|
\subsection{The AD build process in detail |
| 854 |
|
\label{section_ad_build_detail}} |
| 855 |
|
|
| 856 |
|
The {\tt make <MODE>all} target consists of the following procedures: |
| 857 |
|
|
| 858 |
|
\begin{enumerate} |
| 859 |
|
% |
| 860 |
|
\item |
| 861 |
|
A header file {\tt AD\_CONFIG.h} is generated which contains a CPP option |
| 862 |
|
on which code ought to be generated. Depending on the {\tt make} target, |
| 863 |
|
the contents is |
| 864 |
|
\begin{itemize} |
| 865 |
|
\item |
| 866 |
|
{\tt \#define ALLOW\_ADJOINT\_RUN} |
| 867 |
|
\item |
| 868 |
|
{\tt \#define ALLOW\_TANGENTLINEAR\_RUN} |
| 869 |
|
\item |
| 870 |
|
{\tt \#define ALLOW\_ECCO\_OPTIMIZATION} |
| 871 |
|
\end{itemize} |
| 872 |
|
% |
| 873 |
|
\item |
| 874 |
|
A single file {\tt <MODE>\_input\_code.f} is concatenated |
| 875 |
|
consisting of all {\tt .f} files that are part of the list {\bf AD\_FILES} |
| 876 |
|
and all {\tt .flow} files that are part of the list {\bf AD\_FLOW\_FILES}. |
| 877 |
|
% |
| 878 |
|
\item |
| 879 |
|
The AD tool is invoked with the {\bf <MODE>\_<TOOL>\_FLAGS}. |
| 880 |
|
The default AD tool flags in {\tt genmake2} can be overrwritten by |
| 881 |
|
an {\tt adjoint\_options} file (similar to the platform-specific |
| 882 |
|
{\tt build\_options}, see Section ???. |
| 883 |
|
The AD tool writes the resulting AD code into the file |
| 884 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 885 |
|
% |
| 886 |
|
\item |
| 887 |
|
A short sed script {\tt adjoint\_sed} is applied to |
| 888 |
|
{\tt <MODE>\_input\_code\_ad.f} |
| 889 |
|
to reinstate {\bf myThid} into the CALL argument list of active file I/O. |
| 890 |
|
The result is written to file {\tt <MODE>\_<TOOL>\_output.f}. |
| 891 |
|
% |
| 892 |
|
\item |
| 893 |
|
All routines are compiled and an executable is generated |
| 894 |
|
(see Table ???). |
| 895 |
|
% |
| 896 |
|
\end{enumerate} |
| 897 |
|
|
| 898 |
|
\subsubsection{The list AD\_FILES and {\tt .list} files} |
| 899 |
|
|
| 900 |
|
Not all routines are presented to the AD tool. |
| 901 |
|
Routines typically hidden are diagnostics routines which |
| 902 |
|
do not influence the cost function, but may create |
| 903 |
|
artificial flow dependencies such as I/O of active variables. |
| 904 |
|
|
| 905 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FILES} |
| 906 |
|
which contains all routines that are shown to the AD tool. |
| 907 |
|
This list is put together from all files with suffix {\tt .list} |
| 908 |
|
that {\tt genmake2} finds in its search directories. |
| 909 |
|
The list file for the core MITgcm routines is in {\tt model/src/} |
| 910 |
|
is called {\tt model\_ad\_diff.list}. |
| 911 |
|
Note that no wrapper routine is shown to TAF. These are either |
| 912 |
|
not visible at all to the AD code, or hand-written AD code |
| 913 |
|
is available (see next section). |
| 914 |
|
|
| 915 |
|
Each package directory contains its package-specific |
| 916 |
|
list file {\tt <PKG>\_ad\_diff.list}. For example, |
| 917 |
|
{\tt pkg/ptracers/} contains the file {\tt ptracers\_ad\_diff.list}. |
| 918 |
|
Thus, enabling a package will automatically extend the |
| 919 |
|
{\bf AD\_FILES} list of {\tt genmake2} to incorporate the |
| 920 |
|
package-specific routines. |
| 921 |
|
Note that you will need to regenerate the {\tt Makefile} if |
| 922 |
|
you enable a package (e.g. by adding it to {\tt packages.conf}) |
| 923 |
|
and a {\tt Makefile} already exists. |
| 924 |
|
|
| 925 |
|
\subsubsection{The list AD\_FLOW\_FILES and {\tt .flow} files} |
| 926 |
|
|
| 927 |
|
TAMC and TAF can evaluate user-specified directives |
| 928 |
|
that start with a specific syntax ({\tt CADJ}, {\tt C\$TAF}, {\tt !\$TAF}). |
| 929 |
|
The main categories of directives are STORE directives and |
| 930 |
|
FLOW directives. Here, we are concerned with flow directives, |
| 931 |
|
store directives are treated elsewhere. |
| 932 |
|
|
| 933 |
|
Flow directives enable the AD tool to evaluate how it should treat |
| 934 |
|
routines that are 'hidden' by the user, i.e. routines which are |
| 935 |
|
not contained in the {\bf AD\_FILES} list (see previous section), |
| 936 |
|
but which are called in part of the code that the AD tool does see. |
| 937 |
|
The flow directive tell the AD tool |
| 938 |
|
% |
| 939 |
|
\begin{itemize} |
| 940 |
|
% |
| 941 |
|
\item which subroutine arguments are input/output |
| 942 |
|
\item which subroutine arguments are active |
| 943 |
|
\item which subroutine arguments are required to compute the cost |
| 944 |
|
\item which subroutine arguments are dependent |
| 945 |
|
% |
| 946 |
|
\end{itemize} |
| 947 |
|
% |
| 948 |
|
The syntax for the flow directives can be found in the |
| 949 |
|
AD tool manuals. |
| 950 |
|
|
| 951 |
|
{\tt genmake2} generates a list (or variable) {\bf AD\_FLOW\_FILES} |
| 952 |
|
which contains all files with suffix{\tt .flow} that it finds |
| 953 |
|
in its search directories. |
| 954 |
|
The flow directives for the core MITgcm routines of |
| 955 |
|
{\tt eesupp/src/} and {\tt model/src/} |
| 956 |
|
reside in {\tt pkg/autodiff/}. |
| 957 |
|
This directory also contains hand-written adjoint code |
| 958 |
|
for the MITgcm WRAPPER (see Section ???). |
| 959 |
|
|
| 960 |
|
Flow directives for package-specific routines are contained in |
| 961 |
|
the corresponding package directories in the file |
| 962 |
|
{\tt <PKG>\_ad.flow}, e.g. ptracers-specific directives are in |
| 963 |
|
{\tt ptracers\_ad.flow}. |
| 964 |
|
|
| 965 |
|
\subsubsection{Store directives for 3-level checkpointing} |
| 966 |
|
|
| 967 |
|
The storing that is required at each period of the |
| 968 |
|
3-level checkpointing is controled by three |
| 969 |
|
top-level headers. |
| 970 |
|
|
| 971 |
|
\begin{verbatim} |
| 972 |
|
do ilev_3 = 1, nchklev_3 |
| 973 |
|
# include ``checkpoint_lev3.h'' |
| 974 |
|
do ilev_2 = 1, nchklev_2 |
| 975 |
|
# include ``checkpoint_lev2.h'' |
| 976 |
|
do ilev_1 = 1, nchklev_1 |
| 977 |
|
# include ``checkpoint_lev1.h'' |
| 978 |
|
|
| 979 |
|
... |
| 980 |
|
|
| 981 |
|
end do |
| 982 |
|
end do |
| 983 |
|
end do |
| 984 |
|
\end{verbatim} |
| 985 |
|
|
| 986 |
|
All files {\tt checkpoint\_lev?.h} are contained in directory |
| 987 |
|
{\tt pkg/autodiff/}. |
| 988 |
|
|
| 989 |
|
|
| 990 |
|
\subsubsection{Changing the default AD tool flags: ad\_options files} |
| 991 |
|
|
| 992 |
|
|
| 993 |
|
\subsubsection{Hand-written adjoint code} |
| 994 |
|
|
| 995 |
|
%------------------------------------------------------------------ |
| 996 |
|
|
| 997 |
\subsection{The cost function (dependent variable) |
\subsection{The cost function (dependent variable) |
| 998 |
\label{section_cost}} |
\label{section_cost}} |
| 999 |
|
|
| 1011 |
|
|
| 1012 |
\input{part5/doc_cost_flow} |
\input{part5/doc_cost_flow} |
| 1013 |
|
|
| 1014 |
\subsubsection{genmake and CPP options} |
\subsubsection{Enabling the package} |
| 1015 |
% |
|
|
\begin{itemize} |
|
|
% |
|
|
\item |
|
| 1016 |
\fbox{ |
\fbox{ |
| 1017 |
\begin{minipage}{12cm} |
\begin{minipage}{12cm} |
| 1018 |
{\it genmake}, {\it CPP\_OPTIONS.h}, {\it ECCO\_CPPOPTIONS.h} |
{\it packages.conf}, {\it ECCO\_CPPOPTIONS.h} |
| 1019 |
\end{minipage} |
\end{minipage} |
| 1020 |
} |
} |
| 1021 |
\end{itemize} |
\begin{itemize} |
|
% |
|
|
The directory {\it pkg/cost} can be included to the |
|
|
compile list in 3 different ways (cf. Section \ref{???}): |
|
| 1022 |
% |
% |
| 1023 |
\begin{enumerate} |
\item |
| 1024 |
|
The package is enabled by adding {\it cost} to your file {\it packages.conf} |
| 1025 |
|
(see Section ???) |
| 1026 |
% |
% |
| 1027 |
\item {\it genmake}: \\ |
\item |
| 1028 |
Change the default settings in the file {\it genmake} by adding |
|
| 1029 |
{\bf cost} to the {\bf enable} list (not recommended). |
|
| 1030 |
% |
\end{itemize} |
|
\item {\it .genmakerc}: \\ |
|
|
Customize the settings of {\bf enable}, {\bf disable} which are |
|
|
appropriate for your experiment in the file {\it .genmakerc} |
|
|
and add the file to your compile directory. |
|
|
% |
|
|
\item genmake-options: \\ |
|
|
Call {\it genmake} with the option |
|
|
{\tt genmake -enable=cost}. |
|
| 1031 |
% |
% |
| 1032 |
\end{enumerate} |
|
| 1033 |
N.B.: In general the following packages ought to be enabled |
N.B.: In general the following packages ought to be enabled |
| 1034 |
simultaneously: {\it autodiff, cost, ctrl}. |
simultaneously: {\it autodiff, cost, ctrl}. |
| 1035 |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
The basic CPP option to enable the cost function is {\bf ALLOW\_COST}. |
Parent Directory
| 
Revision Log
| 
Revision Graph
| 
Patch