--- manual/s_phys_pkgs/text/exch2.tex	2004/10/12 17:27:17	1.21
+++ manual/s_phys_pkgs/text/exch2.tex	2009/05/02 02:13:18	1.27
@@ -1,4 +1,4 @@
-% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.21 2004/10/12 17:27:17 edhill Exp $
+% $Header: /home/ubuntu/mnt/e9_copy/manual/s_phys_pkgs/text/exch2.tex,v 1.27 2009/05/02 02:13:18 jmc Exp $
 % $Name:  $
 
 %%  * Introduction
@@ -10,11 +10,11 @@
 %%    o automatically inserted at \section{Reference} 
 
 
-\section{exch2: Extended Cubed Sphere \mbox{Topology}}
+\subsection{exch2: Extended Cubed Sphere \mbox{Topology}}
 \label{sec:exch2}
 
 
-\subsection{Introduction}
+\subsubsection{Introduction}
 
 The \texttt{exch2} package extends the original cubed sphere topology
 configuration to allow more flexible domain decomposition and
@@ -23,7 +23,7 @@
 dimensions of the subdomain.  Furthermore, the tiles can run on
 separate processors individually or in groups, which provides for
 manual compile-time load balancing across a relatively arbitrary
-number of processors. \\
+number of processors.
 
 The exchange parameters are declared in
 \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
@@ -41,7 +41,7 @@
 \file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h}
 file for single-processor execution.
 
-\subsection{Invoking exch2}
+\subsubsection{Invoking exch2}
 
 To use exch2 with the cubed sphere, the following conditions must be
 met:
@@ -49,9 +49,10 @@
 \begin{itemize}
 \item The exch2 package is included when \file{genmake2} is run.  The
   easiest way to do this is to add the line \code{exch2} to the
-  \file{profile.conf} file -- see Section \ref{sect:buildingCode}
-  \sectiontitle{Building the code} for general details.
-  
+  \file{packages.conf} file -- see Section \ref{sect:buildingCode}
+  \sectiontitle{Building the code} for general
+  details.
+
 \item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
   \file{w2\_e2setup.F} must reside in a directory containing files
   symbolically linked by the \file{genmake2} script.  The safest place
@@ -62,21 +63,14 @@
   exist elsewhere in the build path, but they should be left untouched
   to avoid breaking configurations other than the one you intend to
   modify.
-  
+
 \item Files containing grid parameters, named \file{tile00$n$.mitgrid}
   where $n$=\code{(1:6)} (one per subdomain), must be in the working
   directory when the MITgcm executable is run.  These files are
   provided in the example experiments for cubed sphere configurations
-  with 32$\times$32 cube sides -- please contact
-  \begin{rawhtml}
-    <A href="mailto:mitgcm-support@dev.mitgcm.org"> 
-  \end{rawhtml} 
-\begin{verbatim}
-MITgcm-support@mitgcm.org
-\end{verbatim}
-  \begin{rawhtml} </A> \end{rawhtml}
-  if you want to generate files for other configurations.
-  
+  with 32$\times$32 cube sides -- please contact MITgcm support if you
+  want to generate files for other configurations.
+
 \item As always when compiling MITgcm, the file \file{SIZE.h} must be
   placed where \file{genmake2} will find it.  In particular for exch2,
   the domain decomposition specified in \file{SIZE.h} must correspond
@@ -87,12 +81,10 @@
   and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
     Multiprocessing}; a more general background on the subject
   relevant to MITgcm is presented in Section
-  \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying a
-    decomposition}.
+  \ref{sect:specifying_a_decomposition}
+  \sectiontitle{Specifying a decomposition}.
 \end{itemize}
 
-
-
 At the time of this writing the following examples use exch2 and may
 be used for guidance:
 
@@ -107,7 +99,7 @@
 
 
 
-\subsection{Generating Topology Files for exch2}
+\subsubsection{Generating Topology Files for exch2}
 \label{sec:topogen}
 
 Alternate cubed sphere topologies may be created using the Matlab
@@ -117,8 +109,8 @@
 from the Matlab prompt (there are no parameters to pass) generates
 exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
 \file{w2\_e2setup.F} in the working directory and displays a figure of
-the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile}, 
-and \ref{fig:24tile} are examples of the generated diagrams.  The other 
+the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:18tile}, 
+and \ref{fig:48tile} are examples of the generated diagrams.  The other 
 m-files in the directory are
 subroutines called from \file{driver.m} and should not be run ``bare'' except
 for development purposes. \\
@@ -139,39 +131,48 @@
 the tiles into which the subdomains are decomposed, and must evenly
 divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
 The result is a rectangular tiling of the subdomain.  Figure
-\ref{fig:24tile} shows one possible topology for a twenty-four-tile
-cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\
+\ref{fig:48tile} shows one possible topology for a twenty-four-tile
+cube, and figure \ref{fig:6tile} shows one for six tiles. \\
 
 \begin{figure}
 \begin{center}
- \resizebox{4in}{!}{
-  \includegraphics{part6/s24t_16x16.ps}
+ \resizebox{6in}{!}{
+% \includegraphics{part6/s24t_16x16.ps}
+  \includegraphics{part6/adjust_cs.ps}
  }
 \end{center} 
 
 \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
 divided into six 32$\times$32 subdomains, each of which is divided
-into four tiles of width \code{tnx=16} and height \code{tny=16} for a 
-total of twenty-four tiles.  The colored borders of the subdomains 
-represent the parameters \code{nr} (red), \code{nb} (blue), and 
-\code{ng} (green).  } \label{fig:24tile}
+into eight tiles of width \code{tnx=16} and height \code{tny=8} for a 
+total of forty-eight tiles. The colored borders of the subdomains 
+represent the parameters \code{nr} (red), \code{ng} (green), and
+\code{nb} (blue). 
+This tiling is used in the example
+verification/adjustment.cs-32x32x1/
+with the option (blanklist.txt) to remove the land-only 4 tiles 
+(11,12,13,14) which are filled in red on the plot.
+} \label{fig:48tile}
 \end{figure}
 
 \begin{figure}
 \begin{center}
- \resizebox{4in}{!}{
-  \includegraphics{part6/s12t_16x32.ps}
+ \resizebox{6in}{!}{
+% \includegraphics{part6/s12t_16x32.ps}
+  \includegraphics{part6/polarcap.ps}
  }
 \end{center} 
-\caption{Plot of a cubed sphere topology with a 32$\times$192 domain
-divided into six 32$\times$32 subdomains of two tiles each
- (\code{tnx=16, tny=32}).
-} \label{fig:12tile}
+\caption{Plot of a non-square cubed sphere topology with 
+6 subdomains of different size (nr=90,ng=360,nb=90),
+divided into one to four tiles each
+ (\code{tnx=90, tny=90}), resulting in a total of 18 tiles.
+} \label{fig:18tile}
 \end{figure}
 
 \begin{figure}
 \begin{center}
  \resizebox{4in}{!}{
+% \includegraphics{part6/s6t_32x32.ps}
   \includegraphics{part6/s6t_32x32.ps}
  }
 \end{center} 
@@ -193,27 +194,26 @@
 
 
 
-\subsection{exch2, SIZE.h, and Multiprocessing}
+\subsubsection{exch2, SIZE.h, and Multiprocessing}
 \label{sec:exch2mpi}
 
 Once the topology configuration files are created, the Fortran
 \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
 Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
-a decomposition} provides a general description of domain
+  a decomposition} provides a general description of domain
 decomposition within MITgcm and its relation to \file{SIZE.h}. The
-current section specifies constraints that the exch2 package
-imposes and describes how to enable parallel execution with
-MPI. \\
+current section specifies constraints that the exch2 package imposes
+and describes how to enable parallel execution with MPI.
 
 As in the general case, the parameters \varlink{sNx}{sNx} and
 \varlink{sNy}{sNy} define the size of the individual tiles, and so
 must be assigned the same respective values as \code{tnx} and
-\code{tny} in \file{driver.m}.\\
+\code{tny} in \file{driver.m}.
 
 The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
 have no special bearing on exch2 and may be assigned as in the general
-case. The same holds for \varlink{Nr}{Nr}, the number of vertical 
-levels in the model.\\
+case. The same holds for \varlink{Nr}{Nr}, the number of vertical
+levels in the model.
 
 The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
 \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
@@ -221,29 +221,36 @@
 the tiles are stored in the $x$ dimension, and so
 \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
 configured by exch2 cannot be split up accross processors without
-regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\
+regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well.
 
 The number of tiles MITgcm allocates and how they are distributed
 between processors depends on \varlink{nPx}{nPx} and
 \varlink{nSx}{nSx}.  \varlink{nSx}{nSx} is the number of tiles per
-processor and \varlink{nPx}{nPx} is the number of processors.  The total
-number of tiles in the topology minus those listed in
-\file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to 
+processor and \varlink{nPx}{nPx} is the number of processors.  The
+total number of tiles in the topology minus those listed in
+\file{blanklist.txt} must equal \code{nSx*nPx}.  Note that in order to
 obtain maximum usage from a given number of processors in some cases,
-this restriction might entail sharing a processor with a tile that would 
-otherwise be excluded. \\
-
-The following is an example of \file{SIZE.h} for the twelve-tile
-configuration illustrated in figure \ref{fig:12tile} running on 
-one processor: \\
+this restriction might entail sharing a processor with a tile that
+would otherwise be excluded because it is topographically outside of
+the domain and therefore in \file{blanklist.txt}.  For example,
+suppose you have five processors and a domain decomposition of
+thirty-six tiles that allows you to exclude seven tiles.  To evenly
+distribute the remaining twenty-nine tiles among five processors, you
+would have to run one ``dummy'' tile to make an even six tiles per
+processor.  Such dummy tiles are \emph{not} listed in
+\file{blanklist.txt}.
+
+The following is an example of \file{SIZE.h} for the six-tile
+configuration illustrated in figure \ref{fig:6tile} 
+running on one processor:
 
 \begin{verbatim}
       PARAMETER (
-     &           sNx =  16,
+     &           sNx =  32,
      &           sNy =  32,
      &           OLx =   2,
      &           OLy =   2,
-     &           nSx =  12,
+     &           nSx =   6,
      &           nSy =   1,
      &           nPx =   1,
      &           nPy =   1,
@@ -252,16 +259,16 @@
      &           Nr  =   5)
 \end{verbatim}
 
-The following is an example for the twenty-four-tile topology in
-figure \ref{fig:24tile} running on six processors:
+The following is an example for the forty-eight-tile topology in
+figure \ref{fig:48tile} running on six processors:
 
 \begin{verbatim}
       PARAMETER (
      &           sNx =  16,
-     &           sNy =  16,
+     &           sNy =   8,
      &           OLx =   2,
      &           OLy =   2,
-     &           nSx =   4,
+     &           nSx =   8,
      &           nSy =   1,
      &           nPx =   6,
      &           nPy =   1,
@@ -271,10 +278,7 @@
 \end{verbatim}
 
 
-
-
-
-\subsection{Key Variables}
+\subsubsection{Key Variables}
 
 The descriptions of the variables are divided up into scalars,
 one-dimensional arrays indexed to the tile number, and two and
@@ -284,7 +288,7 @@
 arrays to individual tiles, and the arrays indexed by tile and
 neighbor to relationships between tiles and their neighbors. \\
 
-\subsubsection{Scalars}
+Scalars:
 
 The number of tiles in a particular topology is set with the parameter
 \code{NTILES}, and the maximum number of neighbors of any tiles by
@@ -298,20 +302,20 @@
 of tiles in the $x$ and $y$ global indices.  For example, the default
 setup of six tiles (Fig. \ref{fig:6tile}) has
 \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
-topology of twenty-four square tiles, four per subdomain (as in figure
-\ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and
-\code{exch2\_domain\_nyt=2}.  Note that these parameters express the
+topology of forty-eight tiles, eight per subdomain (as in figure
+\ref{fig:48tile}), will have \code{exch2\_domain\_nxt=12} and
+\code{exch2\_domain\_nyt=4}.  Note that these parameters express the
 tile layout in order to allow global data files that are tile-layout-neutral.
 They have no bearing on the internal storage of the arrays.  The tiles
 are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
 $x$ axis, and the $y$ axis variable \varlink{bj}{bj} is assumed to 
 equal \code{1} throughout the package. \\
 
-\subsubsection{Arrays indexed to tile number}
+Arrays indexed to tile number:
 
 The following arrays are of length \code{NTILES} and are indexed to
 the tile number, which is indicated in the diagrams with the notation
-\code{tn}.  The indices are omitted in the descriptions. \\
+\textsf{t}$n$.  The indices are omitted in the descriptions. \\
 
 The arrays \varlink{exch2\_tnx}{exch2_tnx} and
 \varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of
@@ -339,11 +343,11 @@
 
 The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
 the subdomain of each tile, in a range \code{(1:6)} in the case of the
-standard cube topology and indicated by \textbf{\textsf{fn}} in
-figures \ref{fig:12tile} and \ref{fig:24tile}. The
-\varlink{exch2\_nNeighbours}{exch2_nNeighbours} variable contains a
-count of the neighboring tiles each tile has, and sets the bounds for
-looping over neighboring tiles.  And
+standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
+figures \ref{fig:6tile} and
+\ref{fig:48tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
+contains a count of the neighboring tiles each tile has, and sets 
+the bounds for looping over neighboring tiles.
 \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
 tile, and is used in interprocess communication.  \\
 
@@ -360,7 +364,7 @@
 corners of the cube. \\
 
 
-\subsubsection{Arrays Indexed to Tile Number and Neighbor}
+Arrays Indexed to Tile Number and Neighbor:
 
 The following arrays have vectors of length \code{MAX\_NEIGHBOURS} and
 \code{NTILES} and describe the orientations between the the tiles. \\
@@ -463,8 +467,8 @@
 \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
 bounds of the edge segment of the neighbor tile \code{N}'s subdomain
 that gets exchanged with the local tile \code{T}.  To take the example
-of tile \code{T=2} in the twelve-tile topology
-(Fig. \ref{fig:12tile}): \\
+of tile \code{T=2} in the forty-eight-tile topology
+(Fig. \ref{fig:48tile}): \\
 
 \begin{verbatim}
        exch2_itlo_c(4,2)=17
@@ -504,7 +508,7 @@
 width of \code{T}'s northern edge, expanded by one into the halo. \\
 
 
-\subsection{Key Routines}
+\subsubsection{Key Routines}
 
 Most of the subroutines particular to exch2 handle the exchanges
 themselves and are of the same format as those described in
@@ -534,3 +538,10 @@
 \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
 the work using the variables discussed above. \\
 
+\subsubsection{Experiments and tutorials that use exch2}
+\label{sec:pkg:exch2:experiments}
+
+\begin{itemize}
+\item{Held Suarez tutorial, in tutorial\_held\_suarez\_cs verification directory, 
+described in section \ref{sect:eg-hs} }
+\end{itemize}
