/[MITgcm]/manual/s_phys_pkgs/text/exch2.tex
ViewVC logotype

Diff of /manual/s_phys_pkgs/text/exch2.tex

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph | View Patch Patch

revision 1.24 by molod, Mon Jul 18 20:45:27 2005 UTC revision 1.27 by jmc, Sat May 2 02:13:18 2009 UTC
# Line 23  into any number of tiles that divide eve Line 23  into any number of tiles that divide eve
23  dimensions of the subdomain.  Furthermore, the tiles can run on  dimensions of the subdomain.  Furthermore, the tiles can run on
24  separate processors individually or in groups, which provides for  separate processors individually or in groups, which provides for
25  manual compile-time load balancing across a relatively arbitrary  manual compile-time load balancing across a relatively arbitrary
26  number of processors. \\  number of processors.
27    
28  The exchange parameters are declared in  The exchange parameters are declared in
29  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}  \filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h}
# Line 44  file for single-processor execution. Line 44  file for single-processor execution.
44  \subsubsection{Invoking exch2}  \subsubsection{Invoking exch2}
45    
46  To use exch2 with the cubed sphere, the following conditions must be  To use exch2 with the cubed sphere, the following conditions must be
47  met: \\  met:
48    
49  $\bullet$ The exch2 package is included when \file{genmake2} is run.  \begin{itemize}
50    The easiest way to do this is to add the line \code{exch2} to the  \item The exch2 package is included when \file{genmake2} is run.  The
51    \file{profile.conf} file -- see Section    easiest way to do this is to add the line \code{exch2} to the
52    \ref{sect:buildingCode} \sectiontitle{Building the code} for general    \file{packages.conf} file -- see Section \ref{sect:buildingCode}
53    details. \\    \sectiontitle{Building the code} for general
54      details.
55    
56  $\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and  \item An example of \file{W2\_EXCH2\_TOPOLOGY.h} and
57    \file{w2\_e2setup.F} must reside in a directory containing files    \file{w2\_e2setup.F} must reside in a directory containing files
58    symbolically linked by the \file{genmake2} script.  The safest place to    symbolically linked by the \file{genmake2} script.  The safest place
59    put these is the directory indicated in the \code{-mods=DIR} command    to put these is the directory indicated in the \code{-mods=DIR}
60    line modifier (typically \file{../code}), or the build directory.    command line modifier (typically \file{../code}), or the build
61    The default versions of these files reside in \file{pkg/exch2} and    directory.  The default versions of these files reside in
62    are linked automatically if no other versions exist elsewhere in the    \file{pkg/exch2} and are linked automatically if no other versions
63    build path, but they should be left untouched to avoid breaking    exist elsewhere in the build path, but they should be left untouched
64    configurations other than the one you intend to modify.\\    to avoid breaking configurations other than the one you intend to
65      modify.
66  $\bullet$ Files containing grid parameters, named  
67    \file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain),  \item Files containing grid parameters, named \file{tile00$n$.mitgrid}
68    must be in the working directory when the MITgcm executable is run.    where $n$=\code{(1:6)} (one per subdomain), must be in the working
69    These files are provided in the example experiments for cubed sphere    directory when the MITgcm executable is run.  These files are
70    configurations with 32$\times$32 cube sides    provided in the example experiments for cubed sphere configurations
71    -- please contact MITgcm support if you want to generate    with 32$\times$32 cube sides -- please contact MITgcm support if you
72    files for other configurations. \\    want to generate files for other configurations.
73    
74  $\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must  \item As always when compiling MITgcm, the file \file{SIZE.h} must be
75    be placed where \file{genmake2} will find it.  In particular for    placed where \file{genmake2} will find it.  In particular for exch2,
76    exch2, the domain decomposition specified in \file{SIZE.h} must    the domain decomposition specified in \file{SIZE.h} must correspond
77    correspond with the particular configuration's topology specified in    with the particular configuration's topology specified in
78    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain    \file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}.  Domain
79    decomposition issues particular to exch2 are addressed in Section    decomposition issues particular to exch2 are addressed in Section
80    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}    \ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2}
81    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and Multiprocessing}; a more    and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and
82    general background on the subject relevant to MITgcm is presented in      Multiprocessing}; a more general background on the subject
83    Section \ref{sect:specifying_a_decomposition}    relevant to MITgcm is presented in Section
84    \sectiontitle{Specifying a decomposition}.\\    \ref{sect:specifying_a_decomposition}
85      \sectiontitle{Specifying a decomposition}.
86    \end{itemize}
87    
88  At the time of this writing the following examples use exch2 and may  At the time of this writing the following examples use exch2 and may
89  be used for guidance:  be used for guidance:
# Line 106  m-file Line 109  m-file
109  from the Matlab prompt (there are no parameters to pass) generates  from the Matlab prompt (there are no parameters to pass) generates
110  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and  exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and
111  \file{w2\_e2setup.F} in the working directory and displays a figure of  \file{w2\_e2setup.F} in the working directory and displays a figure of
112  the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:12tile},  the topology via Matlab -- figures \ref{fig:6tile}, \ref{fig:18tile},
113  and \ref{fig:24tile} are examples of the generated diagrams.  The other  and \ref{fig:48tile} are examples of the generated diagrams.  The other
114  m-files in the directory are  m-files in the directory are
115  subroutines called from \file{driver.m} and should not be run ``bare'' except  subroutines called from \file{driver.m} and should not be run ``bare'' except
116  for development purposes. \\  for development purposes. \\
# Line 128  The parameters \code{tnx} and \code{tny} Line 131  The parameters \code{tnx} and \code{tny}
131  the tiles into which the subdomains are decomposed, and must evenly  the tiles into which the subdomains are decomposed, and must evenly
132  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.  divide the integer assigned to \code{nr}, \code{nb} and \code{ng}.
133  The result is a rectangular tiling of the subdomain.  Figure  The result is a rectangular tiling of the subdomain.  Figure
134  \ref{fig:24tile} shows one possible topology for a twenty-four-tile  \ref{fig:48tile} shows one possible topology for a twenty-four-tile
135  cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\  cube, and figure \ref{fig:6tile} shows one for six tiles. \\
136    
137  \begin{figure}  \begin{figure}
138  \begin{center}  \begin{center}
139   \resizebox{4in}{!}{   \resizebox{6in}{!}{
140    \includegraphics{part6/s24t_16x16.ps}  % \includegraphics{part6/s24t_16x16.ps}
141      \includegraphics{part6/adjust_cs.ps}
142   }   }
143  \end{center}  \end{center}
144    
145  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain
146  divided into six 32$\times$32 subdomains, each of which is divided  divided into six 32$\times$32 subdomains, each of which is divided
147  into four tiles of width \code{tnx=16} and height \code{tny=16} for a  into eight tiles of width \code{tnx=16} and height \code{tny=8} for a
148  total of twenty-four tiles.  The colored borders of the subdomains  total of forty-eight tiles. The colored borders of the subdomains
149  represent the parameters \code{nr} (red), \code{nb} (blue), and  represent the parameters \code{nr} (red), \code{ng} (green), and
150  \code{ng} (green).  } \label{fig:24tile}  \code{nb} (blue).
151    This tiling is used in the example
152    verification/adjustment.cs-32x32x1/
153    with the option (blanklist.txt) to remove the land-only 4 tiles
154    (11,12,13,14) which are filled in red on the plot.
155    } \label{fig:48tile}
156  \end{figure}  \end{figure}
157    
158  \begin{figure}  \begin{figure}
159  \begin{center}  \begin{center}
160   \resizebox{4in}{!}{   \resizebox{6in}{!}{
161    \includegraphics{part6/s12t_16x32.ps}  % \includegraphics{part6/s12t_16x32.ps}
162      \includegraphics{part6/polarcap.ps}
163   }   }
164  \end{center}  \end{center}
165  \caption{Plot of a cubed sphere topology with a 32$\times$192 domain  \caption{Plot of a non-square cubed sphere topology with
166  divided into six 32$\times$32 subdomains of two tiles each  6 subdomains of different size (nr=90,ng=360,nb=90),
167   (\code{tnx=16, tny=32}).  divided into one to four tiles each
168  } \label{fig:12tile}   (\code{tnx=90, tny=90}), resulting in a total of 18 tiles.
169    } \label{fig:18tile}
170  \end{figure}  \end{figure}
171    
172  \begin{figure}  \begin{figure}
173  \begin{center}  \begin{center}
174   \resizebox{4in}{!}{   \resizebox{4in}{!}{
175    % \includegraphics{part6/s6t_32x32.ps}
176    \includegraphics{part6/s6t_32x32.ps}    \includegraphics{part6/s6t_32x32.ps}
177   }   }
178  \end{center}  \end{center}
# Line 188  by their tile number in the topology, se Line 200  by their tile number in the topology, se
200  Once the topology configuration files are created, the Fortran  Once the topology configuration files are created, the Fortran
201  \code{PARAMETER}s in \file{SIZE.h} must be configured to match.  \code{PARAMETER}s in \file{SIZE.h} must be configured to match.
202  Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying  Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying
203  a decomposition} provides a general description of domain    a decomposition} provides a general description of domain
204  decomposition within MITgcm and its relation to \file{SIZE.h}. The  decomposition within MITgcm and its relation to \file{SIZE.h}. The
205  current section specifies constraints that the exch2 package  current section specifies constraints that the exch2 package imposes
206  imposes and describes how to enable parallel execution with  and describes how to enable parallel execution with MPI.
 MPI. \\  
207    
208  As in the general case, the parameters \varlink{sNx}{sNx} and  As in the general case, the parameters \varlink{sNx}{sNx} and
209  \varlink{sNy}{sNy} define the size of the individual tiles, and so  \varlink{sNy}{sNy} define the size of the individual tiles, and so
210  must be assigned the same respective values as \code{tnx} and  must be assigned the same respective values as \code{tnx} and
211  \code{tny} in \file{driver.m}.\\  \code{tny} in \file{driver.m}.
212    
213  The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}  The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy}
214  have no special bearing on exch2 and may be assigned as in the general  have no special bearing on exch2 and may be assigned as in the general
215  case. The same holds for \varlink{Nr}{Nr}, the number of vertical  case. The same holds for \varlink{Nr}{Nr}, the number of vertical
216  levels in the model.\\  levels in the model.
217    
218  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},  The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy},
219  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of  \varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of
# Line 210  tiles and how they are distributed on pr Line 221  tiles and how they are distributed on pr
221  the tiles are stored in the $x$ dimension, and so  the tiles are stored in the $x$ dimension, and so
222  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as  \code{\varlink{nSy}{nSy}=1} in all cases.  Since the tiles as
223  configured by exch2 cannot be split up accross processors without  configured by exch2 cannot be split up accross processors without
224  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\  regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well.
225    
226  The number of tiles MITgcm allocates and how they are distributed  The number of tiles MITgcm allocates and how they are distributed
227  between processors depends on \varlink{nPx}{nPx} and  between processors depends on \varlink{nPx}{nPx} and
# Line 227  thirty-six tiles that allows you to excl Line 238  thirty-six tiles that allows you to excl
238  distribute the remaining twenty-nine tiles among five processors, you  distribute the remaining twenty-nine tiles among five processors, you
239  would have to run one ``dummy'' tile to make an even six tiles per  would have to run one ``dummy'' tile to make an even six tiles per
240  processor.  Such dummy tiles are \emph{not} listed in  processor.  Such dummy tiles are \emph{not} listed in
241  \file{blanklist.txt}.\\  \file{blanklist.txt}.
242    
243    The following is an example of \file{SIZE.h} for the six-tile
244  The following is an example of \file{SIZE.h} for the twelve-tile  configuration illustrated in figure \ref{fig:6tile}
245  configuration illustrated in figure \ref{fig:12tile} running on  running on one processor:
 one processor: \\  
246    
247  \begin{verbatim}  \begin{verbatim}
248        PARAMETER (        PARAMETER (
249       &           sNx =  16,       &           sNx =  32,
250       &           sNy =  32,       &           sNy =  32,
251       &           OLx =   2,       &           OLx =   2,
252       &           OLy =   2,       &           OLy =   2,
253       &           nSx =  12,       &           nSx =   6,
254       &           nSy =   1,       &           nSy =   1,
255       &           nPx =   1,       &           nPx =   1,
256       &           nPy =   1,       &           nPy =   1,
# Line 249  one processor: \\ Line 259  one processor: \\
259       &           Nr  =   5)       &           Nr  =   5)
260  \end{verbatim}  \end{verbatim}
261    
262  The following is an example for the twenty-four-tile topology in  The following is an example for the forty-eight-tile topology in
263  figure \ref{fig:24tile} running on six processors:  figure \ref{fig:48tile} running on six processors:
264    
265  \begin{verbatim}  \begin{verbatim}
266        PARAMETER (        PARAMETER (
267       &           sNx =  16,       &           sNx =  16,
268       &           sNy =  16,       &           sNy =   8,
269       &           OLx =   2,       &           OLx =   2,
270       &           OLy =   2,       &           OLy =   2,
271       &           nSx =   4,       &           nSx =   8,
272       &           nSy =   1,       &           nSy =   1,
273       &           nPx =   6,       &           nPx =   6,
274       &           nPy =   1,       &           nPy =   1,
# Line 268  figure \ref{fig:24tile} running on six p Line 278  figure \ref{fig:24tile} running on six p
278  \end{verbatim}  \end{verbatim}
279    
280    
   
   
   
281  \subsubsection{Key Variables}  \subsubsection{Key Variables}
282    
283  The descriptions of the variables are divided up into scalars,  The descriptions of the variables are divided up into scalars,
# Line 295  and \varlink{exch2\_domain\_nyt}{exch2_d Line 302  and \varlink{exch2\_domain\_nyt}{exch2_d
302  of tiles in the $x$ and $y$ global indices.  For example, the default  of tiles in the $x$ and $y$ global indices.  For example, the default
303  setup of six tiles (Fig. \ref{fig:6tile}) has  setup of six tiles (Fig. \ref{fig:6tile}) has
304  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A  \code{exch2\_domain\_nxt=6} and \code{exch2\_domain\_nyt=1}.  A
305  topology of twenty-four square tiles, four per subdomain (as in figure  topology of forty-eight tiles, eight per subdomain (as in figure
306  \ref{fig:24tile}), will have \code{exch2\_domain\_nxt=12} and  \ref{fig:48tile}), will have \code{exch2\_domain\_nxt=12} and
307  \code{exch2\_domain\_nyt=2}.  Note that these parameters express the  \code{exch2\_domain\_nyt=4}.  Note that these parameters express the
308  tile layout in order to allow global data files that are tile-layout-neutral.  tile layout in order to allow global data files that are tile-layout-neutral.
309  They have no bearing on the internal storage of the arrays.  The tiles  They have no bearing on the internal storage of the arrays.  The tiles
310  are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the  are stored internally in a range from \code{\varlink{bi}{bi}=(1:NTILES)} in the
# Line 337  files. \\ Line 344  files. \\
344  The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of  The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of
345  the subdomain of each tile, in a range \code{(1:6)} in the case of the  the subdomain of each tile, in a range \code{(1:6)} in the case of the
346  standard cube topology and indicated by \textbf{\textsf{f}}$n$ in  standard cube topology and indicated by \textbf{\textsf{f}}$n$ in
347  figures \ref{fig:12tile} and  figures \ref{fig:6tile} and
348  \ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}  \ref{fig:48tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours}
349  contains a count of the neighboring tiles each tile has, and sets  contains a count of the neighboring tiles each tile has, and sets
350  the bounds for looping over neighboring tiles.  the bounds for looping over neighboring tiles.
351  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each  \varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each
# Line 460  Finally, \varlink{exch2\_itlo\_c}{exch2_ Line 467  Finally, \varlink{exch2\_itlo\_c}{exch2_
467  \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index  \varlink{exch2\_jthi\_c}{exch2_jthi_c} hold the location and index
468  bounds of the edge segment of the neighbor tile \code{N}'s subdomain  bounds of the edge segment of the neighbor tile \code{N}'s subdomain
469  that gets exchanged with the local tile \code{T}.  To take the example  that gets exchanged with the local tile \code{T}.  To take the example
470  of tile \code{T=2} in the twelve-tile topology  of tile \code{T=2} in the forty-eight-tile topology
471  (Fig. \ref{fig:12tile}): \\  (Fig. \ref{fig:48tile}): \\
472    
473  \begin{verbatim}  \begin{verbatim}
474         exch2_itlo_c(4,2)=17         exch2_itlo_c(4,2)=17
# Line 531  issue with the scalar case. These subrou Line 538  issue with the scalar case. These subrou
538  \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of  \code{EXCH2\_SEND\_RX1} and \code{EXCH2\_SEND\_RX2}, which do most of
539  the work using the variables discussed above. \\  the work using the variables discussed above. \\
540    
541    \subsubsection{Experiments and tutorials that use exch2}
542    \label{sec:pkg:exch2:experiments}
543    
544    \begin{itemize}
545    \item{Held Suarez tutorial, in tutorial\_held\_suarez\_cs verification directory,
546    described in section \ref{sect:eg-hs} }
547    \end{itemize}
Legend:
Removed from v.1.24  
changed lines
  Added in v.1.27
  ViewVC Help
Powered by ViewVC 1.1.22