1 |
% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.12 2004/03/16 21:52:15 afe Exp $ |
2 |
% $Name: $ |
3 |
|
4 |
%% * Introduction |
5 |
%% o what it does, citations (refs go into mitgcm_manual.bib, |
6 |
%% preferably in alphabetic order) |
7 |
%% o Equations |
8 |
%% * Key subroutines and parameters |
9 |
%% * Reference material (auto generated from Protex and structured comments) |
10 |
%% o automatically inserted at \section{Reference} |
11 |
|
12 |
|
13 |
\section{exch2: Extended Cubed Sphere \mbox{Topology}} |
14 |
\label{sec:exch2} |
15 |
|
16 |
|
17 |
\subsection{Introduction} |
18 |
|
19 |
The \texttt{exch2} package extends the original cubed |
20 |
sphere topology configuration to allow more flexible domain |
21 |
decomposition and parallelization. Cube faces (also called |
22 |
subdomains) may be divided into any number of tiles that divide evenly |
23 |
into the grid point dimensions of the subdomain. Furthermore, the |
24 |
individual tiles can run on separate processors in different |
25 |
combinations, and whether exchanges between particular tiles occur |
26 |
between different processors is determined at runtime. This |
27 |
flexibility provides for manual compile-time load balancing across a |
28 |
relatively arbitrary number of processors. \\ |
29 |
|
30 |
The exchange parameters are declared in |
31 |
\filelink{pkg/exch2/W2\_EXCH2\_TOPOLOGY.h}{pkg-exch2-W2_EXCH2_TOPOLOGY.h} |
32 |
and assigned in |
33 |
\filelink{pkg/exch2/w2\_e2setup.F}{pkg-exch2-w2_e2setup.F}. The |
34 |
validity of the cube topology depends on the \file{SIZE.h} file as |
35 |
detailed below. The default files provided in the release configure a |
36 |
cubed sphere topology of six tiles, one per subdomain, each with |
37 |
32$\times$32 grid points, all running on a single processor. Both |
38 |
files are generated by Matlab scripts in |
39 |
\file{utils/exch2/matlab-topology-generator}; see Section |
40 |
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} |
41 |
for details on creating alternate topologies. Pregenerated examples |
42 |
of these files with alternate topologies are provided under |
43 |
\file{utils/exch2/code-mods} along with the appropriate \file{SIZE.h} |
44 |
file for single-processor execution. |
45 |
|
46 |
\subsection{Invoking exch2} |
47 |
|
48 |
To use exch2 with the cubed sphere, the following conditions must be |
49 |
met: \\ |
50 |
|
51 |
$\bullet$ The exch2 package is included when \file{genmake2} is run. |
52 |
The easiest way to do this is to add the line \code{exch2} to the |
53 |
\file{profile.conf} file -- see Section |
54 |
\ref{sect:buildingCode} \sectiontitle{Building the code} for general |
55 |
details. \\ |
56 |
|
57 |
$\bullet$ An example of \file{W2\_EXCH2\_TOPOLOGY.h} and |
58 |
\file{w2\_e2setup.F} must reside in a directory containing code |
59 |
linked when \file{genmake2} runs. The safest place to put these |
60 |
is the directory indicated in the \code{-mods=DIR} command line |
61 |
modifier (typically \file{../code}), or the build directory. The |
62 |
default versions of these files reside in \file{pkg/exch2} and are |
63 |
linked automatically if no other versions exist elsewhere in the |
64 |
link path, but they should be left untouched to avoid breaking |
65 |
configurations other than the one you intend to modify.\\ |
66 |
|
67 |
$\bullet$ Files containing grid parameters, named |
68 |
\file{tile00$n$.mitgrid} where $n$=\code{(1:6)} (one per subdomain), |
69 |
must be in the working directory when the MITgcm executable is run. |
70 |
These files are provided in the example experiments for cubed sphere |
71 |
configurations with 32$\times$32 cube sides and are non-trivial to |
72 |
generate -- please contact MITgcm support if you want to generate |
73 |
files for other configurations. \\ |
74 |
|
75 |
$\bullet$ As always when compiling MITgcm, the file \file{SIZE.h} must |
76 |
be placed where \file{genmake2} will find it. In particular for |
77 |
exch2, the domain decomposition specified in \file{SIZE.h} must |
78 |
correspond with the particular configuration's topology specified in |
79 |
\file{W2\_EXCH2\_TOPOLOGY.h} and \file{w2\_e2setup.F}. Domain |
80 |
decomposition issues particular to exch2 are addressed in Section |
81 |
\ref{sec:topogen} \sectiontitle{Generating Topology Files for exch2} |
82 |
and \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and MPI}; a more |
83 |
general background on the subject relevant to MITgcm is presented in |
84 |
Section \ref{sect:specifying_a_decomposition} |
85 |
\sectiontitle{Specifying a decomposition}.\\ |
86 |
|
87 |
As of the time of writing the following examples use exch2 and may be |
88 |
used for guidance: |
89 |
|
90 |
\begin{verbatim} |
91 |
verification/adjust_nlfs.cs-32x32x1 |
92 |
verification/adjustment.cs-32x32x1 |
93 |
verification/aim.5l_cs |
94 |
verification/global_ocean.cs32x15 |
95 |
verification/hs94.cs-32x32x5 |
96 |
\end{verbatim} |
97 |
|
98 |
|
99 |
|
100 |
|
101 |
\subsection{Generating Topology Files for exch2} |
102 |
\label{sec:topogen} |
103 |
|
104 |
Alternate cubed sphere topologies may be created using the Matlab |
105 |
scripts in \file{utils/exch2/matlab-topology-generator}. Running the |
106 |
m-file |
107 |
\filelink{driver.m}{utils-exch2-matlab-topology-generator_driver.m} |
108 |
from the Matlab prompt (there are no parameters to pass) generates |
109 |
exch2 topology files \file{W2\_EXCH2\_TOPOLOGY.h} and |
110 |
\file{w2\_e2setup.F} in the working directory and displays a figure of |
111 |
the topology via Matlab. The other m-files in the directory are |
112 |
subroutines of \file{driver.m} and should not be run ``bare'' except |
113 |
for development purposes. \\ |
114 |
|
115 |
The parameters that determine the dimensions and topology of the |
116 |
generated configuration are \code{nr}, \code{nb}, \code{ng}, |
117 |
\code{tnx} and \code{tny}, and all are assigned early in the script. \\ |
118 |
|
119 |
The first three determine the size of the subdomains and |
120 |
hence the size of the overall domain. Each one determines the number |
121 |
of grid points, and therefore the resolution, along the subdomain |
122 |
sides in a ``great circle'' around an axis of the cube. At the time |
123 |
of this writing MITgcm requires these three parameters to be equal, |
124 |
but they provide for future releases to accomodate different |
125 |
resolutions around the axes to allow (for example) greater resolution |
126 |
around the equator.\\ |
127 |
|
128 |
The parameters \code{tnx} and \code{tny} determine the dimensions of |
129 |
the tiles into which the subdomains are decomposed, and must evenly |
130 |
divide the integer assigned to \code{nr}, \code{nb} and \code{ng}. |
131 |
The result is a rectangular tiling of the subdomain. Figure |
132 |
\ref{fig:24tile} shows one possible topology for a twentyfour-tile |
133 |
cube, and figure \ref{fig:12tile} shows one for twelve tiles. \\ |
134 |
|
135 |
\begin{figure} |
136 |
\begin{center} |
137 |
\resizebox{4in}{!}{ |
138 |
\includegraphics{part6/s24t_16x16.ps} |
139 |
} |
140 |
\end{center} |
141 |
|
142 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
143 |
divided into six 32$\times$32 subdomains, each of which is divided into four tiles |
144 |
(\code{tnx=16, tny=16}) for a total of twentyfour tiles. |
145 |
} \label{fig:24tile} |
146 |
\end{figure} |
147 |
|
148 |
\begin{figure} |
149 |
\begin{center} |
150 |
\resizebox{4in}{!}{ |
151 |
\includegraphics{part6/s12t_16x32.ps} |
152 |
} |
153 |
\end{center} |
154 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
155 |
divided into six 32$\times$32 subdomains of two tiles each |
156 |
(\code{tnx=16, tny=32}). |
157 |
} \label{fig:12tile} |
158 |
\end{figure} |
159 |
|
160 |
\begin{figure} |
161 |
\begin{center} |
162 |
\resizebox{4in}{!}{ |
163 |
\includegraphics{part6/s6t_32x32.ps} |
164 |
} |
165 |
\end{center} |
166 |
\caption{Plot of a cubed sphere topology with a 32$\times$192 domain |
167 |
divided into six 32$\times$32 subdomains with one tile each |
168 |
(\code{tnx=32, tny=32}). This is the default configuration. |
169 |
} |
170 |
\label{fig:6tile} |
171 |
\end{figure} |
172 |
|
173 |
|
174 |
Tiles can be selected from the topology to be omitted from being |
175 |
allocated memory and processors. This tuning is useful in ocean |
176 |
modeling for omitting tiles that fall entirely on land. The tiles |
177 |
omitted are specified in the file |
178 |
\filelink{blanklist.txt}{utils-exch2-matlab-topology-generator_blanklist.txt} |
179 |
by their tile number in the topology, separated by a newline. \\ |
180 |
|
181 |
|
182 |
|
183 |
|
184 |
\subsection{exch2, SIZE.h, and multiprocessing} |
185 |
\label{sec:exch2mpi} |
186 |
|
187 |
Once the topology configuration files are created, the Fortran |
188 |
\code{PARAMETER}s in \file{SIZE.h} must be configured to match. |
189 |
Section \ref{sect:specifying_a_decomposition} \sectiontitle{Specifying |
190 |
a decomposition} provides a general description of domain |
191 |
decomposition within MITgcm and its relation to \file{SIZE.h}. The |
192 |
current section specifies certain constraints the exch2 package |
193 |
imposes as well as describes how to enable parallel execution with |
194 |
MPI. \\ |
195 |
|
196 |
As in the general case, the parameters \varlink{sNx}{sNx} and |
197 |
\varlink{sNy}{sNy} define the size of the individual tiles, and so |
198 |
must be assigned the same respective values as \code{tnx} and |
199 |
\code{tny} in \file{driver.m}.\\ |
200 |
|
201 |
The halo width parameters \varlink{OLx}{OLx} and \varlink{OLy}{OLy} |
202 |
have no special bearing on exch2 and may be assigned as in the general |
203 |
case. The same holds for \varlink{Nr}{Nr}, the number of vertical |
204 |
levels in the model.\\ |
205 |
|
206 |
The parameters \varlink{nSx}{nSx}, \varlink{nSy}{nSy}, |
207 |
\varlink{nPx}{nPx}, and \varlink{nPy}{nPy} relate to the number of |
208 |
tiles and how they are distributed on processors. When using exch2, |
209 |
the tiles are stored in single dimension, and so |
210 |
\code{\varlink{nSy}{nSy}=1} in all cases. Since the tiles as |
211 |
configured by exch2 cannot be split up accross processors without |
212 |
regenerating the topology, \code{\varlink{nPy}{nPy}=1} as well. \\ |
213 |
|
214 |
The number of tiles MITgcm allocates and how they are distributed |
215 |
between processors depends on \varlink{nPx}{nPx} and |
216 |
\varlink{nSx}{nSx}. \varlink{nSx}{nSx} is the number of tiles per |
217 |
processor and \varlink{nPx}{nPx} the number of processors. The total |
218 |
number of tiles in the topology minus those listed in |
219 |
\file{blanklist.txt} must equal \code{nSx*nPx}. \\ |
220 |
|
221 |
The following is an example of \file{SIZE.h} for the twelve-tile |
222 |
configuration illustrated in figure \ref{fig:12tile} running on |
223 |
one processor: \\ |
224 |
|
225 |
\begin{verbatim} |
226 |
PARAMETER ( |
227 |
& sNx = 16, |
228 |
& sNy = 32, |
229 |
& OLx = 2, |
230 |
& OLy = 2, |
231 |
& nSx = 12, |
232 |
& nSy = 1, |
233 |
& nPx = 1, |
234 |
& nPy = 1, |
235 |
& Nx = sNx*nSx*nPx, |
236 |
& Ny = sNy*nSy*nPy, |
237 |
& Nr = 5) |
238 |
\end{verbatim} |
239 |
|
240 |
The following is an example for the twentyfour-tile topology in figure |
241 |
\ref{fig:24tile} running on six processors: |
242 |
|
243 |
\begin{verbatim} |
244 |
PARAMETER ( |
245 |
& sNx = 16, |
246 |
& sNy = 16, |
247 |
& OLx = 2, |
248 |
& OLy = 2, |
249 |
& nSx = 4, |
250 |
& nSy = 1, |
251 |
& nPx = 6, |
252 |
& nPy = 1, |
253 |
& Nx = sNx*nSx*nPx, |
254 |
& Ny = sNy*nSy*nPy, |
255 |
& Nr = 5) |
256 |
\end{verbatim} |
257 |
|
258 |
|
259 |
|
260 |
|
261 |
|
262 |
\subsection{Key Variables} |
263 |
|
264 |
The descriptions of the variables are divided up into scalars, |
265 |
one-dimensional arrays indexed to the tile number, and two and three |
266 |
dimensional arrays indexed to tile number and neighboring tile. This |
267 |
division reflects the functionality of these variables: The |
268 |
scalars are common to every part of the topology, the tile-indexed |
269 |
arrays to individual tiles, and the arrays indexed by tile and |
270 |
neighbor to relationships between tiles and their neighbors. \\ |
271 |
|
272 |
\subsubsection{Scalars} |
273 |
|
274 |
The number of tiles in a particular topology is set with the parameter |
275 |
\code{NTILES}, and the maximum number of neighbors of any tiles by |
276 |
\code{MAX\_NEIGHBOURS}. These parameters are used for defining the |
277 |
size of the various one and two dimensional arrays that store tile |
278 |
parameters indexed to the tile number and are assigned in the files |
279 |
generated by \file{driver.m}.\\ |
280 |
|
281 |
The scalar parameters \varlink{exch2\_domain\_nxt}{exch2_domain_nxt} |
282 |
and \varlink{exch2\_domain\_nyt}{exch2_domain_nyt} express the number |
283 |
of tiles in the $x$ and $y$ global indices. For example, the default |
284 |
setup of six tiles (Fig. \ref{fig:6tile}) has \code{exch2\_domain\_nxt=6} and |
285 |
\code{exch2\_domain\_nyt=1}. A topology of twenty-four square tiles, |
286 |
four per subdomain (as in figure \ref{fig:24tile}), will have |
287 |
\code{exch2\_domain\_nxt=12} and \code{exch2\_domain\_nyt=2}. Note |
288 |
that these parameters express the tile layout to allow global data |
289 |
files that are tile-layout-neutral and have no bearing on the internal |
290 |
storage of the arrays. The tiles are internally stored in a range |
291 |
from \code{(1:\varlink{bi}{bi})} the $x$ axis, and $y$ axis variable |
292 |
\varlink{bj}{bj} is generally ignored within the package. \\ |
293 |
|
294 |
\subsubsection{Arrays Indexed to Tile Number} |
295 |
|
296 |
The following arrays are of length \code{NTILES}, are indexed to the |
297 |
tile number, and the indices are omitted in their descriptions. \\ |
298 |
|
299 |
The arrays \varlink{exch2\_tnx}{exch2_tnx} and |
300 |
\varlink{exch2\_tny}{exch2_tny} express the $x$ and $y$ dimensions of |
301 |
each tile. At present for each tile \texttt{exch2\_tnx=sNx} and |
302 |
\texttt{exch2\_tny=sNy}, as assigned in \file{SIZE.h} and described in |
303 |
section \ref{sec:exch2mpi} \sectiontitle{exch2, SIZE.h, and |
304 |
multiprocessing}. Future releases of MITgcm are to allow varying tile |
305 |
sizes. \\ |
306 |
|
307 |
The location of the tiles' Cartesian origin within a subdomain are |
308 |
determined by the arrays \varlink{exch2\_tbasex}{exch2_tbasex} and |
309 |
\varlink{exch2\_tbasey}{exch2_tbasey}. These variables are used to |
310 |
relate the location of the edges of different tiles to each other. As |
311 |
an example, in the default six-tile topology (Fig. \ref{fig:6tile}) |
312 |
each index in these arrays is set to \code{0} since a tile occupies |
313 |
its entire subdomain. The twentyfour-tile case discussed above will |
314 |
have values of \code{0} or \code{16}, depending on the quadrant the |
315 |
tile falls within the subdomain. The elements of the arrays |
316 |
\varlink{exch2\_txglobalo}{exch2_txglobalo} and |
317 |
\varlink{exch2\_txglobalo}{exch2_txglobalo} are similar to |
318 |
\varlink{exch2\_tbasex}{exch2_tbasex} and |
319 |
\varlink{exch2\_tbasey}{exch2_tbasey}, but locate the tiles within the |
320 |
global address space, similar to that used by global files. \\ |
321 |
|
322 |
The array \varlink{exch2\_myFace}{exch2_myFace} contains the number of |
323 |
the subdomain of each tile, in a range \code{(1:6)} in the case of the |
324 |
standard cube topology and indicated by \textbf{\textsf{f}}$n$ in |
325 |
figures \ref{fig:12tile} and |
326 |
\ref{fig:24tile}. \varlink{exch2\_nNeighbours}{exch2_nNeighbours} |
327 |
contains a count of how many neighboring tiles each tile has, and is |
328 |
used for setting bounds for looping over neighboring tiles. |
329 |
\varlink{exch2\_tProc}{exch2_tProc} holds the process rank of each |
330 |
tile, and is used in interprocess communication. \\ |
331 |
|
332 |
|
333 |
The arrays \varlink{exch2\_isWedge}{exch2_isWedge}, |
334 |
\varlink{exch2\_isEedge}{exch2_isEedge}, |
335 |
\varlink{exch2\_isSedge}{exch2_isSedge}, and |
336 |
\varlink{exch2\_isNedge}{exch2_isNedge} are set to \code{1} if the |
337 |
indexed tile lies on the edge of a subdomain, \code{0} if not. The |
338 |
values are used within the topology generator to determine the |
339 |
orientation of neighboring tiles, and to indicate whether a tile lies |
340 |
on the corner of a subdomain. The latter case requires special |
341 |
exchange and numerical handling for the singularities at the eight |
342 |
corners of the cube. \\ |
343 |
|
344 |
|
345 |
\subsubsection{Arrays Indexed to Tile Number and Neighbor} |
346 |
|
347 |
The following arrays are all of size |
348 |
\code{MAX\_NEIGHBOURS}$\times$\code{NTILES} and describe the |
349 |
orientations between the the tiles. \\ |
350 |
|
351 |
The array \code{exch2\_neighbourId(a,T)} holds the tile number |
352 |
\code{Tn} for each of the tile number \code{T}'s neighboring tiles |
353 |
\code{a}. The neighbor tiles are indexed \code{(1:MAX\_NEIGHBOURS)} |
354 |
in the order right to left on the north then south edges, and then top |
355 |
to bottom on the east and west edges. Maybe throw in a fig here, eh? |
356 |
\\ |
357 |
|
358 |
\sloppy |
359 |
The \code{exch2\_opposingSend\_record(a,T)} array holds the index |
360 |
\code{b} in \texttt{exch2\_neighbourId(b,Tn)} that holds the tile |
361 |
number \code{T}. In other words, |
362 |
\begin{verbatim} |
363 |
exch2_neighbourId( exch2_opposingSend_record(a,T), |
364 |
exch2_neighbourId(a,T) ) = T |
365 |
\end{verbatim} |
366 |
This provides a back-reference from the neighbor tiles. \\ |
367 |
|
368 |
The arrays \varlink{exch2\_pi}{exch2_pi} and |
369 |
\varlink{exch2\_pj}{exch2_pj} specify the transformations of variables |
370 |
in exchanges between the neighboring tiles. These transformations are |
371 |
necessary in exchanges between subdomains because a physical vector |
372 |
component in one direction may map to one in a different direction in |
373 |
an adjacent subdomain, and may be have its indexing reversed. This |
374 |
swapping arises from the ``folding'' of two-dimensional arrays into a |
375 |
three-dimensional cube. |
376 |
|
377 |
The dimensions of \code{exch2\_pi(t,N,T)} and \code{exch2\_pj(t,N,T)} |
378 |
are the neighbor ID \code{N} and the tile number \code{T} as explained |
379 |
above, plus a vector of length 2 containing transformation factors |
380 |
\code{t}. The first element of the transformation vector indicates |
381 |
the factor \code{t} by which variables representing the same |
382 |
\emph{physical} vector component of a tile \code{T} will be multiplied |
383 |
in exchanges with neighbor \code{N}, and the second element indicates |
384 |
the transform to the physical vector in the other direction. To |
385 |
clarify (hopefully), \code{exch2\_pi(1,N,T)} holds the transform of |
386 |
the $i$ component of a vector variable in tile \code{T} to the $i$ |
387 |
component of tile \code{T}'s neighbor \code{N}, and |
388 |
\code{exch2\_pi(2,N,T)} holds the transform of \code{T}'s $i$ |
389 |
components to the neighbor \code{N}'s $j$ component. \\ |
390 |
|
391 |
Under the current cube topology, one of the two elements of |
392 |
\code{exch2\_pi} or \code{exch2\_pj} for a given tile \code{T} and |
393 |
neighbor \code{N} will be \code{0}, reflecting the fact that the two |
394 |
vector components are orthogonal. The other element will be \code{1} |
395 |
or \code{-1}, depending on whether the components are indexed in the |
396 |
same or opposite directions. For example, the transform vector of the |
397 |
arrays for all tile neighbors on the same subdomain will be |
398 |
\code{(1,0)}, since all tiles on the same subdomain are oriented |
399 |
identically. A vector direction that corresponds to the orthogonal |
400 |
dimension with the same index direction in a particular tile-neighbor |
401 |
orientation will have \code{(0,1)}, whereas those in the opposite |
402 |
index direction will have \code{(0,-1)}. \\ |
403 |
|
404 |
|
405 |
\varlink{exch2\_oi}{exch2_oi}, |
406 |
\varlink{exch2\_oj}{exch2_oj}, \varlink{exch2\_oi\_f}{exch2_oi_f}, and |
407 |
\varlink{exch2\_oj\_f}{exch2_oj_f} |
408 |
|
409 |
|
410 |
|
411 |
|
412 |
This needs some diagrams. \\ |
413 |
|
414 |
|
415 |
{\footnotesize |
416 |
\begin{verbatim} |
417 |
C exch2_pi :: X index row of target to source permutation |
418 |
C :: matrix for each neighbour entry. |
419 |
C exch2_pj :: Y index row of target to source permutation |
420 |
C :: matrix for each neighbour entry. |
421 |
C exch2_oi :: X index element of target to source |
422 |
C :: offset vector for cell-centered quantities |
423 |
C :: of each neighbor entry. |
424 |
C exch2_oj :: Y index element of target to source |
425 |
C :: offset vector for cell-centered quantities |
426 |
C :: of each neighbor entry. |
427 |
C exch2_oi_f :: X index element of target to source |
428 |
C :: offset vector for face quantities |
429 |
C :: of each neighbor entry. |
430 |
C exch2_oj_f :: Y index element of target to source |
431 |
C :: offset vector for face quantities |
432 |
C :: of each neighbor entry. |
433 |
\end{verbatim} |
434 |
} |
435 |
|
436 |
|
437 |
|
438 |
\subsection{Key Routines} |
439 |
|
440 |
|
441 |
|
442 |
\subsection{References} |