1 |
% $Header: /u/u3/gcmpack/manual/part6/exch2.tex,v 1.4 2004/01/29 17:55:35 afe Exp $ |
2 |
% $Name: $ |
3 |
|
4 |
%% * Introduction |
5 |
%% o what it does, citations (refs go into mitgcm_manual.bib, |
6 |
%% preferably in alphabetic order) |
7 |
%% o Equations |
8 |
%% * Key subroutines and parameters |
9 |
%% * Reference material (auto generated from Protex and structured comments) |
10 |
%% o automatically inserted at \section{Reference} |
11 |
|
12 |
|
13 |
\section{exch2: Extended Cubed Sphere Exchange} |
14 |
\label{sec:exch2} |
15 |
|
16 |
|
17 |
\subsection{Introduction} |
18 |
|
19 |
The exch2 package is an extension to the original cubed sphere exchanges |
20 |
to allow more flexible domain decomposition and parallelization. Cube faces |
21 |
(subdomains) may be divided into whatever number of tiles that divide evenly |
22 |
into the grid point dimensions of the subdomain. Furthermore, the individual |
23 |
tiles may be run on separate processors in different combinations, |
24 |
and whether exchanges between particular tiles occur between different |
25 |
processors is determined at runtime. |
26 |
|
27 |
The exchange parameters are declared in {\em W2\_EXCH2\_TOPOLOGY.h} and |
28 |
assigned in {\em w2\_e2setup.F}, both in the |
29 |
{\em pkg/exch2} directory. The validity of the cube topology depends |
30 |
on the {\em SIZE.h} file as detailed below. Both files are generated by |
31 |
Matlab scripts and |
32 |
should not be edited. The default files provided in the release set up |
33 |
a cube sphere arrangement of six tiles, one per subdomain, each with 32x32 grid |
34 |
points, running on a single processor. |
35 |
|
36 |
\subsection{Key Variables} |
37 |
|
38 |
The descriptions of the variables are divided up into scalars, |
39 |
one-dimensional arrays indexed to the tile number, and two-dimensional |
40 |
arrays indexed to tile number and neighboring tile. This division |
41 |
actually reflects the functionality of these variables, not just the |
42 |
whim of some FORTRAN enthusiast. |
43 |
|
44 |
\subsubsection{Scalars} |
45 |
|
46 |
The number of tiles in a particular topology is set with the parameter |
47 |
{\em NTILES}, and the maximum number of neighbors of any tiles by |
48 |
{\em MAX\_NEIGHBOURS}. These parameters are used for defining the size of |
49 |
the various one and two dimensional arrays that store tile parameters |
50 |
indexed to the tile number. |
51 |
|
52 |
The scalar parameters {\em exch2\_domain\_nxt} and |
53 |
{\em exch2\_domain\_nyt} express the number of tiles in the x and y global |
54 |
indices. For example, the default setup of six tiles has |
55 |
{\em exch2\_domain\_nxt=6} and {\em exch2\_domain\_nyt=1}. A topology of |
56 |
twenty-four square (in gridpoints) tiles, four (2x2) per subdomain, will |
57 |
have {\em exch2\_domain\_nxt=12} and {\em exch2\_domain\_nyt=2}. Note |
58 |
that these parameters express the tile layout to allow global data files that |
59 |
are tile-layout-neutral and have no bearing on the internal storage of the |
60 |
arrays. The tiles are internally stored in a range from {\em 1,bi} (in the |
61 |
x axis) and y-axis variable {\em bj} is generally ignored within the package. |
62 |
|
63 |
\subsubsection{One-Dimensional Arrays} |
64 |
|
65 |
The following arrays are of size {\em NTILES}, are indexed to the tile number, |
66 |
and the indices are omitted in their descriptions. |
67 |
|
68 |
The arrays {\em exch2\_tnx} and {\em exch2\_tny} |
69 |
express the x and y dimensions of each tile. At present for each tile |
70 |
{\em exch2\_tnx = sNx} |
71 |
and {\em exch2\_tny = sNy}, as assigned in {\em SIZE.h}. Future releases of |
72 |
MITgcm are to allow varying tile sizes. |
73 |
|
74 |
The location of the tiles' Cartesian origin within a subdomain are determined |
75 |
by the arrays {\em exch2\_tbasex} and {\em exch2\_tbasey}. These variables |
76 |
are used to relate the location of the edges of the tiles to each other. As |
77 |
an example, in the default six-tile topology (the degenerate case) |
78 |
each index in these arrays are |
79 |
set to 0. The twenty-four, 32x32 cube face case discussed above will have |
80 |
values of 0 or 16, depending on the quadrant the tile falls within the |
81 |
subdomain. {\em exch2\_myFace} contains the number of the |
82 |
cubeface/subdomain of each tile, numbered 1-6 in the case of the standard |
83 |
cube topology. |
84 |
|
85 |
The arrays {\em exch2\_txglobalo} and {\em exch2\_txglobalo} are similar to |
86 |
{\em exch2\_tbasex} and {\em exch2\_tbasey}, but locate the tiles within |
87 |
the global address space, similar to that used by global files. |
88 |
|
89 |
The arrays {\em exch2\_isWedge}, {\em exch2\_isEedge}, {\em exch2\_isSedge}, |
90 |
and {\em exch2\_isNedge} are set to 1 if the indexed tile lies on the edge |
91 |
of a subdomain, 0 if not. The values are used within the topology generator |
92 |
to determine the orientation of neighboring tiles and to indicate whether |
93 |
a tile lies on the corner of a subdomain. The latter case indicates |
94 |
special exchange and numerical handling for the singularities at the eight |
95 |
corners of the cube. {\em exch2\_isNedge} contains a count of how many |
96 |
neighboring tiles each tile has, and is used for setting bounds for looping |
97 |
over neighboring tiles. {\em exch2\_tProc} holds the process rank of each tile, |
98 |
and is used in interprocess communication. |
99 |
|
100 |
\subsubsection{Two-Dimensional Arrays} |
101 |
|
102 |
The following arrays are all of size {\em MAX\_NEIGHBOURS}x{\em NTILES} and |
103 |
describe the orientations between the the tiles. |
104 |
|
105 |
The array {\em exch2\_neighbourId(a,T)} holds the tile number $T_{n}$ for each tile |
106 |
{\em T}'s neighbor tile {\em a}, and {\em exch2\_opposingSend\_record(a,T)} holds |
107 |
the index c in {\em exch2\_neighbourId(b,$T_{n}$)} that holds the tile number T. |
108 |
In other words, |
109 |
|
110 |
\begin{verbatim} |
111 |
exch2_neighbourId( exch2_opposingSend_record(a,T), exch2_neighbourId(a,T) ) = T |
112 |
\end{verbatim} |
113 |
|
114 |
{\em exch2\_neighbourId(exch2\_opposingSend\_record(a,T),exch2\_neighbourId(a,T))=T}. |
115 |
This is to provide a backreference from the neighbor tiles. |
116 |
|
117 |
|
118 |
// |
119 |
|
120 |
\begin{verbatim} |
121 |
|
122 |
|
123 |
C exch2_neighbourId :: Tile number for each neighbour entry. |
124 |
C exch2_opposingSend_record :: Record for entry in target tile send |
125 |
C :: list that has this tile and face |
126 |
C :: as its target. |
127 |
C exch2_pi :: X index row of target to source permutation |
128 |
C :: matrix for each neighbour entry. |
129 |
C exch2_pj :: Y index row of target to source permutation |
130 |
C :: matrix for each neighbour entry. |
131 |
C exch2_oi :: X index element of target to source |
132 |
C :: offset vector for cell-centered quantities |
133 |
C :: of each neighbor entry. |
134 |
C exch2_oj :: Y index element of target to source |
135 |
C :: offset vector for cell-centered quantities |
136 |
C :: of each neighbor entry. |
137 |
C exch2_oi_f :: X index element of target to source |
138 |
C :: offset vector for face quantities |
139 |
C :: of each neighbor entry. |
140 |
C exch2_oj_f :: Y index element of target to source |
141 |
C :: offset vector for face quantities |
142 |
C :: of each neighbor entry. |
143 |
\end{verbatim} |
144 |
|
145 |
|
146 |
|
147 |
|
148 |
\subsection{Key Routines} |
149 |
|
150 |
|
151 |
|
152 |
\subsection{References} |