ViewVC logotype

Annotation of /MITgcm/doc/notes

Parent Directory Parent Directory | Revision Log Revision Log | View Revision Graph Revision Graph

Revision 1.3 - (hide annotations) (download)
Fri Apr 24 03:12:35 1998 UTC (25 years, 7 months ago) by cnh
Branch: MAIN
CVS Tags: checkpoint46n_post, checkpoint51k_post, checkpoint62v, checkpoint47e_post, checkpoint57m_post, checkpoint52l_pre, checkpoint44e_post, checkpoint62u, hrcube4, hrcube5, checkpoint46l_post, checkpoint57g_pre, checkpoint46g_pre, checkpoint47c_post, release1_p13_pre, checkpoint62t, checkpoint50c_post, checkpoint57s_post, checkpoint58b_post, checkpoint57b_post, checkpoint46f_post, checkpoint52d_pre, checkpoint57g_post, checkpoint48e_post, checkpoint56b_post, checkpoint50c_pre, checkpoint57y_post, checkpoint44f_post, checkpoint46b_post, checkpoint52j_pre, checkpoint43a-release1mods, checkpoint51o_pre, checkpoint65z, checkpoint65x, checkpoint65y, checkpoint54d_post, checkpoint65r, checkpoint65s, checkpoint65p, checkpoint65q, checkpoint65v, checkpoint65w, checkpoint65t, checkpoint65u, checkpoint65j, checkpoint65k, checkpoint65h, checkpoint65i, checkpoint65n, checkpoint54e_post, checkpoint65l, checkpoint65m, checkpoint65b, checkpoint65c, checkpoint65a, checkpoint65f, checkpoint65g, checkpoint65d, checkpoint65e, ecco_c50_e32, ecco_c50_e33, ecco_c50_e30, ecco_c50_e31, release1_p13, checkpoint40pre3, checkpoint62c, checkpoint40pre1, checkpoint51l_post, checkpoint40pre7, checkpoint40pre6, checkpoint48i_post, checkpoint57r_post, checkpoint40pre9, checkpoint40pre8, checkpoint46l_pre, checkpoint57d_post, checkpoint57i_post, checkpoint52l_post, checkpoint52k_post, chkpt44d_post, checkpoint59, checkpoint58, checkpoint55, checkpoint54, checkpoint57, checkpoint56, checkpoint51, checkpoint50, checkpoint53, checkpoint52, release1_p8, release1_p9, checkpoint50d_post, release1_p1, release1_p2, release1_p3, release1_p4, release1_p5, release1_p6, release1_p7, checkpoint58f_post, checkpoint52f_post, checkpoint57n_post, checkpoint58d_post, checkpoint62s, checkpoint58a_post, checkpoint62r, checkpoint62q, checkpoint50b_pre, checkpoint44e_pre, checkpoint62p, checkpoint57z_post, checkpoint54f_post, checkpoint51f_post, checkpoint62a, release1_b1, checkpoint62g, checkpoint62f, checkpoint62e, checkpoint62d, checkpoint62k, checkpoint62j, checkpoint62i, checkpoint62h, checkpoint62o, checkpoint62n, checkpoint62m, checkpoint62l, ecco_c51_e34d, ecco_c51_e34e, ecco_c51_e34f, ecco_c51_e34g, checkpoint62w, ecco_c51_e34a, ecco_c51_e34b, ecco_c51_e34c, checkpoint62z, checkpoint62y, checkpoint62x, checkpoint58y_post, checkpoint48b_post, checkpoint43, checkpoint51d_post, checkpoint48c_pre, checkpoint47d_pre, checkpoint51t_post, checkpoint58t_post, checkpoint38, checkpoint51n_post, release1_chkpt44d_post, checkpoint55i_post, checkpoint58m_post, checkpoint57l_post, checkpoint52i_pre, hrcube_1, hrcube_2, hrcube_3, checkpoint51s_post, checkpoint47a_post, checkpoint57t_post, checkpoint55c_post, checkpoint48d_pre, checkpoint51j_post, checkpoint47i_post, checkpoint11, checkpoint10, checkpoint13, checkpoint63g, checkpoint15, checkpoint14, checkpoint17, checkpoint52e_pre, checkpoint19, checkpoint18, checkpoint57v_post, branch-atmos-merge-shapiro, checkpoint57f_post, checkpoint52e_post, checkpoint51n_pre, checkpoint47d_post, icebear5, icebear4, icebear3, icebear2, checkpoint53d_post, checkpoint46d_pre, checkpoint40pre2, checkpoint64, checkpoint65, checkpoint60, checkpoint61, checkpoint62, checkpoint63, checkpoint57a_post, checkpoint48d_post, release1-branch_tutorials, checkpoint57h_pre, checkpoint66g, checkpoint66f, checkpoint66e, checkpoint66d, checkpoint66c, checkpoint66b, checkpoint66a, checkpoint66o, checkpoint66n, checkpoint66m, checkpoint66l, checkpoint66k, checkpoint66j, checkpoint66i, checkpoint66h, checkpoint48f_post, checkpoint45d_post, checkpoint52b_pre, checkpoint54b_post, checkpoint46j_pre, checkpoint58w_post, checkpoint57h_post, checkpoint51l_pre, checkpoint52m_post, checkpoint57y_pre, redigm, chkpt44a_post, checkpoint55g_post, checkpoint44h_pre, checkpoint48h_post, checkpoint51q_post, checkpoint40pre4, ecco_c50_e29, checkpoint51b_pre, checkpoint46a_post, checkpoint47g_post, checkpoint52b_post, checkpoint52c_post, checkpoint46j_post, checkpoint28, checkpoint29, checkpoint51h_pre, checkpoint20, checkpoint21, checkpoint22, checkpoint23, checkpoint24, checkpoint25, checkpoint46k_post, checkpoint27, ecco_c50_e28, chkpt44c_pre, checkpoint58o_post, checkpoint48a_post, checkpoint45a_post, checkpoint57c_post, checkpoint50f_post, checkpoint50a_post, checkpoint50f_pre, checkpoint58p_post, checkpoint58q_post, branch-atmos-merge-freeze, branch-atmos-merge-start, checkpoint52f_pre, checkpoint55d_post, ecco_c44_e19, ecco_c44_e18, ecco_c44_e17, ecco_c44_e16, checkpoint5, checkpoint4, checkpoint7, checkpoint6, checkpoint1, checkpoint3, checkpoint2, checkpoint9, checkpoint8, release1_p12, checkpoint58e_post, release1_p10, release1_p11, release1_p16, release1_p17, release1_p14, release1_p15, checkpoint47j_post, pre38tag1, checkpoint54a_pre, checkpoint63p, checkpoint63q, checkpoint63r, checkpoint63s, checkpoint63l, checkpoint63m, checkpoint63n, checkpoint63o, checkpoint63h, checkpoint63i, checkpoint63j, checkpoint63k, checkpoint63d, checkpoint63e, checkpoint63f, ecco_c50_e33a, checkpoint53c_post, checkpoint63a, checkpoint63b, checkpoint63c, checkpoint55d_pre, kloop1, checkpoint57c_pre, checkpoint58r_post, checkpoint55j_post, kloop2, branch-exfmods-tag, checkpoint44g_post, branchpoint-genmake2, checkpoint26, checkpoint54a_post, checkpoint46e_pre, checkpoint55h_post, checkpoint58n_post, checkpoint51r_post, checkpoint48c_post, checkpoint45b_post, checkpoint46b_pre, checkpoint51i_post, checkpoint57e_post, release1-branch-end, checkpoint12, c37_adj, release1_final_v1, checkpoint55b_post, checkpoint51b_post, checkpoint16, checkpoint51c_post, checkpoint46c_pre, checkpoint53a_post, checkpoint46, checkpoint65o, checkpoint47b_post, checkpoint44b_post, checkpoint59q, checkpoint59p, checkpoint59r, checkpoint59e, checkpoint59d, checkpoint59g, checkpoint59f, checkpoint59a, checkpoint55f_post, checkpoint59c, checkpoint59b, checkpoint59m, checkpoint59l, checkpoint59o, checkpoint59n, checkpoint59i, checkpoint59h, checkpoint59k, ecco_c51_e34, checkpoint46h_pre, checkpoint52d_post, checkpoint53g_post, checkpoint46m_post, checkpoint57p_post, checkpint57u_post, checkpoint46a_pre, checkpoint50g_post, checkpoint45c_post, checkpoint57q_post, ecco_ice2, ecco_ice1, checkpoint44h_post, pre38-close, eckpoint57e_pre, checkpoint46g_post, release1_p12_pre, checkpoint58k_post, checkpoint39, checkpoint52a_pre, checkpoint33, checkpoint32, checkpoint31, checkpoint30, checkpoint37, checkpoint36, checkpoint35, checkpoint34, checkpoint62b, ecco_c44_e22, checkpoint58v_post, checkpoint50h_post, checkpoint52i_post, checkpoint50e_pre, checkpoint50i_post, ecco_c44_e25, checkpoint51i_pre, checkpoint52h_pre, checkpoint56a_post, checkpoint64y, checkpoint64x, checkpoint58l_post, checkpoint64z, checkpoint53f_post, checkpoint64q, checkpoint64p, checkpoint64s, checkpoint64r, checkpoint64u, checkpoint64t, checkpoint64w, checkpoint64v, checkpoint64i, checkpoint64h, checkpoint64k, checkpoint64j, checkpoint64m, checkpoint64l, checkpoint64o, checkpoint64n, checkpoint64a, checkpoint64c, checkpoint64b, checkpoint64e, checkpoint64d, checkpoint64g, checkpoint64f, checkpoint40pre5, checkpoint57h_done, checkpoint52j_post, checkpoint47f_post, checkpoint50e_post, chkpt44a_pre, checkpoint46i_post, ecco_c44_e23, ecco_c44_e20, ecco_c44_e21, ecco_c44_e26, ecco_c44_e27, ecco_c44_e24, checkpoint57j_post, checkpoint57f_pre, checkpoint61f, checkpoint46c_post, checkpoint58g_post, ecco-branch-mod1, ecco-branch-mod2, ecco-branch-mod3, ecco-branch-mod4, ecco-branch-mod5, branch-netcdf, checkpoint50d_pre, checkpoint58x_post, branch-atmos-merge-zonalfilt, checkpoint61n, checkpoint52n_post, checkpoint53b_pre, checkpoint46e_post, checkpoint59j, release1_beta1, checkpoint58h_post, checkpoint56c_post, checkpoint58j_post, checkpoint51e_post, checkpoint44b_pre, checkpoint42, checkpoint57a_pre, checkpoint40, checkpoint41, checkpoint55a_post, checkpoint47, checkpoint44, checkpoint45, checkpoint48, checkpoint49, checkpoint57o_post, checkpoint46h_post, checkpoint51o_post, checkpoint61q, checkpoint57k_post, checkpoint51f_pre, chkpt44c_post, checkpoint48g_post, checkpoint53b_post, checkpoint47h_post, checkpoint52a_post, checkpoint57w_post, checkpoint61e, branch-point-rdot, checkpoint44f_pre, branch-atmos-merge-phase5, branch-atmos-merge-phase4, branch-atmos-merge-phase7, branch-atmos-merge-phase6, branch-atmos-merge-phase1, checkpoint58i_post, branch-atmos-merge-phase3, branch-atmos-merge-phase2, checkpoint51g_post, ecco_c52_e35, checkpoint57x_post, checkpoint46d_post, checkpoint50b_post, checkpoint58c_post, checkpoint58u_post, release1-branch_branchpoint, checkpoint51m_post, checkpoint53d_pre, checkpoint58s_post, checkpoint55e_post, checkpoint61g, checkpoint61d, checkpoint54c_post, checkpoint61b, checkpoint61c, checkpoint61a, checkpoint51a_post, checkpoint61o, checkpoint61l, checkpoint61m, checkpoint61j, checkpoint61k, checkpoint61h, checkpoint61i, checkpoint61v, checkpoint61w, checkpoint61t, checkpoint61u, checkpoint61r, checkpoint61s, checkpoint61p, checkpoint51p_post, checkpoint61z, checkpoint61x, checkpoint61y, checkpoint51u_post, HEAD
Branch point for: c24_e25_ice, branch-exfmods-curt, checkpoint7-4degree-ref, release1_final, release1-branch, branch-genmake2, release1, branch-nonh, tg2-branch, ecco-branch, branch-rdot, release1_50yr, netcdf-sm0, icebear, checkpoint51n_branch, branch-atmos-merge, pre38, release1_coupled
Changes since 1.2: +6 -1 lines
New FAQ entry

1 cnh 1.3 $Header: /u/gcmpack/models/MITgcmUV/doc/notes,v 1.2 1998/04/24 02:36:52 cnh Exp $
2 cnh 1.2
3 cnh 1.1 Miscellaneous notes relating to MITgcm UV
4     =========================================
5 cnh 1.3
6     This files form is close to that of an FAQ. If you are having
7     a problem getting the model to behave as you might expect they
8     may be some helpful clues in this file.
10 cnh 1.1
11     o Something really weird is happening - variables keep
12     changing value!
14     Apart from the usual problems of out of bounds array refs.
15     and various bugs itis important to be sure that "stack"
16     variables really are stack variables in multi-threaded execution.
17     Some compilers put subroutines local variables in static storage.
18     This can result in an apparently private variable in a local
19     routine being mysteriously changed by concurrently executing
20     thread.
22     =====================================
24     o Something really weird is happening - the code gets stuck in
25     a loop somewhere!
27     The routines in barrier.F should be compiled without any
28     optimisation. The routines check variables that are updated by other threads
29     Compiler optimisations generally assume that the code being optimised
30     will obey the sequential semantics of regular Fortran. That means they
31     will assume that a variable is not going to change value unless the
32     code it is optimising changes it. Obviously this can cause problems.
34     =====================================
36     o Is the Fortran SAVE statement a problem.
38     Yes. On the whole the Fortran SAVE statement should not be used
39     for data in a multi-threaded code. SAVE causes data to be held in
40     static storage meaning that all threads will see the same location.
41     Therefore, generally if one thread updates the location all other threads
42     will see it. Note - there is often no specification for what should happen
43     in this situation in a multi-threaded environment, so this is
44     not a robust machanism for sharing data.
45     For most cases where SAVE might be appropriate either of the following
46     recipes should be used instead. Both these schemes are potential
47     performance bottlenecks if they are over-used.
48     Method 1
49     ********
50     1. Put the SAVE variable in a common block
51     2. Update the SAVE variable in a _BEGIN_MASTER, _END_MASTER block.
52     3. Include a _BARRIER after the _BEGIN_MASTER, _END_MASTER block.
53     e.g
54     C nIter - Current iteration counter
55     COMMON /PARAMS/ nIter
56     INTEGER nIter
58     _BEGIN_MASTER(myThid)
59     nIter = nIter+1
60     _END_MASTER(myThid)
61     _BARRIER
63     Note. The _BARRIER operation is potentially expensive. Be conservative
64     in your use of this scheme.
66     Method 2
67     ********
68     1. Put the SAVE variable in a common block but with an extra dimension
69     for the thread number.
70     2. Change the updates and references to the SAVE variable to a per thread
71     basis.
72     e.g
73     C nIter - Current iteration counter
74     COMMON /PARAMS/ nIter
77     nIter(myThid) = nIter(myThid)+1
79     Note. nIter(myThid) and nIter(myThid+1) will share the same
80     cache line. The update will cause extra low-level memory
81     traffic to maintain cache coherence. If the update is in
82     a tight loop this will be a problem and nIter will need
83     padding.
84     In a NUMA system nIter(1:MAX_NO_THREADS) is likely to reside
85     in a single page of physical memory on a single box. Again in
86     a tight loop this would cause lots of remote/far memory references
87     and would be a problem. Some compilers provide a machanism
88     for helping overcome this problem.
90     =====================================
92     o Can I debug using write statements.
94     Many systems do not have "thread-safe" Fortran I/O libraries.
95     On these systems I/O generally orks but it gets a bit intermingled!
96     Occaisionally doing multi-threaded I/O with an unsafe Fortran I/O library
97     will actual cause the program to fail. Note: SGI has a "thread-safe" Fortran
98     I/O library.
100     =====================================
102     o Mapping virtual memory to physical memory.
104     The current code declares arrays as
105     real aW2d (1-OLx:sNx+OLx,1-OLy:sNy+OLy,nSx,nSy)
106     This raises an issue on shared virtual-memory machines that have
107     an underlying non-uniform memory subsystem e.g. HP Exemplar, SGI
108     Origin, DG, Sequent etc.. . What most machines implement is a scheme
109     in which the physical memory that backs the virtual memory is allocated
110     on a page basis at
111     run-time. The OS manages this allocation and without exception
112     pages are assigned to physical memory on the box where the thread
113     which caused the page-fault is running. Pages are typically 4-8KB in
114     size. This means that in some environments it would make sense to
115     declare arrays
116     real aW2d (1-OLx:sNx+OLx+PX,1-OLy:sNy+OLy+PY,nSx,nSy)
117     where PX and PY are chosen so that the divides between near and
118     far memory will coincide with the boundaries of the virtual memory
119     regions a thread works on. In principle this is easy but it is
120     also inelegant and really one would like the OS/hardware to take
121     care of this issue. Doing it oneself requires PX and PY to be recalculated whenever
122     the mapping of the nSx, nSy blocks to nTx and nTy threads is changed. Also
123     different PX and PY are required depending on
124     page size
125     array element size ( real*4, real*8 )
126     array dimensions ( 2d, 3d Nz, 3d Nz+1 ) - in 3d a PZ would also be needed!
127     Note: 1. A C implementation would be a lot easier. An F90 including allocation
128     would also be fairly straightforward.
129     2. The padding really ought to be between the "collection" of blocks
130     that all the threads using the same near memory work on. To save on wasted
131     memory the padding really should be between these blocks. The
132     PX, PY, PZ mechanism does this three levels down on the heirarchy. This
133     wastes more memory.
134     3. For large problems this is less of an issue. For a large problem
135     even for a 2d array there might be say 16 pages per array per processor
136     and at least 4 processors in a uniform memory access box. Assuming a
137     sensible mapping of processors to blocks only one page (1.5% of the
138     memory) referenced by processors in another box.
139     On the other hand for a very small per processor problem size e.g.
140     32x32 per processor and again four processors per box as many as
141     50% of the memory references could be to far memory for 2d fields.
142     This could be very bad!
144     =====================================
146     =====================================
148     =====================================
150     =====================================
152     =====================================
154     =====================================

  ViewVC Help
Powered by ViewVC 1.1.22