high_res_cube/eddy_flux/data_transfer.txt


Getting the "cube_5" data from NAS (NASA Ames)
==============================================

Due mostly to the latency of long-distance connections, its rather
inefficient to copy large files one at a time using a single "scp"
command [1].  In order to speed-up and automate the process of
transferring a large number of large files, a multi-threaded scp
command wrapper called "mt_scp" was written.  To build mt_scp, issue
the following commands

   export CVSROOT=':pserver:cvsanon@mitgcm.org:/u/gcmpack'
   cvs login
     <== "cvsanon"
   cvs co development/edhill/mt_scp
   cd development/edhill/mt_scp/
   ./configure 
   make

at which point the directory should contain an "mt_scp" executable.
Copy that executable to a suitable install location or wherever you
need to use it.

To use mt_scp, it is best to first setup so-called "password-less"
(public-key) SSH authentication between the two hosts.  The
configuration of public-key SSH authentication is described on many
helpful on-line sites [2] and takes only a few minutes.

To use mt_scp, some example commands are:

  #  Get a description of all the mt_scp options
  #
  ./mt_scp

  #  Copy all files matching the pattern "Stave.00*" at the 
  #  location lou.nas.nasa.gov:/ecco/menemenl/cube5/output 
  #  using 8 concurrent threads (note how the file pattern
  #  or "glob" has been escaped so that expansion only 
  #  on the "remote" side):
  #
  ./mt_scp -n 8 -p 'Stave\*' lou.nas.nasa.gov /ecco/menemenl/cube5/output


Please send questions about mt_scp to Ed Hill <eh3@mit.edu>.


References
==========

[1] Speedups of 10x--11.5x have been routinely observed:
      http://mitgcm.org/eh3/lou.nas.nasa.gov/
      http://mitgcm.org/eh3/lou.nas.nasa.gov/lou_data_xfer.png

[2] http://www-106.ibm.com/developerworks/library/l-keyc.html
    http://cfm.gs.washington.edu/security/ssh/client-pkauth/

1
2	Getting the "cube_5" data from NAS (NASA Ames)
3	==============================================
4
5	Due mostly to the latency of long-distance connections, its rather
6	inefficient to copy large files one at a time using a single "scp"
7	command [1]. In order to speed-up and automate the process of
8	transferring a large number of large files, a multi-threaded scp
9	command wrapper called "mt_scp" was written. To build mt_scp, issue
10	the following commands
11
12	export CVSROOT=':pserver:cvsanon@mitgcm.org:/u/gcmpack'
13	cvs login
14	<== "cvsanon"
15	cvs co development/edhill/mt_scp
16	cd development/edhill/mt_scp/
17	./configure
18	make
19
20	at which point the directory should contain an "mt_scp" executable.
21	Copy that executable to a suitable install location or wherever you
22	need to use it.
23
24	To use mt_scp, it is best to first setup so-called "password-less"
25	(public-key) SSH authentication between the two hosts. The
26	configuration of public-key SSH authentication is described on many
27	helpful on-line sites [2] and takes only a few minutes.
28
29	To use mt_scp, some example commands are:
30
31	# Get a description of all the mt_scp options
32	#
33	./mt_scp
34
35	# Copy all files matching the pattern "Stave.00*" at the
36	# location lou.nas.nasa.gov:/ecco/menemenl/cube5/output
37	# using 8 concurrent threads (note how the file pattern
38	# or "glob" has been escaped so that expansion only
39	# on the "remote" side):
40	#
41	./mt_scp -n 8 -p 'Stave\*' lou.nas.nasa.gov /ecco/menemenl/cube5/output
42
43
44	Please send questions about mt_scp to Ed Hill <eh3@mit.edu>.
45
46
47	References
48	==========
49
50	[1] Speedups of 10x--11.5x have been routinely observed:
51	http://mitgcm.org/eh3/lou.nas.nasa.gov/
52	http://mitgcm.org/eh3/lou.nas.nasa.gov/lou_data_xfer.png
53
54	[2] http://www-106.ibm.com/developerworks/library/l-keyc.html
55	http://cfm.gs.washington.edu/security/ssh/client-pkauth/
56