[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]

(This Webpage Page in No Frames Mode)

Collaborative Computational Project Number 14

for Single Crystal and Powder Diffraction

CCP14

Rsync software for Co-operative Incremental and full Mirroring

The CCP14 Homepage is at http://www.ccp14.ac.uk

[Back to CCP14 Web/Config Main Page]

[Why use Rsync | Where to get Rsync | Compiling Rsync]
[Setting up Rsync Server | Setting up Rsync Client | Daily AutoMirroring Scripts]
[Possibilities for the Future - Ultra High Effeciency Mirroring]
[Rsync for Windows] | [Rsync for Win NT running in Deamon Mode - 1] | [Rsync for Win NT running in Deamon Mode - 2]
[Getting Rsync working with ssh/SecureShell for UNIX] | [Getting Rsync working under Secure-Shell for Windows]


The following was for installation on an SGI O2 running IRIX but should work on most UNIX systems will very little change.

Why Use Rsync

The Rsync program is a free, open source program for UNIX (and possibly for Windows(?)) that enables fast incremental file transfer. It is written by the same people who have done the "Samba" program. Thus the software can be used for automatic incremental mirroring over slow internet links (and fast internet links as well) using a custom written rsync algorithm/protocol and its own uniquely assinged "port". For a "no change" file system such as the (30th Nov 1998) 2.1 Gig Crystallographic CCP14 Mirror; a backup using rsync takes ~4.5 minutes as opposed to ~14.5 minutes using WGET/mirror style FTP mirroring/backup. Rsync only tries to transfer the changed part of the files. Thus if only 100 lines of a 100 Meg file have changed, rsync will only try and transfer the changes; not the entire 100 Meg file.

Thus:

One work of warning is that the present set of rsync programs (1998 to Jan 2000 and beyond?) can be a bit finicky when it comes to transferring large chunks of data. Possible due to memory requirements(?). Thus it can be better to split the mirroring into smaller sections. While a pain, this gets the job done.

Relevant pages:

Refer down to the bottom of this page for hints on using ssh/Secure Shell properly with rsync

Also, it is possible that if you are getting the following type of problem. this can be due to compiling up rsync with GCC on IRIX. Switch to CC and things should be OK.
cu11 87% ./rsync.ccp14.nrc
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums
ERROR: out of memory in generate_sums

With the corresponding error in the Syslog file on the server
(this is running in deamon mode)
Jan 29 02:18:27 6T:sv1 rsyncd[1803123]: transfer 
    interrupted (code 12) at io.c(167)

There is an intension to implement a more effecient incremental system for backups going into the rsync program.

Also, note that the method for doing excludes changed with 2.3.0 such that a ** is required to recurse all subdirectories.


Where to get Rsync

Rsync is obtainable as source code or binaries via:


Compiling and Installing the Web Server

As noted before, GCC can give problems with IRIX so it is best to use the IRIX cc compiler.

From: [tridge@linuxcare.com]
To: Multiple recipients of list RSYNC [rsync@samba.org]
Subject: rsync 2.4.0 released
X-URL:  http://lists.samba.org/
Date:   Sat, 29 Jan 2000 23:34:21 +1100

I've just released rsync 2.4.0. The number of test reports I got
tailed off and none of them were negative so I decided to do a
release. See http://rsync.samba.org/ for the release notes and
source.

Note that to _really_ fix the problem with rsync over ssh you'll also
have to compile sshd to use socketpair() instead of pipe(). I have a
little demo program that shows that sshd can definately lock up under
bi-directional transfers when using pipe(). I believe that is the
primary cause of rsync over ssh problems.

To compile openssh with socketpair instead of pipes, just remove the
line:
        #define USE_PIPES 1
from near the bottom of includes.h

Cheers, Tridge

Refer down to the bottom of this page for hints on using ssh/Secure Shell properly with rsync


Setting up Rsync Server

(Note: If doing this again from scratch: rsync can be installed in daemon mode from the command line or a start up script (rsync --daemon) - getting around the need to use inetd.con - or use /usr/local/bin/rsync --daemon --address=xxx.xxx.xxx.xxx if you only want to bind to a specified IP address)


Setting up Rsync Client

Example Partial CCP14 Mirroring Script

(Where the mirror server does not room for the entire mirror)

#!/sbin/csh

#!/bin/bash

#Secure copying updates to the remote clients
#scp rsync.ccp14.nrc guest@cu11.aecl.ca:/usr/people/guest/ccp14mirroring/rsync.c
cp14.nrc
#scp report_template_nrc.txt guest@cu11.aecl.ca:/usr/people/guest/ccp14mirroring
/report_template_nrc.txt

# scp rsync.ccp14 lzc@ftp.minerals.csiro.au:/usr/local/sys/rsync.ccp14

# You should CHANGE THE NEXT 3 LINES to suit your local setup
setenv  CCP14WEBDIR   /usr/people/guest/web_area/ccp14
setenv  REPORTDIR     /usr/people/guest/ccp14mirroring


setenv RSYNC_PASSWORD insert_password_here

# export RSYNC_PASSWORD=insert_password_here
# original script kept on sv1.ccp14.ac.uk
#   /web_disc/ccp14/cron_scripts/remote

#rsync -rlpt  -e ssh -v --stats -z --safe-links --delete \

cp $REPORTDIR/report_template_nrc.txt $REPORTDIR/report.txt

set DATE=(`date`)

#  DATE=`date`  # bash style

# -v   verbose
# -q   quiet
# --dry-run
# --delete-after  delete old file after downloading the update


sed "/Mirror_of_CCP14_Started/s/NOT_FINISHED/         $DATE/" \ 
  $REPORTDIR/report.txt > $REPORTDIR/report.txt.new

mv  $REPORTDIR/report.txt.new   $REPORTDIR/report.txt

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete  \
--include "/ansto/" \
--include "/ansto/**" \
--include "/axes/" \
--include "/axes/**" \
--include "/briantoby/" \
--include "/briantoby/**" \
--include "/crush/" \
--include "/crush/**" \
--include "/crystals/" \
--include "/crystals/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/
ccp14/ftp-mirror/ > $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete  \
--include "/debvin/" \
--include "/debvin/**" \
--include "/dirdif/" \
--include "/fit2d/**" \
--include "/fit2d/" \
--include "/dirdif/**" \
--include "/fullprof/" \
--include "/fullprof/**" \
--include "/gsas/" \
--include "/gsas/**" \
--include "/jana/" \
--include "/jana/**" \
--include "/krumm/" \
--include "/krumm/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/
ccp14/ftp-mirror/ >> $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete  \
--include "/ill-hewat/" \
--include "/ill-hewat/**" \
--include "/lauecell-ravelli/" \
--include "/lauecell-ravelli/**" \
--include "/lapod-langford/" \
--include "/lapod-langford/**" \
--include "/llnlrupp/" \
--include "/llnlrupp/**" \
--include "/lutterotti/" \
--include "/lutterotti/**" \
--include "/nardelli/" \
--include "/nardelli/**" \
--include "/ornl-ortep/" \
--include "/ornl-ortep/**" \
--include "/platon-spek/" \
--include "/platon-spek/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/
ccp14/ftp-mirror/ >> $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete  \
--include "/powdcell/" \
--include "/powdcell/**" \
--include "/profil/" \
--include "/profil/**" \
--include "/prometheus/" \
--include "/prometheus/**" \
--include "/rietan/" \
--include "/rietan/**" \
--include "/weber/" \
--include "/weber/**" \
--include "/wget/" \
--include "/wget/**" \
--include "/xnd/" \
--include "/xnd/**" \
--include "/xray-villers/" \
--include "/xray-villers/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/ccp14/ftp-mirror/ $CCP14WEBDIR/ccp/
ccp14/ftp-mirror/ > $REPORTDIR/Log.ccp14

set DATE=(`date`)


#  DATE=`date`  # bash style


sed "/FTP_mirror/s/NOT_FINISHED/Finished $DATE/" $REPORTDIR/report.txt > $REPORT
DIR/report.txt.new
mv   $REPORTDIR/report.txt.new   $REPORTDIR/report.txt

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete \
--include "/armel/" \
--include "/armel/**" \
--include "/bgmn/" \
--include "/bgmn/**" \
--include "/briantoby/" \
--include "/briantoby/**" \
--include "/caos/" \
--include "/caos/**" \
--include "/crush/" \
--include "/crush/**" \
--include "/crys-r-shirley/" \
--include "/crys-r-shirley/**" \
--include "/crystals/" \
--include "/crystals/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m
irrors/ >>  $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete \
--include "/dirdif/" \
--include "/dirdif/**" \
--include "/dplot/" \
--include "/dplot/**" \
--include "/fit2d/**" \
--include "/fit2d/" \
--include "/ethz/" \
--include "/ethz/**" \
--include "/farrugia/" \
--include "/farrugia/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m
irrors/ >>  $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete \
--include "/hugorietveld/" \
--include "/hugorietveld/**" \
--include "/jana/" \
--include "/jana/**" \
--include "/krumm/" \
--include "/krumm/**" \
--include "/ill-hewat/" \
--include "/ill-hewat/**" \
--include "/lapod-langford/" \
--include "/lapod-langford/**" \
--include "/llnlrupp/" \
--include "/llnlrupp/**" \
--include "/lutterotti/" \
--include "/lutterotti/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m
irrors/ >>  $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete \
--include "/nardelli/" \
--include "/nardelli/**" \
--include "/ndragoe/" \
--include "/ndragoe/**" \
--include "/ornl-ortep/" \
--include "/ornl-ortep/**" \
--include "/ortex/" \
--include "/ortex/**" \
--include "/patsee/" \
--include "/patsee/**" \
--include "/pfe/" \
--include "/pfe/**" \
--include "/pki/" \
--include "/pki/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m
irrors/ >>  $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete \
--include "/platon-spek/" \
--include "/platon-spek/**" \
--include "/plotr/" \
--include "/plotr/**" \
--include "/powdcell/" \
--include "/powdcell/**" \
--include "/rietan/" \
--include "/rietan/**" \
--include "/valence/" \
--include "/valence/**" \
--include "/weber/" \
--include "/weber/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m
irrors/ >>  $REPORTDIR/Log.ccp14

/usr/local/bin/rsync -rlpt -e ssh  -v --stats -z --safe-links  --delete \
--include "/winmprof/" \
--include "/winmprof/**" \
--include "/wget/" \
--include "/wget/**" \
--include "/xnd/" \
--include "/xnd/**" \
--include "/xray-villers/" \
--include "/xray-villers/**" \
--include "/xtalview-mcree/" \
--include "/xtalview-mcree/**" \
--exclude "*" \
rsyncmir@www.ccp14.ac.uk::ccp14/web_live/ccp/web-mirrors/ $CCP14WEBDIR/ccp/web-m
irrors/ >>  $REPORTDIR/Log.ccp14

set DATE=(`date`)

#  DATE=`date`  # bash style

sed "/Web_Mirror/s/NOT_FINISHED/Finished $DATE/" $REPORTDIR/report.txt > $REPORT
DIR/report.txt.new
mv   $REPORTDIR/report.txt.new   $REPORTDIR/report.txt

/usr/local/bin/rsync -rlpt -e ssh  -v --stats  -z --safe-links  --delete   \
rsyncmir@www.ccp14.ac.uk::ccp14/nrc/ $CCP14WEBDIR/nrc/  >>  $REPORTDIR/Log.ccp14

set DATE=(`date`)

#  DATE=`date`  # bash style


sed "/HTML_Index_files/s/NOT_FINISHED/Finished $DATE/" $REPORTDIR/report.txt > $
REPORTDIR/report.txt.new
mv   $REPORTDIR/report.txt.new   $REPORTDIR/report.txt


/usr/sbin/Mail -s "AECL_daily_rsync_mirroring_Results `date`" ccp14@dl.ac.uk < /
usr/people/guest/ccp14mirroring/report.txt

#  /usr/ucb/Mail -s "CSIRO_daily_rsync_mirroring_Results `date`" ccp14@dl.ac.uk
< /usr/people/guest/ccp14mirroring/report.txt


# -n, --dry-run
# --progress


Setting up Rsync Client


Starting up Rsync so that it only binds to one IP address

From: [tim.conway@philips.com]
To: Lachlan Cranswick [l.m.d.cranswick@dl.ac.uk]
Cc: rsync@lists.samba.org
Subject: Re: rsync: network "bind interfaces only" command for rsync?
Date: Tue, 14 May 2002 07:44:37 -0600

Might the --address= option below, in conjunction with your IP filtering, 
do what you want?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
     --address
          By default rsync will bind to the wildcard address when
          run  as  a daemon with the --daemon option or when con-
          necting to a rsync server. The --address option  allows
          you  to  specify a specific IP address (or hostname) to
          bind to. This makes virtual hosting  possible  in  con-
          junction with the --config option.

SunOS 5.7           Last change: 25 Jan 2002                   17

User Commands                                            rsync(1)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Tim Conway
tim.conway@philips.com
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(nnnnnnnnnnnn, 
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), 
".\n" '
"There are some who call me.... Tim?"


Possibilities for the Future - Ultra High Effeciency Mirroring

From the Rsync Mailing List:

Forwarded message:
>From rsync@samba.anu.edu.au  Wed Dec  9 00:38:15 1998
Message-Id: >19981209003703Z12609010-1431+100@samba.anu.edu.au<
Errors-To: listproc-errors@samba.anu.edu.au
Reply-To: tridge@samba.org
Originator: rsync@samba.anu.edu.au
Sender: rsync@samba.anu.edu.au
Precedence: bulk
From: Andrew Tridgell >tridge@samba.org<
To: Multiple recipients of list >rsync@samba.anu.edu.au<
Subject: file list cache
X-Listprocessor-Version: 6.0c -- ListProcessor by Anastasios Kotsikonas
X-URL: http://samba.anu.edu.au/listproc
X-Comment: Discussion of rsync package
Date: 	Wed, 9 Dec 1998 11:37:43 +1100

While writing up part of rsync for my thesis it occurred to me that we
can improve the mirroring performance considerably by cacheing the
file list on the client.

It is quite common that the file list is the most network intensive
part of a rsync mirror run, often making up over 90% of the bytes on
the wire. It is usually about 10-14 bytes per file so if you are
mirroring large numbers of files (say millions) it can soon grow to be
a significant network cost.

I think the solution is a --cache option that works like this:

- the first time it is used the current behaviour is unchanged, except
  that the client saves the file list as a single file in
  /tmp/rsync-cache-$USER/XXXXX.cache where XXXXX is a hash of the
  commandline options. (obviously the directory would need to be able
  to be chosen with an option as well, just defaulting to this)

- the next time through rsync detects this file and the client and
  server use the rsync algorithm to update the cache to the correct
  file list. The idea is that the file list won't change much between
  runs so it is an ideal candidate for rsync. The client sends the
  checksum signatures of the current cache file to the server and the
  server sends back the rsync differences to bring the cache up to
  date.

The nice thing about this system is that it avoids the network traffic
associated with transferring the file list in most cases yet it
guarantees that the client has an accurate file list. 

If we combine this with an option to create/use a file list in a file
on the server then I think many of the current mirroring issues will
be solved.

This will require quite a bit of work in the code and probably won't
happen till next year, but I thought I'd mention it now. I thought the
idea of using rsync to update the rsync file lists is quite nice :)


Rsync for Windows (including binaries)

Installing ssh and rsync on a Windows machine

http://optics.ph.unimelb.edu.au/help/rsync/

Compiled with cygwin-b20.

--
Robert Scholten                   Tel:   +61 3 9344 5457
School of Physics                 Fax:   +61 3 9347 4783
University of Melbourne           email: r.scholten@physics.unimelb.edu.au
Victoria 3010  AUSTRALIA          http://www.ph.unimelb.edu.au/~scholten/

How to setup Rsync as an NT Service: http://members.home.net/cbollerud2/projects/rsync/NTService.html


From: JF Paradis <jfparadis@mec.ca>
To: rsync@lists.samba.org
Subject: Re: Another way to setup Rsync as an NT Service
Date: Thu, 22 Nov 2001 13:09:18 -0500

Here is how:

Use the KILL.EXE from the NT Resource kit.

On my systems, I use the NT scheduler to run
batch files (using the default AT or the GUI
WINAT.EXE from the NT resource kit).

At the start of each batch file, I use:
c:\bin\KILL.EXE rsync

This ensure that rsync is not already running!

Here is my setup:
1. No need to install cygwin
2. I have FS1 & FS2 to backup on FSX
3. The same client batch file is used (use %COMPTUERNAME%)
4. I backup the client e: drive to the server .bak folders

Put into c:\bin of all machines:
- rsync.exe
- cygwin1.dll
- kill.exe
- srvany.exe
- instsrv.exe

Along with these text files (copy everything between
the tags):

<rsync.bat>
SET OPTION=--compress --recursive --times --delete --bwlimit=75
SET SELECT=--exclude "*bak" --exclude "*~"
SET SOURCE=/cygdrive/e/
SET SERVER=FSX::%COMPUTERNAME%
c:\bin\kill.exe rsync
c:\bin\rsync.exe %OPTION% %SELECT% %SOURCE% %SERVER%
<EOF>

<rsybcd.conf>
use chroot = false
strict modes = false
read only = false
[FS1]
path = /cygdrive/e/fs1.bak
hosts allow = fs1
[FS2]
path = /cygdrive/e/fs2.bak
hosts allow = fs2
<EOF>

<rsyncd_install.bat>
c:\bin\instsrv.exe rsyncd "c:\bin\srvany.exe"
c:\bin\rsyncd.reg
pause
<EOF>

<rsyncd_remove.bat>
net stop rsyncd
c:\bin\kill.exe rsync
c:\bin\instsrv.exe rsyncd remove
pause
<EOF>

<rsyncd_restart.bat>
net stop rsyncd
c:\bin\kill.exe rsync
net start rsyncd
pause
<EOF>

<rsyncd_stop.bat>
net stop rsyncd
c:\bin\kill.exe rsync
pause
<EOF>

<rsyncd.reg>
REGEDIT4

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\rsyncd]

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\rsyncd\Parameters]
"AppDirectory"="c:\\program files\\rsync"
"Application"="c:\\program files\\rsync\\rsync.exe"
"AppParameters"="--daemon --config=rsyncd.conf"
<EOF>

<rsyncd_schedule.bat>
at \\fs1 10:00pm /every:m,t,w,th,f,s,su c:\bin\rsync.bat
at \\fs2 11:00pm /every:m,t,w,th,f,s,su c:\bin\rsync.bat
<EOF>

Then double-click on thse for the initial setup:
rsync_install.bat
rsync_restart.bat
rsync_schedule.bat

Use these to manage the service:
rsync_stop.bat
rsync_restart.bat

Use this to uninstall:
rsync_remove.bat

Easy, simple and clean!

JF


Rsync for Windows NT running in Deamon Mode

Refer: http://rsync.samba.org/rsync/nt.html

Sender: rsync@samba.anu.edu.au
From: "Mike McHenry" [mmchen@minn.net]
To: Multiple recipients of list RSYNC [rsync@samba.anu.edu.au] 
Subject: Rsync 2.3.1 WinNT binaries and instructions available
X-URL:  http://samba.anu.edu.au/listproc
Date:   Fri, 15 Oct 1999 02:53:33 +1000


Hello all,

I have created Windows NT binaries for rsync 2.3.1 and have decided to make
them available for others to download. These binaries have been tested on
Windows NT Server 4.0 SP5 and WILL run in daemon mode if you follow my
instructions below. I make no guarantees about these binaries, they have
however been working for me for weeks on several NT machines.

Binaries at ftp://ftp.minn.net/usr/mmchen/

Instructions for running in daemon mode:

1. You will need two files, rsync.exe and cygwin1.dll. Place rsync.exe
anywhere you like (I chose c:\program files\rsync\rsync.exe) and put
cygwin1.dll in c:\winnt\system32

2. You will need a program from the NT Server resource kit called
srvany.exe. This program allows you to run any executable as a service. If
you simply install the entire service pack it will be located in c:\ntreskit

3. Create a service for rsync by typing the following:
  instsrv Rsync "C:\ntreskit\srvany.exe"

4. You should now have a new service called Rsync and you can verify by
looking in Start->Control Panel->Services DON'T START IT YET!

5. If you want to run rsync in daemon mode you will need a configuration
file. Here is the one I use, call it rsyncd.conf and place it in the same
directory as rsync (C:\Program files\rsync\rsyncd.conf)
  use chroot = false
  strict modes = false
  hosts allow = *

  [backup]
     path = /
     read only = yes
     list = no

This example configuration will make one big anonymous anonymous rsync area
available, I use this to backup my NT machines from a central Unix machine.
This configuration might not be ideal for you, change to suit your tastes.
The first two lines are important for rsync to work on Windows NT however.

6. You are going to need to hack some keys in the registry to make it work.
Don't do this unless you are comfortable with the changes! Run regedit32 and
add the following keys and values (quotation marks ARE IMPORTANT):
  HKEY_LOCAL_MACHINE->SYSTEM->CurrentControlSet->Services->Rsync
    Edit->Add Key->  Key Name: Paramaters
    Edit->Add Value->  Value Name: AppDirectory  Value: "C:\program
files\rsync"
    Edit->Add Value->  Value Name: Application  Value: "C:\program
files\rsync\rsync.exe"
    Edit->Add Value->  Value Name: AppParamters  Value: --config="C:\program
files\rsync\rsyncd.conf" --daemon

7. That's it, you should be able to start and stop the rsync service at will
using the Services Control Panel. When running with the above configuration
you should be able to test by attempting to telnet to port 873 from a remote
machine.
  telnet rsync.server.com 873 (replacing rsync.server.com with your own
server's address)
You should get a connection to the rsync daemon running on your NT box.

8. If you have problems you are on your own, sorry, I have enough to do :) I
would suggest triple-checking your spelling on EVERYTHING (filenames,
configs, reg keys). If you have any comments or suggestions I would be happy
to hear them at mmchen@minn.net.

Mike McHenry
 Systems Administrator
 MinnNet Communications, Inc.


Delivered-To: rsync@lists.samba.org
From: Bill Campbell [bill@celestial.com]
To: rsync@lists.samba.org
Subject: Re: rsync: Re: using rsync to backup windows workstations
Date: Wed, 14 Nov 2001 11:45:20 -0800

On Wed, Nov 14, 2001 at 10:15:59AM -0500, Thomas Lambert wrote:
>I have about 12 NT servers that I want to backup 1 directory to my Linux
>Server.  The problem I am having is trying to get it not to ask for the
>user's password.  I am trying to use ssh.  I just installed the latest
>cygwin on my NT server (1.3.4).  Rsync is 2.5.2.  What I have tried was on
>the NT machine, I ran ssh-keygen (no passphrase) and then added the
>identity.pub to the Linux box in the /home/thomas/.ssh/authorized_keys.

Instead of using ssh, why don't you use straight rsync, with rsync running
in server mode on the Linux box.  You should probably put separate entries
in /etc/rsyncd.conf for each Windows box something like the one below that
I use to backup our local hardware suffering from the Microsoft Virus, W2K.

[dumbo_backup]
	uid = root
	gid = root
	path = /rd0/backups/dumbo
	read only = false
	use chroot = false
	comment = Backup for Dumbo Win2K
	hosts allow = 192.168.253.28 
	hosts deny = *

Bill
--
INTERNET:   bill@Celestial.COM  Bill Campbell; Celestial Software LLC
UUCP:               camco!bill  PO Box 820; 6641 E. Mercer Way
FAX:            (206) 232-9186  Mercer Island, WA 98040-0820; (206) 236-1676
URL: http://www.celestial.com/

Breathe fire, slay dragons, and take chances. Failure is temporary, regret
is eternal.


Getting Rsync to work on Win NT in Services/deamon mode

Refer: http://v.iki.fi/~vherva/cygwin-rsync/

From: Ville Herva [vherva@niksula.hut.fi] 
To: Multiple recipients of list RSYNC [rsync@samba.org] 
Subject: [PATCH] Cygwin tweaks and some questions 
X-URL:  http://lists.samba.org/ 
X-Comment: Discussion of rsync package 
X-Mailer: Mutt 1.0pre3i 
Date:   Sat, 15 Apr 2000 04:39:41 +1000 


[I understand perfectly if you don't want this stuff into the mainline
tree, but perhaps somebody finds it useful. Anyhow, I would be glad if
somebody savvy commented especially the second point.]


I just compiled rsync for CYGWIN, and found that while it compiles
cleanly out of the box (newest (almost) cygwin1.dll snapshot and
gcc-2.95.2), some tweaks are necessary to get most out of it. (2.4.3 was
only released during my tweaking so I've been lazy enough not to upgrade.
This patch is against 2.4.2).


Firstly, the only sensible way to run something as "daemon" under NT is to
run it as service. One can use SrvAny.exe from the NT resource kit for
this. However, as it is, rsync immediately forks another process, and net
stop has not desired effect, since srvany can't kill the rsync daemon. I
added --dont-fork option so rsync can be forced not to fork when run as
daemon. This way starting and stopping services works right.


Secondly, if I connect to rsync daemon from another machine and hit ctrl-c
at the client end during transfer, the rsync daemon exists (not just the
connection handler process, but every rsync). I guess this is because
rsync gets sigpipe signal, which is handled by sig_int, which in turn
sends sigusr to the parent. I have no idea why this should be. I changed
sigpipe handler to SIG_IGN, and I now get the behaviour I want: if there
is a network error (such as premature socket closing) the rysnc daemon
won't die, and I can reconnect to it. I didn't do this cleanly: the
connection handler process should in fact exit (perhaps after some clean
up), but not the parent. This is not a problem, however, since the parent
will reap the connection handler after a minute. Am I missing something
here, or why is sigpipe handled by sig_int?


Third, under CYGWIN, it makes little sense to force attribute check on the
password file. It can be made to work if env var CYGWIN contains "ntea"
(nt extended attribute mode -- CYGWIN emulates unix permissions through a
hack), but frankly, it is a MAJOR pain in the ass to get working
especially when trying to run rsync as a NT service. Besides, CYGWIN
permissions won't give you any added security -- the security should be
handled with NTFS ACL's. So I added a #ifndef __CYGWIN__'s around the
check.


Lastly, the original reason I began compiling my own version was that I
had a binary version that did not force binary open (I got corrupted .doc
files etc in the transmission.) I found out that the mainline rsync source
(as of 2.4.2) already has that O_BINARY bit in do_open, so no problem
here. It works without, if you mount everything with -b (binary) in
CYGWIN or force that with "set CYGWIN=BINMODE", but O_BINARY is the (most)
correct way to do it -- it works regardless how the user has mounted his
filesystems. However, if the mounts are binary, the configuration files
are not read right: if, for example, the user makes a password file with
notepad, he'll get an additional CR into the password (due to missing
CR/LF->LF conversion). So I added open(O_TEXT) forcing to every config
file open (#ifdef CYGWIN'ed, of course). Now the users shouldn't have to
worry about mount types.


I put the patch at [http://v.iki.fi/~vherva/cygwin-rsync/].


[OT] In case you are wondering: I'm implementing NT workstation backups
with rsync here. One Linux server with two 34GB ide disk softraided
together and e2compr patch thrown on top of that. Then I just take
incremental backups with rsync so that every file is a hardlink to
yesterday's copy unless it's changed. (Basicly, cp -al yesterday
today;rsync rsync://nt/dir today/) Add e2compr to that, and the backup
won't really take that much space anymore. Just the directory structure
and changed files (compressed) per backup. The nice thing is that I can
export the whole beast with samba, and the users can access their
yesterdays or monday's tree transparently without hassle. No need to
disturb operator and to showel tapes just to find out that, gee, we have
no backup of THAT file. Of course, there are certain gotchas, but overall,
it seems to work quite nicely.


Getting Rsync working happily with Secure Shell for UNIX

Also refer: Rsync/ssh backup scripts: ftp://ftp.stearns.org/pub/rsync-backup/

From: "Adye, TJ (Tim) " [T.J.Adye@rl.ac.uk]
To: "'John Horne'" [J.Horne@plymouth.ac.uk]
Cc: "'rsync@samba.org'" [rsync@samba.org]
Subject: RE: SSH with non-default key
Date: Fri, 1 Dec 2000 14:13:33 -0000

Hi John,

> I am trying to work out how I can use rsync with ssh but also 
> specifying the
> key to use. I want to run rsync/ssh via cron but need to use 
> different keys
> from the same account. With just ssh this is not a problem - 
> the -o option takes care of it.

Which version of ssh is this? I use Unix ssh 1.2.x, and the option is -i. I
run rsync with

  rsync -e"ssh -x -a -i IdFile"

(-x -a turns off unnecessary X11 and agent forwarding for speed in setting
up). If you want to specify the identity file as an option with -o, you need
to use -o IdentityFile=IdFile (or if you really like to play with quotes, -o
'IdentityFile IdFile').

Actually I use this so often that I have a wrapper script that allows the -i
option (and several others) to be specified directly and then builds its own
ssh command.

> However, I can find no option to rsync except perhaps by
> using '-e ssh'. Changing this to something like '-e "ssh 
> -o'IdentityFile2
> ~/.ssh/newkey'"' doesn't work (yes, you need to escape some 
> of the quotes).

What's this IdentityFile2 option? That's a new one to me.

> Setting the environment variable didn't seem to work either 
> (I may have got
> things wrong, but it just didn't like it). It seems that 
> rsync keeps seing
> the -o option (or rather the key I specify) and complaining 
> about it.

Tim.

==============================  cut here  ==============================
Tim Adye, BaBar/DELPHI Groups, Particle Physics Dept.,     _   /|
          Rutherford Appleton Laboratory, UK.              \'o.O'   Oop!
e-mail:   T.J.Adye@rl.ac.uk or VXCERN::ADYE (HEPNET/SPAN)  =(___)=  Ack!
WWW:      http://hepwww.rl.ac.uk/Delphi/Adye/homepage.html    U  Thphft!


From: John Horne [J.Horne@plymouth.ac.uk]
To: "rsync@samba.org" [rsync@samba.org]
Subject: RE: SSH with non-default key
Date: Fri, 01 Dec 2000 16:01:58 -0000 (GMT)

On 01-Dec-00 at 14:59:38 John Horne wrote:
> On 01-Dec-00 at 14:13:33 Adye, TJ (Tim) wrote:
>>> I am trying to work out how I can use rsync with ssh but also 
>>> specifying the key to use. I want to run rsync/ssh via cron but need to
>>> use different keys from the same account. With just ssh this is not a
>>> problem - the -o option takes care of it.
>> 
>> Which version of ssh is this? I use Unix ssh 1.2.x, and the option is -i.
>> I run rsync with
>> 
>>   rsync -e"ssh -x -a -i IdFile"
>> 
Okay I've got this fixed now :-) I've been trying to use something like:

  -e "ssh -o'IdentityFile2 ~/.ssh/xxkey'"

for some reason something doesn't like the arrangement of quotes (I have
tried swapping them around but to no effect). I don't know if its rsync, ssh
or the Korn shell I'm using (on a Sun system). Anyway, by using:

  -e "ssh -oIdentityFile2=~/.ssh/xxkey"

this works fine :-) The first form is valid, but as said something doesn't
like it.

Thanks for your help,

John.
PS - for those reading this and use openSSH from cron and kept getting the
'Pseudo-tty will not be allocated...' message back (very annoying), I have
received a patch for this since I pointed out on the developers list that
the '-T' option was supposed to prevent a tty being allocated (and hence the
message was not necessary). Hopefully the next release of openSSH will have
this fixed once and for all :-)

------------------------------------------------------------------------
John Horne, University of Plymouth, UK           Tel: +44 (0)1752 233914
E-mail: jhorne@plymouth.ac.uk
PGP key available from public key servers


From: Dave Dykstra [dwd@bell-labs.com]
To: Ernie [ernie.cline@ipgdirect.com]
Cc: rsync@us5.samba.org
Subject: Re: strange rsync problem
Date: Fri, 1 Dec 2000 15:52:47 -0600

On Fri, Dec 01, 2000 at 01:41:53PM -0500, Ernie wrote:
> >  You can specify an explicit path to rsync on the client side with --rsync-path.
> 
> Oh, duh, i should have remembered that.  I did use the --rsync-path paramter
> with no success.

I presume by "no success" you mean that it still hung, right?  I would
think it should at least have had success finding the remote rsync.  The
rsync-path parameter is really a shell command for whatever login shell
you're using on the remote side so you may have some success debugging it
by using something like
    --rsync-path "set -x; /usr/local/bin/rsync"
or
    --rsync-path "strace -o /tmp/rsync.strace /usr/local/bin/rsync"
You can also turn on ssh debugging with
    -e "ssh -v"


> Hm.  I am running rsync 2.4.6, with OpenSSH 2.3.0p1.  Both boxes have this
> config, and are running linux:


- Dave Dykstra


Why Rsync may not be copying files over

From: David Bolen [db3l@fitlinxx.com]
To: "'John Horne'" [J.Horne@plymouth.ac.uk]
Cc: rsync@us5.samba.org
Subject: RE: Changed file not copied
Date: Fri, 1 Dec 2000 13:03:38 -0500

John Horne [J.Horne@plymouth.ac.uk] writes:

> Okay I think I've solved this :-) The rsync with the '--size-only'
> option updates the modification time for the file but doesn't copy
> it.

Actually, I believe it's less related to "--size-only" rather than
your use of the "-t/--times" option (implicitly since you used "-a").

Thus, "--size-only" says not to bother transferring the file contents
even if the date is different but the size is the same, but the "-t"
still transfers over the appropriate timestamp information.  This can
be a good way to initially sync up two systems that weren't previously
mirrored with maintenance of timestamp information, but otherwise have
the same content.

> Hence when I run the command without that option the size and time
> are equal - hence the file is not copied despite being different. I
> see that there are options to ignore the time as well, so that would
> get round it on the second rsync command. Not a problem, I just
> found it a bit confusing :-)

In addition to ignoring the timestamp ("-I/--ignore-times"), and
depending on your environment and size of the files involved, you can
also use the "-c/--checksum" option to have rsync compute a file
checksum to determine if a file should be transferred.  While this can
be slow, if the actual rsync processing of the file (e.g., block
checksums) and transmission of that information (particularly on a
slow link) would be lengthy, it's a more efficient way to guarantee
you only bother sending a file if it is different.

-- David

/-----------------------------------------------------------------------\
 \               David Bolen            \   E-mail: db3l@fitlinxx.com  /
  |             FitLinxx, Inc.            \  Phone: (203) 708-5192    |
 /  860 Canal Street, Stamford, CT  06902   \  Fax: (203) 316-5150     \
\-----------------------------------------------------------------------/


[Why use Rsync | Where to get Rsync | Compiling Rsync]
[Setting up Rsync Server | Setting up Rsync Client | Daily AutoMirroring Scripts]
[Possibilities for the Future - Ultra High Effeciency Mirroring]
[Rsync for Windows] | [Rsync for Win NT running in Deamon Mode - 1] | [Rsync for Win NT running in Deamon Mode - 2]
[Getting Rsync working with ssh/SecureShell for UNIX] | [Getting Rsync working under Secure-Shell for Windows]

[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]

(This Webpage Page in No Frames Mode)

If you have any queries or comments, please feel free to contact the CCP14