[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]
(This Webpage Page in No Frames Mode)

Collaborative Computational Project Number 14

for Single Crystal and Powder Diffraction

CCP14

Server Security Information

Secure Routine Windows to UNIX Web updating using tunnelling via Teraterm and Rsync

The CCP14 Homepage is at http://www.ccp14.ac.uk
[Back to CCP14 Web/Config Main Page]
[Security Links Homepage] | [To FTP secure shell Tunnelling Page] | [To X secure shell Tunnelling] | [Routine Windows to UNIX Web updating using Rsync] | [Secure Routine Windows to UNIX Web updating using tunnelling via Teraterm and Rsync]

What the point of this?

The point of this is to be able do very easy web updating (even over tortuously slow phone lines) in a convenient, timely, effective and secure manner. FTP can be quite painful in this regard but Rsync can be superb. If you change a line of a 500kB ASCII file, rsync will only need to send around 2kB of the "changes" over - and it will compress, and run via secure shell if desired. Combined with the easy to setup teraterm, rsync can be tunnelled so that nothing can be sniffed over the network.

The following is based around the excellent "Installing ssh and rsync on a Windows machine" by Robert Scholten at http://optics.ph.unimelb.edu.au/help/rsync/

However, a big difference is that this goes via "teraterm/tssh" as I could not get the default compiled ssh to work.

Notes:

Relevant pages:

Installing the Rsync deamon on the webserver

Install and configure Rsync on your UNIX (VMS?) webserver. You can use the following information.

Methods may vary, but put in a user specific config in the /etc/rsyncd.conf file. One for each user. The other trick is to have the hosts allow for the webserver - as this is where the connections will look like they are coming from due to the tunnelling. 

motd file = /etc/motd
     max connections = 25
     syslog facility = local3
#  Normally you would set read only = yes - but we want the user to be
#     able to write files to his area.

[joe_blogs_web]
   comment = updating Joe Blogs's webpage via rsync
   path = /path/to/web_area/for/joe
   read only = no
   list = yes
   uid = real_user_id
   gid = real_user_group
   hosts allow = www.webserver.ac.uk
   auth users = rsync_specific_username
   secrets file = /etc/rsyncd.secrets

NOTE: If you change the rsyncd.secrets file, then (in inetd mode) REHUP Inetd on the server using /etc/killall -HUP inetd to get this changed noticed. Otherwise, you may get an auth error, even if you are inserting the correct rsync password.

Downloading and installing rsync for Windows

Look at the instructions at Robert Scholten's "Installing ssh and rsync on a Windows machine" by Robert Scholten at http://optics.ph.unimelb.edu.au/help/rsync/

Download the rsync/ssh package from http://optics.ph.unimelb.edu.au/help/rsync/rsync_package.zip

Extra the zip so that it creates subdirectories in the c:\etc and c:\rsync area. This is important if you are trying to use the secureshell option that comes with this distribution of "rsync for Windows". If you extract these into other directories such as c:\rsync\etc and c:\rsync\sync, the following secure shell part will not work. This is to take into account some UNIX nuances of this particular Windows implementation of rsync.

In the c:\rsync directory, rename one of the rsync_version.exe to rsync.exe. At this date, I used the rsync2.4.3.exe version. If you get problems in file transfers, you may have to try another version.

ren rsync2.4.3.exe rsync.exe

Installing Teraterm and tssh for windows

(easy to do and it runs via a setup program)

Install the tssh secureshell plugin for Teraterm. (download the zip file and extract the files into the teraterm directory)

Customising Teraterm and tssh for windows

This might be a good time to customise teraterm and have the screen font, size, list of machines you commonly connect to, etc that can make you happy. Select setup, save setup to save these as the defaults. (it is also possible to manually edit the teraterm.ini file in the teraterm executable directory if you wish)

Customising Teraterm

Setting up the rsync tunnelling in teraterm for windows

Now to setup the Port Forwarding that allows FTP to go via the Secure Shell port.

Go into the Setup, SSH Forwarding whereby you should see the following window ready to have the relevant information inserted into it. (you can see that this is already setup for FTP tunnelling and X-session tunnelling)

SSH Forwarding Window

Using the Add icon, we want to tell teraterm to forward/tunnel the rsync port (port 873) (using connecting to the www.ccp14.ac.uk server as an example) by adding port 873 and the machine we wish to rsync to in the Forward Local Port (port 873) area of the window.

Telling Teraterm to forward/tunnel the FTP control port

This gives the following SSH Forwarding setup that we should save as a Teraterm default by doing Setup, Save Setup and select the teraterm.ini file.

SSH Forwarding Window with setup information

Setting up rsync command file

The method to use to start with is to always try this with --dry-run option first to see what will happen. It is possible to do nasty things with rsync - it will obey what you tell it to do (which may not be what you think you requested). Also, don't forget to turn off the --dry-run option when you want the files to actually transfer across.

Another hint is don't use the --delete option. One tired move with this option and you may accidentally delete a portion of the web-area that you did not mean to. If you need to elliminate files, it might be best to do this using a GUI FTP program.

Following is an example DOS style bat batch file to run various options depending on what is required. (it has a lot of the common tools "REMMED" ready for cut and paste) Once you get into a routine, you may create you own discrete batch files. If doing a set of work on a certain directory system, extend a command line for that directory. --exclude "*.pfe" is the backup file name that the freeware PFE for Windows text editor creates when editing the HTM and HTML file. So we don't want to transfer the backup files over. The "\\d/" stands for D: drive on the hard-disk.

We specify the rsync area we want to update (in this case, joe_blogs_web), the rsync specific user name, and the option.

The remaining trick is to specify 127.0.0.1 (localhost) as the machine we want to connect to. This tells Teraterm to use the tunnelling.

@echo off
REM c:\rsync\rsync -av --delete "\\d/" torte:/rsync/data2/scholten/schizo/d >>c:\rsync\rsync.log
REM netjam --progress --safe-links --dry-run
REM http://optics.ph.unimelb.edu.au/help/rsync/#install
REM http://www.ccp14.ac.uk/ccp14admin/rsync/index.html --exclude "*.gif" --exclude "*.jpg"
set CYGWIN=tty binmode
set TERM=ansi
REM set RSYNC_PASSWORD=plaintextpassword
REM >>c:\rsync\rsync.log
set RSYNC_RSH=c:\rsync\ssh.exe
REM echo Drive D >>c:\rsync\rsync.log

c:\rsync\gnudate

REM Books test
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --delete --stats -e ssh -z "\\d/lachlan/web/books/" rsync_user_name@127.0.0.1::joe_blogs_web/books/

REM root area HTML changes (what's new). Currently running
c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --exclude "*.gif" --exclude "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@127.0.0.1::joe_blogs_web/

REM Images test
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/images/" rsync_user_name@127.0.0.1::joe_blogs_web/images/

REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@127.0.0.1::joe_blogs_web/

REM Complete update
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@127.0.0.1::joe_blogs_web/

REM Root area and tutorial area
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/tutorial/" --include "/tutorial/**" --exclude "**" --stats -e ssh -z "\\c/lachlan/ccp14/web_live/" rsync_user_name@127.0.0.1::joe_blogs_web/

REM Root area and tutorial directory and no subdirectories
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/tutorial/" --exclude "**" --stats -e ssh -z "\\c/lachlan/ccp14/web_live/" rsync_user_name@127.0.0.1::joe_blogs_web/

REM ccp14admin security area + rsync + ccp14admin index page
REM c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.txt" --include "*.zip" --include "*gz" --include "*Z" --include "*.html" --include "*.htm" --include "*.gif" --include "*.jpg" --include "/rsync/" --include "/rsync/**" --include "/security/" --include "/security/**" --exclude "**" --stats -z "\\c/lachlan/ccp14/web_live/ccp14admin/" rsync_user_name@127.0.0.1::joe_blogs_web/ccp14admin/

c:\rsync\gnudate

Running Rsync

In this example, we have modified files (this tutorial) in ccp14admin araa, So the DOS style BATCH program we run has an rsync command line that only updates the HTML files in the root web-area, and not in any subdirectory.

c:\rsync\rsync -rlpt -vv --exclude "*.pfe" --include "*.html" --include "*.htm" --exclude "*.gif" --exclude "*.jpg" --exclude "**" --stats -e ssh -z "\\d/lachlan/web/" rsync_user_name@127.0.0.1::joe_blogs_web/

Rsync will then prompt for the "rsync" password corresponding to this username and the transfer will continue.

You can avoid having to type in a password by using the following declaration in your batch file (but this is not a good habit to get into):

  • set RSYNC_PASSWORD=plaintextpassword

In the following screen dump, rsync has determined which files have been altered and is loading up the "differences".

Rsync running

To repeat the uploading exercise, have a DOS window open with doskey installed, and just press the "up arrow" key and press enter to run the previous command.

In the following screen dump, rsync has started, prompted for the password (which the user enters); then determins what has changed, and updates the file. Note that even though the whatsnew.htm file is 47 kB, rsync only had to transfer 808 bytes to update the remote file on the web server to match the local version. This means it takes far less time and bandwidth.

Rsync running

[Back to CCP14 Web/Config Main Page]
[Security Links Homepage] | [To FTP secure shell Tunnelling Page] | [To X secure shell Tunnelling] | [Routine Windows to UNIX Web updating using Rsync] | [Secure Routine Windows to UNIX Web updating using Teraterm and Rsync]
[CCP14 Home: (Frames | No Frames)]
CCP14 Mirrors: [UK] | [CA] | [US] | [AU]
(This Webpage Page in No Frames Mode)
If you have any queries or comments, please feel free to contact the CCP14