Rsync Synchronize Directories One Way

  1. Rsync Two Way Sync
  2. Rsync Synchronize Directories One Way Transfer
  3. Rsync Sync Two Directories
  4. Rsync Directories Linux
  5. Rsync Directory Sync

Rsync is a one way sync and will not alter the source. You can safely use a userid which can only read the source file, but has no write privileges. However, this is not necessary. Rsync will recover any partially completed transfers when you rerun it. Rsync will only delete files on the destinations if you use one of delete options. The above cron and rsync command simply syncing “/var/www/” from the main web server to a backup server in every 5 minutes. You can change the time and folder location configuration according to your needs. To be more creative and customize with Rsync and Cron command, you can check out our more detailed articles at. Rsync is an open source software that can be used to synchronize files/folders from local to local computers, or local to remote computers and vice versa. The notable feature of the rsync is we can transfer encrypted files using SSH, and SSL encrypted transfer can be done using Stunnel.

Rsync is a command-line tool in Linux that is used to copy files from a source location to a destination location. You can copy files, directories, and entire file system and keep in sync the files between different directories. It does more than just copying the files. It reduces the amount of data sent over the network by only sending the files which are new or updated. That’s why is it considered a great utility for file copying and backing up. Rsync also supports the copying of files to a remote system over SSH.

In this article, we will explain how to use rsync to copy files with some practical examples. We will also explain the use of rsync in the copying of a single file, multiple files, and directory to both local and remote systems. We will be covering how to use Rsync to:

  • Copy file/directory within the local system
  • Copy file/directory from local to the remote system
  • Copy file/directory from the remote system to local

We have explained the commands and procedure mentioned in this article on a Ubuntu 20.04 LTS. You will need the command-line Terminal for copying files using rsync. To open the command line Terminal application, use the Ctrl+Alt+T keyboard shortcut.

Rsync sync two directories

Installing Rsync

Rsync comes pre-installed in Ubuntu 20.04 LTS. However, if it is accidentally removed from the system, you can install it as follows:

Here are some of the commonly used options that can be used with rsync:

–a: archive mode
–v: shows details of the copying process
–p: shows the progress bar
–r: copies data recursively
–z: compresses data
–q: suppress output

Copy files and directories locally

The general syntax of rsync is:

Copy a single file locally

If you want to copy a file from one location to another within your system, you can do so by typing rsync followed by the source file name and the destination directory.

For instance, to copy a single file “file1.txt” from the current location that is Home directory to ~/Documents directory, the command would be:

Note: Instead of “/home/tin/file1.txt”, we can also type “file1” as we are currently working in the home directory. Also, we can use ~/Documents instead of mentioning the full path as “/home/tin/Documents”.

Copy multiple files locally

If you want to copy multiple files at once from one location to another within your system, you can do so by typing rsync followed by source files name and the destination directory.

For instance, to copy the files “file2.txt” and “file3.txt” from the Home directory to ~/Documents directory, the command would be:

$ rsync /home/tin/file2.txt /home/tin/file3.txt /home/tin/Documents

To copy all the files with the same extension, you can use asterisk(*) symbol instead of individually specifying the file names. For instance, to copy all the files from the Home directory ending with “.zip” extension to the ~/Documents directory, we will use:

Copy directories locally

If you want to copy a directory with its sub-directory and all contents from one location to another within your system, you can do so as by typing rsync followed by the source and destination directory.

For instance, to copy the “sample” directory to “test/” directory within the Home directory, the command would be:

Note: Specifying “/” after the source directory only copies the contents of the directory. If we do not specify the “/”after the source directory, the source directory will also be copied to the destination directory.

Like in the following output, you can see that our source directory “sample” is also copied to the destination directory (as we have used “sample” instead of “sample/” as a source directory).

Rsync supports incremental transfer as you can see in the above output that it only transfers the files which are new or updated.

One

Rsync Two Way Sync

Copy files based on maximum size

While copying, we can also specify the maximum size of files that can be copied with the “–max-size” option. For instance, to copy the files no larger than 2000k from ~/Downloads to ~/Documents directory, the command would be:

$ rsync -a--max-size=2000k /home/tin/Downloads/*/home/tin/Documents

This command will copy all the files from ~/Downloads to ~/Documents directory except the ones larger than 2000k.

Copy files based on minimum size

Similarly, we can also specify the minimum size of files that can be copied with the “–min-size” option. For instance, to copy the files not lesser than 5M from ~/Downloads to ~/Documents directory, the command would be:

$ rsync -a--min-size=5M /home/tin/Downloads//home/tin/Documents

This command will copy all the files from ~/Downloads to the ~/Documents directory except the ones lesser than 5M.

Exclude files

While copying files from one directory to another, you can exclude some files using “—exclude” option followed by the file name or the extension of the file type.

For instance, to exclude all the .zip files while copying files from ~/Downloads to ~/Documents directory, the command would be:

$ rsync -a--exclude='*.zip'/home/tin/Downloads//home/tin/Documents

Copy files and directories remotely

With Rsync, you can copy a single file, multiple files, and directories to a remote system. For copying files and directories remotely, you will require:

  • Rsync installed on both local and remote system
  • SSH access to the remote system
  • Remote user password
Rsync Synchronize Directories One Way

Before copying files using rsync, make sure you can access the remote system via SSH:


The general syntax for copying files and directories to a remote system using rsync:

$ rsync [option][source][remoteuser@remoteip:destination]

For instance, to copy a single file “file.txt” from ~/Downloads to remote system ~/Documents directory, the command would be:

$ rsync -v ~/Downloads/file1.txt umara@192.168.72.164:~/Documents

Similarly, to copy the “~/Downloads/files” directory in a local system with its sub-directory and all contents to the “~/Downloads/samples” directory at the remote system:

$ rsync -rv ~/Downloads/files umara@192.168.72.164:~/Downloads/samples

You can also copy the files and directories from the remote machine to your local machine. For instance, to copy a “file2.txt” from the remote system to the local system desktop:

$ rsync -v umara@192.168.72.164:~/Downloads/file2.txt ~/Desktop/

Similarly, to copy a directory from the remote system to the local system

$ rsync -rv umara@192.168.72.164:~/Downloads/sample ~/Desktop/test

You can also use all the options like –max-size, –min-size, –exclude and others while copying files to/from the remote system.

Rsync is a great tool for copying files from source to destination either locally or remotely. It includes a large set of options that gives you flexibility and control while transferring files to/from the remote system.

Rsync

Rsync Synchronize Directories One Way Transfer

Synchronizing files

For those people with more than one computer, synchronizing them is a common problem. How do you keep up-to-date, how do you make sure you've got the most recent changes you made, and most importantly of all, how do you avoid accidentally copying the wrong version of a file and losing hours or days of work?

When I just had a laptop, this was no problem - all the files were on there. Then it sadly died, and I got the Shuttle desktop - also no problem since all the files were on there. Now I've got this new netbook, and the problem has suddenly become real. How can I work on files on the netbook and make sure that I don't have to spend hours copying them (and just them!) back onto the desktop?

There are of course many products on the market to help with this problem, it's the same problem experienced with synchronizing appointments and contact information on mobile phones, for example. And a similar problem to that of organising off-site backups of recently-changed files. In this page we'll have a look at a command line tool called rsync.

Obviously the end goal is to set up a synchronization process so that it automatically does its stuff over the network. It should be fast, efficient, and secure. But in order to get there we'll need a few baby steps first, and this means using rsync to synchronize the contents of two folders on the same machine. Once that's done, then we can look at the networking aspects later.

Locally running rsync

In order to demonstrate how rsync works in a simple way, we'll construct two directories of files on the same machine, and use rsync to synchronize them. Here's an outline of the two directories. We can imagine that in the future, one directory will be on a desktop machine and the other will be on a laptop or netbook. Files have been edited on both machines and we want to efficiently synchronize them so we've got the latest versions on both systems and (hopefully) don't lose work.

Desktopfile1.txt
This is the main version of the first file.
It's quite simple.
file2.txt
This is the second file.
This won't be modified by either side.
file3.txt
This one will be changed on the host but not on the laptop.
See, now the main copy has been changed on the desktop.
file4.txtfile5.txt
This file only exists on the host, so it's like it's been deleted from the second set.
Laptopfile1.txt
This is the second version of the first file.
It's now been edited, with some extra text added to it.
file2.txt
This is the second file.
This won't be modified by either side.
file3.txt
This one will be changed on the host but not on the laptop.
file4.txt
file5.txt

What's important to note here is that rsync is not a version control system. So if we compare two files, rsync will take the later of them. It won't be able to check whether both files have been edited or not - if this is the case then the one which was saved more recently wins, and any of the edits in the other version will be lost.

Similarly, if we look at file5.txt, rsync will not be able to tell whether the file was created on the desktop, or whether it used to be on both systems and has now been deleted from the laptop. So there's no way for rsync to know whether to delete the one on the desktop, or copy it to the laptop. Obviously, copying to the laptop is the safer thing to do, but then that might be frustrating if you explicitly wanted to delete it and it comes back again (and again) after each synchronize. But that's the price you pay for using a synchronizing tool instead of a version control tool. Pick what you want to use.

Ok, now we've got our two directories, what do we want to do? Let's synchronize in both directions, firstly from the laptop to the desktop, and then from the desktop to the laptop. In this case the order doesn't really matter. The options -tuv here specify to preserve timestamps, only update files (copy if newer) and be verbose about it.

Note that file1.txt (edited on the laptop) and file4.txt (created on the laptop) are successfully copied to the desktop. Also, file3.txt (edited later on the desktop) and file5.txt (created on the desktop / deleted on the laptop) are copied to the laptop. Also note that file2.txt was recognised to be the same on both and so wasn't transferred at all. The whole thing is done using clever comparison algorithms and checksums so that the transfers are very efficient. They can also be compressed too, if you want.

Obviously, if you really did want to delete file5.txt, you'd have to manually delete it on both the laptop and the desktop before the next time rsync is run.

Additional options

The commands shown above only synchronize the files in the given directories, but none of their subdirectories. Fortunately, this is easily done by adding the option r to the commands, and then all subdirectories will be properly handled.

But perhaps the files in one of the directories are under cvs or subversion, in which case there will be lots of other little files there which don't need synchronizing. You can easily omit them from consideration by simply adding the option C as well (that's a capital letter C).

Instead of just synchronizing every file, we could say we only want to synchronize .txt files, simply by putting laptop/*.txt instead. However, that would stop us from doing a recursive synchronize, with the option r, because the subdirectories wouldn't match our filter of *.txt - irritating. Instead you'd have to work with a more complicated and awkward set of include and exclude filters, such as:This could also be useful for specifying just java files or just html files, for example.

Another useful option while you're experimenting with rsync is a so-called 'dry run', which doesn't actually copy any files, but just confirms what would be synchronized if it were run properly. This option can be called with --dry-run or in short form -n and is especially useful with the v option for verbose output. If the output is what you expect, then can simply repeat the command without the n option.

TODO: rsync between machines with a USB stick

Over the network

Of course, if all rsync could do was synchronize folders on the same machine, it wouldn't solve our problem with the laptop and the desktop. Somehow we've got to get it working over the network. And in this particular case it's going to be a wireless network.

There are some intermediate steps to go through first though, to make sure that our synchronization will be secure and simple.

Setting up ssh

Rsync sync two directories

We'll use ssh (secure shell) to connect from the laptop to the desktop. The laptop already has the openssh-client installed by default, so all we need to do is install the ssh server on the desktop. The package we need is conveniently called ssh so aptitude can get it simply from the repositories.

To test this out, we try the following from the laptop:

Where here obviously the user and the IP address must match the details of the server. If your username is the same on both machines then you don't need to give it here, just the IP address will do. Then simply enter the password for that user on that machine, and you're in with a console prompt - you can list files and directories as if you were sitting in front of the other machine, and of course edit them using a console editor such as vi.

At this point it doesn't seem too impressive - it's not much different from having access to a shared network drive which is old news. However, with ssh you really are logged into the remote machine, and can even start programs, for example try playing an ogg file with totem from the command line - what do you think will happen?

A neat trick is to pipe the graphical output from the X server on the remote machine. This needs the -X option when starting ssh, so the command would look like this:

Then you can start for example OpenOffice Writer (oowriter) from the console, and even though it's not even installed on this laptop, it runs on the remote machine and shows the display on the laptop. Very clever. When you save the file, of course, because it's running on the remote machine it'll save it to the file system on the remote machine too.

Using scp

Accessing the desktop from the command line is great, but what about copying files from one machine to the other? That's the next step, using scp or 'secure copy':

Rsync Sync Two Directories

The previous command copies a single file from the local machine (in this case the laptop) onto the specified remote machine (here the desktop). The new file will be created in the home directory in this example.

A second example shows copying a file from the other machine into the current directory on the local machine - in this case from the desktop onto this laptop:

Fixing computer names

Up until now we've referred to the other machine using its IP address, which is a bit clumsy. First we have to find out what this address is, and because it's obtained via DHCP from the router, it might change next time we try the same command. So it's good to have a way to refer to the machines by name instead.

One way to solve this is to configure the desktop with a fixed IP instead of using DHCP. Then this fixed IP could be stored in the laptop and associated with a memorable name. This is a little ugly though, because as soon as you get more than one fixed IP address, you have to worry about multiple configuration places and possibility of conflicts.

A more elegant solution is to go into the router config, and set up an IP address 'reservation' so that it still uses DHCP, but the router recognises the MAC address of the desktop and always gives it the same, specified ip address. So the IP address configuration is in one place, on the router, and other clients can still use DHCP without conflicts.

You still need to let the laptop know about this though, so you need to add a line to its /etc/hosts file specifying the IP address and its associated name. You can then ping and ssh just using the name in the hosts file. This should be the same as the hostname of the other machine for convenience but this isn't required.

So the ssh and scp commands now look like this:

Now we're nearly there, we just need to run rsync over ssh and we've got our synchronization working!

rsync over ssh

Rsync Directories Linux

For this we just need to add the options -e ssh to specify that the rsync connection should go via ssh to the other machine. Then we specify the paths as before to go in both directions:

Using an authentication key

When using scp or rsync over ssh, obviously you need to authenticate yourself somehow, and the default way to do this is using a password. But if you're doing a lot of copying and rsyncing, typing in the same password dozens of times can get annoying. Instead, you can set up an authentication key, so that you're automatically authenticated without having to enter a password.

The basic way it works, is that on the client, you generate a two-part key, consisting of a private part and a public part. The private part stays on the client machine, but the public part (which isn't secret) gets copied to the host machine. In our case that's the netbook being the client and the desktop being the host. It sounds a little risky to remove the password protection on the desktop, but with a little thought it's clear that it's not a problem.

Only the netbook has the private part of the key, so if somebody hasn't got the netbook, they can't get into the host without a password. The other users of the netbook haven't got read rights to get to the private key so they can't use it either. And if the netbook is stolen, you can simply revoke the authentication by removing the public key from the desktop. Only if the netbook is silently completely compromised does it pose a problem for the desktop, but then there are more serious implications if that happens. They'd probably have difficulties getting into the wireless network anyway, as the desktop isn't visible outside.

You may also wonder why it's not a problem for somebody else to initiate such a scheme and get themselves automatically authenticated like this. The trick is that it requires somebody to be able to plant their public key in the user home of the desktop. And if they can do that, then they're already in the desktop anyway.

So how to set it up? Basically it's as simple as running a command ssh-keygen on the client machine, and then copying (for example, using scp) the public keyfile across to the host's home directory. There's a concise walkthrough at Wikipedia, and everything else you need is in man ssh-keygen.

And tada! No more password required for ssh, or scp, or rsync ! Result!

Diff & Patch >

Page last updated: October 2012

Rsync Directory Sync