Question

24

[root@centos /]# rsync -av --exclude thumbs /storage root@xx.27.1.xx:/storage
root@xx.27.1.xx's password: 
building file list ...

I've been sitting for about an hour... it's 135GB of images and folders

/storage is a mounted ext3 scsi drive.

Is it normal for rsync to be sitting this long calculating the files/directories?

Tombart

2,5614 gold badges30 silver badges54 bronze badges

asked Dec 2, 2010 at 19:14

6

Answer 1

28

rsync 2.x does build a full file list up front.

Add the -P option and you'll see a progress indicator.

If you think it really is hanging, in another terminal find the pid of rsync and then

Run strace -p PID and see what it's doing. (Hit ^C to stop.)
Run ps -o wchan PID to see where it is in the kernel.

answered Dec 2, 2010 at 22:54

4

Answer 2

16

You should upgrade to rsync 3.0.x where you will get the benefit of incremental file lists, explained here. I am rsyncing millions of images (~200GB total) and saw an enormous speedup when going from rsync 2.x to 3.x.

Even so, it will probably take a long time to go through all that data. In my case it still takes over an hour between two pretty powerful DL380 G5 servers.

answered Dec 2, 2010 at 22:58

2

Answer 3

3

You noted earlier that du was hanging as well. Something is up with this filesystem, and rsync is hitting whatever du was.

One quick thing you can check is for kernel error messages indicating a disk problem. Type dmesg and see what's at the bottom.

answered Dec 2, 2010 at 19:46

Answer 4

1

Based on this and your other question of du /storage hanging makes me think that neither du or rsync is the problem but rather there are some issues with /storage.

answered Dec 2, 2010 at 23:42

Answer 5

1

The hang after the message "building file list..." can be caused by MTU mismatch, e.g. if you've set up the network interfaces with MTU 9000 but not the switch in between, small packets (like connecting to the rsync server) will pass through but not bigger ones (like sending the file list).

answered May 18, 2018 at 9:00

1

You must log in to answer this question.