Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Rsync hangs on larger folders (between ext - vfat)
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
kistian
n00b
n00b


Joined: 09 Nov 2015
Posts: 13

PostPosted: Wed Dec 30, 2015 7:16 pm    Post subject: [SOLVED] Rsync hangs on larger folders (between ext - vfat) Reply with quote

I have a problem that I can't solve

when I do an rsync for do a backup, it hangs when the cache covers the total amount of RAM

I have 8Gb of ram

example of I do:
Code:
rsync -aAXv /home/fotos/2014 /data/fotos/2014


rsync does until suddenly hangs, I've been checking the RAM and when fully match full of cache due to rsync

it stills without error log or something, I've tried to rsync dividing a folder to various subfolders and do individuals rsyncs and it works, but, on an entery and large folder it hangs

NOTE: the /home is another disk than /data mount point


Last edited by kistian on Fri Jan 08, 2016 9:21 am; edited 1 time in total
Back to top
View user's profile Send private message
kistian
n00b
n00b


Joined: 09 Nov 2015
Posts: 13

PostPosted: Mon Jan 04, 2016 4:22 pm    Post subject: Reply with quote

nobody knows what's the problem?
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Mon Jan 04, 2016 4:38 pm    Post subject: Reply with quote

kistian wrote:
nobody knows what's the problem?

kistian ... from the above, no ... buffer/cache is available should something require it, so, there is nothing pointing to an issue ... other than the transfer hanging.

You need to provide more information ... rsync version, kernel version, filesystem(s) in use, 'emerge --info', 'free -m', etc, etc.

best ... khay
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Mon Jan 04, 2016 5:06 pm    Post subject: Reply with quote

kistian you use a swap file or swap partition ?

same like khayyam wrote: more information is needed
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Logicien
Veteran
Veteran


Joined: 16 Sep 2005
Posts: 1366
Location: Montréal

PostPosted: Mon Jan 04, 2016 6:06 pm    Post subject: Reply with quote

I often use Rsync. I transfer large files localy and through my local network. Through the network, I use the -z option to compress data, localy no. I put a slash at the end of the paths. Rarely I have problems. So you can try
Code:
rsync -aAXvz /home/fotos/2014/ /data/fotos/2014/

or like this for more informations
Code:
rsync -aAXvz --delete-after --progress --stats /home/fotos/2014/ /data/fotos/2014/

Compressing data may reduce the cache used and of course reduce the amount of data transfer between the source and the destination medias. If one of the disks is faster than the other, it have to wait. It's cache can come full as the source of the transfer. Just a though probably not related with the cache you are talking about but, if one of the disks or the two stop to respond, Rsync will stop too eventually.
_________________
Paul
Back to top
View user's profile Send private message
kistian
n00b
n00b


Joined: 09 Nov 2015
Posts: 13

PostPosted: Tue Jan 05, 2016 12:44 pm    Post subject: Reply with quote

thanks to all for your help

the rsync usually hangs when the RAM cames with unaviable space, yes, I have Swap partition, this is the free befor starts rsync and on the hangs moment (or 10 second later), it hangs on 11,3 Gb transfered (27G total)
before rsync:
Code:
gentoo kistian # free -h
              total        used        free      shared  buff/cache   available
Mem:           7,8G        1,5G        1,2G        327M        5,0G        5,9G
Swap:          3,7G         49M        3,7G

on hangs:
Code:
gentoo kistian # free -h
              total        used        free      shared  buff/cache   available
Mem:           7,8G        1,4G        120M        329M        6,2G        6,0G
Swap:          3,7G         49M        3,7G


my kernel is: 4.1.12-gentoo
rsync version: 3.1.2 (i've tested the 3.1.1 and has the same result)

Code:
gentoo kistian # du -bsh /home/kistian/fotos/2014/
27G     /home/kistian/fotos/2014/


filesystems in use (mount -l):
Code:
/dev/sda2 on / type ext4 (rw,noatime,data=ordered)
devtmpfs on /dev type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1022236,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run type tmpfs (rw,nodev,relatime,size=817904k,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
/dev/sda1 on /boot type ext3 (rw,noatime,data=ordered)
/dev/sdb1 on /home type ext3 (rw,noatime,data=ordered) [home]
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
/dev/sda9 on /data type vfat (rw,nosuid,nodev,noexec,noatime,fmask=0000,dmask=0000,allow_utime=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)


emerge --info
Code:
Portage 2.2.24 (python 3.4.3-final-0, default/linux/amd64/13.0/desktop/plasma, gcc-4.9.3, glibc-2.21-r1, 4.1.12-gentoo x86_64)
=================================================================
System uname: Linux-4.1.12-gentoo-x86_64-Intel-R-_Core-TM-_i5_CPU_650_@_3.20GHz-with-gentoo-2.2
KiB Mem:     8179004 total,     64680 free
KiB Swap:    3905532 total,   3855232 free
Timestamp of repository gentoo: Mon, 04 Jan 2016 16:00:01 +0000
sh bash 4.3_p42
ld GNU ld (Gentoo 2.25.1 p1.1) 2.25.1
app-shells/bash:          4.3_p42::gentoo
dev-java/java-config:     2.2.0::gentoo
dev-lang/perl:            5.20.2::gentoo
dev-lang/python:          2.7.10-r1::gentoo, 3.4.3::gentoo
dev-util/cmake:           3.3.1-r1::gentoo
dev-util/pkgconfig:       0.28-r2::gentoo
sys-apps/baselayout:      2.2::gentoo
sys-apps/openrc:          0.18.4::gentoo
sys-apps/sandbox:         2.6-r1::gentoo
sys-devel/autoconf:       2.13::gentoo, 2.69::gentoo
sys-devel/automake:       1.11.6-r1::gentoo, 1.14.1::gentoo, 1.15::gentoo
sys-devel/binutils:       2.25.1-r1::gentoo
sys-devel/gcc:            4.9.3::gentoo
sys-devel/gcc-config:     1.7.3::gentoo
sys-devel/libtool:        2.4.6::gentoo
sys-devel/make:           4.1-r1::gentoo
sys-kernel/linux-headers: 3.18::gentoo (virtual/os-headers)
sys-libs/glibc:           2.21-r1::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

kde
    location: /var/lib/layman/kde
    masters: gentoo
    priority: 1

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt /usr/share/themes/oxygen-gtk/gtk-2.0"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=native -O2 -pipe"
DISTDIR="/usr/portage/distfiles"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://distfiles.gentoo.org"
LANG="en_US.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j5"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli consolekit cracklib crypt cups cxx dbus declarative dri dts dvd dvdr emboss encode exif fam firefox flac fortran gdbm gif glamor gpm gtk iconv ipv6 jpeg kde kipi lcms ldap libnotify mad mmx mmxext mng modules mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf phonon plasma png policykit ppds qml qt3support qt4 qt5 readline sdl seccomp semantic-desktop session spell sse sse2 ssl startup-notification svg tcpd tiff truetype udev udisks unicode upower usb vorbis widgets wxwidgets x264 xattr xcb xcomposite xinerama xml xscreensaver xv xvid zlib" ABI_X86="64" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="kexi words flow plan sheets stage tables krita karbon braindump author" CAMERAS="ptp2" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes mmx mmxext popcnt sse sse2 sse3 sse4_1 sse4_2 ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LINGUAS="es en" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-5" PYTHON_SINGLE_TARGET="python2_7" PYTHON_TARGETS="python2_7 python3_4" RUBY_TARGETS="ruby20 ruby21" SANE_BACKENDS="hp" USERLAND="GNU" VIDEO_CARDS="nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, USE_PYTHON


at this time, i've tried to do an rsync with compression (z value) I think it stills working, but slower, later I see if it hangs and write here the result
Back to top
View user's profile Send private message
kistian
n00b
n00b


Joined: 09 Nov 2015
Posts: 13

PostPosted: Tue Jan 05, 2016 3:41 pm    Post subject: Reply with quote

using z value it hangs also
Back to top
View user's profile Send private message
khayyam
Watchman
Watchman


Joined: 07 Jun 2012
Posts: 6228
Location: Room 101

PostPosted: Tue Jan 05, 2016 3:45 pm    Post subject: Reply with quote

kistian wrote:
the rsync usually hangs when the RAM cames with unaviable space, yes, I have Swap partition, this is the free befor starts rsync and on the hangs moment (or 10 second later), it hangs on 11,3 Gb transfered (27G total)

before rsync:
Code:
gentoo kistian # free -h
              total        used        free      shared  buff/cache   available
Mem:           7,8G        1,5G        1,2G        327M        5,0G        5,9G
Swap:          3,7G         49M        3,7G


on hangs:
Code:
gentoo kistian # free -h
              total        used        free      shared  buff/cache   available
Mem:           7,8G        1,4G        120M        329M        6,2G        6,0G
Swap:          3,7G         49M        3,7G

kistian ... there is nothing wrong with the above ... note how only 1.4G is in use, and 6.0G is 'available'. The buffer/cache is what you'd expect, as "an mb of unused ram is wasted ram", it'd be freed if needed.

kistian wrote:
my kernel is: 4.1.12-gentoo

I wouldn't expect that such an early revision of a major release to be without a fair number of bugs ... if you must use the 4.x series then I'd suggest > 4.3.3 (and updating as and when the kernel is bumped), otherwise use 3.18.x.

kistian wrote:
mount -l:
/dev/sda9 on /data type vfat

I'd imagine this will be the cause ... have you tried to reproduce with another filesystem (such as ext4)?

best ... khay
Back to top
View user's profile Send private message
Syl20
Guru
Guru


Joined: 04 Aug 2005
Posts: 564
Location: France

PostPosted: Tue Jan 05, 2016 5:43 pm    Post subject: Reply with quote

khayyam wrote:
kistian wrote:
mount -l:
/dev/sda9 on /data type vfat

I'd imagine this will be the cause ... have you tried to reproduce with another filesystem (such as ext4)?

VFAT doesn't support permissions. Rsync will probably be more stable if you don't use -p, -o, -g (all implied by -a), -X and -A options.
Back to top
View user's profile Send private message
kistian
n00b
n00b


Joined: 09 Nov 2015
Posts: 13

PostPosted: Fri Jan 08, 2016 9:20 am    Post subject: Reply with quote

CneGroumF wrote:
khayyam wrote:
kistian wrote:
mount -l:
/dev/sda9 on /data type vfat

I'd imagine this will be the cause ... have you tried to reproduce with another filesystem (such as ext4)?

VFAT doesn't support permissions. Rsync will probably be more stable if you don't use -p, -o, -g (all implied by -a), -X and -A options.


yes! that's the problem!

using an rsync -vr it works!


thanks for all, I change the topic and add a "solved" tag
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum