Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Copying to USB NTFS kills firefox responsiveness
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Mon Dec 26, 2011 11:57 am    Post subject: Copying to USB NTFS kills firefox responsiveness Reply with quote

Whilst backing up my photo collection to a spare drive I had lying around, Using 'cp -vur source /mnt/disk/'

Formatted to NTFS, using FUSE ntfs-3g, connected via a 7 port USB 2 hub. I have 8G of memory with a 64 bit kernel (Linux lyalls-pc 3.0.6-gentoo #1 SMP Tue Oct 18 20:30:00 CST 2011 x86_64 Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz GenuineIntel GNU/Linux)

Copying goes swimmingly, for a while, until my 8G of RAM is soaked up by system buffers, then, whilst the USB disk light flashes often, and TOP shows IO waits in the order of 50%, Idle time in the order of no less than 50%, firefox stops responding. So badly, that I have to kill the copy, wait for the buffers to flush out (which takes quite some time) before firefox resumes responding. (even the animated emoticons on the left of this post stopped animating).

Other apps seem to be fine, thunderbird, for instance.

Any suggestions as to how I diagnose what the problem is?

I have full debug symbols in my entire system (split debug is a wonderful thing).

I tried running firefox with strace, it spends it's entire life polling, except when this 'seizure' starts, when I see mmap, followed by an munmap with an invalid argument, followed by another munmap. This mapping and unmapping is just constant. Additionally a few 'futex' calls seem to have fun also.
Code:
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f480000
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f470000
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f460000
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f450000
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff823300000
munmap(0x7ff823300000, 0)               = -1 EINVAL (Invalid argument)
munmap(0x7ff823400000, 1048576)         = 0
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f440000
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f430000
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f420000
mmap(NULL, 65536, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff83f410000
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ff823100000
munmap(0x7ff823100000, 0)               = -1 EINVAL (Invalid argument)
munmap(0x7ff823200000, 1048576)         = 0


Anyway, advice on how I go about diagnosing this problem would be appreciated.
_________________
...Lyall
Back to top
View user's profile Send private message
SlashBeast
Moderator
Moderator


Joined: 23 May 2006
Posts: 2843

PostPosted: Mon Dec 26, 2011 12:35 pm    Post subject: Reply with quote

The problem is that your system buffer a lot of data in ram and flush them all when you it hit X Megabytes.

Try if it fix the issue for you:
Code:
sysctl -w vm.dirty_bytes=52428800
sysctl -w vm.dirty_background_bytes=26214400


It will start flushing when there is 25M written data in cache and force flush when there is 50M. Default values are 10% of installed ram and 20% of installed ram.
_________________
BitBucket -- better-initramfs
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Mon Dec 26, 2011 2:32 pm    Post subject: Reply with quote

@SlashBeast:

thanks for those settings ! :)

gotta see if that also makes things more responsive for me
_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
SlashBeast
Moderator
Moderator


Joined: 23 May 2006
Posts: 2843

PostPosted: Mon Dec 26, 2011 2:59 pm    Post subject: Reply with quote

Another tip is mount usb storage with 'flush' mount option. vfat support it, not sure about ntfs. Since I use pmount for external usb storage, I have written a patch https://github.com/slashbeast/localpatch-patches/blob/master/sys-apps/pmount/pmount-mount-vfat-with-flush-option.patch

flush just 'flush' each file after its transfer is over, before next file is sent.
_________________
BitBucket -- better-initramfs
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Mon Dec 26, 2011 10:47 pm    Post subject: Reply with quote

I fail to see how shrinking the buffers or forcing direct flush is going to help.
The problem lies in the fact that the USB IO or buffer subsystem to USB seems to preempt all IO, including SATA, leaving firefox IO starved. FF seems to do a lot of IO whilst it apparently does nothing, as I saw from the strace output.

Reducing the buffers will simply make it happen quicker and forcing flush will simply slow down the entire transfer and I have over 400G to copy.
Still, I will give the buffers a whack and see what happens.

FYI: I notice that FF 'pauses' for around 1/2 a second periodically, and it's this 1/2 second pause which is exacerbated by the IO starvation.
I only have AddBlock, ChatZilla, DownThemAll, FlashBlock, ShowIP, TabMixPlus addons enabled.
FF version is 8.0.
_________________
...Lyall
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5735

PostPosted: Mon Dec 26, 2011 10:54 pm    Post subject: Reply with quote

Try doing "echo never > /sys/kernel/mm/transparent_hugepage/enabled". Known problem.
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Tue Dec 27, 2011 2:17 am    Post subject: Reply with quote

Absolutely the behavior I have seen, awesome.
Will give the setting a whack and report back.
edit: Once I find a disk that doesn't fail on me...
_________________
...Lyall
Back to top
View user's profile Send private message
lyallp
Veteran
Veteran


Joined: 15 Jul 2004
Posts: 1400
Location: Adelaide/Australia

PostPosted: Tue Dec 27, 2011 8:25 am    Post subject: Reply with quote

Huge improvement, by disabling transparent hugepage!

Copy to a USB mounted drive screaming along, along with a VirtualBox VM running flat out, hammering an NTFS filesystem via FUSE and ntfs-3g and Firefox hardly pauses for breath.

Thanks!
_________________
...Lyall
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum