Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Reduce writes on Gentoo install
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
Marcih
Apprentice
Apprentice


Joined: 19 Feb 2018
Posts: 152

PostPosted: Fri Mar 23, 2018 7:58 pm    Post subject: [SOLVED] Reduce writes on Gentoo install Reply with quote

I have an SSD but I want to keep it for as long as I can (I don't like buying new stuff) and don't really mind waiting during compilations or boot all that much. My system is a Intel Core i5 6500 with 8GiB of memory, a 120GB SSD (which currently houses my boot, swap and root partitions) and a 1TB dusty spinny thingy (mounted at /home). I've already mounted /var/tmp/portage to tmpfs as per the Gentoo Wiki and I'm preparing to rearrange my partitions to be:
Code:
SSD partition 1: /boot, ~100MB
SDD partition 2: /, rest of the disk
HDD partition 1: swap, 16GiB?
HDD partition 2: /home, rest of the disk

Anything else I should(n't) do? Would having my /usr folder on my HDD be any use?

Thanks :)
_________________
Bones McCracker wrote:
It wouldn't be so bad, if it didn't suck.

===
I refuse to move on from GTK+2.


Last edited by Marcih on Fri Mar 30, 2018 6:59 pm; edited 1 time in total
Back to top
View user's profile Send private message
John R. Graham
Administrator
Administrator


Joined: 08 Mar 2005
Posts: 10261
Location: Somewhere over Atlanta, Georgia

PostPosted: Fri Mar 23, 2018 8:31 pm    Post subject: Reply with quote

If you want to reduce the writes during Portage's package compiles, you should mount /var/tmp/portage (or maybe /var/tmp) on the rotating rust as well.

- John
_________________
I can confirm that I have received between 0 and 499 National Security Letters.
Back to top
View user's profile Send private message
toralf
Developer
Developer


Joined: 01 Feb 2004
Posts: 3686
Location: Hamburg

PostPosted: Fri Mar 23, 2018 8:47 pm    Post subject: Reply with quote

my tmpfs was too small for at least one package, I circumvented that with :
Code:
$> grep tmpdir /etc/portage/package.env/z_misc
dev-qt/qtwebengine        tmpdir

$> cat /etc/portage/env/tmpdir
PORTAGE_TMPDIR=/var
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1746

PostPosted: Fri Mar 23, 2018 9:21 pm    Post subject: Reply with quote

8GB RAM is enough to build pretty much anything in RAM only, just make sure your portage's temporary directory is on tmpfs (and bump its size from default 50% RAM to 85 or 90%).
However, there is another thing you may want to consider: SSD are made for serving high IOps and high bandwidth. If you're not gonna use that IO and bandwidth, you'd probably be better with a couple of high capacity HDDs
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5761

PostPosted: Fri Mar 23, 2018 9:28 pm    Post subject: Reply with quote

Also turn on laptop mode. The disk won't be written to unless the buffers fill beyond a certain amount, or if it stays idle long enough.
/etc/sysctl.d/whatever.conf:
vm.dirty_expire_centisecs=6000
vm.dirty_ratio=80
vm.dirty_writeback_centisecs=3000
vm.laptop_mode=15
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43186
Location: 56N 3W

PostPosted: Fri Mar 23, 2018 9:36 pm    Post subject: Reply with quote

Marcih,

Put /usr/portage/distfiles on the rotating rust.
The files here are large and sequentially accessed. That's something that rotating rust is quite good at.
Likewise save your binary packages (if any) to rotating rust.

The portage repository is a large number of small files, so that should go on the SSD.
However it will waste a lot of space. If space on the SSD is at a premium, look into putting the repo into a squashfs file.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
keet
Guru
Guru


Joined: 09 Sep 2008
Posts: 529

PostPosted: Sat Mar 24, 2018 12:42 am    Post subject: Reply with quote

I switch DISTDIR to /tmp and mount /tmp as tmpfs. Of all of my packages, Icedtea needs the most, 8.5GB. Fortunately, my laptop has 12GB of R.A.M., so I give /tmp 9GB.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17922

PostPosted: Sat Mar 24, 2018 5:12 am    Post subject: Reply with quote

NeddySeagoon wrote:
The portage repository is a large number of small files, so that should go on the SSD.
However it will waste a lot of space. If space on the SSD is at a premium, look into putting the repo into a squashfs file.
Any reason to not go with tmpfs, squashfs and rotating rust? xaviermiller does the tmpfs/squashfs thing, which I have on my todo list. I'm still using rotary-platter storage, but still like the idea of minimizing writes to SSD when reasonably possible. I plan to incorporate fixed-platter (SSD) "soon."
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43186
Location: 56N 3W

PostPosted: Sat Mar 24, 2018 10:43 am    Post subject: Reply with quote

pjp,

That works too. I've not considered the repo in squashfs moved to tmpfs on demand.
That saves IO bandwidth but not writes as the writes occur when the squashfs is written.
You could go one step further and not save a local copy of the tree at all.
That's a bit radical for me.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Tony0945
Advocate
Advocate


Joined: 25 Jul 2006
Posts: 3076
Location: Illinois, USA

PostPosted: Sat Mar 24, 2018 2:25 pm    Post subject: Reply with quote

NeddySeagoon wrote:
pjp,

That works too. I've not considered the repo in squashfs moved to tmpfs on demand.
That saves IO bandwidth but not writes as the writes occur when the squashfs is written.
You could go one step further and not save a local copy of the tree at all.
That's a bit radical for me.

My setup is a central server that the other machines synchronize with via http-replicator.
I've considered mounting /usr/portage as a network drive instead, saving all those sync's.
Would you care to comment?

P.S. I really like the distfiles on hard drive idea. They take a lot of space and are only read during actually emerge.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43186
Location: 56N 3W

PostPosted: Sat Mar 24, 2018 2:41 pm    Post subject: Reply with quote

Tony0945,

I use http-replicator and run my own rsync server.
I don't share things with NFS as I have three zones on my internal network with varying degrees of trust.
An NFS share across all three internal zones breaks the physical security.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
szatox
Veteran
Veteran


Joined: 27 Aug 2013
Posts: 1746

PostPosted: Sat Mar 24, 2018 3:18 pm    Post subject: Reply with quote

Quote:
I've considered mounting /usr/portage as a network drive instead, saving all those sync's.
I trust my machines enough to share portage over NFS in RW mode, including distfiles and packages, so I only download everything once, and sometimes even reuse binary packages.
It works really well, as long as you can trust your network - and your network is reasonably fast.

In case if limited trust, you can still share it over NFS in read only mode and patch your clients with overlayfs, so they would only download sources not yet available (but not store them on NFS server, as it would not accept writes). Obviously, this approach takes significantly more effort.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13842

PostPosted: Sat Mar 24, 2018 4:13 pm    Post subject: Reply with quote

I too use the approach that xaviermiller describes, though mine dates to when webrsync was the only way to validate the tree. I can clean up and publish my sync script if there is interest.
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17922

PostPosted: Mon Mar 26, 2018 2:31 am    Post subject: Reply with quote

Split off [split] Happy trolling :)
_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
Marcih
Apprentice
Apprentice


Joined: 19 Feb 2018
Posts: 152

PostPosted: Thu Mar 29, 2018 2:52 pm    Post subject: Reply with quote

Wow, I didn't expect this many responses! Usually when I start a topic it's a ghost town. :lol:

John,
that's already taken care of. I've mentioned this in my OP, I've done what szatox suggested and mounted /var/tmp/portage to tmpfs. I could however have a backup partition on the rotating rust as a backup solution in case I ever have a package requiring more than I can fit in the tmpfs, like toralf suggested.

szatox,
I know, I've already been informed I'm an idiot :P and I was already aware that SSDs excell at this. Even then, I can't help the feeling that I'm wasting precious write cycles just compiling software...

NeddySeagoon wrote:
You could go one step further and not save a local copy of the tree at all.

Neddy,
I was actually considering having my own rsync mirror that would sync the repos periodically (run "emerge --sync" as a cron job every 24 hours?) so that I can keep all my Gentoo boxes reasonably up to date simultaneously without hogging the bandwidth. Would it be possible to take it that step further and have the repo tree on the server itself and have the machines fetch and temporarily save to e.g. tmpfs when needed?
Just to clarify, the SSD storage is in no way at premium, I'm just bored. :D

I'll repeat the question I asked in the OP again: What about mounting other directories like /usr and /var/log? /usr is a place where a lot of writes occur too (I'm not too sure what exactly resides there though, and the Filesystem Hierarchy Standard doesn't help either) and so is /var/log and I highly doubt I stand to benefit in any way in logs being written to an SSD.
_________________
Bones McCracker wrote:
It wouldn't be so bad, if it didn't suck.

===
I refuse to move on from GTK+2.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 13842

PostPosted: Fri Mar 30, 2018 2:23 am    Post subject: Reply with quote

Why let the clients fetch the tree at all? Make the local rsync mirror an NFS server, then have the clients NFS mount the Portage tree so that they never keep it locally.

I would put /usr on the SSD. I still remember the first time I watched a configure script run on a system that had /usr on SSD, after years of having used a spinning drive for /usr. I was amazed at how rapidly configure ran through all its tests, since all the files it needed were on the SSD and thus read very very quickly. Remember that the purpose of your SSD is to make your life better than if you had only the spinning drive. The more effort you spend not using the SSD, the less it serves that purpose.
Back to top
View user's profile Send private message
Marcih
Apprentice
Apprentice


Joined: 19 Feb 2018
Posts: 152

PostPosted: Fri Mar 30, 2018 9:53 am    Post subject: Reply with quote

Hu wrote:
Why let the clients fetch the tree at all? Make the local rsync mirror an NFS server, then have the clients NFS mount the Portage tree so that they never keep it locally.

Fair point, I'll take a look at that in the near future.
Hu wrote:
I would put /usr on the SSD. [...] Remember that the purpose of your SSD is to make your life better than if you had only the spinning drive.

Of course, an SSD is an upgrade. If I'm not going to use it then there was no reason to upgrade in the first place. Granted, I got the SSD when I was still on Windows and just booting from it cut boot times in less than half. With Gentoo, boot is pretty much instantenous even on a spinning platter, the most time-consuming process is bringing the network interface up.
The reason I was thinking about putting /usr on the spinning rust is because I have more than half of the drive free and I'm looking to fill it up with something while prolonging the SSD's life, and as far as I understand, programs/packages wrire most of their stuff to /usr.

What about logs? Reading and writing them doesn't need a lightning-fast disk access and I can't remember the last time I even opened up one (maybe with the exception of dmesg, that's always handy to read, especially when configuring your kernel).
Hu wrote:
The more effort you spend not using the SSD, the less it serves that purpose.

But it will at least be able to not serve its purpose longer. :P


I've gotten some good suggestions here, thanks to all that answered. I'll mark the thread as solved but you're of course welcome to give more advice. :)
_________________
Bones McCracker wrote:
It wouldn't be so bad, if it didn't suck.

===
I refuse to move on from GTK+2.
Back to top
View user's profile Send private message
P.Kosunen
Guru
Guru


Joined: 21 Nov 2005
Posts: 309
Location: Finland

PostPosted: Fri Mar 30, 2018 1:31 pm    Post subject: Reply with quote

Marcih wrote:
But it will at least be able to not serve its purpose longer. :P

You don't live long enough to see the write cycles to wear out with typical Gentoo usage.
Back to top
View user's profile Send private message
tuggbuss
Apprentice
Apprentice


Joined: 20 Mar 2017
Posts: 214

PostPosted: Fri Mar 30, 2018 2:30 pm    Post subject: Reply with quote

I like the statement in "Improving Performance" (storage devices) in Arch Wiki

Code:
Note: A 32GB SSD with a mediocre 10x write amplification factor, a standard 10000 write/erase cycle, and 10GB of data written per day, would get an 8 years life expectancy. It gets better with bigger SSDs and modern controllers with less write amplification. Also compare [1] when considering whether any particular strategy to limit disk writes is actually needed.


So it's put in perspective.

https://wiki.archlinux.org/index.php/Improving_performance#Storage_devices
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43186
Location: 56N 3W

PostPosted: Fri Mar 30, 2018 5:09 pm    Post subject: Reply with quote

tuggbuss,

That's going back a bit, only 10000 write/erase cycles.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Marcih
Apprentice
Apprentice


Joined: 19 Feb 2018
Posts: 152

PostPosted: Fri Mar 30, 2018 7:07 pm    Post subject: Reply with quote

P.Kosunen, tuggbuss, Neddy,
so what you're saying, even with my obsession of running "emerge --sync && emerge -avuDU --with-bdeps=y --complete-graph @world" every day after I get home, I'm still fine? I remember running "fstrim -v /" and being alarmed after seeing it had like between 1 and 10GiB (I really can't remember) after a couple of days of not running TRIM.
_________________
Bones McCracker wrote:
It wouldn't be so bad, if it didn't suck.

===
I refuse to move on from GTK+2.
Back to top
View user's profile Send private message
bunder
Bodhisattva
Bodhisattva


Joined: 10 Apr 2004
Posts: 5844

PostPosted: Sat Mar 31, 2018 4:46 am    Post subject: Reply with quote

I didn't see it mentioned, but one way to reduce writes is to turn off atime and relatime. Most people don't care about access times anyways, so they're just useless writes.
_________________
Neddyseagoon wrote:
The problem with leaving is that you can only do it once and it reduces your influence.

banned from #gentoo since sept 2017
Back to top
View user's profile Send private message
P.Kosunen
Guru
Guru


Joined: 21 Nov 2005
Posts: 309
Location: Finland

PostPosted: Sat Mar 31, 2018 2:59 pm    Post subject: Reply with quote

Marcih wrote:
so what you're saying, even with my obsession of running "emerge --sync && emerge -avuDU --with-bdeps=y --complete-graph @world" every day after I get home, I'm still fine? I remember running "fstrim -v /" and being alarmed after seeing it had like between 1 and 10GiB (I really can't remember) after a couple of days of not running TRIM.

Check if your drive has been in some write endurance test and how much writes have you managed to get written so far. If my drives last only half write cycles of those tested and i write 10 times more than usual, they will still last for decades or centuries by those numbers. I bet some drives get broken long before write cycles are finished for other reason than too much writing, if that happens i buy new one and restore system from backup.

I do not limit writes on my drives for anything that decrease performance or usability. OFC i limit stupid writes like Firefox session backup too often, it might even slow down other things.
Back to top
View user's profile Send private message
n05ph3r42
Tux's lil' helper
Tux's lil' helper


Joined: 11 Jul 2016
Posts: 103

PostPosted: Tue Apr 03, 2018 8:11 pm    Post subject: Reply with quote

Ugh, https://wiki.gentoo.org/wiki/Zram ?
But u need specific options in kernel.
My /etc/conf.d/zram-init :
Code:

# load zram kernel module on start?
load_on_start=yes

# unload zram kernel module on stop?
unload_on_stop=yes

# Number of devices.
# This value is also passed to the kernel module on modprobe.
num_devices=2

# /tmp ; phy 2G used, expanded to 6G with lz4
# If you use this, you should put zram-init into the boot runlevel,
# and services using /tmp should be in the default runlevel and/or have
# rc_need=zram-init in their /etc/conf.d/... file
type0=/tmp
flag0=ext4
size0=6144
opts0="nodev,nouser,noatime" # e.g. "relatime" or "noatime" are also reasonable choices
mode0=1777
owgr0= # No reason to change the default "root:root"
notr0= # keep the default on linux-3.15 or newer
maxs0=2
algo0=lz4

# /var/tmp ; phy 4G used, expanded to 12G with lz4
type1=/var/tmp
flag1=ext4
size1=12880
opts1="nodev,nouser,noatime" # e.g. "noatime" or "strictatime" are also reasonable choices
mode1=1777
owgr1= # No reason to change the default "root:root"
notr1= # keep the default on linux-3.15 or newer
maxs1=2
algo1=lz4


I run one of my laptops with Samsung SSD 840 PRO Series on this config since 2k14. update @world almost everyday =]
Also I use F2FS for rootfs on SSD.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum