Joined: 26 Jul 2007
|Posted: Sat Apr 04, 2015 5:46 pm Post subject: LVM cache + initramfs as root partition
I've decided to share some experience on LVM cache usage in Gentoo.
That is caching HDD using SSD.
Some details first, I've used kernel v3.19.0, lvm2 v2.02.109 under x64 stable branch.
Kernel was built using genkernel v126.96.36.199 (initramfs).
I have following tree structure:
sdc - HDD
sdb - SSD
sdc1 - /boot
sdc2 - swap
sdc3 - root (/root, /sbin, /lib, /etc )
sdc4 - LVM (/usr, /opt, /var, /home, etc)
Cache itself was made using http://blog-vpodzime.rhcloud.com/?p=45 as a howto.
Cache started working immediately and everything was fine until reboot.
Upon reboot, it appears that LVM cache uses cache_check utility from /sbin and it's a c++ application.
It requires libexpat.so* (from /usr/lib), libgcc_s.so* and libstdc++.so* (from /usr/lib/gcc/...).
And those libs were left under LVM and didn't get in initramfs!
So /usr dir is no longer accessible.
I guess something needs to be done in order to place those libs in initramfs (genkernel bug?).
Should I file a bug for that?
I ended up copying those libs to /lib (non-LVM partition). At least it works now.
Another thing that came up was completely inability to mount cached LVM partition under x86 kernel.
It just barked in /var/log/messages:
device-mapper: cache-policy-mq: couldn't initialize pool of cache entries
device-mapper: table: 254:4: cache: Error creating cache's policy
device-mapper: ioctl: error adding target to table
vmap allocation for size 77336576 failed: use vmalloc=<size> to increase size
vmalloc: allocation failure: 77332480 bytes
... it seems that x32 kernel just does not have enough continuous memory to load cache meta!
Here I must admit that could be my own fault - I made SSD cache as big as 1G (total SSD size 128GB). Official howto recommends something like 1/1000 of total size and I made it 1/100th.
So use x64 kernel
To see actual SSD cache utilization I've made a small bash utility you can get here - https://github.com/MageSlayer/lvmcache-stats/raw/master/lvmcache-stats