Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[Solved] Root Filesystem Read-only / Wrong boot
View unanswered posts
View posts from last 24 hours

Goto page 1, 2  Next  
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Thu Nov 12, 2015 7:19 am    Post subject: [Solved] Root Filesystem Read-only / Wrong boot Reply with quote

Wow, this one is super weird....

I was briefly on linux-4.0.9 then got the upgrade for linux-4.1.12. Did the usual, copied over my old kernel config, built it in the new kernel, copied it to boot and that was that. I did edit my fstab some but I see no major issues there. I continued to use the system but eventually it started lagging after a few days so i restarted.... Now when I start into my new kernel, the filesystem is readonly and I can do nothing as root or normal user. Also, my boot directory is jacked. None of my old kernels are there nor grub the only thing is some old kernel files from 3.1.0 which I havent used since I first turned this box back on about 1.5 years ago....

How did i overwrite my previous /boot with files from over a year and half ago and how can I get it back?
_________________
...and it should be known by now


Last edited by phobos13013 on Sat Dec 05, 2015 6:03 pm; edited 1 time in total
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2613

PostPosted: Thu Nov 12, 2015 7:31 am    Post subject: Reply with quote

Sounds like you forgot to mount /boot before make install modules_install?
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Thu Nov 12, 2015 8:13 am    Post subject: Reply with quote

Good thought, and I am not denying this is not a super basic issue... but I am still not so certain. That said, I can confirm that /boot is still alive and well just not properly mounted (mounting /dev/sda1 shows my correct kernel files). I did just notice that the whole netmount thing that was in the news awhile back was not done, so I added that to boot. But it was not the solution.

I was about to post my fstab but borked the syntax and now my fstab is dead... is there a place I get a template version or do I have to make it from scratch?
_________________
...and it should be known by now
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Thu Nov 12, 2015 8:50 am    Post subject: Reply with quote

I rebuilt the fstab (posted here: http://bpaste.net/show/020ca9f48633), assuming this is correct, but maybe I am not being clear on the primary issue here...., it is that my system is booting read-only and I do not know why. During the boot sequence, I am being told that:

Quote:
/run/lvm/lvmetad.socket: connect failed: No such file or directory
WARNING: Failed to connect to lvmetad. Falling back to internal scanning
Checking local filesystems ...
/dev/sda5 contains a file system with errors, check forced.


the drive is determined to be inconsistent then takes me to the login page

Of course I am certain this is a symptom and not the ailment.

In addition to this, when I do login (read-only) my /boot drive is not there but instead apparently a /boot that is written to the root directory. Why is my boot drive not getting mounted? It historically was in the past....
_________________
...and it should be known by now
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2613

PostPosted: Thu Nov 12, 2015 9:12 am    Post subject: Reply with quote

phobos13013 wrote:
I rebuilt the fstab (posted here: http://bpaste.net/show/020ca9f48633), assuming this is correct,
IMHO the /dev/proc line should be removed. See https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/System#About_fstab

phobos13013 wrote:
Quote:
/run/lvm/lvmetad.socket: connect failed: No such file or directory
WARNING: Failed to connect to lvmetad. Falling back to internal scanning

Are you using LVM? And shouldn't the fstab entries for LVM look like https://wiki.gentoo.org/wiki/Sakaki's_EFI_Install_Guide/Final_Preparations_and_Reboot_into_EFI#Setting_up_the_Mountpoint_Tables?
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Thu Nov 12, 2015 2:39 pm    Post subject: Reply with quote

Well I can comment out proc pretty easy, dont think that is my problem. And I can try to follow this guide for EFI... I hadnt been using it before although I was using LVM with no problems; why is this a problem all of a sudden? I am not really sure what the EFI settings mean, though so do not feel I understand this guide. For instance, what is this mapper directory for? Is there any other explanation for all this other EFI stuff? I am not using an initiramfs and do not really know how or why to set it up. So any guidance towards this is appreciated.
_________________
...and it should be known by now
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2613

PostPosted: Thu Nov 12, 2015 2:55 pm    Post subject: Reply with quote

Forget about EFI. It is LVM where I was trying to point your attention.
But maybe your fstab would also work for LVM. Did you set up for LVM at all?
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Thu Nov 12, 2015 3:04 pm    Post subject: Reply with quote

Well, I have it in my kernel, I guess I am not sure what it is that I need to do to set up for LVM in the first place. The thing is, I have had this same configuration for boot on this system since at least July but upgraded a new kernel and now I am stuck with a non-mounting boot and a read-only root and there is no clear reason why. If you can point me to a way to configure LVM (which I have never done before), its appreciated.
_________________
...and it should be known by now
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7071

PostPosted: Thu Nov 12, 2015 3:09 pm    Post subject: Reply with quote

/dev/sda1 /boot ext2 defaults,noauto 0 2

the noauto mean you will have not /boot mount until you ask it to mount.
that's normal and you should keep it like that, your previous fstab wasn't using it if it was mounting /boot always.

as /boot partition isn't mount, what you see in /boot comes from your /boot subdirectory of your /
so what you see in it, is that times ago, you forget to mount /boot and install a kernel and it was copied there.

if your system works without initram, it mean your system doesn't specially need something to mount /
if your / is mount read only, you have some issue.

you can
Code:
mount -o rw,remount /

to allow you to fix the issue. But as you probability guess, you need to find first, what is your issue :)
per example, lvm is only an issue if you use lvm, something i don't think you do.

And if per example, your kernel lack tmpfs, you cannot mount /run, udev will not works, lvm will complain /run/lvm/lvmetad.socket doesn't exists because nobody could write in /run if /run is not on a tmpfs but in your / that is also ro to makes things worst.

If you are lost with what could be your issue, look in dmesg, because many clues should be there.
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Thu Nov 12, 2015 3:31 pm    Post subject: Reply with quote

This I think has gotten to the root of the matter, PUN INTENDED!

I had seen this remount command somehwere in the forums but hadnt actually tried it yet and unfortunately time is short for the rest of my American day so wont be able to test but I have faith in that temporary fix.

I will explore dmesg and do the basic diagnostic also, but I think you are right LVM complaints are a symptom and not a disease. The question is where is the disease, this I am afraid is above my paygrade... which for Linux is 0. Any other ideas or info is appreciated until my next post which is inevitably forthcoming.
_________________
...and it should be known by now
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Fri Nov 13, 2015 3:20 am    Post subject: Reply with quote

Well, I have finally gotten some time to do some testing.

You are correct in assuming that previously I mounted /boot unabashedly, I hope that hasnt gotten me in to trouble. From now on I will definitely be locking that down. You are also correct that I have a bum /boot from when I was trying to reinstall a kernel to it when I first turned this box back on a few years ago.

The mount -o command worked but none of my major services are working so this just won't do. One weird thing I will say is that when I run the desktop manager after the remount happens is a desktop that is typically setup for a sub instance of X I had setup for vnc and such activities. It is not my primary user desktop.

You actually may be right that I am not using lvm.... but then where did it come from? My understanding is that it is used to access partitions above sda3, am I incorrect here?

regarding /run and tmpfs, at the start of openrc, I get the following info, not sure if it means anything.

Quote:
Mounting /run ...
/run/openrc: creating directory
/run/lock: creating directory
/run/lock: correcting owner
Caching service dependencies ...
Remounting devtmpfs on /dev


The stuff about lvm not working and root inconsistencies follows.

My dmesg is found here... http://bpaste.net/show/61914cbdd3d9
Not seeing anywhere to start; clearly VFS is reporting that root gets mounted ro pretty unceremoniously... There is clearly an ACPI hiccup there but do not see how this can lock me out of my system. So the only thing left to say is HELP!
_________________
...and it should be known by now
Back to top
View user's profile Send private message
Syl20
Guru
Guru


Joined: 04 Aug 2005
Posts: 564
Location: France

PostPosted: Fri Nov 13, 2015 11:56 am    Post subject: Re: Root Filesystem Read-only / Wrong boot Reply with quote

phobos13013 wrote:
Did the usual, copied over my old kernel config,

So you didn't make oldconfig ? All the new or renamed options don't appear in your pasted .config, so I suppose the building process is setting for these options default values that may be different that you expect.
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2613

PostPosted: Fri Nov 13, 2015 12:37 pm    Post subject: Reply with quote

phobos13013 wrote:
My dmesg is found here... http://bpaste.net/show/61914cbdd3d9
I don't see your lvmetad stuff there.

phobos13013 wrote:
Quote:
/run/lvm/lvmetad.socket: connect failed: No such file or directory
WARNING: Failed to connect to lvmetad. Falling back to internal scanning
Did that come before the grub prompt or after?
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7071

PostPosted: Fri Nov 13, 2015 3:53 pm    Post subject: Reply with quote

Code:
Command line: BOOT_IMAGE=/vmlinuz-4.1.12-gentoo root=/dev/sda5 ro


Hmmm, do you know what "ro" could mean?
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Sat Nov 14, 2015 2:44 am    Post subject: Reply with quote

Whoa! All good info folks, I appreciate everyone thinking this through. I have been looking at some or all of these issues.... Please see below for individual responses:

CneGroumF: I have evolved my kernel upgrade process over the years even within the last few years... My current process is to just do a cp of the .config from the old linux directory to the new kernel version then do an eselect to change the kernel version then run make and build the kernel again from the previous .config which I know is valid. I then copy the bzImage to /boot renaming it to linux-version convention. Then I remake grub.cfg. I have no doubt there are all sorts of issues and problems with doing it this way but really it has worked for me for the last few version upgrades (since at least the last few 3.X I was running); I have thought about make oldconfig but just have never run it. I do this, though, for exactly the reason you mention. Awhile ago I rebuit a kernel without copying the oldconfig and had a hell of a time trying to figure out why my X was bad my net was bad and all that.

charles17: Thanks for sticking with me here; yeh good point I do not see any mention of lvm... which doesnt surprise me since it couldnt work on boot and the complaint from openrc is that it cant find lvm. I do not know how to confirm but for some reason when I did the emerge that brought in the new kernel, i want to say that an lvm upgrade was in there... but it could just be priming bias from the current conundrum. I am musing at this point actually. Anyway, the lvm complaint is after grub, its during the drive mouning after openrc picks up the initscript.

krinn: I know what you are thinking and I had this same idea, since I did notice that... But from what I can tell, that is normal behavior for my system. First looking back at all my grub.cfg (from past kernels also), this is also the case on the boot script line. My understanding from the research i did after seeing that is the initial action of the boot script is to mount all drives ro until later in the script where it then makes them rw, it has something to do with timing of services and such... I am not an expert on any of this so may be talking out of my lower regions but they are probably more informed than my upper regions anyway. Whatever the made up excuse, the fact is I went into my grub.cfg and removed any mention of ro after the script line for the boot location and it had zero effect. That said, now that I think about it, maybe that line isnt the same as the grub.cfg line and there is somewhere else that is telling this thing to boot ro... I just do not understand why it would have happened all of a sudden.. and where for that matter the thing is that would say to do that.

Any other suggestions are appreciated.
_________________
...and it should be known by now
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2613

PostPosted: Sun Nov 15, 2015 7:34 am    Post subject: Reply with quote

phobos13013 wrote:
I am musing at this point actually. Anyway, the lvm complaint is after grub, its during the drive mouning after openrc picks up the initscript.

What about doing an interactive boot (have rc_interactive="YES" in /etc/rc.conf) in order to see which of the init scripts does this?
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Tue Nov 24, 2015 5:13 am    Post subject: Reply with quote

I am bumping this looking for some fresh ideas on this... I have an interactive boot set currently, but dont know what if any of the services are contributing to this. Not sure which services to add or deny to triage it. So far what I have tried comes out the same, read-only.
_________________
...and it should be known by now
Back to top
View user's profile Send private message
charles17
Advocate
Advocate


Joined: 02 Mar 2008
Posts: 2613

PostPosted: Tue Nov 24, 2015 9:48 am    Post subject: Reply with quote

phobos13013 wrote:
I have an interactive boot set currently,

Press the i button very early at boot, so it should prompt you for starting or skipping each service (initscript) separately.
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7071

PostPosted: Tue Nov 24, 2015 12:39 pm    Post subject: Reply with quote

phobos13013 wrote:

krinn: I know what you are thinking and I had this same idea, since I did notice that... But from what I can tell, that is normal behavior for my system.


If you ask kernel to boot ro, it will boot ro, it's only people that ask a first boot ro, then boot gets remount to rw later (that's for people that remount their system because of initram or temporary boot usage that will be remount later)
So if your root is not remount, it should remains ro (edit: i have checked, and openrc service will remount any ro to rw, check rc-update | grep root and make sure it is start at boot, and look if it works)

phobos13013 wrote:
That said, now that I think about it, maybe that line isnt the same as the grub.cfg line and there is somewhere else that is telling this thing to boot ro

Command line can be force thru kernel options. So even removing ro from grub may not change it.
Code:
grep "CMDLINE=" /usr/src/linux/.config
CONFIG_CMDLINE="clocksource=hpet"
grub.cfg: kernel /vmlinuz root=/dev/sdb2
[    0.000000] Kernel command line: clocksource=hpet root=/dev/sdb2


You can either clean grub config and check dmesg to see if ro has really disappears, or check your kernel CMDLINE to see if its enable and have ro set.
If you use an initramfs, make sure it remount it rw.
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Tue Nov 24, 2015 3:33 pm    Post subject: Reply with quote

Quote:
So if your root is not remount, it should remains ro (edit: i have checked, and openrc service will remount any ro to rw, check rc-update | grep root and make sure it is start at boot, and look if it works)


So this is the rub... yes rc-update has root on boot and it always has, so not sure why my most recent update would change any of this. BUT no it is not being remount as rw for some reason later down the sequence. I did an interactive mount and it is certainly being requested to mount then remount but its not happening, nonetheless.

Quote:
You can either clean grub config and check dmesg to see if ro has really disappears, or check your kernel CMDLINE to see if its enable and have ro set.


Well if this is the solution, I think im too dumb to implement this without some additional info. I run the grep command and this is returned, CONFIG_FB_CMDLINE=y

dmesg does not seem to be reporting a remount and i do not have an initiramfs. Something changed in terms of my boot expectation and I am not clear when, how or why.
_________________
...and it should be known by now
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7071

PostPosted: Tue Nov 24, 2015 4:08 pm    Post subject: Reply with quote

phobos13013 wrote:
I did an interactive mount and it is certainly being requested to mount then remount but its not happening, nonetheless.

Have an rc.log to see the reason?

Quote:

dmesg does not seem to be reporting a remount and i do not have an initiramfs. Something changed in terms of my boot expectation and I am not clear when, how or why.

You will not see a remount in dmesg, it will appears as a mount info, the kernel mount boot, it doesn't remount it itself. that's openrc that do that.
You may check dmesg still to see if the mount options were properly tried, some limits may prevent a remount or bad option.
Speaking of bad options i've look at your fstab, and i could tell you that noatimes is not noatime
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43178
Location: 56N 3W

PostPosted: Tue Nov 24, 2015 7:18 pm    Post subject: Reply with quote

phobos13013,

post your dmesg after root comes up read only.
I suspect that when the root fs is checked, it may have some damage that cannot be fixed on the fly.

-- edit --

Logical Volume Manager is not used to access Logical Partitions. The naming is an unfortunate coincidence.
A Logical Volumes are like partitions but they can be dynamically resized, providing the filesystem supports the operation too.
Unlike a partition, where all the blocks are in one contiguious chunk on the disk, a logical volume need not be in one piece and can even be spread over several disks.
Logical Volume manager hides all this from you.

In short, if you didn't set up your install on logical volumes, you are not using logical volume manager.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
phobos13013
Apprentice
Apprentice


Joined: 22 Sep 2004
Posts: 255
Location: NOLA

PostPosted: Wed Nov 25, 2015 8:01 am    Post subject: Reply with quote

Still truding along here... but I think I had a memory flash that is the cause of what got here, more on that below but let me provide responses first:

krinn:
I am not getting rc.logged right now unless its in /var/log/messages; I cannot confirm this because it is bloated to GBs in size cannot open, will have to scrub it i guess. I am posting dmesg for Neddy below.

NeddySeagoon:
I have the old dmesg from when I first got this error towards the beginning of the thread (see the only other bpaste link). Go ahead and compare it to the most recent one here: https://bpaste.net/show/91c716edf624. But I think you are right, it is seeing some kind of error to the root drive which is triggering e2fsck but it cannot resolve it. Before it was getting through the whole check, but now it panics at like 18% and continues with the boot. Is there a way to fix this my transfering my root partition to another one and try to boot to that one?

I have changed some kernel and config settings, but nothing has helped. There is one last possiblity here. I recently installed a file manager that I have not used before as a test. I was transferring files to backup and then deleteing them from the hdd. There are some partitions on here that I used to not be able to modify contents of and I am wondering if the file manager was able to exceed its permissions which borked the drive. If this is the issue, is there anything that can be done to salvage this short of formatting, if that?
_________________
...and it should be known by now
Back to top
View user's profile Send private message
krinn
Watchman
Watchman


Joined: 02 May 2003
Posts: 7071

PostPosted: Wed Nov 25, 2015 11:19 am    Post subject: Reply with quote

About my query for rc.log, the file is alone and made up by openrc if you have enable the option.
That's in /etc/rc.conf
Code:
# rc_logger launches a logging daemon to log the entire rc process to
# /var/log/rc.log
# NOTE: Linux systems require the devfs service to be started before
# logging can take place and as such cannot log the sysinit runlevel.
#rc_logger="YES"

I'm seeing i didn't think about it, but of course creating that /var/log/rc.log on a ro fs to see where rc report problems is chiken/egg problem
It might be possible to gave it some tmpfs location to bypass the limitation, but i never tried something like that myself.

As usual, solution always comes when NeddySeagoon comes in, i really start to think it's not coincidence :)
Code:
[code][  266.923622] EXT4-fs (sda5): warning: mounting fs with errors, running e2fsck is recommended[/code]

Didn't saw it because it appears at end of dmesg (and really late), that must be you are remounting it rw by hands.
Use a livecd and try fix it from there as from your fstab, sda5 hold your /
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Wed Nov 25, 2015 12:02 pm    Post subject: Reply with quote

phobos13013 wrote:
Well I can comment out proc pretty easy, dont think that is my problem. And I can try to follow this guide for EFI... I hadnt been using it before although I was using LVM with no problems; why is this a problem all of a sudden? I am not really sure what the EFI settings mean, though so do not feel I understand this guide. For instance, what is this mapper directory for? Is there any other explanation for all this other EFI stuff? I am not using an initiramfs and do not really know how or why to set it up. So any guidance towards this is appreciated.

If you're using LVM without initramfs, you probably need these settings worked out by saellaven in /etc/lvm/lvm.conf (if you don't already):
Code:
obtain_device_list_from_udev = 0
udev_sync = 1
udev_rules = 0

I wouldn't worry about the warning for lvmetad; though i do recall thinking it seemed like one we might want as a dependency in any eventuality, it's not strictly needed, and is for networked filesystems iirc (could well be wrong on its actual purpose; was a while ago.)

WRT logging output in early init, you can try the same thing I did with lvm, for a previous bug.

Though it seems your issue may be a fsck required; touch /forcefsck if you had root.. ;)

Sorry I haven't been updating that thread; was thinking to leave Gentoo (again) thanks to developer idiocy (again.)
Forums users persuaded me otherwise, just by being themselves (again.)
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum