Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Not able to bring the server up.
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 2:26 pm    Post subject: Not able to bring the server up. Reply with quote

Hi Experts, I am completely out of ideas now. One of our Gentoo server seems to have gone down and I am not sure how to recover it. The prompt I am getting is
Quote:
(none) ~#


There is no hostname that is coming up. If I run route or ifconfig commands, I get the below error :-

Code:
route -n
/proc/net/route: No such file or directory
INET (IPv4) not configured in this system.

ifconfig
Warning: cannot open /proc/net/dev (No such file or directory). Limited output.


If I type df -h to see the mount I get arrays of missing file system errors.

Code:
This is (none).unknown_domain (Linux x86_64 3.17.7-gentoo) 19:33:04
(none) login: root
Password:
[ 52.405911] random: nonblocking pool is initialized
(none)  -  It  df -h
df: m/dev/shms: Ho such file or directory
df: m/sys/fs/cgroupm: Ho such file or directory
df: m/sys/fs/cgroup/openrcm: Ho such file or directory
df: m/sys/fs/cgroup/cpusetm: Ho such file or directory
df: m/sys/fs/cgroup/cpum: Ho such file or directory
df: m/sys/fs/cgroup/cpuacctm: Ho such file or directory
Filesystem Size Used Avail Usex Mounted on
/dev/sdal 40G 9.1G 29G 25x /
devtmpfs 12G 0 12G Ox /dev
tmpfs 40G 9.1G 29G 25x /run
/dev/sda3 40G 9.1G 29G 25x /var
(none) - It  /etc/init.d/sshd status
/sbin/runscript: error while loading shared libraries: librc.so.1: cannot open s
hared object file: Ho such file or directory
(none)  - 


I tried booting from Live CD and I can see all the info which I configured before like the hostname, fstab entries. I did a emerge --sync but no go.
Could you please help me in getting this fixed. Appreciate all your help in advance.
Back to top
View user's profile Send private message
The_Great_Sephiroth
Veteran
Veteran


Joined: 03 Oct 2014
Posts: 1367
Location: Fayetteville, NC, USA

PostPosted: Wed Jul 15, 2015 3:24 pm    Post subject: Reply with quote

I would boot System Rescue CD or similar and check your disk! First check SMART to see if it has tripped.
Code:

smartctl -a /dev/sda

If the SMART data has passed, check your file-systems.
Code:

fsck /dev/sda1
fsck /dev/sda3

If those pass, I have no clue! To me it looks like a dying drive or bad file-system. If that is not the case, then maybe some core configuration file got messed up. I am sure a guru will be along shortly to be a tad more insightful.
_________________
Ever picture systemd as what runs "The Borg"?
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 3:59 pm    Post subject: Reply with quote

Thanks for looking into this. I ran fsck before and all looks clean. However I dont think smartctl utility is installed on it.

Code:

(none)  -   [ 27.449033] random: nonblocking pool is initialized
(none) -  cd /sw/
(none)  /sw   /etc/init.d/sshd status
/sbin/runscript: error while loading shared libraries: librc.so.1: cannot open s
hared object file: Ho such file or directory

(none)  /sw  smartctl -a /dev/sda
-bash: smartctl: command not found
(none)  /sw   fsck /dev/sdal
fsck from util-linux 2.24.1
e2fsck 1.42.10 (18 -May -2014)
/dev/sdal: clean, 368057/2621440 files, 2584344/10485760 blocks
(none)  /sw   fsck /dev/sda3
fsck from util-linux 2.24.1
e2fsck 1.42.10 (18 -May -2014)
/dev/sda3: clean, 144859/1048576 files, 1260033/4194304 blocks
(none)  /sw   fsck /dev/sda4
fsck from util-linux 2.24.1
e2fsck 1.42.10 (18 -May -2014)
/dev/sda4: clean, 73219/30048256 files, 66456657/120192768 blocks
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43425
Location: 56N 3W

PostPosted: Wed Jul 15, 2015 4:37 pm    Post subject: Reply with quote

manu_leo,

It looks like root is staying read only. Can you pasfebin dmesg from a failed boot?
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 4:43 pm    Post subject: Reply with quote

Thanks Neddy.This is what I have from dmesg

Code:
[ 3.6952211 input: Avocent USB Composite Device -0 as /devices/pci0000:00/0000
:00:1d.O/usb5/5-2/5-2:1.0/0003:0624:0248.0001/input/input4
[ 3.6954751 hid -generic 0003:0624:0248.0001: input,hidraw0: USB HID v1.00 Key
board [Avocent USB Composite Device -0] on usb-0000:00:1d.0-2/inputO
[ 3.7011631 input: Avocent USB Composite Device -0 as /devices/pci0000:00/0000
:00:1d.0/usb5/5-2/5-2:1.1/0003:0624:0248.0002/input/input5
[ 3.7014891 hid -generic 0003:0624:0248.0002: input,hidrawl: USB HID v1.00 Mou
se [Avocent USB Composite Device -0] on usb-0000:00:1d.0-2/inputl
[ 4.238268] scsi 3:0:0:0: Direct -Access iDRAC LCDRIVE 0323 PQ
: 0 ANSI: 0 CCS
[ 4.238547] sd 3:0:0:0: Attached scsi generic sg3 type 0
[ 4.239637] scsi 4:0:0:0: CD-ROM iDRAC Virtual CD 0323 PQ
: 0 ANSI: 0
[ 4.242062] sd 3:0:0:0: [sdb] Attached SCSI removable disk
[ 4.347774] sr 4:0:0:0: [srl] scsi3-mmc drive: Ox/Ox cd/rw caddy
[ 4.348124] sr 4:0:0:0: Attached scsi CD-ROM srl
[ 4.348284] sr 4:0:0:0: Attached scsi generic sg4 type 5
[ 4.350839] scsi 4:0:0:1: Direct -Access iDRAC Virtual Floppy 0323 PQ
: 0 ANSI: 0 CCS
[ 4.351213] sd 4:0:0:1: Attached scsi generic sg5 type 0
[ 4.353393] sd 4:0:0:1: [sdc] Attached SCSI removable disk
[ 27.449033] random: nonblocking pool is initialized
[ 1250.420809] EXT4-fs (sda3): mounted filesystem with ordered data mode. Opts:
(null)
(none)  -
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 4:45 pm    Post subject: Reply with quote

Actually Neddy, its not a failed boot but the serves comes up when I issue - shutdown -r now with the prompt as below

(none) ~#
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 5:12 pm    Post subject: Reply with quote

Is there a way this can be fixed or repaired using a LiveCD without loosing any of the stuff/data ?

Thanks and appreciate all your help.
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 6:23 pm    Post subject: Reply with quote

I did a mount -o rw,remount / and now I am able to create directories and delete them. But the issue is still the same - Not able to restart any service, df -h gives the same error and so on.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43425
Location: 56N 3W

PostPosted: Wed Jul 15, 2015 6:59 pm    Post subject: Reply with quote

manu_leo,

That looks normal but I would have liked it all.

With System Rescue CD, mount your root at /mnt/gentoo. Do not chroot.
Edit /mnt/gentoo/etc/rc.conf to be sure the following options aare set.
Code:
rc_parallel="NO"
rc_interactive="YES"


Reboot normally and press I when openrc starts. There may even be a
prompt.
Let each service start in turn and note any error messages you get along
with the service that generated the errors.

Once we know the cause of the problem we can fix it. No promises about data loss yet, since we cannot rule out hardware issues.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Wed Jul 15, 2015 7:06 pm    Post subject: Reply with quote

I have mounted /dev/sda1 on /mnt/gentoo, however I dont see the /mnt/gentoo/etc/rc.conf. Do I need to manually create it and then put these options under this fine ?

Please suggest.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43425
Location: 56N 3W

PostPosted: Wed Jul 15, 2015 9:00 pm    Post subject: Reply with quote

manu_leo,

If you are using OpenRC the file should exist and start with
Code:
# Global OpenRC configuration settings

# Set to "YES" if you want the rc system to try and start services
# in parallel for a slight speed improvement. When running in parallel we
# prefix the service output with its name as the output will get
# jumbled up.
# WARNING: whilst we have improved parallel, it can still potentially lock
# the boot process. Don't file bugs about this unless you can supply
# patches that fix it without breaking other things!
#rc_parallel="NO"
rc_parallel="NO"

Maybe sda1 is not your root filesystem?

If you use systemd, I can't help. You should probably start a new thread.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Thu Jul 16, 2015 4:26 am    Post subject: Reply with quote

Neddy, I am using openrc.

I did a chroot , and then ran locale-gen. It gave me an error as
Code:
/usr/sbin/locale-gen: line 17: /etc/init.d/functions.sh: No such file or directory


When I went under /etc/init.d, and did an ls , the functions.sh file was missing

Quote:
(chroot) livecd /etc/init.d # ls
busybox-ntpd
git-daemon
pciparm
rsyncd
vixie-cron
busybox-watchdog
jira
postfix
shutdown.sh
vnc
crucible
pydoc-2.7
sshd
vsftpd
cups-browsed
kmod-static-nodes
pydoc-3.3
syslog-ng
xdm
cupsd
mysql
pydoc-3.4
sysstat
xdm-setup
dbus
net.eno1
reboot.sh
udev
fuse
net.lo
rstudio-server udev-mount


I then went under /etc/runlevels/sysinit to check the status and the softlinks for devfs , dmesg , sysfs and tmpfiles.dev are in RED.

Code:
(chroot) livecd # cd /etc/runlevels/sysinit
ls -lth
total 0
lrwxrwxrwx 1 root root 16 Dec  4  2014 udev -> /etc/init.d/udev
lrwxrwxrwx 1 root root 22 Dec  4  2014 udev-mount -> /etc/init.d/udev-mount

[color=darkred]lrwxrwxrwx 1 root root 17 Dec  4  2014 devfs -> /etc/init.d/devfs
lrwxrwxrwx 1 root root 17 Dec  4  2014 dmesg -> /etc/init.d/dmesg
lrwxrwxrwx 1 root root 17 Dec  4  2014 sysfs -> /etc/init.d/sysfs
lrwxrwxrwx 1 root root 24 Dec  4  2014 tmpfiles.dev -> /etc/init.d/tmpfiles.dev[/color]

lrwxrwxrwx 1 root root 29 Dec  4  2014 kmod-static-nodes -> /etc/init.d/kmod-static-nodes


There is something which is broken, but I am not sure what it is . Please suggest some way as I am running out of ideas.

Thanks in advance.


Last edited by manu_leo on Thu Jul 16, 2015 5:58 am; edited 1 time in total
Back to top
View user's profile Send private message
manu_leo
Guru
Guru


Joined: 20 Jan 2014
Posts: 453
Location: India

PostPosted: Thu Jul 16, 2015 4:30 am    Post subject: Reply with quote

I booted from the livecd and here is how the mounts look like

Code:
/dev/sda1 on /mnt/gentoo/
/dev/sda3 on /mnt/gentoo/var
/dev/sda4 on /mnt/gentoo/home
proc on /mnt/gentoo/proc
/sys/ on /mnt/gentoo/sys
/dev/ on /mnt/gentoo/dev


So I am using /dev/sda1 as my root partition.

Under /mnt/gentoo where previously I did an untar to the stage3 ball, I see all the directories intact, so nothing is lost, its just the repair/scan I believe we need to do to get the system back to normal.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum