Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Xen kernel load fails
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Thu Nov 17, 2016 2:25 am    Post subject: [SOLVED] Xen kernel load fails Reply with quote

After much effort, I was able to get an instance of Gentoo running on a UEFI based system (a simple Supermicro (Intel Atom) server with 64GB ram & 4TB hard drive). Now I am trying to install Xen referencing the Gentoo Xen Wiki.

My Gentoo kernel was compiled with genkernel following the AMD64 Handbook. Once I overcame the boot loading issues, I then modified the kernel in accordance with the suggestions of the Gentoo Xen Wiki. That kernel, when loaded by itself, works fine.

I installed xen and xen-tools and they created kernels under /boot (below represents the current state):
Quote:
zeta jlpoole # ls -la /boot
total 35034
drwxr-xr-x 5 root root 1536 Dec 31 1969 .
drwxr-xr-x 23 root root 4096 Nov 11 16:07 ..
drwxr-xr-x 4 root root 512 Nov 12 16:42 efi
drwxr-xr-x 6 root root 1024 Nov 16 09:59 grub
-rwxr-xr-x 1 root root 23285812 Nov 16 09:59 initramfs-genkernel-x86_64-4.4.26-gentoo
-rwxr-xr-x 1 root root 0 Nov 16 04:03 .keep
-rwxr-xr-x 1 root root 5620880 Nov 16 09:52 kernel-genkernel-x86_64-4.4.26-gentoo
drwxr-xr-x 2 root root 512 Nov 12 00:16 lost+found
-rwxr-xr-x 1 root root 3286053 Nov 16 09:52 System.map-genkernel-x86_64-4.4.26-gentoo
-rwxr-xr-x 1 root root 949 Nov 16 04:02 xen-4.7.1.config
-rwxr-xr-x 1 root root 917876 Nov 16 04:03 xen-4.7.1.gz
-rwxr-xr-x 1 root root 917876 Nov 16 04:03 xen-4.7.gz
-rwxr-xr-x 1 root root 917876 Nov 16 04:03 xen-4.gz
-rwxr-xr-x 1 root root 917876 Nov 16 04:03 xen.gz
zeta jlpoole #


When Xen did not load, I upgraded xen and xen tools to the high watermark, as I read some hint that there were problems with Xen and EUFI and grub2, but they were aware of it and it was being worked on. So I have currently installed:
Code:
zeta jlpoole # eix -I xen
[I] app-emulation/xen
     Available versions:  4.6.3-r3^st ~4.6.4^st ~4.7.0-r3^st (~)4.7.1^st {custom-cflags debug efi flask}
     Installed versions:  4.7.1^st(08:03:39 PM 11/15/2016)(efi flask -custom-cflags -debug)
     Homepage:            http://xen.org/
     Description:         The Xen virtual machine monitor

[I] app-emulation/xen-tools
     Available versions:  4.6.3-r2(0/4.6)^t ~4.6.4(0/4.6)^t ~4.7.0-r2(0/4.7)^t (~)4.7.1(0/4.7)^t {api custom-cflags debug doc flask hvm ocaml ovmf +pam pygrub python qemu +qemu-traditional screen sdl static-libs system-qemu system-seabios PYTHON_TARGETS="python2_7"}
     Installed versions:  4.7.1^t(08:01:41 PM 11/15/2016)(flask hvm ovmf pam qemu qemu-traditional screen sdl -api -custom-cflags -debug -doc -ocaml -pygrub -python -static-libs -system-qemu -system-seabios PYTHON_TARGETS="python2_7")
     Homepage:            http://xen.org/
     Description:         Xen tools including QEMU and xl

Found 2 matches
zeta jlpoole #


The symptoms I experience on an attempt to load Xen are: the echos of the statement loading both kernels appear, then a row across the top of the screen goes white and then the system hangs. Previously, I placed an echo just after loading the initramfs and it was duly executed, so I'm concluding that the image itself has a problem, or that I am not providing a needed parameter. What is helpful to know is that kernel-genkernel-x86_64-4.4.26-gentoo and initramfs-genkernel-x86_64-4.4.26-gentoo do work, as they are used when I boot into a regular non-Xen session.

Here are links to:
/boot/grub/grub.cfg
/boot/xen-4.7.1.config
genkernel session
kernel configuration

Is there something more I can post to my site that would be helpful in gaining insight as to what problem I may be experiencing?

More importantly, what suggestions do you have that I should try to troubleshoot this. I do not know if it is a problem with the xen.gz image or the grub boot loader. I'm hoping the symptoms I have described cause someone to recognize it as relating to something like failing to specify the file system, or root disk. Lastly, would anyone who has successfully launched Xen on a UEFI system with grub2 consider sharing their grub.cfg, or a portion thereof and/or their kernel .config? It would be helpful to be able to compare against a known successful configuration.


Last edited by jlpoole on Sat Dec 10, 2016 1:28 am; edited 1 time in total
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Thu Nov 17, 2016 4:54 am    Post subject: Looks Like Serial Console Time Reply with quote

From https://wiki.xenproject.org/wiki/Xen_Serial_Console
Quote:
If your server is having problems and it crashes, you usually just see an empty VGA console when you go look at it. That doesn't help you at all troubleshooting the actual problem. Serial console to the rescue!


I guess I'm going to learn about serial console monitoring.

I boot into a normal session an install syslog-ng and activated it. Then I shutdown and tried various Xen menus all without success and then booted back into a normal session to see if anything was captured on the logs. There was nothing in between the last shutdown and the start of a regular Gentoo session. Moreover, during the attempt to launch Xen, the disk light had no activity, so the problem was occurring before the disk was even activated.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Thu Nov 17, 2016 7:03 am    Post subject: Ipmi Reply with quote

WOW!

I dug around for cables &etc. to see if I could monitor my computer, and then I learned about Intelligent Platform Management Interface (IPMI) which my server has a feature. This is all new to me and I thought I would post here because I'm believing that I may be able to monitor my server's boot failure on loading the Xen kernel via the network to my workstation using IPMI. Right now, I have a window open on my workstation that depicts the BIOS window on the server and I just took control of it on my workstation.

Yes, for those people who work in data centers this is something you all know for day-to-day use, but for a person such as I who works in software and uses Gentoo during his personal time, this is a remarkable feature (and a horrendous security hazard if not locked down).

If your computer support IPMI and you are having issues with booting, this may be an area worth learning about so you can set up a monitor of what is happening during the boot phase when screens go dark.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Thu Nov 17, 2016 2:43 pm    Post subject: Reply with quote

With Intelligent Platform Management Interface (IPMI) I was able to record an entire sequence from entering "reboot" in a shell to the failure to load Xen:

https://youtu.be/TwWMmU0dYA8

I never knew this type of tool existed and feel like I have been in the Dark Ages. Any rate, there it is for readers who have been like me and unaware of the power of IPMI. Now... to get the Serial Over Lan (SOL) to work.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Wed Dec 07, 2016 2:42 am    Post subject: Re: Xen kernel load fails Reply with quote

jlpoole wrote:
[snip]
More importantly, what suggestions do you have that I should try to troubleshoot this. I do not know if it is a problem with the xen.gz image or the grub boot loader.
[snip]


After consuming about 3 weeks including sending my server back to Supermicro get the Serial Over Lan ("SOL") working, that effort fizzled and was a waste of time, so I obtained a serial null modem cable and now with PuTTY I am monitoring through the hardware serial port. The SOL did not offer anything more than the regular console. So, I tried tinkering with the /boot/grub/grub.cfg and eventually the command line interface in GRUB2 and determined that the problem I am facing is either missing or incorrect GRUB settings or the xen.gz kernel is no good or I am not setting the serial port correctly from the Xen instance.

The regular Gentoo kernel works -- I can launch that and run the machine in a regular Gentoo session. The problem I encounter is when I tried to boot to the Xen kernel -- everything hangs.
I can successfully load the xen.gz kernel, kernel-genkernel-x86_64-4.4.26-gentoo and initramfs-genkernel-x86_64-4.4.26-gentoo manually in the grub command line. But when I go to boot, nothing happens -- nothing posts to the console or serial port.

http://pasik.reaktio.net/xenserialconsolelog.tx suggests that I should be getting entries such as this:

Quote:
(XEN) Xen version 3.4.1 (root@(none)) (gcc version 4.4.0 20090506 (Red Hat 4.4.0-4) (GCC) ) Mon Aug 17 22:24:38 EEST 200
9
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: dom0_mem=1024M loglvl=all guest_loglvl=all com1=19200,8n1 console=com1
...


Since I do not even get that, I'm concluding that the problem is with the xen.gz kernel and/or the configuration of the serial port. I guess I'm so far out of the normal path here, I'm on my own, but document here this long journey.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Sat Dec 10, 2016 1:28 am    Post subject: Success Reply with quote

Great news: I succeeded in launching my dom0 on top of Xen.
See https://bugs.gentoo.org/show_bug.cgi?id=602052

I haven't tested any clients and my way to launching was very dodgy (dropping into an EFI shell from GRUB2), so I do not know launching clients will work -- it's been awhile since I've been playing around in Xen and my droping into an EFI shell may preclude launch other clients?? To be determined. Any rate, getting to the dom0 stage atop a Xen on a machine that is UEFI is a tremendous step forward for me.

I had to patch Xen-4.8.0 and that finally allowed me to build a Xen-4.8.0.efi file which lodges in the EFI (FAT32) portion under /boot. Then I had to copy my kernel and initramfs into the same directory, /boot/efi/gentoo and specify my configuration file on the same command line and Xen, then my Kernel launched and the initramfs loaded and voila!.

So, where shall I share what I learned?
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Sat Dec 10, 2016 6:09 pm    Post subject: Video of launching Xen EFI from EFI Shell Reply with quote

Video from reboot to login depicting how I dropped into the EFI Shell by exiting Grub2's command line

https://www.youtube.com/watch?v=sjHzXfjVnxw
Back to top
View user's profile Send private message
Thoth93
n00b
n00b


Joined: 26 Feb 2017
Posts: 3

PostPosted: Fri Mar 03, 2017 4:37 am    Post subject: Xen + UEFI Reply with quote

I've ran into the same problem. I might have made some progress, I can boot with Grub2 into Xen but get lots
of errors that dont show up when I do BIOS boot.
:arrow: https://forums.gentoo.org/viewtopic-p-8036938.html#8036938
Read the begining ** and *** It also talks about using efibootmgr to add xen.efi to UEFI boot havenot yet tryed
that out. Also just today read about using grub2 to chain load xen.efi, I Plan on trying that out soon. Let me know
if it helps and or what you think.You know how I can get this info into the xen wiki ? Maby we can get this to work
and have them update it.
Back to top
View user's profile Send private message
jlpoole
Guru
Guru


Joined: 01 Nov 2005
Posts: 347
Location: Salem, OR

PostPosted: Fri Mar 03, 2017 4:43 am    Post subject: Re: Xen + UEFI Reply with quote

Thoth93 wrote:
... Let me know
if it helps and or what you think.You know how I can get this info into the xen wiki ? Maby we can get this to work
and have them update it.


I'm afraid I cannot focus on Xen-Gentoo matters for many months, major life event will be consuming my time. Good luck!
Back to top
View user's profile Send private message
83657664
n00b
n00b


Joined: 16 Mar 2011
Posts: 50

PostPosted: Sun Mar 05, 2017 5:23 pm    Post subject: Re: Video of launching Xen EFI from EFI Shell Reply with quote

jlpoole wrote:
Video from reboot to login depicting how I dropped into the EFI Shell by exiting Grub2's command line

https://www.youtube.com/watch?v=sjHzXfjVnxw


how i set a efi shell in grub?
when i type "exit",it just return to motherboard uefi mode,
please teach me how?
there is so little info on efi shell,
better record a youtube video. :lol: :lol: :lol:
thanks!!!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum