Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
upower keeps segfaulting
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
jhon987
Apprentice
Apprentice


Joined: 18 Nov 2013
Posts: 253

PostPosted: Thu Dec 21, 2017 4:08 pm    Post subject: upower keeps segfaulting Reply with quote

since I've upgraded to the latest stable kernel 4.14.8 (now r1) I've encountered all sorts of issues most of which I've been able to solve, but here's one I could use some support:

dmesg gives the following:
Code:
[   27.876763] upowerd[4639]: segfault at 8 ip 000055dde7ae76b0 sp 00007ffdc10e9a30 error 4 in upowerd[55dde7ac7000+36000]
[   27.887427] upowerd[4651]: segfault at 8 ip 00005636844326b0 sp 00007ffd1e325600 error 4 in upowerd[563684412000+36000]
[   27.934996] upowerd[4664]: segfault at 8 ip 000056144f63c6b0 sp 00007ffc5745eef0 error 4 in upowerd[56144f61c000+36000]
[   36.192489] upowerd[4880]: segfault at 8 ip 00005559339a76b0 sp 00007ffe45e7c8c0 error 4 in upowerd[555933987000+36000]
....


in order to sort other issues I had (foolishly) 'make distclean' the kernel and reconfigured it all over again, so I'm not sure whether it's a bug or mis-configuration.
According to this bug report: https://bugs.freedesktop.org/show_bug.cgi?id=102903, it might have something to do with openrc's cgroupv2 feature, though, in my trial and error it seems not - I tried setting rc.conf's rc_cgroup_mode="legacy" but the issue persists.
This thread right here: https://forums.gentoo.org/viewtopic-t-1049614-start-0.html, suggest it might be an issue with upower version 0.99.4 but should be resolved in later versions. I've tried upgrading upower to the latest (unstable) version 0.99.7 but the segfaults are still here.

So what do you think is it: openrc bug? / upower bug? / kernel mis-configuration?
and how can I solve this?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14390

PostPosted: Fri Dec 22, 2017 12:23 am    Post subject: Reply with quote

By definition, no well-formed program will die with a segfault, so there is a defect somewhere. It's possible that this is a kernel bug, but it's more likely that it's an application bug provoked by a change elsewhere (whether kernel, openrc, or some library). What is the backtrace for that crash?

See https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces if you need instructions for collecting a backtrace.
Back to top
View user's profile Send private message
jhon987
Apprentice
Apprentice


Joined: 18 Nov 2013
Posts: 253

PostPosted: Fri Dec 22, 2017 11:11 am    Post subject: Reply with quote

Hi Hu,

Thanks for replaying, I actually use: https://wiki.gentoo.org/wiki/Debugging for debugging as I only want to debug certain programs.
Anyways,

Code:
# gdb /usr/bin/upower
GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/upower...Reading symbols from /usr/lib64/debug//usr/bin/upower.debug...done.
done.
(gdb) run
Starting program: /usr/bin/upower
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffeeb6d700 (LWP 11184)]
[New Thread 0x7fffee36c700 (LWP 11185)]

(upower:11180): UPower-WARNING **: Cannot connect to upowerd: Error calling StartServiceByName for org.freedesktop.UPower: GDBus.Error:org.freedesktop.DBus.Error.Spawn.ChildSignaled: Process org.freedesktop.UPower received signal 11
[Thread 0x7fffee36c700 (LWP 11185) exited]
[Thread 0x7fffeeb6d700 (LWP 11184) exited]
[Inferior 1 (process 11180) exited with code 01]
(gdb) backtrace
No stack.


Now the thing is -

Code:
# rc-service -l
NetworkManager
agetty
alsasound
avahi-daemon
avahi-dnsconfd
binfmt
bootmisc
busybox-ntpd
busybox-watchdog
cgmanager
cgproxy
consolefont
consolekit
cronie
cups-browsed
cupsd
dbus
devfs
device-mapper
dhcpcd
dhcpd
dhcrelay
dhcrelay6
dmcrypt
dmesg
dmeventd
elogind
exim
fsck
fuse
git-daemon
gpm
hostname
hwclock
ip6tables
ipset
iptables
keymaps
killprocs
kmod-static-nodes
lighttpd
local
localmount
loopback
lvm
lvm-monitoring
lvmetad
modules
modules-load
mount-ro
mtab
mysql
mysql-s6
mysql-supervise
net-online
net.enp7s0
net.eth0
net.lo
netmount
nginx
ntp-client
ntpd
numlock
opentmpfiles-dev
opentmpfiles-setup
osclock
pciparm
php-fpm
pktcdvd
procfs
pwcheck
pydoc-2.7
pydoc-3.5
pydoc-3.6
root
rsyncd
runsvdir
s6-svscan
saned
saslauthd
savecache
slapd
sntp
sshd
svnserve
swap
swclock
sysctl
sysfs
sysklogd
termencoding
tor
udev
udev-settle
udev-trigger
urandom
wpa_supplicant
xdm
xdm-setup


there is no upowerd service, so I'm really confused. Shouldn't I have a upowerd service, or maybe because of openrc there is no need for one?
As I mentioned previously, I already re-emerged upower when I installed different versions of it and yet the upowerd daemon hasn't become available on my system.

Any ideas?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14390

PostPosted: Sat Dec 23, 2017 12:20 am    Post subject: Reply with quote

According to your dmesg output, upowerd crashed. Why did you then debug upower, a completely different program? Interestingly, it looks like your attempt to debug upower caused DBus to spawn a upowerd, which promptly crashed. You need to debug the crashing upowerd, not the user interface frontend upower.

I do not know why there is no rcscript for upowerd. Presumably, it is unnecessary since DBus launches it on demand.
Back to top
View user's profile Send private message
jhon987
Apprentice
Apprentice


Joined: 18 Nov 2013
Posts: 253

PostPosted: Sat Dec 23, 2017 2:21 am    Post subject: Reply with quote

upowerd seem to me like it should be the daemon of upower, since:
Code:
~ $ whereis upowerd
upowerd: /usr/share/man/man8/upowerd.8.bz2

is the only result I got, I figured maybe I should check out upower and then maybe have a clue regarding upowerd, however your reply made me reconsider, so now:
Code:
$ locate upowerd
/usr/lib64/debug/usr/lib/upower/upowerd.debug
/usr/lib64/upower/upowerd
/usr/share/man/man8/upowerd.8.bz2


but then backtrace is meaningless:
Code:
# gdb /usr/lib64/upower/upowerd
GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib64/upower/upowerd...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/lib64/upower/upowerd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffee20d700 (LWP 23240)]
[New Thread 0x7fffeda0c700 (LWP 23241)]

(upowerd:23236): GLib-GIO-CRITICAL **: g_dbus_proxy_get_connection: assertion 'G_IS_DBUS_PROXY (proxy)' failed

(upowerd:23236): GLib-GIO-CRITICAL **: g_dbus_connection_signal_subscribe: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(upowerd:23236): GLib-GIO-CRITICAL **: g_dbus_proxy_call_sync_internal: assertion 'G_IS_DBUS_PROXY (proxy)' failed

Thread 1 "upowerd" received signal SIGSEGV, Segmentation fault.
0x00005555555746b0 in ?? ()
(gdb) backtrace
#0  0x00005555555746b0 in ?? ()
#1  0x00007ffff6ec0569 in g_type_create_instance ()
   from /usr/lib64/libgobject-2.0.so.0
#2  0x00007ffff6ea3437 in ?? () from /usr/lib64/libgobject-2.0.so.0
#3  0x00007ffff6ea4e2d in g_object_newv () from /usr/lib64/libgobject-2.0.so.0
#4  0x00007ffff6ea557c in g_object_new () from /usr/lib64/libgobject-2.0.so.0
#5  0x00005555555617d2 in ?? ()
#6  0x00007ffff6ec0569 in g_type_create_instance ()
   from /usr/lib64/libgobject-2.0.so.0
#7  0x00007ffff6ea3437 in ?? () from /usr/lib64/libgobject-2.0.so.0
#8  0x00007ffff6ea4e2d in g_object_newv () from /usr/lib64/libgobject-2.0.so.0
#9  0x00007ffff6ea557c in g_object_new () from /usr/lib64/libgobject-2.0.so.0
#10 0x0000555555562fda in ?? ()
#11 0x0000555555560c34 in ?? ()
#12 0x00007ffff67eb541 in __libc_start_main (main=0x555555560a80, argc=1,
    argv=0x7fffffffdb08, init=<optimized out>, fini=<optimized out>,
    rtld_fini=<optimized out>, stack_end=0x7fffffffdaf8)
    at ../csu/libc-start.c:295
#13 0x0000555555560e5a in ?? ()
(gdb)


even though:
/etc/portage/env/debugsyms
Code:

CFLAGS="${CFLAGS} -ggdb"
CXXFLAGS="${CXXFLAGS} -ggdb"
FEATURES="${FEATURES} splitdebug compressdebug -nostrip"
USE="debug"

/etc/portage/env/installsources
Code:

FEATURES="${FEATURES} installsources"

and lastly, /etc/portage/package.env
Code:
sys-libs/glibc debugsyms installsources
sys-power/upower debugsyms installsources


all that even though, as you can see above, debug symbols are installed:
Code:
/usr/lib64/debug/usr/lib/upower/upowerd.debug

?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14390

PostPosted: Sat Dec 23, 2017 6:19 pm    Post subject: Reply with quote

jhon987 wrote:
upowerd seem to me like it should be the daemon of upower
Agreed.
jhon987 wrote:
Code:
$ locate upowerd
/usr/lib64/debug/usr/lib/upower/upowerd.debug
/usr/lib64/upower/upowerd


but then backtrace is meaningless:
Code:
# gdb /usr/lib64/upower/upowerd
Reading symbols from /usr/lib64/upower/upowerd...(no debugging symbols found)...done.


even though:
/etc/portage/env/debugsyms
Code:
FEATURES="${FEATURES} splitdebug compressdebug -nostrip"
and lastly, /etc/portage/package.env
Code:
sys-libs/glibc debugsyms installsources
sys-power/upower debugsyms installsources

all that even though, as you can see above, debug symbols are installed:
Code:
/usr/lib64/debug/usr/lib/upower/upowerd.debug

?
The debug symbols are installed, but not in the correct location. Interestingly, another thread about a different manifestation of this problem came up very recently. You are seeing another manifestation of kill SYMLINK_LIB=yes support. Symbols were split to /usr/lib64/debug/usr/lib/upower, but gdb searches for them in /usr/lib64/debug/path-to-upowerd, meaning /usr/lib64/debug/usr/lib64/upower/upowerd. Note the difference for the inner lib. You can move the debug symbols to the path gdb expects, use FEATURES=nostrip instead of FEATURES=splitdebug, add a symlink to force gdb to find the symbols anyway, or patch upower to install the files in the correct place. I think changing FEATURES is the simplest solution for your case.
Back to top
View user's profile Send private message
jhon987
Apprentice
Apprentice


Joined: 18 Nov 2013
Posts: 253

PostPosted: Sat Dec 23, 2017 9:36 pm    Post subject: Reply with quote

Thanks Hu for your help!
so, here's the backtrace:
Code:
# gdb /usr/lib64/upower/upowerd
GNU gdb (Gentoo 7.12.1 vanilla) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib64/upower/upowerd...done.
(gdb) run
Starting program: /usr/lib64/upower/upowerd
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffee20d700 (LWP 24531)]
[New Thread 0x7fffeda0c700 (LWP 24532)]

(upowerd:24527): GLib-GIO-CRITICAL **: g_dbus_proxy_get_connection: assertion 'G_IS_DBUS_PROXY (proxy)' failed

(upowerd:24527): GLib-GIO-CRITICAL **: g_dbus_connection_signal_subscribe: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(upowerd:24527): GLib-GIO-CRITICAL **: g_dbus_proxy_call_sync_internal: assertion 'G_IS_DBUS_PROXY (proxy)' failed

Thread 1 "upowerd" received signal SIGSEGV, Segmentation fault.
0x00005555555746b0 in up_backend_inhibitor_lock_take (backend=0x55555578e4a0)
    at up-backend.c:501
501                     g_warning ("Could not acquire inhibitor lock: %s", error->message);
(gdb) backtrace
#0  0x00005555555746b0 in up_backend_inhibitor_lock_take (
    backend=0x55555578e4a0) at up-backend.c:501
#1  0x00007ffff6ec0569 in g_type_create_instance ()
   from /usr/lib64/libgobject-2.0.so.0
#2  0x00007ffff6ea3437 in ?? () from /usr/lib64/libgobject-2.0.so.0
#3  0x00007ffff6ea4e2d in g_object_newv () from /usr/lib64/libgobject-2.0.so.0
#4  0x00007ffff6ea557c in g_object_new () from /usr/lib64/libgobject-2.0.so.0
#5  0x0000555555575619 in up_backend_new () at up-backend.c:697
#6  0x00005555555617d2 in up_daemon_init (daemon=0x5555557a2150)
    at up-daemon.c:1087
#7  0x00007ffff6ec0569 in g_type_create_instance ()
   from /usr/lib64/libgobject-2.0.so.0
#8  0x00007ffff6ea3437 in ?? () from /usr/lib64/libgobject-2.0.so.0
#9  0x00007ffff6ea4e2d in g_object_newv () from /usr/lib64/libgobject-2.0.so.0
#10 0x00007ffff6ea557c in g_object_new () from /usr/lib64/libgobject-2.0.so.0
#11 0x0000555555562fda in up_daemon_new () at up-daemon.c:1169
#12 0x0000555555560c34 in up_state_new () at up-main.c:72
#13 main (argc=<optimized out>, argv=<optimized out>) at up-main.c:238
(gdb)

Here's what valgrind had to say:
Code:

valgrind /usr/lib64/upower/upowerd
==24605== Memcheck, a memory error detector
==24605== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24605== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==24605== Command: /usr/lib64/upower/upowerd
==24605==

(upowerd:24605): GLib-GIO-CRITICAL **: g_dbus_proxy_get_connection: assertion 'G_IS_DBUS_PROXY (proxy)' failed

(upowerd:24605): GLib-GIO-CRITICAL **: g_dbus_connection_signal_subscribe: assertion 'G_IS_DBUS_CONNECTION (connection)' failed

(upowerd:24605): GLib-GIO-CRITICAL **: g_dbus_proxy_call_sync_internal: assertion 'G_IS_DBUS_PROXY (proxy)' failed
==24605== Invalid read of size 8
==24605==    at 0x1286B0: up_backend_inhibitor_lock_take (up-backend.c:501)
==24605==    by 0x5B66568: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B49436: ??? (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4AE2C: g_object_newv (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4B57B: g_object_new (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x1157D1: up_daemon_init (up-daemon.c:1087)
==24605==    by 0x5B66568: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B49436: ??? (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4AE2C: g_object_newv (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4B57B: g_object_new (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x116FD9: up_daemon_new (up-daemon.c:1169)
==24605==    by 0x114C33: up_state_new (up-main.c:72)
==24605==    by 0x114C33: main (up-main.c:238)
==24605==  Address 0x8 is not stack'd, malloc'd or (recently) free'd
==24605==
==24605==
==24605== Process terminating with default action of signal 11 (SIGSEGV)
==24605==  Access not within mapped region at address 0x8
==24605==    at 0x1286B0: up_backend_inhibitor_lock_take (up-backend.c:501)
==24605==    by 0x5B66568: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B49436: ??? (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4AE2C: g_object_newv (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4B57B: g_object_new (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x1157D1: up_daemon_init (up-daemon.c:1087)
==24605==    by 0x5B66568: g_type_create_instance (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B49436: ??? (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4AE2C: g_object_newv (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x5B4B57B: g_object_new (in /usr/lib64/libgobject-2.0.so.0.5000.3)
==24605==    by 0x116FD9: up_daemon_new (up-daemon.c:1169)
==24605==    by 0x114C33: up_state_new (up-main.c:72)
==24605==    by 0x114C33: main (up-main.c:238)


According to the above, if I understand the code correctly, the pointer *state (on line 238) is never being mallocd which means it should be a bug on other people's systems as well, yet it seems I'm the only one experiencing it.
I tried upgrading to upower 0.99.7 and it's the same error there as well.

Well, I guess I'll report it on Gentoo's bugzilla, but I really am baffled how come I'm the only one experiencing it, the only explanation I can think of is that I'm wrong regarding *state needs to be mallocd... either that, ot I'm not the only one, but no one else reported it on Gentoo's bugzilla
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14390

PostPosted: Sun Dec 24, 2017 1:35 am    Post subject: Reply with quote

I have not looked long at the code, but I think you misunderstood the message. I read that output to be that the C expression error->message faults because error is NULL. If I am correct, then either g_dbus_proxy_call_with_unix_fd_list_sync is buggy and failed to set the error pointer, or upowerd is buggy for assuming that g_dbus_proxy_call_with_unix_fd_list_sync would set the error pointer. To determine blame, we would need to know whether g_dbus_proxy_call_with_unix_fd_list_sync is documented to always set the error pointer on failure. You could also take the cynical view and say that upowerd is buggy for trusting that g_dbus_proxy_call_with_unix_fd_list_sync would set the pointer, even if the documentation promises that it will always be set. ;)

Either way, this crash will only affect people for whom g_dbus_proxy_call_with_unix_fd_list_sync returns NULL. When out is not NULL, that error log path is not executed. I don't know why that function returns NULL for you, but if other people do not get NULL, they will not get a crash here.
Back to top
View user's profile Send private message
jhon987
Apprentice
Apprentice


Joined: 18 Nov 2013
Posts: 253

PostPosted: Sun Dec 24, 2017 10:39 am    Post subject: Reply with quote

I always get completely lost when I look at other peoples' code. I filed a bug: https://bugs.gentoo.org/642128
I think that if upowerd is relying on g_dbus_proxy... but can't handle exceptions then it may be a bug in g_dbus but it's therefore also a bug in upowerd as well.

I guess I'm cynical... :wink:
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 14390

PostPosted: Sun Dec 24, 2017 5:23 pm    Post subject: Reply with quote

Hypothetically, if g_dbus_proxy_call_with_unix_fd_list_sync were documented to always either: (a) Return non-NULL or (b) Return NULL and set the error pointer, but never (c) return NULL without setting the error pointer, then it would not be wrong for upowerd to be written as it is. Though, like you, I would take the cynical view and add a check for NULL error anyway, since it's cheap and can provide a much better user experience if it ever happens.
Back to top
View user's profile Send private message
jhon987
Apprentice
Apprentice


Joined: 18 Nov 2013
Posts: 253

PostPosted: Mon Dec 25, 2017 2:02 pm    Post subject: Reply with quote

Agreed.
By the way, if g_dbus_proxy_call_with_unix_fd_list_sync is in fact the culprit then:
Quote:
Returns

NULL if error is set. Otherwise a GVariant tuple with return values. Free with g_variant_unref().

Since: 2.30

https://developer.gnome.org/gio/stable/GDBusProxy.html#g-dbus-proxy-call-with-unix-fd-list-sync

which means that g_dbus might not even be the end of the line, because g_dbus returns a value based on the error's state (whether it's set or not), meaning, not setting the error by itself.
So either you follow the chain of failures until you reach the bug's hive, or you simply make upowerd immune to failures in the external function/library.

Personally, I would start with making upowerd immune and only then I would follow the bug...
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum