Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
Meltdown/Spectre: Unauthorized Disclosure of Kernel Memory
View unanswered posts
View posts from last 24 hours

Goto page Previous  1, 2, 3 ... 18, 19, 20, 21, 22, 23  Next  
This topic is locked: you cannot edit posts or make replies.    Gentoo Forums Forum Index Kernel & Hardware
View previous topic :: View next topic  
Author Message
alex-kas
n00b
n00b


Joined: 27 Jan 2018
Posts: 2

PostPosted: Sat Jan 27, 2018 12:58 pm    Post subject: Reply with quote

+1 to Kajzer,
How binary packages can be controlled?
To NeddySeagoon:
I believe too few species in this world and trust ..., sometimes do not trust even to myself ...
Compiling kernel with gcc 7.3.0 produces to me on kernel-4.14.15:
Code:

cat /sys/devices/system/cpu/vulnerabilities/*
Mitigation: PTI
Vulnerable
Mitigation: Full generic retpoline

And I for sure have not recompiled anything else yet.

I can imagine that generically any kind of a kernel or a hypervisor may do a better job having retpoline protection in it. I thus can imagine that kernel is what awaits for retpoline to protect us in the userland. On the same ground firefox maybe can become safer for running javascripts if the firefox itself has some setting for -mindirect-branch during the emerge ... The same for virtualbox and similar stuff.
The last proposition is however a conjecture. What do you guys think about all of that?
Back to top
View user's profile Send private message
PrSo
Tux's lil' helper
Tux's lil' helper


Joined: 01 Jun 2017
Posts: 128

PostPosted: Sat Jan 27, 2018 1:29 pm    Post subject: Reply with quote

I agree with kajzer.
Previously, at time when I wrote post about compiling everything (@world) with retpoline there was a discussion among kernel people how to resolve mitigation for Spectre.

Since AMD's KVM and Intel's VT are vulnerable to Spectre v2 counter action should be done by authors of VM's, ie. Qemu:
https://www.qemu.org/2018/01/04/spectre/
With mitigation applied to kernel host machines should be safe.

Same regards to web browsers. Firefox applied own patches, and so Chrome. Qtwebengine would have mitigation from version 5.9.4 applied:
https://codereview.qt-project.org/#/c/216729/

https://bugreports.qt.io/browse/QTBUG-65561

To conclude IMHO kernel should be enough but also needed to be supported by patches to critical applications.
Back to top
View user's profile Send private message
tholin
Apprentice
Apprentice


Joined: 04 Oct 2008
Posts: 168

PostPosted: Sat Jan 27, 2018 1:50 pm    Post subject: Reply with quote

alex-kas wrote:
Can anyone explain me why should someone think about recompiling something more than a kernel??? I mean to defend from Specter. I was thinking that having kernel compiled with the retpoline switches I kinda guarantee this kernel will help me protecting my box. Am I wrong?

Compiling the kernel with retpolines will protect the kernel but only the kernel. Applications could still read each others memory but not the kernels memory. Recompiling everything will protect everything (except the firmware so you still need the microcode fix).
alex-kas wrote:
Also, as tholin wrote in the previous post, devs try or at least think to implement ELF changes which would be used by the kernel in order to speedup things IF the program with such an ELF has a bit indicating the retpoline protection switches were used. Is it a joke? I will compile my binary my way and then put whatever bit you want :) And the kernel would blindly think it is a wise and proper app???

The new bit would be use to tell the kernel that a program is protected by itself (with retpolines) so that the kernel doesn't have to protect it (using the new microcode functionality). You could compile a program without retpolines and set the bit. Then your program could be attacked by other program but in that case you have yourself to blame. The bit is only used to say a program i immune to attack, it says nothing about how trustworthy a program is.
alex-kas wrote:
So, back to the question: what is the background for aiming to the whole box re-compilation? And, if yes, how to do the same with external/binary packages??? And if yes, recompile and yes, no control of binary packages then how to protect if these are malicious intruders???

Binary packages without retpolines would have to be protected my the kernel using using the new microcode functionality. This has nothing to do with binary packages being evil.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Sat Jan 27, 2018 2:28 pm    Post subject: Reply with quote

There are a lot of different things mixed here:

The point of spectre v2 is that a malevolent program can steal "any" data from every program it can access (e.g. by calling) which is not compiled with retpoline.

Since the kernel can be called by any program and can access any data, this is the most critical part: If this is not protected against stealing data, every other protection makes no sense.

But you will of course want to protect (i.e. recompile with retpoline) as well all other programs which deal with (or can be used to access) sensible data: All login/encryption/mail/... software
PrSo wrote:
counter action should be done by authors of VM's, ie. Qemu [...]
With mitigation applied to kernel host machines should be safe.

If VM/Qemu/... itself uses retpolines in a proper way (and has no other security hole) the host machines are "safe" in the sense that one virtual machine cannot steal data from another one. But inside one virtual machine programs can happily steal data from each other unless these programs protect themselves against data theft by using retpoline.
Quote:
Same regards to web browsers. Firefox applied own patches, and so Chrome

This is again a different story: Browsers attempt to make it harder to steal data by restricting the possibilities of browsers. So with a "protected" browser it is not so simple for a malevolent website (even if you access it with activated javascript) to read all data it wants from all of your programs which still lack repoline support. However, "not so simple" really means what it says; it is not impossible but only some additional obstacles have been built (e.g. by giving the website less accurate timers).
Back to top
View user's profile Send private message
PrSo
Tux's lil' helper
Tux's lil' helper


Joined: 01 Jun 2017
Posts: 128

PostPosted: Sat Jan 27, 2018 4:17 pm    Post subject: Reply with quote

With the assumption that every program should be treated as "untrusted" with a malicious code inside? Fair enough.

But even author of retpoline patches says:
Quote:
Binaries with shared linkage
While our initial focus has been the protection of operating system and hypervisor-type targets, there are classes of user application for which this coverage is valuable. In these cases, it should be called out that shared linkage and runtimes will lead to frequent additional interactions with indirect branches. Ubiquitous examples include the Program Link Table (PLT) and dynamically loaded standard libraries. We will be publishing additional optimization notes and techniques for these cases.


source:
https://support.google.com/faqs/answer/7625886

As I my understanding is correct Spectre is cpu architectural design flaw, so without "in silicon" change it is impossible to eliminate the threat, and all those mitigations should only minimize the probability of stealing other program memory.

In that case I also wonder if there would be any problems with compile everything with retpoline, similar like with LTO and graphite.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43186
Location: 56N 3W

PostPosted: Sat Jan 27, 2018 4:27 pm    Post subject: Reply with quote

PrSo,

The software fix reduces or eliminates the threat by reducing or eliminating the use of the flawed silicon.
This comes at a performance cost.

Silicon changes will be required so that the retpoline patches are no longer required.
Once revised silicon is available, the retpoline patches will need to be dropped or the lost performance will not be regained.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Sat Jan 27, 2018 6:40 pm    Post subject: Reply with quote

NeddySeagoon wrote:
Once revised silicon is available, the retpoline patches will need to be dropped or the lost performance will not be regained.

The lost performance can never be regained: The whole prediction technique is vulnerable.
If newer processor will be faster then only because of other innovations, but not because of "regaining" the loss...
If I understand correctly, the current retpoline technique does not add additional overhead but just exactly prevents prediction.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 43186
Location: 56N 3W

PostPosted: Sat Jan 27, 2018 7:21 pm    Post subject: Reply with quote

mv,

When you prevent prediction, you lose the performance that correct prediction would have gained.
That's a net loss to CPU throughput.

mv wrote:
The lost performance can never be regained: The whole prediction technique is vulnerable.

That may be an implementation detail. I don't know enough about the implementation of branch prediction to know if it can be made safe.
There will be a lot of effort going into making it work safely.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Pearlseattle
Apprentice
Apprentice


Joined: 04 Oct 2007
Posts: 153
Location: Switzerland

PostPosted: Sat Jan 27, 2018 11:07 pm    Post subject: Reply with quote

tholin wrote:
alex-kas wrote:
Can anyone explain me why should someone think about recompiling something more than a kernel??? I mean to defend from Specter. I was thinking that having kernel compiled with the retpoline switches I kinda guarantee this kernel will help me protecting my box. Am I wrong?

Compiling the kernel with retpolines will protect the kernel but only the kernel. Applications could still read each others memory but not the kernels memory. Recompiling everything will protect everything (except the firmware so you still need the microcode fix).
alex-kas wrote:
Also, as tholin wrote in the previous post, devs try or at least think to implement ELF changes which would be used by the kernel in order to speedup things IF the program with such an ELF has a bit indicating the retpoline protection switches were used. Is it a joke? I will compile my binary my way and then put whatever bit you want :) And the kernel would blindly think it is a wise and proper app???

The new bit would be use to tell the kernel that a program is protected by itself (with retpolines) so that the kernel doesn't have to protect it (using the new microcode functionality). You could compile a program without retpolines and set the bit. Then your program could be attacked by other program but in that case you have yourself to blame. The bit is only used to say a program i immune to attack, it says nothing about how trustworthy a program is.
alex-kas wrote:
So, back to the question: what is the background for aiming to the whole box re-compilation? And, if yes, how to do the same with external/binary packages??? And if yes, recompile and yes, no control of binary packages then how to protect if these are malicious intruders???

Binary packages without retpolines would have to be protected my the kernel using using the new microcode functionality. This has nothing to do with binary packages being evil.


Well, I'm following this whole story since Jan 3 and everything has been very exciting and dynamic and etc... but to sum it up, especially regarding the fixes, I admit that I did not quite understand anything, hehe... .
Therefore, the questions posted by alex-kas kind of mirror my own ones.
I think that I do understand quite well the part involving the kernel being protected (by using retpolines, Kpti, whatever... AND the firware of case of Intel CPUs) => the kernel is protected no-matter-what, and that's it, right?!

But I still do not understand basically "everything else":
1) The new bit would be use to tell the kernel that a program is protected by itself (with retpolines) so that the kernel doesn't have to protect it (using the new microcode functionality)
The kernel would therefore "speculate" (hehe) that the program hasn't been corrupted? Meaning that if I'm a great cracker and I find out that I can (re/over)write program X on my target's server I'll just flip the bit and I'll have access to everything (not related to the kernel)?

2) Binary packages without retpolines would have to be protected my the kernel using using the new microcode functionality. This has nothing to do with binary packages being evil.
Basically same thing as #1, but let's say that somebody flips the retpoline-bit in a binary package (being nVidia drivers, Teamviewer, Chrome, etc...) => then just because of "THAT ONE PACKAGE" the whole environment would potentially be back to stoneage (if anyone uses that SW to find a flaw in the environment being OS or other apps), or not?

1+2)
Ok, summarized, I'm basically asking the following:
in a best-case-scenario, are we back to stoneage every-single-time that an attacker manages to introduce and execute on a host ANY ((pre)compiled) code that has the retpoline-flag switched on (as fake) which in turn woud exploit the meltdown/spectre analysis to gather critical info about the running kernel/programs (to then of course access through "normal" app and/or kernel bugs to then gain control of the target device)?

Thaaaaaaks a lot!
:D

Edit: added some details 3 minutes later
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Sun Jan 28, 2018 5:28 am    Post subject: Reply with quote

Pearlseattle wrote:
I'll just flip the bit and I'll have access to everything (not related to the kernel)?

No, the opposite: If you flip the bit, that program (and everything it can access) can potentially be accessed "from the outside". So if it is a program which is run with root permissions, everything can potentially be accessed "from the outside". But if in attacker can manage to flip the bit in a program which is run with root permissions, he can do everything else with this program, anyway, and in particular directly access and send the data where he wants to.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Sun Jan 28, 2018 5:34 am    Post subject: Reply with quote

NeddySeagoon wrote:
When you prevent prediction, you lose the performance that correct prediction would have gained.
[...]
That may be an implementation detail.

It is not: If you don't lose time in one case but lose it in another, this can always be measured and used to understand which branch you have taken.
The only "solution" is that you do not loose time in any case; but this means simply higher internal parallelism and not prediction.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Sun Jan 28, 2018 6:49 am    Post subject: Reply with quote

The released binuitils-2.30 does not seem to know -z,retpolineplt:
Code:
$ grep -Ri retpo binutils-2.30 || echo :\(
:(

Edit: I guess the newest gold linker from llvm does.
Back to top
View user's profile Send private message
lagalopex
Guru
Guru


Joined: 16 Oct 2004
Posts: 545

PostPosted: Mon Jan 29, 2018 7:31 am    Post subject: Reply with quote

mv wrote:
The released binuitils-2.30 does not seem to know -z,retpolineplt

Maybe have a look at their ml.
tl;dr:
Florian Weimer wrote:
I expect that compiler support for indirect branch rewriting together with -fno-plt is sufficient. There is simply no need to put any code into binutils at this point.

_________________
System: AMD FX 8350, 16GB RAM, NVidia GeForce GTX 750 Ti, Asus M5A99X EVO R2.0
Back to top
View user's profile Send private message
kajzer
Guru
Guru


Joined: 27 Nov 2014
Posts: 434

PostPosted: Mon Jan 29, 2018 2:45 pm    Post subject: Reply with quote

With gcc-7.3.0 :

Code:
$ cat /sys/devices/system/cpu/vulnerabilities/*                                                                                                                                                         
Mitigation: PTI
Vulnerable
Mitigation: Full generic retpoline
Back to top
View user's profile Send private message
gengreen
Tux's lil' helper
Tux's lil' helper


Joined: 23 Dec 2017
Posts: 84

PostPosted: Wed Jan 31, 2018 8:00 pm    Post subject: Reply with quote

I applied the retpoline patch on a kernel 4.9.74 (minipli) and have rebuild my entire system with gcc 7.3.0 p1.0 on my glibc system.

Unlike the most recent kernel, I do not have
Code:
/sys/devices/system/cpu/vulnerabilities/
to give me some information about the effect of retpoline on my system, but the performance did decrease badly, especially with qemu.

I'm currently rebuilding my musl system with gcc 7.3.0 and a retpoline kernel too, I except it will be more or less the same in loss of performance.
Back to top
View user's profile Send private message
Elleni
l33t
l33t


Joined: 23 May 2006
Posts: 858

PostPosted: Thu Feb 01, 2018 7:35 pm    Post subject: Reply with quote

Elleni wrote:
PrSo wrote:
pjp wrote:
PrSo wrote:
pjp wrote:

That sounds to me like CONFIG_PAGE_TABLE_ISOLATION should be enabled for AMD processors. Or at least not setting it with the knowledge of leaving the vulnerability exposed.

So it seems that even if you compile kernel with CONFIG_PAGE_TABLE_ISOLATION=Y PTI is auto-disabled on AMD cpu anyway.
But the underlying issue is still whether or not AMD should have it enabled. From the prior information, the answer appears to be yes.

To enable the functionality, I had to enable the kernel option AND enable it on the kernel command line with "pti=on". After that (and only after that):
Code:
 dmesg |grep -i isol
[    0.000000] Kernel/User page tables isolation: force enabled on command line.
[    0.000000] Kernel/User page tables isolation: enabled
(I got the idea from Naib's post on page 5 of this thread which referenced "pti=off". Thanks Naib!)



I think that you are playing here the advocatus diaboli role.

With the knowledge that the test case provided on wiki page was performed in 2013, and should be mitigated by KAISER (now PTI) I personally think that AMD statement to which you got link in mike155 post is still in power, of course with the assumption that AMD is aware of that vulnerability.

Thomas Lendacky wrote:
AMD processors are not subject to the types of attacks that the kernel page table isolation feature protects against.


I know that this could be some kind of uncomfortable situation but there is nothing more we can do for now than to trust AMD with that. Maybe someone will write PoC on that case in the near future proofing that AMD was duly diligent.

If you think different on that subject please feel free to contact AMD an ask them to resolve your possible concerns.


Mhm, now what is officially recommended on amd ryzen boxes ? Ehable CONFIG_PAGE_TABLE_ISOLATION=Y PTI and as its autodisabled by default enable it on the kernel command line with "pti=on" ? Or is this not required ?


Can anyone say something on amd ryzen really not being affected or may it be advisable to enable autodisabled PTI even on amd ryzen?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Thu Feb 01, 2018 8:13 pm    Post subject: Reply with quote

The new gcc flags cause some problems on x86 (i.e. 32 bit):
gcc-7.3.0 and rust won't compile with -fno-plt (segfault), and firefox won't compile if any of -mindirect-branch=thunk or -mfunction-return=thunk is used.

There might be more (perhaps also for 64 bit...): This git repository will probably be updated regularly.

Edit: Another surprising observation: -mindirect-branch=thunk without -O2 (or better) will not protect you...
Back to top
View user's profile Send private message
kernelOfTruth
Watchman
Watchman


Joined: 20 Dec 2005
Posts: 6108
Location: Vienna, Austria; Germany; hello world :)

PostPosted: Fri Feb 02, 2018 1:08 am    Post subject: Reply with quote

mv wrote:
The new gcc flags cause some problems on x86 (i.e. 32 bit):
gcc-7.3.0 and rust won't compile with -fno-plt (segfault), and firefox won't compile if any of -mindirect-branch=thunk or -mfunction-return=thunk is used.

There might be more (perhaps also for 64 bit...): This git repository will probably be updated regularly.

Edit: Another surprising observation: -mindirect-branch=thunk without -O2 (or better) will not protect you...


*Subscribing*

good to know !


Exactly what I was attempting to do: compile firefox :?

Thanks for saving me (and probably others) some time with troubleshooting to get firefox to compile, mv


I'll wait and see what will be done to get firefox to work with those retpoline flags ...


4.14.16 with gcc 7.3:

Quote:
cat /sys/devices/system/cpu/vulnerabilities/*
Mitigation: PTI
Vulnerable
Mitigation: Full generic retpoline

_________________
https://github.com/kernelOfTruth/ZFS-for-SystemRescueCD/tree/ZFS-for-SysRescCD-4.9.0
https://github.com/kernelOfTruth/pulseaudio-equalizer-ladspa

Hardcore Gentoo Linux user since 2004 :D
Back to top
View user's profile Send private message
Naib
Watchman
Watchman


Joined: 21 May 2004
Posts: 5625
Location: Removed by Neddy

PostPosted: Fri Feb 02, 2018 1:10 pm    Post subject: Reply with quote

Code:
uname -a; for f in /sys/devices/system/cpu/vulnerabilities/*; do echo "${f##*/}:$(cat $f)" ;done
Linux fluidmotion 4.15.0-gentoo #1 SMP PREEMPT Mon Jan 29 23:30:55 GMT 2018 x86_64 AMD Ryzen 5 1600 Six-Core Processor AuthenticAMD GNU/Linux
meltdown:Not affected
spectre_v1:Vulnerable
spectre_v2:Mitigation: Full AMD retpoline


just need to sort out v1
_________________
The best argument against democracy is a five-minute conversation with the average voter
Great Britain is a republic, with a hereditary president, while the United States is a monarchy with an elective king
Back to top
View user's profile Send private message
pjp
Administrator
Administrator


Joined: 16 Apr 2002
Posts: 17918

PostPosted: Fri Feb 02, 2018 3:26 pm    Post subject: Reply with quote

Elleni wrote:
Can anyone say something on amd ryzen really not being affected or may it be advisable to enable autodisabled PTI even on amd ryzen?
I can only add another reference of someone else also expressing interest. Until I learn otherwise, I'm considering it a desirable security setting.

Bug 643228, Comment 11 wrote:
I wish to present something else for consideration. The KAISER patchset was originally intended to harden the implementation of KASLR in the Linux kernel [1] [2]. It was hastily re-purposed to address Meltdown, and re-branded as KPTI in the process. Later, Thomas Lendacky submitted a patch that prevents KPTI from being activated by default for AMD processors - a patch that gentoo-sources is now carrying. AMD's pretext is that their processors are not affected by Meltdown.

My concern over this situation is that it may put AMD processors at a disadvantage in lieu of the security issue that KAISER was originally intended to protect against. That is, KASLR may be unnecessarily weakened on AMD processors, by default. Indeed, the "Practical Timing Side Channel Attacks Against Kernel Space ASLR" whitepaper [3] specifically tested their attacks on AMD processors, which were found to be affected.

Assuming that I'm correct, AMD users who enable both CONFIG_RANDOMIZE_BASE and CONFIG_PAGE_TABLE_ISOLATION will need to explicitly pass "pti=on" as a kernel option in order to harden KASLR, whereas Intel users will not. I realise that this is a less pressing concern then attending to Meltdown, but it struck me as being worthy of mention.

_________________
Those who know what's best for us must rise and save us from ourselves.
Back to top
View user's profile Send private message
Hossie
Tux's lil' helper
Tux's lil' helper


Joined: 08 Dec 2005
Posts: 115

PostPosted: Fri Feb 02, 2018 6:15 pm    Post subject: Reply with quote

So what compilation options for userspace is recommended now?

Code:
-mindirect-branch=thunk -mfunction-return=thunk


?
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Fri Feb 02, 2018 8:13 pm    Post subject: Reply with quote

Hossie wrote:
So what compilation options for userspace is recommended now?

I would guess
Code:
-O2 -fno-plt -mindirect-branch=thunk -mfunction-return=thunk

for gcc. And with the next clang, I guess either
Code:
-O2 -fno-plt -mretpoline

or (if this should be supported by the llvmgold plugin):
Code:
-O2 -Wl,-z,repolineplt -mrepoline
Back to top
View user's profile Send private message
transpetaflops
Tux's lil' helper
Tux's lil' helper


Joined: 16 May 2005
Posts: 136

PostPosted: Fri Feb 02, 2018 8:44 pm    Post subject: Reply with quote

mv wrote:
And with the next clang, I guess either
Code:
-O2 -fno-plt -mretpoline

or (if this should be supported by the llvmgold plugin):
Code:
-O2 -Wl,-z,repolineplt -mrepoline


Just out of curiosity. Do I define CFLAGS/CXXFLAGS specifically for clang somewhere in make.conf? I don't remember seeing it in the handbook.
Back to top
View user's profile Send private message
mv
Watchman
Watchman


Joined: 20 Apr 2005
Posts: 6281

PostPosted: Sat Feb 03, 2018 6:36 am    Post subject: Reply with quote

transpetaflops wrote:
Do I define CFLAGS/CXXFLAGS specifically for clang somewhere in make.conf?

portage does not care which compiler you use, i.e. these variables just refer to your currently configured compiler.
If you want to switch compilers on a per-package bases, you might want to use app-portage/portage-bashrc-mv (from the mv overlay) which a few days ago has been updated to filter by default the correspondingly unknown new flags for gcc/clang.
Back to top
View user's profile Send private message
lagalopex
Guru
Guru


Joined: 16 Oct 2004
Posts: 545

PostPosted: Sat Feb 03, 2018 7:45 am    Post subject: Reply with quote

mv wrote:
I would guess
Code:
-O2 -fno-plt -mindirect-branch=thunk -mfunction-return=thunk

for gcc. And with the next clang, I guess either
Code:
-O2 -fno-plt -mretpoline

or (if this should be supported by the llvmgold plugin):
Code:
-O2 -Wl,-z,repolineplt -mrepoline


That should be described in wiki (Project:Security/Vulnerabilities/Meltdown and Spectre) as well?
Pushing a news item would be nice as well, right?
_________________
System: AMD FX 8350, 16GB RAM, NVidia GeForce GTX 750 Ti, Asus M5A99X EVO R2.0
Back to top
View user's profile Send private message
Display posts from previous:   
This topic is locked: you cannot edit posts or make replies.    Gentoo Forums Forum Index Kernel & Hardware All times are GMT
Goto page Previous  1, 2, 3 ... 18, 19, 20, 21, 22, 23  Next
Page 19 of 23

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum