Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] Problems with the Python update to 3.5
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Thu Nov 30, 2017 8:34 pm    Post subject: [SOLVED] Problems with the Python update to 3.5 Reply with quote

With the recent stabilization of Python 3.5 and the even more recent profile change that makes it the default over 3.4, a newuse emerge update gives a huge list of packages to be rebuilt. This is in principle not a problem (just let it run overnight), if all goes well. However, multiple packages don't build; contextlib2-0.5.4 and qtawesome-0.4.4 for starters:

Code:
 * python3_5: running distutils-r1_run_phase distutils-r1_python_compile
/usr/bin/python3.5 setup.py build
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from setuptools import setup
ImportError: No module named 'setuptools'


Code:
 * python3_5: running distutils-r1_run_phase distutils-r1_python_compile
/usr/bin/python3.5 setup.py build
Traceback (most recent call last):
  File "setup.py", line 8, in <module>
    from distutils.core.command.install import install
ImportError: No module named 'distutils.core.command'; 'distutils.core' is not a package


This probably is Bug 515060 via Bug 639176. But even if I --keep-going, I get problems, with packages seemingly still wanting, requiring, or being blocked by Python 3.4 version, for example:

Code:
emerge: there are no ebuilds built with USE flags to satisfy "dev-python/ipython_genutils[python_targets_python2_7(-)?,python_targets_python3_4(-)?,python_targets_python3_5(-)?,-python_single_target_python2_7(-),-python_single_target_python3_4(-),-python_single_target_python3_5(-)]".
!!! One of the following packages is required to complete your request:
- dev-python/ipython_genutils-0.1.0::gentoo (Change USE: +python_targets_python3_4)
- dev-python/notebook-4.2.3::gentoo (Change USE: -python_targets_python3_4)
(dependency required by "dev-python/notebook-4.2.3::gentoo" [installed])
(dependency required by "dev-python/ipyparallel-5.2.0::gentoo" [installed])
(dependency required by "dev-python/ipython-5.3.0::gentoo[smp]" [installed])
(dependency required by "dev-python/ipykernel-4.5.2::gentoo" [installed])
(dependency required by "dev-python/qtconsole-4.3.0::gentoo" [installed])
(dependency required by "dev-python/spyder-3.2.4::gentoo" [installed])
(dependency required by "@selected" [set])
(dependency required by "@world" [argument])


The exact packages that cause blocks (ipython_genutils-0.1.0 and notebook-4.2.3 here) varies with the --backtrack level. But I've gone to 1000 without successfully overcoming the problem, so I don't think that way lies my path out of this mess.

So I would be very grateful for any pointers to a solution. Given that I don't see the forums aflame with posts about updating issues probably means my setup has something special, but what?
Back to top
View user's profile Send private message
ct85711
Veteran
Veteran


Joined: 27 Sep 2005
Posts: 1635

PostPosted: Thu Nov 30, 2017 9:06 pm    Post subject: Reply with quote

Code:
ImportError: No module named 'setuptools'


This part has been a very common error that people been encountering, in that dev-python/setuptools need to be recompiled. So try recompiling that, and see if that fixes your problem.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Thu Nov 30, 2017 9:54 pm    Post subject: Reply with quote

ct85711 wrote:
Code:
ImportError: No module named 'setuptools'


This part has been a very common error that people been encountering, in that dev-python/setuptools need to be recompiled. So try recompiling that, and see if that fixes your problem.

Yes I tried that. I tried it again more insistently:
Code:
# emerge -q1 --autounmask-backtrack=y --backtrack=1000 setuptools
[ebuild   R   ] dev-python/packaging-16.8
[ebuild   R   ] dev-python/setuptools-34.0.2-r1
[ebuild   R   ] dev-python/certifi-2017.4.17

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

dev-python/setuptools:0

  (dev-python/setuptools-34.0.2-r1:0/0::gentoo, ebuild scheduled for merge) pulled in by
    dev-python/setuptools[python_targets_pypy(-)?,python_targets_pypy3(-)?,python_targets_python2_7(-)?,
python_targets_python3_4(-)?,python_targets_python3_5(-)?,python_targets_python3_6(-)?,
-python_single_target_pypy(-),-python_single_target_pypy3(-),-python_single_target_python2_7(-),
-python_single_target_python3_4(-),-python_single_target_python3_5(-),-python_single_target_python3_6(-)] required by (dev-python/certifi-2017.4.17:0/0::gentoo, ebuild scheduled for merge)

    dev-python/setuptools (Argument)

  (dev-python/setuptools-34.0.2-r1:0/0::gentoo, installed) pulled in by
    dev-python/setuptools[python_targets_python2_7(-)?,python_targets_python3_4(-)?,python_targets_python3_5(-)?,
python_targets_python3_6(-)?,-python_single_target_python2_7(-),-python_single_target_python3_4(-),
-python_single_target_python3_5(-),-python_single_target_python3_6(-)] required by (dev-python/pyflakes-0.8.1:0/0::gentoo, installed)

After that comes plenty more blocker lines.

So now what?

[Moderator edit: broke long whitespace-free lines in code tags to fix thread layout. -Hu]
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5212

PostPosted: Fri Dec 01, 2017 12:25 am    Post subject: Reply with quote

Try adding --with-bdeps=y.
Back to top
View user's profile Send private message
vokiel
n00b
n00b


Joined: 01 Oct 2004
Posts: 69

PostPosted: Fri Dec 01, 2017 12:33 am    Post subject: Reply with quote

I tried bdeps as well.. and the tree is just broken so:

Code:

sudo emerge -av1 --nodeps setuptools


Then oneshot anything else with --nodeps that you need.

I had another bunch to do listed in here: https://forums.gentoo.org/viewtopic-t-1072786.html
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12792

PostPosted: Fri Dec 01, 2017 2:29 am    Post subject: Reply with quote

vokiel: could you explain what sys-apps/portage bug necessitates bypassing the dependency resolver in this case? As I just remarked in your thread, --nodeps should be used sparingly and only when the rationale is well understood. The latter does not seem to be the case here, so recommending --nodeps is premature in my opinion.

OP: per that output, you need to update all the affected packages in one step. The output so far tells you to include dev-python/pyflakes. The output you omitted may tell you to include other packages too.
Back to top
View user's profile Send private message
vokiel
n00b
n00b


Joined: 01 Oct 2004
Posts: 69

PostPosted: Fri Dec 01, 2017 3:09 am    Post subject: Reply with quote

Hu wrote:
vokiel: could you explain what sys-apps/portage bug necessitates bypassing the dependency resolver in this case? As I just remarked in your thread, --nodeps should be used sparingly and only when the rationale is well understood. The latter does not seem to be the case here, so recommending --nodeps is premature in my opinion.

OP: per that output, you need to update all the affected packages in one step. The output so far tells you to include dev-python/pyflakes. The output you omitted may tell you to include other packages too.


No I can't apparently there is no issues, and it's just us poor schmoes that are the problem.

That said my laptop has the exact same issue. It's brand new, one month installation, updated regularly. I have portage and layman installed and attempting to update portage or python leads to a barricade of blockers. There is nothing special about the installation concerning python, I don't develop using that crap. I've not done anything yet so if you need any output that's easy to get, I'm available to test your solution.

p.s. I can't get equery because merging gentoolkit leads to the same issue.

Thanks.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12792

PostPosted: Fri Dec 01, 2017 4:55 am    Post subject: Reply with quote

If you have the same issue, please try the solution I suggested to the original poster and report back.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Fri Dec 01, 2017 7:08 am    Post subject: Reply with quote

Ant P. wrote:
Try adding --with-bdeps=y.

Sadly, I already have in make.conf:
Code:
EMERGE_DEFAULT_OPTS="--misspell-suggestions=n --with-bdeps=y"

So that won't be the solution (tested anyway).
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Fri Dec 01, 2017 7:47 am    Post subject: Reply with quote

Hu wrote:
OP: per that output, you need to update all the affected packages in one step. The output so far tells you to include dev-python/pyflakes. The output you omitted may tell you to include other packages too.

Ok, I added those listed. Th next time, I got a new list, which is longer than my scrollbuffer…

This doesn't surprise me: a simple emerge -DNu world does get portage working, but then it quits on contextlib2 and qtawesome and --keep-going doesn't help because on resume I arrive to blocker hell.

As vokiel said, the tree seems too broken or portage not intelligent enough to have it be able to handle this.
Please advise on using --nodeps to get setuptools installed, as it seems to have worked for vokiel.

@vokiel: It is correct that you managed to get your problem fixed using --nodep, yes?

@all: I guess the safe thing to do is to emerge --nodep steuptools together with its python dependencies that have not yet moved from 3.4 to 3.5?
Back to top
View user's profile Send private message
hceline
n00b
n00b


Joined: 30 Aug 2015
Posts: 26

PostPosted: Fri Dec 01, 2017 7:53 am    Post subject: Reply with quote

I had a similar problem when doing the upgrade, I solved it by setting python2.7 as default with eselect. Do not know if it will help here though.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Fri Dec 01, 2017 8:25 am    Post subject: Reply with quote

hceline wrote:
I had a similar problem when doing the upgrade, I solved it by setting python2.7 as default with eselect. Do not know if it will help here though.

No it does not help, but thanks, for suggesting.
Back to top
View user's profile Send private message
vokiel
n00b
n00b


Joined: 01 Oct 2004
Posts: 69

PostPosted: Fri Dec 01, 2017 1:24 pm    Post subject: Reply with quote

equaeghe wrote:
Hu wrote:
OP: per that output, you need to update all the affected packages in one step. The output so far tells you to include dev-python/pyflakes. The output you omitted may tell you to include other packages too.

Ok, I added those listed. Th next time, I got a new list, which is longer than my scrollbuffer…

This doesn't surprise me: a simple emerge -DNu world does get portage working, but then it quits on contextlib2 and qtawesome and --keep-going doesn't help because on resume I arrive to blocker hell.

As vokiel said, the tree seems too broken or portage not intelligent enough to have it be able to handle this.
Please advise on using --nodeps to get setuptools installed, as it seems to have worked for vokiel.

@vokiel: It is correct that you managed to get your problem fixed using --nodep, yes?

@all: I guess the safe thing to do is to emerge --nodep steuptools together with its python dependencies that have not yet moved from 3.4 to 3.5?


I managed to fix some portage issues by trying to emerge layman with --nodeps and then see where the sync would fail. I would get a runtime error just like you and then I would find the python package corresponding to the module and emerge -av1 --nodeps dev-python/<package>. Yes, It doesn't fix all issues. In my case I don't quite care that much, unless portage isn't running anymore, which is not the case and I managed to update it to 2.3.13.

Syncing works, ... very slowly.

That all said I still have blocker issues when I try cherry picking dev-python/cryptography which requires openssl with -bindist. So it'll fix your individual package updates, but eventually you'll just get the blockers back.
Back to top
View user's profile Send private message
trent
n00b
n00b


Joined: 27 Jul 2007
Posts: 7

PostPosted: Fri Dec 01, 2017 6:14 pm    Post subject: Reply with quote

Hu wrote:

OP: per that output, you need to update all the affected packages in one step. The output so far tells you to include dev-python/pyflakes. The output you omitted may tell you to include other packages too.


I tried this but I had to reiterate the procedure twice. For those who do not want to copy/paste a long list of packages, you could try
Code:
emerge -pv1 setuptools 2>&1 | tee wtf | sed -ne "s/^.*required by (\(.*\)\/\(.*\)-[0-9].*$/\1\/\2/p" | sort | uniq > emerge1


Then you have a list of packages in emerge1 that at least need to be emerged. wtf contains the output of the complete emerge. If it still contains conflicts, repeat.
Code:
emerge -pv1 `cat emerge1` 2>&1 | tee wtf | sed -ne "s/^.*required by (\(.*\)\/\(.*\)-[0-9].*$/\1\/\2/p" | sort | uniq > emerge2
cat emerge1 emerge2 | sort | uniq > emerge3


Reinspect wtf and repeat if necessary. If emerge -p doesn't give any further conflicts do the emerge:

Code:
emerge -v1 `cat emerge6`


or whatever number the last file. Setuptools will be updated and emerge -uND should work.

Anyhow, I totally understand that people are upset by this kind of behaviour. No news item relates to this mess, no pointer to any help.
This comes across like: User, you probably broke it, go google yourself. At bugs.gentoo.org there are bugs about missing dependencies to setuptools but why can't I just reemerge setuptools which was installed on my machine before the python switch. I don't want to go paper chasing forums when updating my system and this is not the first time. This is not user friendly.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5212

PostPosted: Fri Dec 01, 2017 9:05 pm    Post subject: Reply with quote

It seems like part of the problem here is a circular dependency chain (setuptools -> certifi -> setuptools). I've just hit it myself updating one of my machines, after a few others went through without this problem. Doing emerge -1 --nodeps certifi setuptools gets rid of it. My impression is it's safe to do that, since one is only a PDEPEND for the other (not strictly needed for it to work or build).
Back to top
View user's profile Send private message
FoZo
n00b
n00b


Joined: 14 Sep 2013
Posts: 5
Location: Stockholm/Sweden

PostPosted: Fri Dec 01, 2017 10:30 pm    Post subject: Reply with quote

I had the same problem and
Code:
emerge -1 --nodeps certifi setuptools

fix it.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12792

PostPosted: Sat Dec 02, 2017 1:24 am    Post subject: Reply with quote

equaeghe wrote:
As vokiel said, the tree seems too broken or portage not intelligent enough to have it be able to handle this.
I will reserve judgment on that until I see a full error dump, but a conflict list longer than scrollback certainly indicates a problem somewhere. As Ant P. said later in the thread, this seems like a safe case for --nodeps. The most likely failure in this case if that is not the solution for your problem is just a build failure, rather than installing a broken package.
Ant P. wrote:
My impression is it's safe to do that, since one is only a PDEPEND for the other (not strictly needed for it to work or build).
This is exactly the kind of commentary I like to see when discussing a bypass of the dependency resolver. This description both justifies why it is (likely) safe and provides context so that future readers can judge whether their situation is like the one described here.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Sat Dec 02, 2017 8:45 pm    Post subject: Reply with quote

Ant P. wrote:
It seems like part of the problem here is a circular dependency chain (setuptools -> certifi -> setuptools). I've just hit it myself updating one of my machines, after a few others went through without this problem. Doing emerge -1 --nodeps certifi setuptools gets rid of it. My impression is it's safe to do that, since one is only a PDEPEND for the other (not strictly needed for it to work or build).

I needed to add packaging:
Code:
emerge -qva1 --nodeps packaging certifi setuptools

But that doesn't yet seem to fix the problem :-(.

Currently I can already do a non-Deep, non-Newuse emerge -u, so I'm doing that to get a big list of KDE packages out of the way already. Then I'll probably try trent's solution. I agree with him that the Python upgrade was not handled very user-friendly; it seems not to have been tested sufficiently to detect this stuff. Also the combination with packages missing setuptools as a dependency makes this one extra toxic. Bug 515060 should have been a hard blocker for the Python upgrade.

Hu wrote:
equaeghe wrote:
As vokiel said, the tree seems too broken or portage not intelligent enough to have it be able to handle this.

I will reserve judgment on that until I see a full error dump, […]

With all due respect, but how can you say that? (i) the contextlib2 error disappeared after I begged for the package to be fixed: Bug 639176, (ii) the setuptools issue seems to be due to a circular dep that seems insurmountable, (iii) quite experienced people like me need to go on the forums for help, and (iv) the solution may require such convoluted procedures as described by trent. How is that not portage being broken and emerge not being intelligent enough?
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 12792

PostPosted: Sun Dec 03, 2017 12:09 am    Post subject: Reply with quote

equaeghe wrote:
With all due respect, but how can you say that?
Easily. I routinely field error reports that are nothing more than "$FOO doesn't work" with no actual error output, so I make it a point not to take such claims at face value. Once I see the error output and either evaluate it or see an evaluation by someone whose opinion I trust, then I may conclude that yes, $FOO really is broken. Without the error output, I cannot determine whether it is broken for everyone or only for the person reporting it. In the latter case, it could be that they have an edge case that was not properly handled (a bug in $FOO, which needs to be fixed, but is not of the same level as a break in the default case) or it could be that they have some local override that guarantees it will be broken (such as force enabling compiler flags that are known to break things or masking out files associated with necessary functionality). Besides, the latter part of the sentence (which you did not quote) acknowledges that something is definitely broken somewhere. It stops short of assigning blame to any particular component, because I lack sufficient information about your scenario to be certain which component is at fault.
equaeghe wrote:
(i) the contextlib2 error disappeared after I begged for the package to be fixed: Bug 639176,
Good that it was fixed. I did not follow the bug link in the opening of the thread, because it was not marked as relevant to the post to which I was replying. If the fix was a Portage tree change, then for that issue, your conclusion that the Portage tree had a defect (as opposed to local user error) is, in retrospect, clearly correct.
equaeghe wrote:
(ii) the setuptools issue seems to be due to a circular dep that seems insurmountable,
If true, that is bad for everyone. I haven't yet undertaken the upgrades on my system, nor seen error output from anyone that would substantiate (or refute) your analysis.
equaeghe wrote:
(iii) quite experienced people like me need to go on the forums for help, and (iv) the solution may require such convoluted procedures as described by trent. How is that not portage being broken and emerge not being intelligent enough?
I did not recognize your username as someone whom I automatically assume to be an advanced and frequently self-supporting user. As above, since you didn't provide the error output that led to your conclusion, and I haven't personally experienced it, I haven't evaluated that output (nor seen an evaluation from someone whose experience I recognize). I am not stating that your conclusion is definitely wrong or even probably wrong. I am stating that I neither agree nor disagree with it, until such time as either I analyze it or see an analysis from someone who has, through prior activity, convinced me that he/she usually posts correct analyses.
Back to top
View user's profile Send private message
equaeghe
Guru
Guru


Joined: 22 Feb 2005
Posts: 434

PostPosted: Sun Dec 03, 2017 11:11 am    Post subject: Reply with quote

equaeghe wrote:
Currently I can already do a non-Deep, non-Newuse emerge -u, so I'm doing that to get a big list of KDE packages out of the way already. Then I'll probably try trent's solution.

I now have a clean install again!
In the end I did not need to try trent's approach. Namely, Bug 639368 got fixed, I temporarily unmerged spyder and its deps, and after the update of the KDE packages that resulted in a smooth emerge -DNvuqa @world. Remerging spyder then made my system whole again.

I'm not going to mark this as solved, because others have also used this thread to deal with their manifestation of the same problem and aren't through yet. The Moderators can mark as solved as per their good judgment.

Now, off to the profile change.

[Moderator note: the other user's problem appears to be different, as it was not resolved by the suggestions posted here. I moved it to Problems with the Python update to 3.5 (same title, different poster), and marked this thread as solved per comment from equaeghe above. -Hu]
Back to top
View user's profile Send private message
npaust
n00b
n00b


Joined: 23 Apr 2004
Posts: 67
Location: Walla Walla, WA

PostPosted: Tue Dec 05, 2017 8:24 am    Post subject: Reply with quote

So, I was having the problem of python not being able to manage moving to 3.5 on all five of my workstations. I muddled through an answer with two of them, but I think I came up with the quickest answer for the last three.

It looks like you need to update packaging and setuptools first. I did it with the --nodeps option. Now that I'm done, I don't have a good way to check if it would work without that option.
emerge packaging --nodeps
emerge setuptools --nodeps
emerge -uDN world

and away it goes!
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 346

PostPosted: Tue Dec 05, 2017 3:47 pm    Post subject: Reply with quote

npaust wrote:
So, I was having the problem of python not being able to manage moving to 3.5 on all five of my workstations. I muddled through an answer with two of them, but I think I came up with the quickest answer for the last three.

It looks like you need to update packaging and setuptools first. I did it with the --nodeps option. Now that I'm done, I don't have a good way to check if it would work without that option.
emerge packaging --nodeps
emerge setuptools --nodeps
emerge -uDN world

and away it goes!


I had to add six, pyparsing and appdirs between packaging and setuptools.
Quite the nice workaround the blockers!
Back to top
View user's profile Send private message
npaust
n00b
n00b


Joined: 23 Apr 2004
Posts: 67
Location: Walla Walla, WA

PostPosted: Tue Dec 05, 2017 6:48 pm    Post subject: Reply with quote

One other note, I'm still having problems with matplotlib, I've had to update my computers to the 2.1.0 version to get it to build.
Back to top
View user's profile Send private message
transpetaflops
Tux's lil' helper
Tux's lil' helper


Joined: 16 May 2005
Posts: 132

PostPosted: Sat Dec 09, 2017 12:21 am    Post subject: Reply with quote

After switching to Python 3.5 I still have some superfluous references to 3.4 despite having --depclean remove it. Did I miss some step in the upgrade? Can I just manually delete/edit these references to 3.4?

Code:
# eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.5
  [2]   python3.4
  [3]   python2.7 (fallback)

Code:
# cat /etc/env.d/python/config
python3.4

Code:
# grep -v \# /etc/python-exec/python-exec.conf
python3.5
python3.4


Listing of /var/db/pkg to verify that 3.4 is gone.
Code:
# ls -ld /var/db/pkg/dev-lang/python*
drwxr-xr-x 2 root root 4096  8 dec 20.06 /var/db/pkg/dev-lang/python-2.7.14-r1
drwxr-xr-x 2 root root 4096  8 dec 20.08 /var/db/pkg/dev-lang/python-3.5.4-r1
drwxr-xr-x 2 root root 4096  8 dec 19.49 /var/db/pkg/dev-lang/python-exec-2.4.4
Back to top
View user's profile Send private message
transpetaflops
Tux's lil' helper
Tux's lil' helper


Joined: 16 May 2005
Posts: 132

PostPosted: Mon Dec 11, 2017 6:42 pm    Post subject: Reply with quote

I'm answering myself here. The issue was addressed and resolved in https://bugs.gentoo.org/635678

EDIT: Actually not. /etc/env.d/python/config still contains "python3.4". I will continue this in the referenced bug report.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum