Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
stupid bash tricks: multi-emerge-tail
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Unsupported Software
View previous topic :: View next topic  
Author Message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 11
Location: meatspace

PostPosted: Sat Jan 04, 2014 11:54 pm    Post subject: stupid bash tricks: multi-emerge-tail Reply with quote

Here is a very ugly script I've whipped up to watch the good ol' meaningless spew scroll by when doing parallel emerges:

Code:

#!/bin/bash

[[ $( id -u ) -eq 0 ]] || {
   [[ -z "$I_WANT_TO_BE_ME" ]] && {
      echo 'becoming root via sudo and reinvoking myself...' >&2
      echo '(or perhaps you [[ -z "$I_WANT_TO_BE_ME" ]] ?)' >&2
      echo >&2
      SCRIPT=`realpath -s $0`
      echo "running: ${SCRIPT} $*" >&2
      echo >&2
      sleep 0.5
      exec sudo "${SCRIPT}" "$@"
      exit 0
   }
}

delay=2.5

vartmpportage="$( portageq envvar PORTAGE_TMPDIR )/portage"

notfirst=
bam=poo
# turns out it surprisingly difficult to figure out what to test here...
# while [[ $( ps -eo args | grep '^/usr/bin/python.*/emerge' ) ]] ; do
while [[ 1 ]] ; do
   oldbam=bam
   bam=
   if [[ $notfirst ]] ; then
      sleep ${delay}
   else
      notfirst=exactly
   fi
   active_emerges=$( qlop -cC | grep '^[[:space:]]\*' | sed 's/^[[:space:]]*\*[[:space:]]*//' | sed 's/[[:space:]][[:space:]]*/\n/g' | sort )
   lines=5
   if [[ ${active_emerges} ]] ; then
      lines=$(( ( $( tput lines ) / $( echo ${active_emerges} | wc -w ) ) - 2 ))
   fi
   (( lines < 2 )) && lines=2
   for merge in ${active_emerges} ; do
      hdr=$( echo -e "\033[1m[[ ${merge} ]]\033[0m" )
      cols=$(tput cols)
      thelines="$( tail -n ${lines} "${vartmpportage}"/${merge}/temp/build.log | fold -w ${cols} | tail -n ${lines} )"
      # if bam already nonempty, add a newline for aesthetic purposes
      if [[ ${bam} ]] ; then
         bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
      else
         bam="${hdr}"
      fi
      bam=$( echo "${bam}" ; echo "${thelines}" )
   done
   if [[ ${oldbam} != ${bam} ]] ; then
      clear
      echo "${bam:-"nuthin'"}"
   fi
done

echo "looks like emerge finished."

_________________
-gmt
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Tue Jan 07, 2014 2:40 am    Post subject: Reply with quote

You might like to check out update which does a similar thing when using --jobs, and also filters emerge output when not.

It's an awful lot bigger than yours, but started out along the same lines, only there was no --jobs back then, and indeed no --keep-going, which was the original problem to fix.

Your grep | sed | sed can all be done with one sed afaict:
Code:
sed -n '/^[[:blank:]]*\*[[:blank:]]*/{s///;s/[[:blank:]][[:blank:]]*//;p;}'

should give you the basis, though I'm not sure what's going on with \n in there. sed is line-based, so you might as well use [[:blank:]] which is slightly quicker. The only way you could have newlines is if you used the n command. I have a feeling the last substitution may be better as s/[[:blank:]].*/;

Code:
lines=$(( ( $( tput lines ) / $( echo ${active_emerges} | wc -w ) ) - 2 ))
makes little sense to me: you'd be better off using an array for active imo, and then ${#active[@]} is the number of elements in the array, or build it with active[n++]=$pkg to get rid of the echo | wc -w subshell.

Code:
bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
is much easier as
Code:
bam+="
$hdr"
or I use an $EOL readonly so:
Code:
bam+=$EOL$hdr
(shells do not field-split assignment, same as case.)

subshells are the thing which slows stuff down the most, ime, especially with bash which is quite a heavy program in comparison to smaller shells. #bash and #sed on IRC: chat.freenode.net are your friends.

HTH,
steveL
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 11
Location: meatspace

PostPosted: Tue Jan 07, 2014 9:22 am    Post subject: Reply with quote

steveL wrote:
You might like to check out update which does a similar thing when using --jobs, and also filters emerge output when not.

It's an awful lot bigger than yours, but started out along the same lines, only there was no --jobs back then, and indeed no --keep-going, which was the original problem to fix.

That looks pretty cool. Many's time time I considered building such a beast myself but, well there's only room in my brain for so many half-finished projects :)

Your grep | sed | sed can all be done with one sed afaict:
Code:
sed -n '/^[[:blank:]]*\*[[:blank:]]*/{s///;s/[[:blank:]][[:blank:]]*//;p;}'


should give you the basis, though I'm not sure what's going on with \n in there. sed is line-based, so you might as well use [[:blank:]] which is slightly quicker. The only way you could have newlines is if you used the n command. I have a feeling the last substitution may be better as s/[[:blank:]].*/;



Since I wrote this code (if you can even call it that hehe) I finally broke down and learned a few basics of how sed works... I'm a way better sed scripter now but I still think I pretty much suck at it :)

To be clear, I'm well aware that this is not anything even vaguely resembling decently written.

I occasionally throw together a crappy little script like this as a one-off but end up addicted to it... this was one of those.

Definitely one from the Jackson Pollack school of software design... Don't think I've used qlop or tput before or since -- today I have no idea what they are (of course I could ask Google but then I'd probably have to forget something to make room for it -- with my luck, it'd be the knowledge of where my genitals are, or how to put on socks, so I'm not gonna risk it :D )

steveL wrote:

subshells are the thing which slows stuff down the most, ime, especially with bash which is quite a heavy program in comparison to smaller shells. #bash and #sed on IRC: chat.freenode.net are your friends.


Many ACKs above. I fucking /love/ #bash. They are shockingly nice there, and as often as not, happy to explore whatever impossibly obscure corner case I've decided I need to obsess over on any given 3am coding bender run amok. And subshells are clearly from the devil, I'm fully on that page with you.

Dunno about other shells. Bash seems pretty damn fast to me ... but I don't have meaningful amounts of experience with anything else and anyhow, my computer has rack ears instead of a megapixel camera on both ends -- I should probably bear in mind that most of theirs will go way slower than mine does when I'm expecting people to consume code I've written...
_________________
-gmt
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 11
Location: meatspace

PostPosted: Tue Jan 07, 2014 9:25 am    Post subject: Reply with quote

steveL wrote:
check out update


Looks kinda cool, btw! Or if not cool, useful. I'll check it out.
_________________
-gmt
Back to top
View user's profile Send private message
steveL
Watchman
Watchman


Joined: 13 Sep 2006
Posts: 5153
Location: The Peanut Gallery

PostPosted: Wed Jan 08, 2014 6:01 am    Post subject: Reply with quote

gmt wrote:
Since I wrote this code (if you can even call it that hehe) I finally broke down and learned a few basics of how sed works... I'm a way better sed scripter now but I still think I pretty much suck at it :)

To be clear, I'm well aware that this is not anything even vaguely resembling decently written.

I occasionally throw together a crappy little script like this as a one-off but end up addicted to it... this was one of those.

Definitely one from the Jackson Pollack school of software design... Don't think I've used qlop or tput before or since -- today I have no idea what they are (of course I could ask Google)

Use the manpages silly ;) With portage-utils (q*) they're pretty minimal, but they don't change much, and are designed for use in scripts. They're all the same binary q, it's multi-call, built for use in rescue situations. Though algorithmically it had a lot of problems last time I looked (over 3 years ago) it's always been very quick. So, you end up just playing with it each time you're writing something that uses it, to check the output. That's a good habit to get into in any case, since a lot of scripting is wrapping other commands; basically building a bit of knowledge in around them, to make routine tasks routine so complex issues are easier to handle.

Hmm tput and terminal handling is a big one: but you're already using #bash so you're in the right place. There's ##tty as well, though it tends to be a bit more on the code side. You'll find some stuff in the libIgli file that's part of update that deals with tput and colours. $CUP is used to move up the terminal for example, and the --jobs filtering uses a lot of that.

As for stuff you've thrown together, don't worry about it: I know how you feel and that's exactly how I started out. update was simply an exercise to learn bash with, and the issue at the time was no --keep-going, but for me it was the pages of spew from every compile output meaning I couldn't see what was happening. So filtering emerges (build-logs essentially) became the major focus, til --jobs came along. It's still used when -j1 or we're installing eg toolchain stuff, where we do one at a time before the main build.

You can try that out using --test for a package you've already installed: it'll just work from the build-log to show you a compacted display as if it had run that compile now (only a lot quicker ofc.:)
Quote:
Dunno about other shells. Bash seems pretty damn fast to me ... but I don't have meaningful amounts of experience with anything else and anyhow, my computer has rack ears instead of a megapixel camera on both ends -- I should probably bear in mind that most of theirs will go way slower than mine does when I'm expecting people to consume code I've written...

I was amazed how far I could push my old 32-bit 1G machine with update; I kept waiting for it to blow up on the script, but it never did. But yeah, most of the "optimisation" in shellscript comes from not shelling out, paradoxically, for minor things that can be done by the runtime. Bash is definitely a step up if you need to write larger glue apps; arrays for instance are essential.
Quote:
Looks kinda cool, btw! Or if not cool, useful. I'll check it out.

Heh thanks :) That's another reason i haven't reworked it and split it into different files: it's had to stay in use, and I don't have the time. Also at some point we'll reimplement it in C or something, so it's easier to keep it as is and save the rewrite for another language, preferably by parsing the bash (which is a much more fun use of time;)

Feel free to patch it and play with it: you can't hurt anyone else's machine ;) and in any case any mistakes you make are unlikely to lead to damage to your machine. For a start it'll likely crash on you immediately, and if not you just hit CTRL-C at any point and it'll bail. Most errors occur in the logic around emerge, not during the actual emerge itself. It's got various safety checks; criticalPkg comes to mind, but there are lots of places where it will abort and give you a stack trace if you pass it dodgy data. And it won't downgrade your toolchain unless you force it to. ;) Having said that, you can also use it for ROOT and PORTAGE_CONFIGROOT builds (an area that has been expanded but not fully-tested to my knowledge) or in a chroot (I did that for 6 months when we were working on ABI upgrades in /etc/warning for expat; from binhosts as well which was kinda spooky: just like a binary distro ;)

Looking at your script, you're outputting from the build-logs: that would be nice to incorporate into the current display which is simply a list of packages and where they are in the process. Something I'd like, so if you fancy it, play with runInstall() which is the multi-job, terminal-aware function. Don't worry too much about keeping it in sync with runInstallPipe() just get something displaying how you'd like it (if you want) and i'll sync anything that needs it (unlikely since the other function doesn't display much by design, and there's no terminal.) If not, no bother, but don't hold back from patching if you feel like it (or asking me to, assuming you have an idea that we can reasonably implement, and you're willing to test.)

People sometimes get put off by how large it is, but each piece is not complex, and that's more an indication of how ad-hoc it is, which should make you feel more relaxed about changing it. Most of the tricky code is in support functions to make handling the tree reasonable, and then higher-layers build on top of that. Oh and some faking of associative arrays, from bash-3 days ;) Tricky stuff is more likely to be found in the lib file, unless it's to do with the portage tree, and most of that is only complex if you're trying to rework version comparison, or something fundamental.

The rest is just dealing with output from various programs and deciding what to do about it. The interesting things from our end are usually making it do something specific for a package, when a hook is not sufficient. The two main mechanisms for that are the warning file, and something being in toolchain (an area I've been working on for last year on and off, and wanted for several more: update -a --toolchain; I'm still working on the last bits for that, around perl, but it tests the version of the compiler used to build stuff now, for example.)

Hmm that went on a bit longer than I intended; blame old age ;)
Regards,
steveL.
Back to top
View user's profile Send private message
Maitreya
Guru
Guru


Joined: 11 Jan 2006
Posts: 426

PostPosted: Wed Jan 08, 2014 1:27 pm    Post subject: Reply with quote

That is some nice Fu there. Here I am just using a one-liner :
Code:

watch -n 3  -c "ccache -s && genlop -c && tail /var/tmp/portage/*/*/temp/build.log"
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 11
Location: meatspace

PostPosted: Thu Jan 30, 2014 10:26 am    Post subject: Re: stupid bash tricks: multi-emerge-tail Reply with quote

gmt wrote:
Here is a very ugly script


This updated version uses lsof to implement a considerably better qlop than qlop is. Oh, since my script stopped working I was forced to figure out what qlop is btw :) It's part of portage-utils. The problem is, it assumes that portage will present its $0 to /proc/${PID} in a certain format that it sometimes doesn't. I should probably fix qlop while I'm at it but I'm pretty lazy...

Code:

#!/bin/bash

[[ $( id -u ) -eq 0 ]] || {
        [[ -z "$I_WANT_TO_BE_ME" ]] && {
                echo 'becoming root via sudo and reinvoking myself...' >&2
                echo '(or perhaps you [[ -z "$I_WANT_TO_BE_ME" ]] ?)' >&2
                echo >&2
                SCRIPT=`realpath -s $0`
                echo "running: ${SCRIPT} $*" >&2
                echo >&2
                sleep 0.5
                exec sudo "${SCRIPT}" "$@"
                exit 0
        }
}

delay=2.5

vartmpportage="$( portageq envvar PORTAGE_TMPDIR )/portage"

notfirst=
bam=poo
# turns out it surprisingly difficult to figure out what to test here...
# while [[ $( ps -eo args | grep '^/usr/bin/python.*/emerge' ) ]] ; do
while [[ 1 ]] ; do
        oldbam=bam
        bam=
        if [[ $notfirst ]] ; then
                sleep ${delay}
        else
                notfirst=exactly
        fi
        # active_emerges=$( qlop -cC | grep '^[[:space:]]\*' | sed 's/^[[:space:]]*\*[[:space:]]*//' | sed 's/[[:space:]][[:space:]]*/\n/g' | sort )
        active_emerges=$( lsof "${vartmpportage}"/*/*/temp/build.log 2>/dev/null | sed -e 's|^.*/\([^/]*/[^/]*\)/temp/build.log|\1|' | tail -n +2 | sort -u )
        lines=5
        if [[ ${active_emerges} ]] ; then
                lines=$(( ( $( tput lines ) / $( echo ${active_emerges} | wc -w ) ) - 2 ))
        fi
        (( lines < 2 )) && lines=2
        for merge in ${active_emerges} ; do
                hdr=$( echo -e "\033[1m[[ ${merge} ]]\033[0m" )
                cols=$(tput cols)
                thelines="$( tail -n ${lines} "${vartmpportage}"/${merge}/temp/build.log | fold -w ${cols} | tail -n ${lines} )"
                # if bam already nonempty, add a newline for aesthetic purposes
                if [[ ${bam} ]] ; then
                        bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
                else
                        bam="${hdr}"
                fi
                bam=$( echo "${bam}" ; echo "${thelines}" )
        done
        if [[ ${oldbam} != ${bam} ]] ; then
                clear
                echo "${bam:-"nuthin'"}"
        fi
done

echo "looks like emerge finished."

_________________
-gmt
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 11
Location: meatspace

PostPosted: Thu Jan 30, 2014 11:00 pm    Post subject: Re: stupid bash tricks: multi-emerge-tail Reply with quote

gmt wrote:
Here is a very ugly scrip


Discovered there is vastly superior implementation of this at: https://github.com/mgorny/portage-jobsmon

Unfortunately, it does suffer, somewhat, (as does mine, still, but not nearly so much after I put in the lsof hack) from false negatives when finding the running emerges.

Meanwhile, I figured out a way to split lines without getting confused by color escape codes. Unfortunately it requires some perl modules for which there are no ebuilds:

Code:

#!/bin/bash

[[ $( id -u ) -eq 0 ]] || {
   [[ -z "$I_WANT_TO_BE_ME" ]] && {
      echo 'becoming root via sudo and reinvoking myself...' >&2
      echo '(or perhaps you [[ -z "$I_WANT_TO_BE_ME" ]] ?)' >&2
      echo >&2
      SCRIPT=`realpath -s $0`
      echo "running: ${SCRIPT} $*" >&2
      echo >&2
      sleep 0.5
      exec sudo "${SCRIPT}" "$@"
      exit 0
   }
}

delay=2.5

vartmpportage="$( portageq envvar PORTAGE_TMPDIR )/portage"

notfirst=
bam=poo
# turns out it surprisingly difficult to figure out what to test here...
# while [[ $( ps -eo args | grep '^/usr/bin/python.*/emerge' ) ]] ; do
while [[ 1 ]] ; do
   oldbam=bam
   bam=
   if [[ $notfirst ]] ; then
      sleep ${delay}
   else
      notfirst=exactly
   fi
   active_emerges=()
   while read mrg; do
      active_emerges+=("${mrg}")
   done < <( lsof "${vartmpportage}"/*/*/temp/build.log 2>/dev/null | sed -e 's|^.*/\([^/]*/[^/]*\)/temp/build\.log|\1|' | tail -n +2 | sort -u )
   lines=5
   active_logs=()
   for merge in "${active_emerges[@]}"; do
      active_logs+=("${vartmpportage}"/${merge}/temp/build.log)
   done
   if (( ${#active_logs[@]} )); then
      lines=$(( ( $( tput lines ) / ${#active_emerges[@]} ) - 2 ))
      if (( lines > 8 )) ; then
         poss_active_logs=()
         for amerge in "${active_emerges[@]}" ; do
            poss_active_logs+=("${vartmpportage}"/${amerge}/temp/build.log)
            sublogs=()
            while read nulog; do
               sublogs+=("${nulog}")
            done < <( find "${vartmpportage}"/${amerge}/temp -name 'build-*.log' 2>/dev/null )
            poss_active_logs+=("${sublogs[@]}")
         done
         poss_lines=$(( ( $( tput lines ) / ${#poss_active_logs[@]} ) - 2 ))
         if (( poss_lines >= 3 )) ; then
            lines=${poss_lines}
            active_logs=("${poss_active_logs[@]}")
         fi
      fi
   fi
   (( lines < 2 )) && lines=2
   for log in "${active_logs[@]}" ; do
      merge="${log#${vartmpportage}/}"
      merge="${merge%.log}"
      merge="${merge/\/temp\/build/}"
      
      hdr=$( echo -e "\033[1m[[ ${merge} ]]\033[0m" )
      cols=$(tput cols)
      thelines="$( tail -n ${lines} "${log}" | perl -e "use Text::ANSI::Util; binmode(STDOUT, \":utf8\"); while(<STDIN>) { print Text::ANSI::Util::ta_mbwrap(\$_, ${cols}); }" | tail -n ${lines} )"
      # if bam already nonempty, add a newline for aesthetic purposes
      if [[ ${bam} ]] ; then
         bam=$( echo "${bam}" ; echo ; echo "${hdr}" )
      else
         bam="${hdr}"
      fi
      bam=$( echo "${bam}" ; echo "${thelines}" )
   done
   if [[ ${oldbam} != ${bam} ]] ; then
      clear
      echo "${bam:-"nuthin'"}"
   fi
done

echo "looks like emerge finished."


This requires perl modules:

dev-perl/Text-ANSI-Util/Text-ANSI-Util-0.12.ebuild:
Code:

# $Header: $

EAPI=4

MODULE_AUTHOR="SHARYANTO"
MODULE_VERSION=${PV}
inherit perl-module

DESCRIPTION="Terminal control using ANSI escape sequences"

LICENSE="GPL-1"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

SRC_TEST="do"

RDEPEND="dev-perl/Data-Dump
      dev-perl/Text-WideChar-Util"


and dev-perl/Text-WideChar-Util/Text-WideChar-Util-0.08.ebuild
Code:

# Copyright 1999-2013 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI=4

MODULE_AUTHOR="SHARYANTO"
MODULE_VERSION=${PV}
inherit perl-module

DESCRIPTION="Terminal control using ANSI escape sequences"

LICENSE="GPL-1"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

SRC_TEST="do"

RDEPEND="dev-perl/Unicode-LineBreak"


Anyhow, I doubt I'll work on it any more now that I've discovered mgorny's script which has a far more "practical" (iow: not shit) implementation based on python and a real curses library. Going forward it seems more sensible to contribute to that than to continue to dink around with this.
_________________
-gmt
Back to top
View user's profile Send private message
gmt
n00b
n00b


Joined: 15 Feb 2013
Posts: 11
Location: meatspace

PostPosted: Thu May 09, 2019 12:59 am    Post subject: Re: stupid bash tricks: multi-emerge-tail Reply with quote

gmt wrote:
...doubt I'll work on it any more now that I've discovered mgorny's script


So much for that. I've continued to maintain and improve this thing... I'd might as well post the latest, since the old post probably doesn't work, these days, and what I do have works nicely. I find it to be a super-convenient fix for parallel portage's attempts to deprive me of the Gentoo experience of gobs of gibberish scrolling past faster than I can read it:

Here's a link to a "screen shot"

It is still a pretty poor example of bash scripting (a bit less so than it used to be) but the performance/results are much better:

/usr/bin/multi-ebuild-tail:
#!/bin/bash

: ${PORTAGEY_ROOTISH_UID:=$( id -u ${I_JUST_WANT_TO_BE:-$(id -un root)} )}
: ${PORTAGEY_ROOTISH_UID:=0}


#: ${MULTI_ABI:=hellno}
: ${MULTI_ABI:=fuckyeah}
case ${MULTI_ABI} in
   no|No|NO|false|False|FALSE|nope|Nope|NOPE|0) MULTI_ABI=hellno ;;
esac

: ${DEFDELAY:=2}
: ${DELAY:=${DEFDELAY:-2}}
: ${FORCED:=0}

pqvars() {
   [[ -n ${VARTMPPORTAGE} ]] && return 0
   : ${VARTMPPORTAGE:=$(portageq envvar PORTAGE_TMPDIR)/portage}
   : ${VARTMPPORTAGE:=/var/tmp/portage}
   : ${VARTMPPORTAGE2:=${VARTMPPORTAGE%/tmp/portage}/slow/portage}
}

active_logs=()
old_active_logs=()
older_active_logs=()

notfirst=0
bam=poo

# of the given filenames, pick the one most recently modified
# if several files modified within the same most-recent second,
# pick the one second-to-last one provided
most_recently_modified_file() {
   local reord=("${@:$(($#)):1}" "${@:1:$(($#-1))}")
   local best_guess=""
   local best_guess_modified_unixtime=0
   local logfile logfile_modified_unixtime
   for logfile in "${reord[@]}"; do
      logfile_modified_unixtime=$(stat -c %Y "${logfile}")
      if [[ ${best_guess_modified_unixtime} -le ${logfile_modified_unixtime} ]]; then
         best_guess=${logfile}
         best_guess_modified_unixtime=${logfile_modified_unixtime}
      fi
   done
   echo "${best_guess}"
}

usage() {
   local thisprog="${0##*/}"
   pqvars
   echo
   echo "${thisprog}: sort-of tail multiple simultaneous emerges at the console"
   echo
   echo "  ${thisprog} [-f|--force] [-dN.N|--delay=N.N]"
   echo
   echo "    -f: Force all files matching the shell globs:"
   echo
   echo "          \"${VARTMPPORTAGE}\"/*/*/temp/build.log"
   echo
   echo "        or"
   echo
   echo "          \"${VARTMPPORTAGE2}\"/*/*/temp/build.log"
   echo
   echo "        to be treated as active logs.  Otherwise ${thisprog} will"
   echo "        only treat the subset of these files which lsof reports as"
   echo "        currently active as active logs.  The lsof tests are quite"
   echo "        costly so this will improve performance and provide feedback"
   echo "        for merges that failed, or which portage has put on ice for"
   echo "        the time being, which may be preferable in some circumstances."
   echo
   echo "    -dN.N: Any reasonable positive decimal value may be provided attached"
   echo "           to this argument; when provided this will replace the default"
   echo "           \"delay\" period of ${DEFDELAY}.  Note that the delay is not"
   echo "           a period; ${thisprog} simply sleeps for the \"delay\" period"
   echo "           between each processing cycle.  Since the processing time will"
   echo "           depend on a number of factors, there is no way to provide true"
   echo "           periodic screen updates in ${thisprog}."
   echo
}

processargs() {
   local thisarg
   for thisarg in "$@"; do
      case ${thisarg} in
      -h|--help)
         usage
         exit 0
         ;;
      -f|--force)
         FORCED=1
         ;;
      -d?*|--delay=?*)
         thisarg2=${thisarg#-}
         thisarg3=${thisarg2#-}
         thisarg4=${thisarg3#d}
         thisarg5=${thisarg4#elay=}
         DELAY=${thisarg5}
         ;;
      *)
         echo "Error: argument \"${thisarg}\" not grokked." >&2
         echo >&2
         usage >&2
         exit 1
         ;;
      esac
   done
}

processargs "$@"

if [[ ! $(id -u) -eq ${PORTAGEY_ROOTISH_UID} ]]; then
   PORTAGEY_ROOTISH_UN=$(id -un ${PORTAGEY_ROOTISH_UID})
   echo "be-sudo-coming $(id -un ${PORTAGEY_ROOTISH_UN}) and reinvoking ${0} ..." >&2
   echo '(or... perhaps you want to be someone else?  Then:' >&2
   echo >&2
   echo "    export I_JUST_WANT_TO_BE=\"[ other_guy's_username | other_gal's_user_id ]\"" >&2
   echo >&2
   echo "(note: users of all POSIX-compliant genders are supported by either syntax)" >&2
   echo >&2
   LE_SCRIPT=$(realpath -s ${0})
   case ${LE_SCRIPT} in
      *\ *) LE_SCRIPT_SH_REP="\"${LE_SCRIPT}\"" ;;
      *) LE_SCRIPT_SH_REP=${LE_SCRIPT} ;;
   esac
   echo -n "Ctrl-C to abort..."
   sleep 0.75
   echo
   exec sudo -u ${PORTAGEY_ROOTISH_UN} "${LE_SCRIPT}" "$@"
   exit $?
fi

pqvars

# turns out it surprisingly difficult to figure out what to test here...
while [[ 1 ]] ; do
   oldbam="${bam}"
   bam=""
   if [[ $notfirst -eq 1 ]] ; then
      sleep ${DELAY}
   else
      notfirst=1
   fi
   active_logs=()
   for f in "${VARTMPPORTAGE}"/*/*/temp/build.log "${VARTMPPORTAGE2}"/*/*/temp/build.log; do
      if [[ -f ${f} ]]; then
         if [[ ${FORCED} -eq 1 ]]; then
            active_logs+=("${f}")
         elif lsof -- "${f}" &>/dev/null; then
            active_logs+=("${f}")
         fi
      fi
   done
   lines=5
   logs_to_display=()
   if [[ ${#active_logs[*]} -gt 0 || ${#old_active_logs[*]} -gt 0 || ${#older_active_logs[*]} -gt 0 ]]; then
      candidate_raw_candidate_logs=("${active_logs[@]}" "${old_active_logs[@]}" "${older_active_logs[@]}")
      raw_candidate_logs=()
      candidate_sublogs=()
      for candidate in "${candidate_raw_candidate_logs[@]}"; do
         for athing in "${raw_candidate_logs[@]}"; do
            [[ ${candidate} == ${athing} ]] && break 2
         done
         raw_candidate_logs+=("${candidate}")
      done
      if [[ ${MULTI_ABI} != hellno ]]; then
         for candidate in "${raw_candidate_logs[@]}"; do
            for candidate_sublog in "${candidate%/build.log}"/build-*.log; do
               # candidate sublogs still use lsof for now :(
               if lsof -- "${candidate_sublog}" &>/dev/null; then
                  case ${candidate_sublog} in
                     */build.log)
                        :;
                        ;;
                     *)
                        
                        candidate_sublogs+=("${candidate_sublog}")
                        active_logs+=("${candidate_sublog}")
                        ;;
                  esac
               fi
            done
         done
         #orig_candidate_sublogs=("${candidate_sublogs[@]}")
      fi
      candidate_logs=()
      # make sure these things actually exist as they could have finished
      # (they still could, but, whatchagonnado?  the point is to not fuck
      # up the lines calculation due to useless entries.
      for raw_candidate in "${raw_candidate_logs[@]}"; do
         [[ -f "${raw_candidate}" ]] && candidate_logs+=("${raw_candidate}")
      done

      #orig_candidate_logs=("${candidate_logs[@]}")

      # sort the logs alphabetically by filename
      merged_candidate_logs_pre=()
      readarray -t merged_candidate_logs_pre < <( for log in "${candidate_logs[@]}" "${candidate_sublogs[@]}"; do echo "${log}"; done | sort -d )

      # push substrings to the bottom
      pending=""
      merged_candidate_logs=()
      for nextpending in "${merged_candidate_logs_pre[@]}"; do
         nextpending=${nextpending%.log}
         if [[ -n ${pending} ]]; then
            if [[ "${nextpending}" == "${pending}"-* ]] ; then
               merged_candidate_logs+=( "${nextpending}.log" )
            else
               merged_candidate_logs+=( "${pending}.log" )
               pending=${nextpending}
            fi
         else
            pending=${nextpending}
         fi
      done
      [[ -n ${pending} ]] && merged_candidate_logs+=( "${pending}.log" )

      # for each master, filter "slave" logs from the list, gathering into a "slavebundle".
      # for each "slavebundle", preserve the most recently modified logfile only.  collect
      # all selected logs into filtered_merged_candidate_logs (including non-bundled logs).
      filtered_merged_candidate_logs=()
      slavebundle=()
      bundle_master=""
      for this_log in "${merged_candidate_logs[@]}"; do
         if [[ -n ${bundle_master} ]]; then
            slavebundle+=("${this_log}")
            if [[ ${this_log} == ${bundle_master} ]]; then
               filtered_merged_candidate_logs+=("$(most_recently_modified_file "${slavebundle[@]}")")
               bundle_master=""
            fi
         else
            this_file="${this_log##*/}"
            this_dir="${this_log%/*}"
            if [[ ${this_file} == build-?* ]]; then
               bundle_master="${this_dir}/build.log"
               slavebundle=("${this_log}")
            else
               filtered_merged_candidate_logs+=("${this_log}")
            fi
         fi
      done

      # calculate lines-per-log (aka ${lines}).  crop if we have more logs than fit comfortably on the screen
      for this_candidate_log in "${filtered_merged_candidate_logs[@]}"; do
         possible_lines=$(( ( $( tput lines ) + 1 ) / ( ${#logs_to_display[@]} + 1 ) ))
         [[ ${possible_lines} -le 3 ]] && break
         lines=${possible_lines}
         logs_to_display+=("${this_candidate_log}")
      done
   fi

   # two lines are wasted per. build -- except for the first log tailed,
   # for which only one line is wasted, because we don't need to make extra space
   # to separate the logs; but we already accounted for that one extra line, when
   # we added one to the numerator in the division test, above.
   lines=$(( lines - 2 ))

#   echo
#   for var in active_logs old_active_logs older_active_logs candidate_raw_candidate_logs raw_candidate_logs candidate_logs candidate_sublogs merged_candidate_logs_pre merged_candidate_logs filtered_merged_candidate_logs logs_to_display; do
#      declare -p "${var}" | sed 's/\[[[:digit:]]*\]=//g'
#   done
#   echo
#    echo "log_to_display -> merge mappings"
#   echo "================================"
#   for log in "${logs_to_display[@]}" ; do
#      merge="${log#${VARTMPPORTAGE2}/}"
#      merge="${merge#${VARTMPPORTAGE}/}"
#      merge="${merge%.log}"
#      merge="${merge/\/temp\/build/}"
#      echo "log=\"${log}\" -> merge=\"${merge}\""
#   done
#   sleep 20

   # buffer lines for printing
   nextcr=""
   for log in "${logs_to_display[@]}"; do
      merge="${log#${VARTMPPORTAGE2}/}"
      merge="${merge#${VARTMPPORTAGE}/}"
      merge="${merge%.log}"
      if [[ ${merge} != *build ]]; then
         merge="${merge/\/temp\/build-/ <}>"
      else
         merge="${merge/\/temp\/build/}"
      fi
      bam+="${nextcr}"
      nextcr=$'\n\n'
      hdr=$( echo -e "\033[1m[[ $(
         [[ "${merge}" ]] && echo "${merge}" || echo "${log}" # failsafe for if (when, ime) crappy path->package-ish mapping hacks break
      ) ]]\033[0m" )
      cols=$(tput cols)
      thelines="$( tail -n ${lines} "${log}" | perl -e "use Text::ANSI::WideUtil; binmode(STDOUT, \":utf8\"); while(<STDIN>) { print Text::ANSI::WideUtil::ta_mbwrap(\$_, ${cols}); }" 2>>/tmp/meterr.log | tail -n ${lines} )"
      bam+="${hdr}"$'\n'"${thelines}"
   done

   # prepend the magic to clear the screen; if no text to display, replace with no activity message.
   bam=$(clear ; echo -n "${bam:-"no recent build.log activity..."}")

   # bam!  clear screen and dump the whole next screen image to console as fast as bash can print it
   if [[ ${oldbam} != ${bam} ]] ; then
      echo -n "${bam}"
   fi
   older_active_logs=("${old_active_logs[@]}")
   old_active_logs=("${active_logs[@]}")
done


It now requires three out-of-tree ebuilds (and, therefore, some kind of overlay), unfortunately:

dev-perl/Text-ANSI-Util/Text-ANSI-Util-0.22.ebuild:
# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6

DIST_AUTHOR=PERLANCAR
DIST_VERSION=0.22
inherit perl-module

DESCRIPTION="Used to justify strings to various alignment styles"
LICENSE="ISC"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE=""

RDEPEND="
        virtual/perl-Exporter
        >=virtual/perl-Scalar-List-Utils-1.460.0-r1"
DEPEND="${RDEPEND}
        >=dev-perl/Module-Build-0.280.0
"


dev-perl/Text-ANSI-WideUtil/Text-ANSI-WideUtil-0.22.ebuild:
# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6

DIST_AUTHOR=PERLANCAR
DIST_VERSION=0.22
inherit perl-module

DESCRIPTION="Used to justify strings to various alignment styles"
LICENSE="ISC"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE=""

RDEPEND="
        virtual/perl-Exporter
        >=dev-perl/Text-ANSI-Util-${PV}
        >=dev-perl/Text-WideChar-Util-0.16"
DEPEND="${RDEPEND}
        >=dev-perl/Module-Build-0.280.0
"


dev-perl/Text-WideChar-Util/Text-WideChar-Util-0.16.ebuild:
# Copyright 1999-2016 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2

EAPI=6

DIST_AUTHOR=PERLANCAR
DIST_VERSION=0.16
inherit perl-module

DESCRIPTION="Used to justify strings to various alignment styles"
LICENSE="ISC"
SLOT="0"
KEYWORDS="~amd64 ~x86"
IUSE=""

RDEPEND="
        virtual/perl-Exporter
        dev-perl/Unicode-LineBreak"
DEPEND="${RDEPEND}
        >=dev-perl/Module-Build-0.280.0
"

_________________
-gmt
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Unsupported Software All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum