Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] How to autodetect changes in the Gentoo handbook
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Installing Gentoo
View previous topic :: View next topic  
Author Message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 546

PostPosted: Tue Nov 15, 2016 8:59 am    Post subject: [SOLVED] How to autodetect changes in the Gentoo handbook Reply with quote

Hi,

A few years ago I implemented a simple BASH script to try to quickly detect changes within the Gentoo Handbook.
It wasn't great but at least it helped decrease the time spent to review the whole manual.
For instance, the Gentoo Handbook used to emerge vixie-cron as an example cron daemon but then switched to cronie.
I wouldn't have detected this change easily if it wasn't for this script.

Basically, what I did was:
1) wget http://www.gentoo.org/doc/en/handbook/handbook-x86.xml?style=printable&full=1
2) converted to text with html2text
3) diff previous_copy_in_local_dir new_copy
4) review changes and mv new_copy previous_copy_in_local_dir

It's been a while since the last time I checked the handbook with this method but now I realize that the new Handbook has changed completely and does not support viewing the whole document in one single page.

I also noticed that wiki.gentoo.org allows to view changes (diffs) of all the docs on that site:

https://wiki.gentoo.org/wiki/Special:RecentChanges

However, there quite a few limitations to the web interface.
If I select Namespace:Handbook, Language:English and "show last 500 changes in last 30 days", I get NO results.
I'm not going to be reviewing the handbook every month so is there a way to view the last changes over the past 12 months?

I then tried to access the following URL hoping to get data from the last 5 years:
https://wiki.gentoo.org/index.php?namespace=560&tagfilter=&translations=filter&users=&trailer=%2Fen&days=1825&limit=500&title=Special%3ARecentChanges
However, I'm getting the same page with NO results.
Does that mean that the English version of the Handbook has not had any changes over the last 5 years on wiki.gentoo.org?

I finally tried this URL (no language filter):
https://wiki.gentoo.org/index.php?namespace=560&tagfilter=&translations=filter&users=&trailer=&days=1825&limit=500&title=Special%3ARecentChanges
but the oldest changes are dated Oct. 23rd 2016.
Does this mean that the Handbook was first edited on Oct. 23rd of this year only?


Last edited by Vieri on Sun Jan 08, 2017 9:52 pm; edited 1 time in total
Back to top
View user's profile Send private message
audiodef
Watchman
Watchman


Joined: 06 Jul 2005
Posts: 6313
Location: /usr/lib64/lv2

PostPosted: Tue Nov 29, 2016 9:31 pm    Post subject: Reply with quote

I would find such a script useful, myself. I would write it in perl, though, and what I'd do is wget the entire handbook locally and then have my perl script diff between my local copy and the current online copy. You can manipulate the output to be as pretty and useful as you need it to be. After each time I use it and am satisfied I'm up-to-date, I'd have the script wget a fresh copy. Repeat as needed over time.
_________________
Gentoo Studio: A Gentoo-based, professional digital audio workstation OS.
Back to top
View user's profile Send private message
324874
Apprentice
Apprentice


Joined: 26 Jul 2014
Posts: 168

PostPosted: Wed Nov 30, 2016 9:40 am    Post subject: Gentoo wiki Reply with quote

No need!
Back to top
View user's profile Send private message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 546

PostPosted: Tue Dec 20, 2016 7:22 am    Post subject: Re: Gentoo wiki Reply with quote

324874 wrote:
No need!


Thanks for the input but as I stated in my first post, there doesn't seem to be any change at all to the English manual.
So either the Gentoo wiki is too "young" and there really are no changes or I'm doing it wrong.
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Wed Dec 21, 2016 7:03 am    Post subject: Reply with quote

I do not get the point of reinstalling with each new hardware.

MY installation is several years old and was moved from several different hardware platforms.

My Backup strategy is basically moving the hole installation and securing it at a point of time.
Back to top
View user's profile Send private message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 546

PostPosted: Wed Dec 21, 2016 7:30 am    Post subject: Reply with quote

Roman_Gruber wrote:
I do not get the point of reinstalling with each new hardware.

MY installation is several years old and was moved from several different hardware platforms.

My Backup strategy is basically moving the hole installation and securing it at a point of time.


Sorry, but I don't get it. The goal of the original post is to be automatically informed of any change within the Gentoo handbook. Even if you don't reinstall from scratch, you still might want to know that, for example, Gentoo "officially" changed or extended the way /etc/conf.d/net works, or that it prefers to install cronie instead of vixie-cron, etc. So all this information should be taken into account even if you keep up to date with frequent "emerge world"s.

Then there are cases where you installed a Gentoo box for a given purpose at a given point in time, without updating regularly, and after a few years you realize you can't easily upgrade and that the best way is to reinstall from scratch.
At this point, it's nice to know your "install script" is "in sync" with the Gentoo handbook instructions.

Anyway, I've come to the conclusion that the Gentoo handbook on the newly-installed wiki has not undergone any changes in the English version.

Thanks
Back to top
View user's profile Send private message
Roman_Gruber
Advocate
Advocate


Joined: 03 Oct 2006
Posts: 3806
Location: Austro Bavaria

PostPosted: Wed Dec 21, 2016 8:02 am    Post subject: Reply with quote

The gentoo handbooks are just a suggestion and very very often "correct".

When you lack the motivation to keep up to date with gnu linux you may go the binary path. Gentoo will not be the correct distro for yourself than. The basic concepts have not changed over those years.


Quote:
Then there are cases where you installed a Gentoo box for a given purpose at a given point in time, without updating regularly, and after a few years you realize you can't easily upgrade and that the best way is to reinstall from scratch.


Just don*t. If so you are better off with a binary distro like linux mint or something else. I updated some outdated boxes, it's not worth it. For such use cases something else fits better the bill.

--

The gentoo handbook had some changes over the few years I used it. UEFI was introduced. SYSTEMD was introduced. config files have changed, but these reflects changes in the opinion of the developers of htose software pieces. And guys do not care for the FHS anymore.
Back to top
View user's profile Send private message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 546

PostPosted: Wed Dec 21, 2016 8:36 am    Post subject: Reply with quote

Roman_Gruber wrote:

When you lack the motivation to keep up to date with gnu linux you may go the binary path. Gentoo will not be the correct distro for yourself than. The basic concepts have not changed over those years.


I've been using Gentoo on *many* systems for 13 years now... I've left behind only a couple of them so, as you've also experienced it, it's really hard to upgrade an old server. The issue here isn't binary vs. source. The issue is that since the Gentoo handbook is the main and only system installation guide (whereas other distros may have installers), it is important for users/admins to keep track of changes to this document. Besides, updating really old system from scratch was just an example. Even if you do have an updated system, you still need to keep track of changes/suggestions in the Gentoo handbook each time you emerge world...

Roman_Gruber wrote:

The gentoo handbook had some changes over the few years I used it. UEFI was introduced. SYSTEMD was introduced. config files have changed, but these reflects changes in the opinion of the developers of htose software pieces. And guys do not care for the FHS anymore.


Right, but these changes (uefi, systemd...) occured before or after the Gentoo Handbook was published on wiki.gentoo.org? I guess they happened before because the Wiki revision history doesn't show these changes.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 5761

PostPosted: Thu Dec 22, 2016 3:01 am    Post subject: Reply with quote

I just use https://wiki.gentoo.org/wiki/Quick_Installation_Checklist.

It is hard to upgrade a severely neglected Gentoo install, but that's why we have `eselect news` and why emerge notifies of unread items.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Tue Dec 27, 2016 11:15 am    Post subject: Re: Gentoo wiki Reply with quote

Vieri wrote:
Thanks for the input but as I stated in my first post, there doesn't seem to be any change at all to the English manual.
So either the Gentoo wiki is too "young" and there really are no changes or I'm doing it wrong.

Most of the handbook pages are "transcluded" (look at the source of the page or the history). For example, look at the source code of Handbook:AMD64/Installation/Base:
Quote:
{{Handbook:Parts/Installation/Base}}

The transcluded page is https://wiki.gentoo.org/wiki/Handbook:Parts/Installation/Base. Finally, you can see the changes in the history of the transcluded pages.
Back to top
View user's profile Send private message
dufeu
l33t
l33t


Joined: 30 Aug 2002
Posts: 896
Location: US-FL-EST

PostPosted: Fri Dec 30, 2016 10:56 pm    Post subject: Re: Gentoo wiki Reply with quote

324874 wrote:
No need!
Tracking changes and Watchlist are pretty much useless for anyone except Wiki editors/admins. The problem with both is that we're looking for a specific kind of change while Tracking and Watchlist attempt to display all changes including typo fixes, grammatical corrections, internal comments, yada-yada-yada. In addition to attempting to point out all changes, they also provide a lot of meta information. It becomes a chore to separate the wheat from the chaff.

Vieri wrote:
A few years ago I implemented a simple BASH script to try to quickly detect changes within the Gentoo Handbook.
It wasn't great but at least it helped decrease the time spent to review the whole manual...

The OP goes on to give some albeit trivial examples but does make a point here.

Some rather significant {from the perspective of a system admin or savvy end-user} changes to the Handbook include dealing with changes in determining NIC names {eth0 to by-path}, deprecating hard drive designations from /dev/sd{letter}{#} to /dev/disk/by-partuid {by-uuid, by-path, by-partlabel}, the continually increasing number of kernel CONFIG values that are required, changes to /etc/fstab, the addition of support for 'system-md', changes in the way you chroot into a new system you're building. A lot of changes in kernel are directly reflected in changes in the Handbook. Those are just the first things that come immediately to mind.

I used to do something similar as the OP in that I'd print the "manual as a single page" and compare it visually side by side with previous incarnations.

What the OP and I are looking for aren't which typos got fixed or which paragraphs got deleted or added or what poor grammatical constructs were changeg. We're looking for changes which have a direct impact on how we manage our current systems in addition to how to build a new system. For system admin purposes, Tracking and Watchlist are mostly noise. It's actually easier to re-read the manual from beginning to end instead.

Moving the Handbook from it's prior incarnation to the Wiki is both good and bad. As the OP points out,
  • all the pre-wiki historical changes are no longer available for research.

  • There is no longer a way to capture the entire handbook as a single {printed} document.

I too, would be very interested in being able to print the Handbook as a single document again. I will say that being able to print the manual to PDF in 4 single page parts is a big return towards previous functionality.

YMMV and all that.
_________________
People whom think M$ is mediocre, don't know the half of it.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Thu Jan 05, 2017 5:42 pm    Post subject: Reply with quote

dufeu wrote:
It becomes a chore to separate the wheat from the chaff. [...] We're looking for changes which have a direct impact on how we manage our current systems in addition to how to build a new system.

I reviewed all the history pages of the handbook in the first part and I saw the interesting changes easily.

dufeu wrote:
I too, would be very interested in being able to print the Handbook as a single document again.

We can display one page per part and print it (go on the "Page Tabs" bar to "Tools" and choose "Printable version").

We can learn how to use the wiki by reading the webpage "Help:Contents".
Back to top
View user's profile Send private message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 546

PostPosted: Thu Jan 05, 2017 9:49 pm    Post subject: Reply with quote

Thanks helecho!
Back to top
View user's profile Send private message
cwr
Veteran
Veteran


Joined: 17 Dec 2005
Posts: 1969

PostPosted: Fri Jan 06, 2017 1:04 pm    Post subject: Reply with quote

In fact I've just been bitten by a handbook change I didn't notice - although I found it in the end.
I was using an older version for an install, and a number of minor but significant items had changed.
Some sort of handbook diff would be helpful; if the handbook is generated from text or Markdown that's
straightforward, but from html, not so much.

Will
Back to top
View user's profile Send private message
dcljr
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2005
Posts: 139
Location: Austin, TX

PostPosted: Sat Jan 07, 2017 4:04 pm    Post subject: Re: How to autodetect changes in the Gentoo handbook Reply with quote

Apologies if I repeat something someone else has already said...

Vieri wrote:
If I select Namespace:Handbook, Language:English and "show last 500 changes in last 30 days", I get NO results.

Don't select "English". That filtering feature doesn't work with English because the English copy of the handbook is not stored under "/en" subpages (as are, say, Spanish under "/es").

Vieri wrote:
I then tried to access the following URL hoping to get data from the last 5 years:

That is way beyond the capabilities of the wiki's "RecentChanges". Can't go beyond 500 changes or 30 days.

dufeu wrote:
Tracking changes and Watchlist are pretty much useless for anyone except Wiki editors/admins. The problem with both is that we're looking for a specific kind of change while Tracking and Watchlist attempt to display all changes including typo fixes, grammatical corrections, internal comments, yada-yada-yada.

I don't see how this is made easier by 'diff'ing the entire Handbook in plain text (aside from "internal comments")…

[digression]As with software updates, the longer you wait to update your world (check for changes in the Handbook) the more likely you'll be bitten by multiple major changes that cause you great pains (diffs where it's hard to see what's actually changed — say, when whole paragraphs are moved around). With that in mind, a "seasonal" check (4 times per year) doesn't seem too unreasonable.[/digression]

FYI, the last I checked (admittedly, many months ago) the Actual Content of the Handbook was contained in these platform-independent pages:

and a just handful of more-specfic pages for each architecture. For amd64, they are (/were):
EDIT: Removed 2 pages that weren't real content.

(You can look through the current list of all Handbook pages at https://wiki.gentoo.org/index.php?title=Special%3AAllPages&namespace=560&hideredirects=1.)

At the risk of being pedantic, what I do is "watch" these pages with "Email me when a page or a file on my watchlist is changed" selected in my preferences. Whenever one of the pages is changed, I get an e-mail containing a link to a diff of the changes. I can follow the link immediately or after any amount of time (even months or years, so long as the page isn't deleted — although note that e-mails about that page stop until you actually visit it again). And then the process starts again (i.e., the next time a page is changed, I get an e-mail…).

If you can deal with wikicode in yours diffs, this is probably the most convenient way of keeping up with changes on a semi-regular basis.

Of course, if you simply must have plain text in your diffs, it probably wouldn't be too difficult to partially automate the process by handing off the e-mails to a script that changes the URL contained within to a "printable" version (e.g., https://wiki.gentoo.org/wiki/Handbook:AMD64/Blocks/Bootloaderhttps://wiki.gentoo.org/index.php?title=Handbook:AMD64/Blocks/Bootloader&printable=yes); then calls, say, 'links -dump' on the URLs; and finally calls 'diff' on the resulting text file(s).

Alternatively, of course, you can just change your previous method to use four different pages instead of just one, as alluded to by dufeu and helecho. [grin]

cwr wrote:
Some sort of handbook diff would be helpful

Perhaps you could lobby the developers to maintain (on the wiki) a "ChangeLog" for the Handbook, listing (and perhaps linking to diffs for?) major changes to the installation process (e.g., the switch to GRUB2).
Back to top
View user's profile Send private message
cwr
Veteran
Veteran


Joined: 17 Dec 2005
Posts: 1969

PostPosted: Sat Jan 07, 2017 4:27 pm    Post subject: Reply with quote

A possible way ahead is putting the manual under a version control system; I haven't
followed the recent debates on the use of git in the portage tree, but once that's set up
adding the manual shouldn't be overly difficult.

Will
Back to top
View user's profile Send private message
dcljr
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2005
Posts: 139
Location: Austin, TX

PostPosted: Sun Jan 08, 2017 2:52 am    Post subject: Re: How to autodetect changes in the Gentoo handbook Reply with quote

BTW…
dcljr wrote:
Whenever one of the pages is changed, I get an e-mail containing a link to a diff of the changes. I can follow the link immediately or after any amount of time

I should point out that there are links in the e-mail to just the latest change and to all the changes since the page was last visited. The latter is the link I'm talking about — it allows you to visit after any length of time has passed and see all the changes made in the intervening time. Just wanted to clarify that.
Back to top
View user's profile Send private message
Mr. T.
Guru
Guru


Joined: 26 Dec 2016
Posts: 477

PostPosted: Sun Jan 08, 2017 7:32 am    Post subject: Reply with quote

You will waste your time if you reinvent the wheel. The wiki explains how to use its features (@dcljr) and these features are sufficient to know
the changes (@cwr). :x

P.S: @Vieri: could you modify the title of the thread, please? (I suppose your issue is solved)

[SOLVED] Handbook: view the changes?
Back to top
View user's profile Send private message
dcljr
Tux's lil' helper
Tux's lil' helper


Joined: 20 Aug 2005
Posts: 139
Location: Austin, TX

PostPosted: Sun Jan 08, 2017 11:21 am    Post subject: Reply with quote

helecho wrote:
You will waste your time if you reinvent the wheel. The wiki explains how to use its features (@dcljr) and these features are sufficient to know
the changes (@cwr). :x

Just trying to be helpful. Take a chill pill…
Back to top
View user's profile Send private message
Vieri
Guru
Guru


Joined: 18 Dec 2005
Posts: 546

PostPosted: Sun Jan 08, 2017 9:51 pm    Post subject: Reply with quote

Every single feedback has been helpful...

Thanks!
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Installing Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum