I may have years of experience with Linux, I may be capable of pulling simple hacks in the kernel source code, but when it comes to practical server maintenance / distro upgrades, I always feel like a complete noob.
I've been a RedHat/Fedora user for some time, and also knew some basics of Debian - but now I tend to lean in favour of Debian for several reasons. It's got this nice "vanilla feel", overall it tends to work pretty well without distro-proprietary bugs, and it works down to 486 computers, which makes it an ideal vehicle for my hardware test suite... And it's not the hopeless dinosaur it used to be - not anymore.
Anyway - the old Debian distro on our mail server was becoming
a pain. The few bits of software that I compile from source
were starting to have issues with the old Libc and compilers,
and trying to upgrade individual components from source would
quickly end up in the classic dependency hell. Some useful
system components (such as mdadm) also deserved an upgrade.
Small partial updates (such as the latest vanilla 2.4 kernel)
also ran out of their steam and glory... the time has come
for an apt-get dist-upgrade.
For the basic distro-upgrade stuff, all you really need is a line or
two per each distro version that you have to deal with (switch between).
Don't be surprised if you have to switch back-n-forth between say
Etch to Lenny, back to Etch and back to Lenny again... just to
accomplish the upgrade. The dependencies can force you into that.
For every version, you should have a line or two, pointing to the
main and contrib branches. You can add security updates later on
as a final touch, after the dust settles.
My /etc/apt/sources.list looks like this:
#deb http://archive.debian.org/debian Debian-3.0 main contrib
#deb-src http://archive.debian.org/debian Debian-3.0 main contrib
#deb http://archive.debian.org/debian Debian-3.1 main contrib
#deb-src http://archive.debian.org/debian Debian-3.1 main contrib
#deb http://archive.debian.org/debian Debian-4.0 main contrib
#deb-src http://archive.debian.org/debian Debian-4.0 main contrib
##deb http://archive.debian.org/debian Debian-4.0/updates main contrib
##deb-src http://archive.debian.org/debian Debian-4.0/updates main contrib
deb ftp://ftp.cz.debian.org/debian/ lenny main contrib
deb-src ftp://ftp.cz.debian.org/debian/ lenny main contrib
#deb ftp://ftp.cz.debian.org/debian/ squeeze main contrib
#deb-src ftp://ftp.cz.debian.org/debian/ squeeze main contrib
apt-get update).
At the time I was upgrading, my machine reported itself as Etch (4.0),
and it was shortly after the new stable Squeeze (6.0) got released
and Lenny (5.0) became old-stable, and Etch became unsupported
-> its repo moved to archive.debian.org.
For the stable and old-stable releases, the master repo resides at
ftp://ftp.cz.debian.org/debian/ ,
but you can (and you should!) use some regional mirror nearby,
both to offload the global master and to get faster downloads for yourself.
My closest mirror is at ftp.cz.debian.org - all I had to do in sources.list
was insert a ".cz." as the third-level domain.
While fighting with the upgrade process, I concluded that it's perhaps a good habit, in terms of staying out of trouble, to only ever have a single Debian version pointed to (repo active) in the sources.list. Which is easily accomplished by commenting out all the others. Merely commenting the others out proves convenient whenever the dependency hell during the upgrade forces you to flip between nearby versions.
Whenever you modify the set of repos linked to by sources.list,
even just switch distro versions up or down, you must perform an
apt-get update to re-create your local index of packages
available (and perhaps to rebuild their dependency map).
To install an individual package, perform apt-get install <package> .
To upgrade an individual package to latest version, perform apt-get install <package> .
To upgrade all packages to their latest versions, perform apt-get upgrade .
I obviously started by making an off-line backup of the whole system, in case I needed to roll back everything later that evening. Boot from a favourite live CD, mount all the partitions under /mnt/source, add an empty drive under /mnt/backup, cp -dpR. Piece of cake.
As you're getting ready for the upgrade, you'll probably google
a bit, to learn your way about the task. The fist thing Google reveals
is something like
# update /etc/apt/sources.list to point to the new stable version
apt-get update
apt-get install apt
apt-get dist-upgrade # upgr. whole distro, handle non-trivial configuration changes
In my case, the machine was originally installed as Sarge
(3.0) or maybe even Woody (2.0). Two other admins have
carved their signatures into the distro before my upgrade attempt.
I myself have upgraded the hardware under the distro months
ago - I prepared a newer machine with two spacious IDE drives,
set up a MD RAID (mirror) with header v0.90 for RAID autodetect,
partitioned the RAID from a live CD, installed Grub v1 by bare hand,
inserted a very new 2.4 kernel compiled elsewhere, and finally
I copied all the data using just "cp -dpR" or some such, from the
old drive. That worked just fine.
I mean to say that the Debian install had been through quite some
individual history :-) when I decided to upgrade the distro again,
this time myselfs.
When I approached the machine, /etc/debian_version said 4.0.
So I pointed sources.list to Lenny (5.0) and followed the sequence of
commands up to apt-get dist-upgrade.
Quite to my surprise, that did nothing at all! 0 packages upgraded,
all of them already at the current version. ... ***What***?
Whatever I tried, apt-get refused to install new versions of packages.
On packages that were not outright wrong (non-existent package name)
it would say something like "that package does exist in the database,
but I have nowhere to download it from".
I discovered that my APT version was 0.4.5 or so, compiled in 2001.
(apt-get --version)
Looked more like Sarge or even Woody, rather than Etch.
Now it's 2011, for god's sake...
But even when I pointed sources.list back to Etch (4.0),
apt-get install apt would not upgrade apt to a newer
version (I expected at least 0.5). Even more interestingly, apt-get
was refusing to install other Etch packages, such as "aptitude",
which would've been another possible workaround.
Only when I pointed sources.list back to *Sarge* (3.0) did I get
some responses: aptitude got installed, but didn't help.
I was suspecting some problem with the old version of APT not able
to parse the new package index data for Etch/Lenny etc. And, being
unable to get a new APT via apt-get, I went hunting for a way to
install new APT from source!
It took me a while to find a source tarball of Lenny-level APT 0.7.20 at
http://packages.debian.org/lenny/
which seems to be a somewhat cryptic site. Then I also installed
a fresh libcurl from source, because the Sarge-level libcurl wasn't
good enough (APT needs it for HTTPS support, it would seem = somewhat
useless, but no way to ./configure --without-curl).
I did manage to port APT 0.7.20 to my Etch/Sarge hybrid - I had to
clean up some modern yet marginal elements of C++ that didn't compile
on the old compiler (string::clear() and operator "or" for instance),
add some missing includes, make a symlink to libstdc++ and some such.
Finally, I installed the APT binaries by hand, because the compilation
shipwrecked on some documentation conversion issues (gettext too old).
So, after three hours of detective work, I had a new APT, and it was
executable on my system. Another apt-get update, and voila:
It still didn't work!!!
I was about to call it a day, reboot and go home
(it was about midnight), as I didn't get pretty much anywhere.
I still googled a bit running on the last bits
of my motivation, when I stumbled over some barely relevant
forum posting which mentioned:
"it doesn't work even if you remove /etc/apt/preferences?"
So I did exactly that and
Bingo!!!
Suddenly apt-get started to work the way I'd expect it to.
More or less, anyway.
I pointed sources.list to Lenny (5.0) and fired up
apt-get dist-upgrade.
That did replace a few packages, then got as far as the glibc.
At that point, it said something like "we are about to install a new glibc.
In order to do that, you need to upgrade your kernel first, from 2.4 to 2.6,
as the new Glibc has NPTL. I can tell you right here that it won't work with
Lenny kernels, as those already need the newer libc. So you need to add Etch
to your sources.list and start with some Etch-level 2.6 kernel".
Well I tried some Lenny-level 2.6 kernel first anyway
(apt-get install linux-image-2.6, then pick a particular
flavour from the list offered), and it failed exactly the way the
warning had said.
So I added Etch (4.0) to sources.list, so that both Etch and Lenny were uncommented
at the same time - and tried upgrading the kernel. For some reason, that didn't
quite want to work. I had weird errors, dependencies failing etc.
I learned to use apt-get -f install, which did help
a bit / in some cases, but I couldn't get past the installation of a kernel.
Next, I removed Lenny (5.0) from sources.list - so that only Etch (4.0)
packages remained in play. That cleaned up some garbage from view,
and finally I got that darn 2.6 kernel to install.
From then on, things started to develop more positively.
Still at Etch-level, I over-installed APT with the proper Etch
version from the repo.
Then I switched sources.list from Etch-only to Lenny-only,
and now the dist-upgrade went ahead much further. I upgraded the glibc,
the kernel again, APT again (back to 0.7.20, this time from a binary .DEB).
Perhaps at this point it's time to draw some conclusion, sum it up
into a morale of some sorts. Let me put it this way:
When upgrading from Etch (kernel 2.4) to Lenny (kernel 2.6),
start by checking that APT works the way it should.
If necessary, remove /etc/apt/preferences.
Next, make sure that APT is at Etch level.
Next, as you'll be upgrading across the NPTL compatibility boundary,
first upgrade the kernel to the latest 2.6 available for Etch.
Only then, shift your sources.list to Lenny and try a dist-upgrade.
Not sure if this "always start by the kernel" is a general advice,
applicable to future Debian releases. Maybe not Lenny to Squeeze.
Maybe the point is to look for important milestones that break
compatibility between the kernel and libc (user space).
Some minor gotchas still remained to be solved.
First there was some conflict between two packages, both providing
the same documentation file (passwd and some PAM-modules),
which was evaluated by APT as a no-go and made the dist-upgrade
fail. I managed to solve this relatively quickly by an explicit call to
dpkg -i --force-overwrite <package> .
(Got the full package pathname from the error message printed by apt-get.)
Next, as soon as Perl got upped to Lenny level, apt-get
(probably running some Perl scripts) started bitching
over and over about some Locale settings being wrong.
I first tried setting some things manually, but then I gave up.
Googled a bit and found the canonical solution:
apt-get install locales # perhaps already installed
dpkg-reconfigure locales # and pick the locales desired
After that, Perl and apt-get stopped whining about Locale.
And some really small snippets to finish with:
Between Etch and Lenny, Postfix gets upgraded past v2.4.
This brings some minor configuration changes (to master.cf).
It seems appropriate to run "postfix upgrade-configuration"
manually after the dist-upgrade.
I also lost /usr/sbin/amavisd-new somewhere along the way.
Once I realized that, I just ran "apt-get install amavis" and got it back.
Also the structure of Amavis config files has changed,
so that I had to transmogrify the old config into new config
semi-manually (copy and paste individual blocks),
mostly because the direct access to clamd doesn't work
(had to switch back to clamdscan). Then I realized
the new config has prevented .EXE and .ZIP files from being sent,
so I removed them from the blacklist.
The new Postfix build from the DEB probably knows SMTP Auth.
And, Thunderbird as a client is configured by default to ask for SMTP Auth.
If you don't properly configure the Auth in the Postfix, this leads
to an authentication failure. There are two ways to solve this:
configure SMTP Auth source in the Postfix server, or configure the
Thunderbird not to ask for SMTP auth.
An upgrade from Cyrus 2.1 to 2.2 might theoretically require
some manual conversion of DB formats for its various indexes,
but practically I didn't need pretty much any conversion.
I have a limited number of users, and the DB files
were mostly plain text -> nothing to convert there.
Again the process was fairly well guided.
I also got SpamAssassin derailed somehow along the way,
during the upgrade... the remedy was as simple as copying
back the old config (or commenting out two lines in
SpamAssassin's config file, if memory serves, as the
init script would LOUDLY suggest).
Squirrelmail somehow lost the ability to speak our local language.
After some inspection, I found out that all that is needed is a
dpkg-reconfigure locales
followed by
apt-get install squirrelmail-locales
and check the $squirrelmail_default_language and $default_charset
in /usr/share/squirrelmail/config/config.php.
See also aptitude search squirrelmail .
The one thing that I've learned very early and for sure is:
when apt-get dist-upgrade fails, don't worry.
The packages already updated are not rolled back.
The particular package that failed to install may be
in a somewhat inconsistent state, but apt-get -f install
should take care of that one. After you remove the problem,
run apt-get dist-upgrade again and it'll start
approximately from the point where it last stopped. Or sometimes,
it will start elsewhere - the decision where to start seems
to be somewhat random. Only if the install failure affects
a key common dependency (single package), apt-get will fail
repeatedly right after you start it, until you solve the problem.
Overall I was amazed at how much abuse the APT/DPKG packaging system can tolerate, without bringing the system into a coma. Just consider that you're messing with crucial system packages and even the Glibc library, the process often fails somewhere halfway there, and still the overall outcome is a responsive system, not a hint of some difficulty.
The addition of a new kernel (the old ones remain in /boot,
along with their initrd images) is equally seamless.
The package addition also entails automagical creation of an initrd,
all you have to do is add the respective entry to your bootloader
(if you have your own bootloader config, which APT knows about).
My existing 2.4 kernel was monolithic, without an initrd, with MD RAID
compiled in - I was a little worried if the 2.6 kernel installed
via APT would reflect the root on an MD device.
Quite to my surprise, it worked perfectly seamlessly. The MD drivers
are modules, but were automatically (silently) added to the respective
initrd upon the installation of the new kernel.
Amazing, impressive.
After modding /etc/apt/sources.list +
"apt-get update" + "apt-get install apt" + "apt-get install aptitude",
I first installed the squeeze kernel = 2.6.32-5.
This entailed the infamous and somewhat pointless change of
HDD device names (hda->sda) - but as I'd installed on
/dev/md0,1,2,3 with kernel-space auto-detect (long-ago),
I wasn't very afraid... Otherwise I'd have to mod the fstab and
maybe the bootloader cfg. That initial step went just fine.
During "apt-get dist-upgrade", there were lots of "warning: missing architecture" from APT (essentially several such lines for every package). Maybe they started pouring after the libc upgrade, which came fairly early on. Or after an APT upgrade. They mentioned some ancient packages still present in /var/lib/dpkg/available.
When asked whether to switch from Bash to Dash, reluctantly
I chose "yes".
Hmm: pop-before-smtp is moving to a new TCP port? That's
configured in /etc/defaults/pop-before-smtp and /etc/postfix/main.cf.
And besides, perhaps I should switch to SASL Auth for Postfix in the
first place...
Then I faced a message saying "Unable to migrate to dependency-based boot system". Some useful info followed - essentially a list of some old packages, which kept init scripts around that were missing the newly required LSB tags. Finally, there was a line saying "To reattempt the migration process after the problems have been fixed, run 'dpkg-reconfigure sysv-rc'". Very helpful. And a link to a Debian Wiki entry on dependency-based booting. My offending packages seemd pretty harmless, but I skipped this just to save time. My small single-CPU system with slow drives likely wouldn't benefit from parallel init anyway.
Apache comes with a new default config, ho hum, no problem,
let's try to keep the old one for example. Its only task is
to serve SquirrelMail, anyway.
Locale broken again, I already know what to do about that one...
Again the same changes to spamassassin, amavis and Cyrus... (no, no, no).
Heh - an update to initramfs-tools, or rather a post-update script,
has tried to update an old/unused lilo.conf, and was failing at that.
So I just renamed lilo.conf to some other name, and the dist-upgrade
finished (this was the very last package).
Ahh well. TLS crypto used by Cyrus. Several messages in /var/log/mail.err:
Fatal error: tls_start_servertls() failed
TLS server engine: No CA file specified. Client side certs may not work
unable to get private key from '/etc/ssl/private/cyrus-global.key
TLS server engine: cannot load cert/key data, may be a cert/key mismatch?
error initializing TLS
Fatal error: tls_init() failed
The one about "No CA file specified" is usually harmless, just a warning.
Moreover, it can be zapped by uncommenting the following line in /etc/imapd.conf:
"tls_ca_file: /etc/ssl/certs/cyrus-imapd-ca.pem"
Except that, the filename has a typo, and results in yet another error message,
saying "cannot load CA data" - this one pops up in mail.log.
So the line in imapd.conf should actually read:
tls_ca_file: /etc/ssl/certs/cyrus-imapd.ca.pem
Well at least I've learned how to properly strace cyrus (I ended up
just stracing "/etc/init.d/cyrus2.2 start" explicitly, as the internal
hooks in /etc/imapd.conf didn't want to work).
The other messages are more serious. Effectively IMAPs and POP3s are now defunct.
It turns out that the key file is okay, but Cyrus cannot read it anymore
for some reason, after the upgrade. In /etc/imapd.conf you can see:
tls_key_file: /etc/ssl/private/cyrus-global.key
Looking on disk, you can see that cyrus-global.key is
a symlink to ../certs/server.pem (inherited from some old Cyrus install).
The trouble is, that /etc/ssl/private (the directory) has ownership
root/ssl-cert - i.e. only root and the "ssl-cert" group can enter that
directory. And, Cyrus is neither for some reason: neither suid root,
nor a member of the ssl-cert group. Some say it is a big no-no to add
the "cyrus" user to the ssl-cert group. Rather, the private key file
should be put someplace where Cyrus can read it. But voila,
the symlinked target directory is readable to anyone and the
target file is actually owned by cyrus/mail.
So the simple remedy is: in /etc/imapd.conf, turn the original
line into a comment:
#tls_key_file: /etc/ssl/private/cyrus-global.key
and make a copy of that line and redirect it straight to the
symlink's target:
tls_key_file: /etc/ssl/certs/server.pem
Voila, the users can log in again.
In my case, the cert and key files are rather stale,
I wasn't the one who created them, I have renumbered IP addresses
in the meantime... the certs/keys no good anymore, I should generate
a new self-signed certificate at the very least. Which implies
further trouble: my server has three distinct IP addresses
and is serving its SSL-borne services on all three of them
(two public uplinks and a private subnet) using a single instance
of the Cyrus master.
Which is a problem for SSL (designed to run on a single IP addr).
Ahh well...
I finished off by dpkg-reconfigure locales and by checking
that SquirrelMail can still run in our local language.
With respect to Squirrelmail, I have noticed some glitches in the
"vacation message" plugin. Firstly, Proftpd (needed for this plugin)
kept failing to start (via inetd), because it doesn't recognize the
following option anymore, in /etc/prodftpd/proftpd.conf:
DisplayFirstChdir .message
Just comment that line out.
Secondly, the "auto-response" messages previously stored in iso-8859-2
somehow got displayed in UTF-8 without conversion, leading to garbled
national characters. Which entailed quite some fun to get sorted :-)
The whole debugging procedure is perhaps worth mentioning, even if it's
just a bit of newbie stalking documented. Fun to read, if nothing else :-)
telnet mail 80
GET http://mail/squirrelmail/src/webmail.php HTTP/1.0
(and press ENTER twice)
<?php phpinfo(); ?>$default_charset = 'iso-8859-2';
in /usr/share/squirrelmail/config/config.php - to no avail.
(Squirrelmail's global config.)
CharsetDefault ISO-8859-2 to the Directory
section in /etc/apache2/conf.d/squirrelmail.conf - to no avail.
(Apache's config for the SquirrelMail virtual host.)cd /etc/apache2/mods-enabled/ln -s ../mods-available/charset_lite.load charset_lite.load/etc/init.d/apache2 restartAddDefaultCharset ISO-8859-2 in
/etc/apache2/conf.d/charset - to no avail.
(Apache's global config.)
dpkg-reconfigure locales and changed the system-wide
default locale from cs_CZ.UTF-8 to cs_CZ (=Latin2) and rebooted
- to no avail.
default_charset = "iso-8859-2"cstocs -i utf8 il2 *.po
(apt-get install cstocs first, if you don't have it yet)
sed -i -e 's/[Uu][Tt][Ff]-8/iso-8859-2/g' *.po
rm *.mofor file in *.po; do msgfmt -o `echo $file | sed -e 's/\.po$/\.mo/'` $file; donecd /usr/share/squirrelmail/localemv cs_CZ cs_CZ-utf8mkdir cs_CZmkdir cs_CZ/LC_MESSAGEScp cs_CZ-utf8/setup.php cs_CZ/cp $SM_LOC_SRC/cs_CZ/LC_MESSAGES/*.mo ./cs_CZ/LC_MESSAGES/$languages['cs_CZ']['CHARSET'] = 'iso-8859-2';$languages['cs_CZ']['LOCALE'] = array('cs_CZ.ISO-8859-2', 'cs_CZ.ISO8859-2', 'cs_CZ');functions/i18n.php$languages['cs_CZ']['CHARSET'] = 'iso-8859-2';$languages['cs_CZ']['LOCALE'] = array('cs_CZ.ISO-8859-2', 'cs_CZ.ISO8859-2', 'cs_CZ');$languages['cs_CZ']['NAME'] = 'Czech';(... making a backup of the whole /etc/ ... good. )
Let's start with apt-get update, just for a good measure, still with the apt sources
pointing to squeeze.
Uh oh: NO_PUBKEY 8B48AD6246925553
Remedy:
gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
gpg -a --export 8B48AD6246925553 | apt-key add -
(Mind the "-" at the end of the line.)
Turns out to be some key for Wheezy. Which would get imported by apt-get update.
Which however cannot work, for lack of the key. Ahh well :-)
Good. The next one up is apt-get upgrade. Again, just for a good measure.
Some message about ISC DHCPd whirred past. Ahh well.
Maybe it said that dhcpd was in fact already stopped - that would make sense, it is stopped.
Next, the post-install script for Grub asked, which device is my boot device... fine.
I'll quote this one verbatim:
cups (1.4.4-7+squeeze2) stable-security; urgency=high
In order to mitigate a privilege escalation from the lpadmin to root
(CVE-2012-5519), the /etc/cups/cupsd.conf configuration file is split
in two configuration files:
* /etc/cups/cupsd.conf can be edited by members of the lpadmin group
through the cups web interface;
* /etc/cups/cups-files.conf can only be edited by root;
Many sensitive configuration statements can now only be set in
cups-files.conf. No statements have been moved automatically. Please
check the respective manpages.
Other than that, there were no objections. Good!
Some authors
recommend to follow up with further steps
(with sources still pointing to squeeze):
apt-get dist-upgrade
dpkg --audit
dpkg --get-selections | grep hold
aptitude ...and press "g".
Top it off by a reboot, just in case.
Next, point the sources to Wheezy - edit /etc/apt/sources.list and make it look e.g. like this:
deb http://ftp.cz.debian.org/debian/ wheezy main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ wheezy main non-free contrib
deb http://ftp.cz.debian.org/debian/ wheezy-updates main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ wheezy-updates main non-free contrib
#deb http://security.debian.org/debian-security wheezy/updates main contrib non-free
#the security stuff may be offline, if you're two Debian versions behind stable
Basically, you need to replace any occurrence of "squeeze" with "wheezy" :-)
Let's update the list of packages, from the new sources:
apt-get update
Uh oh... some key missing again...
gpg --keyserver pgpkeys.mit.edu --recv-key 7638D0442B90D010
gpg -a --export 7638D0442B90D010 | apt-key add -
Hmm. This is a key for Jessie (= Debian 8). Interesting.
Always "one key ahead" seems to be missing :-)
Now for the upgrade.
First step (the easy one):
apt-get upgrade
apt-get dist-upgrade(... making a backup of the whole /etc/ ... good. )
Let's start with apt-get update, just for a good measure, still with the apt sources pointing to squeeze.
Good. The next one up is apt-get upgrade. Again, just for a good measure.
Further steps to remove any left-over lint
(with sources still pointing to wheezy):
apt-get dist-upgrade
dpkg --audit
...hmm... some packages are missing .md5sums checksums.
aptitude install debsums
debsums -l
cd /var/cache/apt/archives
aptitude download binutils
aptitude reinstall binutils
aptitude remove libdb4.5 old, superseded, unused - be careful
aptitude remove dhcp3-client pre-squeeze version
aptitude remove dhcp3-common pre-squeeze version
dpkg --audit
...good.
dpkg --get-selections | grep hold
aptitude ...and press "g".
Next, point the sources to Jessie - edit /etc/apt/sources.list and make it look e.g. like this:
deb http://ftp.cz.debian.org/debian/ jessie main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ jessie main non-free contrib
deb http://ftp.cz.debian.org/debian/ jessie-updates main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ jessie-updates main non-free contrib
deb http://security.debian.org/debian-security jessie/updates main contrib non-free
Basically, you need to replace any occurrence of "wheezy" with "jessie" :-)
Update the list of packages, from the new sources:
apt-get update
No surprises.
Now for the upgrade.
First step (the easy one):
apt-get upgrade
apt-get dist-upgrade
On one machine, during the dist-upgrade, the package containing a new udev or something
has complained that my running kernel lacks some features required for the new udev (or something)
and that if the dist-upgrade proceeds, it will render the machine unbootable.
I scratched my head... I did have a vanilla 3.3.0. installed on the machine, based on
squeeze config - and this was already past wheezy, with Wheezy 3.2.0. installed,
but not selected as a default in grub. So I tried to change the default boot profile in the
debianese grub2. I noticed grub-set-default, noticed its warning to change the recommendation
to set GRUB_DEFAULT=saved in /etc/default/grub (I did), and I did "grub-set-default"
using the lengthy UUID string (difficult to count the entries in the generated grub.cfg, right?),
and i also removed the old /boot/grub/device.map... I rebooted, and ended up with
a grub rescue prompt :-(
Boot from installer CD, mount / and /boot, mount --bind /dev /proc /sys, chroot,
revert all the silly changes, update-grub2, and also grub-install on the physical
drives just for a good measure (grub-install objected against installing onto /dev/md0).
Voila, system bootable again. Selected the Debianese 3.2.0 kernel manually,
to go ahead with the dist-upgrade. Morale: use an ordinal number to select a default
profile in /etc/default/grub. If you stick to the canonical naming of kernel versions
in file names under /boot/, the boot profiles are ordered in descending order
by kernel version (the latest gets profile No. 0 = first in the queue, top of the stack).
After this intermezzo, apt-get dist-upgrade complained about some messed up dependencies.
Lazy as I am, I just started "aptitude" and pressed "g" twice. It did mention some of the
messed up dependencies (that apt-get had complained about) and corrected them.
Back to apt-get dist-upgrade - now it ran fine.
Ehh... LPD not listening on the socket. There's no /var/lib/cups/daemon/cups-lpd. What ho?
Ahh well... cups not installed. aptitude install cups. cups-lpd now back in place.
But, LPD still not listening on the socket! Openbsd-inetd installed, configured...
Heh Samba complains that cups itself does not listen on its socket! What ho?
Cups-daemon is only half-installed. aptitude install cups-daemon. Cups finally running.
But... LPD still not listening! Inetd... Inetd not running! Grr... Openbsd-inetd not really installed!
Aptitude install openbsd-inetd... Inetd finally running, LPD and TFTP finally listening again.
Try to print something... theres NOTHING on the printer! Fumbling in the cups logs... "job-name : bad UTF-8 sequence".
Reported by cups-lpd, generated within cups/ipp.c, ippValidateAttribute().
Hacking this in the source was over my head (decipher the precise call stack,
hack it where it hurts, understand all the consequences) - so I followed my
decade-old gut feeling and GUTTED THE WHOLE CUPS THING!
Refusing a print job on the grounds of "job name" (doesn't get printed anywhere)
not being proper UTF-8, which gets triggered by any print job from Windows
which has localized characters in the job name, even though the job name actually
gets printed allright in the logs and displayed in a terminal, that's a sin against
"be liberal in what you accept". Doesn't look like free software. After all,
there's an Apple logo on top of all cups manpages... enough of this.
Back to LPR. Or rather, LPRNG. Didn't have the old recipes backed up,
but the printcap generated by cups from its printers.conf was a good start,
I merely corrected a thing or two and added some two further tags as a frosting,
then I had to learn the hard way about how to tell lpd to listen on a TCP socket
and actually accept TCP sessions from outside... (funny thing: you can see
lpd listening on TCP:515 in netstat, you can see some SYN and RST packets
actually exchanged with clients on port 515, but no jobs accepted and netstat -t
doesn't even show any sessions established - it's down to LPD security setup, stupid!)
-- and THEN my centralized network printing was finally back to normal :-)
One last touch: change a single word in smb.conf, to tell Samba to print using
lprng, rather than cups. All the shared print queues resurfaced in Samba.
Another last touch, actually: finally I could remove the 10-minute cron job,
re-enabling paused cups queues.
Yeah. Better than taking a shower :-)
BTW, do you know where LPRNG logs its activity? Job submissions etc?
They're not in syslog or /var/log, they're under /var/spool/lpd/<queue name>/
(= among the print queue files) and there are actually *several* log files.
And, the log entries only appear several seconds after the job has actually
been forwarded to the printer... Two useful commands: 'lpstat all' (shows last job)
and 'lpc status all' (shows actual status information you'd expect).
Btw, you may want to add :jobs_done=0: into your printcap (per queue),
otherwise the nightly cron job of lprng (checkpc) will complain about
'unremoved' stale jobs in the queue. That's a feature of lprng, it
keeps the last job in the directory after it has been successfully
processed... also Samba may have a problem interpreting the queue status,
unless you specify jobs_done=0.
Side note: after I disabled IPv6 on the box (just to make sure) using a sysctl,
Exim4 started complaining that it's unable to bind to ::1 = the IPv6 loopback.
After lots of searching what helped was the following change
in /etc/exim4/update-exim4.conf.conf :
#dc_local_interfaces='127.0.0.1 ; ::1'
dc_local_interfaces='127.0.0.1'
/etc/init.d/exim4 force-stop
update-exim4.conf
/etc/init.d/exim4 start
Later on, I discovered further subtle changes in the behavior of Samba.
Firstly, I noticed that, on a first logon attempt of a client after
the client's reboot or suspend/wakeup, Samba would kick the client
(auth failed). If you tried again, Samba would accept the second
attempt just fine. For a human user this hurts but is acceptable,
for automation scripts (batch files that map drives) this is a showstopper.
I later found other people's reports that if they kill idmapd,
the problem appears gone. I considered that a clumsy workaround
(especially considering centralized LDAP-based auth) and dug deeper.
Solution:
add the following line to smb.conf:
idmap config * : range = 1000-1999999
I have yet to find exactly what it means / how the ranges
are established, but it looks more plausible than killing idmapd.
Secondly, some of my users have notified me that after the upgrade,
some .EXE files are no longer executable straight from a Samba share.
Files that previously used to be executable.
As I also moved the files to a bigger MD RAID volume
with a different FS (EXT3->EXT4), I considered some
mischief in privileges correlating to the filesystem,
and sure enough there were some old/odd ownerships
and privileges across the directory tree, inherited through
the ages of upgrades (RedHat, Fedora, Debian) but at a closer
look I found out, that Samba indeed changed its behavior too:
until Samba version 3.6, it would automatically flag
all files as Executable (to Windows clients) - which it
no longer does. No longer automatically, unless you specify
the following in smb.conf:
acl allow execute always
Samba no longer likes "printing = none". And even though this yields just a warning, smbd actually fails to start, saying that it could not open /etc/samba/smb.conf. Out of which it has already parsed several dozen rows...
The solution is: just comment out "printing = none". Voila, Samba is willing to start.
Actually when this happens during "aptitude upgrade", the Samba package will end up in a half-installed state (unconfigured). To get out of this, just stop all the samba daemons (smbd and nmbd) and try another "aptitude install samba".
While trying to figure out this problem, I've learned a nice
trick to make smbd more talkative. Run the daemon by hand, using
smbd -F -S --debuglevel=5
Actually the range is 0 to 10.
You can also run this through strace if the debug alone is not enough.
apt-get update apt-get upgrade apt-get dist-upgrade dpkg --audit aptitude install debsums debsums -l dpkg --get-selections | grep hold
apt-get update apt-get upgrade
Now for the serious step, where some fun is most likely to be expected:
apt-get dist-upgrade
Yup, that's all. A reboot went without any problem.
One final step:
apt-get autoremove
Let's try samba. Hmm... cannot connect with my old password. What ho?
Nothing in the usual logs...(I should probably take another look at Samba logging)
Let me try the trick with smbd -F -S --debuglevel=5 once again...
NTLMv1 passwords NOT PERMITTED for user what_have_you
Good... so I need "ntlm auth = yes" in smb.conf.
apt-get update apt-get upgrade # Should I have done this? Oops :-) too late #apt-get dist-upgrade #dpkg --audit #aptitude install debsums #debsums -l #dpkg --get-selections | grep hold # edit the /etc/apt/sources.list apt-get update apt-get install apt apt-get dist-upgrade
Gotta check the status of systemd-timesyncd and possibly zap it. I used to have ntpd installed.
Samba has some news:
May 03 20:31:26 file systemd[1]: /lib/systemd/system/smbd.service:9: PIDFile= references path below legacy directory /var/run/, updating /var/run/samba/smbd.pid › /run/samba/smbd.pid; please update the unit file accordingly.Same thing for nmbd. You just need to update the two unit files = remove the leading /var from the path in those unit files.
systemctl restart nmbdYup, running again.
systemctl restart smbd
VIM needs "syntax on" and "set mouse=" to protect my sanity.
Needs them in /etc/vim/vimrc.
If I try to put them in /etc/vim/vimrc.local, I get an error message on every start of vim. Go figure. Maybe next time.
/etc/sshd_config got auto-replaced for me... ahh well.
/etc/initramfstools/conf.d/resume now wants to set the RESUME variable to a particular swap partition.
On a server, I can set it to RESUME=none. On a laptop, I'd use blkid | grep swap to learn the right UUID.
Irqbalance has fallen out of favor? Debian 10 says that v1.5 is available. Upstream is at 1.9 as of this writing...
Otherwise, so far so good! I did some removal of old kernels, and a few restarts, and everything seems fine.
dpkg --audit
aptitude install debsums
debsums -l
dpkg --get-selections | grep hold
apt-get autoremove
apt clean
Everything seems squeaky clean.
# This is appropriate after you've lived with the current version for some time. # I.e., *before* you switch the pointers to the next version to upgrade to. apt-get dist-upgrade dpkg --audit aptitude install debsums debsums -l dpkg --get-selections | grep hold # and resolve any pending junk # now finally edit the /etc/apt/sources.list, point to the next version. # Apparently, for Bullseye, instead of security/updates you now need to write bullseye-security/updates . apt-get update apt-get install apt
Meh, apt complains that there are some conflicting dependencies,
and suggests to uninstall exim4 to satisfy these.
Nah... is this EXIM still a maintained package?
Yes it is. So instead, let's start by upgrading Exim first and foremost.
And to be on the safe side, start by making a backup of your /etc/exim4/ directory.
After that:
apg-get install exim4-daemon-lightThis has auto-installed exim4-base and exim4-config as dependencies. And about 80 MB of other dependencies get upgraded, among them a new libc, and some that were needed for the new apt.
After that, the following step (re-tried) only installs some 4 remaining dependencies :-)
And then of course the total dist-upgrade may follow...
apt-get install apt apt-get dist-upgrade # and a restart dpkg --audit aptitude install debsums debsums -l apt-get autoremove apt cleanAll of that went pretty much without a hitch :-)
Oh... we have an update: /var/named needs to have correct privileges (bind.bind).