Debian, apt-get dist-upgrade, practical notes

by Frank Rysanek [ rysanek AT fccps DOT cz ]

Intro

I may have years of experience with Linux, I may be capable of pulling simple hacks in the kernel source code, but when it comes to practical server maintenance / distro upgrades, I always feel like a complete noob.

I've been a RedHat/Fedora user for some time, and also knew some basics of Debian - but now I tend to lean in favour of Debian for several reasons. It's got this nice "vanilla feel", overall it tends to work pretty well without distro-proprietary bugs, and it works down to 486 computers, which makes it an ideal vehicle for my hardware test suite... And it's not the hopeless dinosaur it used to be - not anymore.

Anyway - the old Debian distro on our mail server was becoming a pain. The few bits of software that I compile from source were starting to have issues with the old Libc and compilers, and trying to upgrade individual components from source would quickly end up in the classic dependency hell. Some useful system components (such as mdadm) also deserved an upgrade. Small partial updates (such as the latest vanilla 2.4 kernel) also ran out of their steam and glory... the time has come for an apt-get dist-upgrade.

APT basics; what you should have in /etc/apt/sources.list

For the basic distro-upgrade stuff, all you really need is a line or two per each distro version that you have to deal with (switch between). Don't be surprised if you have to switch back-n-forth between say Etch to Lenny, back to Etch and back to Lenny again... just to accomplish the upgrade. The dependencies can force you into that.
For every version, you should have a line or two, pointing to the main and contrib branches. You can add security updates later on as a final touch, after the dust settles.

My /etc/apt/sources.list looks like this:

#deb http://archive.debian.org/debian Debian-3.0 main contrib
#deb-src http://archive.debian.org/debian Debian-3.0 main contrib

#deb http://archive.debian.org/debian Debian-3.1 main contrib
#deb-src http://archive.debian.org/debian Debian-3.1 main contrib

#deb http://archive.debian.org/debian Debian-4.0 main contrib
#deb-src http://archive.debian.org/debian Debian-4.0 main contrib
##deb http://archive.debian.org/debian Debian-4.0/updates main contrib
##deb-src http://archive.debian.org/debian Debian-4.0/updates main contrib

deb ftp://ftp.cz.debian.org/debian/ lenny main contrib
deb-src ftp://ftp.cz.debian.org/debian/ lenny main contrib

#deb ftp://ftp.cz.debian.org/debian/ squeeze main contrib
#deb-src ftp://ftp.cz.debian.org/debian/ squeeze main contrib
      

This is after a successful upgrade to Lenny (5.0) - note that I have only the lines relevant to Lenny uncommented.
Actually a single line should be enough, the one starting with "deb ". The lines starting with deb-src refer to "source code packages", i.e. if you wanted to recompile some original Debian package from source. If you don't know what you'd need this for, you can safely remove/comment those lines (for a faster apt-get update).

At the time I was upgrading, my machine reported itself as Etch (4.0), and it was shortly after the new stable Squeeze (6.0) got released and Lenny (5.0) became old-stable, and Etch became unsupported -> its repo moved to archive.debian.org.
For the stable and old-stable releases, the master repo resides at ftp://ftp.cz.debian.org/debian/ , but you can (and you should!) use some regional mirror nearby, both to offload the global master and to get faster downloads for yourself. My closest mirror is at ftp.cz.debian.org - all I had to do in sources.list was insert a ".cz." as the third-level domain.

While fighting with the upgrade process, I concluded that it's perhaps a good habit, in terms of staying out of trouble, to only ever have a single Debian version pointed to (repo active) in the sources.list. Which is easily accomplished by commenting out all the others. Merely commenting the others out proves convenient whenever the dependency hell during the upgrade forces you to flip between nearby versions.

Whenever you modify the set of repos linked to by sources.list, even just switch distro versions up or down, you must perform an apt-get update to re-create your local index of packages available (and perhaps to rebuild their dependency map).

To install an individual package, perform apt-get install <package> .
To upgrade an individual package to latest version, perform apt-get install <package> .
To upgrade all packages to their latest versions, perform apt-get upgrade .

Stage 1 : Etch to Lenny

I obviously started by making an off-line backup of the whole system, in case I needed to roll back everything later that evening. Boot from a favourite live CD, mount all the partitions under /mnt/source, add an empty drive under /mnt/backup, cp -dpR. Piece of cake.

As you're getting ready for the upgrade, you'll probably google a bit, to learn your way about the task. The fist thing Google reveals is something like

      # update /etc/apt/sources.list to point to the new stable version
      apt-get update
      apt-get install apt
      apt-get dist-upgrade    # upgr. whole distro, handle non-trivial configuration changes
      

While this is essentially correct, you may find yourself stuck and forced to learn more :-)

In my case, the machine was originally installed as Sarge (3.0) or maybe even Woody (2.0). Two other admins have carved their signatures into the distro before my upgrade attempt. I myself have upgraded the hardware under the distro months ago - I prepared a newer machine with two spacious IDE drives, set up a MD RAID (mirror) with header v0.90 for RAID autodetect, partitioned the RAID from a live CD, installed Grub v1 by bare hand, inserted a very new 2.4 kernel compiled elsewhere, and finally I copied all the data using just "cp -dpR" or some such, from the old drive. That worked just fine.
I mean to say that the Debian install had been through quite some individual history :-) when I decided to upgrade the distro again, this time myselfs.

When I approached the machine, /etc/debian_version said 4.0. So I pointed sources.list to Lenny (5.0) and followed the sequence of commands up to apt-get dist-upgrade.
Quite to my surprise, that did nothing at all! 0 packages upgraded, all of them already at the current version. ... ***What***?

Whatever I tried, apt-get refused to install new versions of packages. On packages that were not outright wrong (non-existent package name) it would say something like "that package does exist in the database, but I have nowhere to download it from".
I discovered that my APT version was 0.4.5 or so, compiled in 2001. (apt-get --version) Looked more like Sarge or even Woody, rather than Etch. Now it's 2011, for god's sake...
But even when I pointed sources.list back to Etch (4.0), apt-get install apt would not upgrade apt to a newer version (I expected at least 0.5). Even more interestingly, apt-get was refusing to install other Etch packages, such as "aptitude", which would've been another possible workaround.
Only when I pointed sources.list back to *Sarge* (3.0) did I get some responses: aptitude got installed, but didn't help.
I was suspecting some problem with the old version of APT not able to parse the new package index data for Etch/Lenny etc. And, being unable to get a new APT via apt-get, I went hunting for a way to install new APT from source!
It took me a while to find a source tarball of Lenny-level APT 0.7.20 at http://packages.debian.org/lenny/ which seems to be a somewhat cryptic site. Then I also installed a fresh libcurl from source, because the Sarge-level libcurl wasn't good enough (APT needs it for HTTPS support, it would seem = somewhat useless, but no way to ./configure --without-curl). I did manage to port APT 0.7.20 to my Etch/Sarge hybrid - I had to clean up some modern yet marginal elements of C++ that didn't compile on the old compiler (string::clear() and operator "or" for instance), add some missing includes, make a symlink to libstdc++ and some such. Finally, I installed the APT binaries by hand, because the compilation shipwrecked on some documentation conversion issues (gettext too old).
So, after three hours of detective work, I had a new APT, and it was executable on my system. Another apt-get update, and voila:

It still didn't work!!!

I was about to call it a day, reboot and go home (it was about midnight), as I didn't get pretty much anywhere. I still googled a bit running on the last bits of my motivation, when I stumbled over some barely relevant forum posting which mentioned:
"it doesn't work even if you remove /etc/apt/preferences?"

So I did exactly that and
Bingo!!!
Suddenly apt-get started to work the way I'd expect it to.
More or less, anyway.

I pointed sources.list to Lenny (5.0) and fired up apt-get dist-upgrade.
That did replace a few packages, then got as far as the glibc.
At that point, it said something like "we are about to install a new glibc. In order to do that, you need to upgrade your kernel first, from 2.4 to 2.6, as the new Glibc has NPTL. I can tell you right here that it won't work with Lenny kernels, as those already need the newer libc. So you need to add Etch to your sources.list and start with some Etch-level 2.6 kernel".
Well I tried some Lenny-level 2.6 kernel first anyway (apt-get install linux-image-2.6, then pick a particular flavour from the list offered), and it failed exactly the way the warning had said.
So I added Etch (4.0) to sources.list, so that both Etch and Lenny were uncommented at the same time - and tried upgrading the kernel. For some reason, that didn't quite want to work. I had weird errors, dependencies failing etc. I learned to use apt-get -f install, which did help a bit / in some cases, but I couldn't get past the installation of a kernel.
Next, I removed Lenny (5.0) from sources.list - so that only Etch (4.0) packages remained in play. That cleaned up some garbage from view, and finally I got that darn 2.6 kernel to install.
From then on, things started to develop more positively.
Still at Etch-level, I over-installed APT with the proper Etch version from the repo.
Then I switched sources.list from Etch-only to Lenny-only, and now the dist-upgrade went ahead much further. I upgraded the glibc, the kernel again, APT again (back to 0.7.20, this time from a binary .DEB).

Perhaps at this point it's time to draw some conclusion, sum it up into a morale of some sorts. Let me put it this way:
When upgrading from Etch (kernel 2.4) to Lenny (kernel 2.6), start by checking that APT works the way it should. If necessary, remove /etc/apt/preferences. Next, make sure that APT is at Etch level. Next, as you'll be upgrading across the NPTL compatibility boundary, first upgrade the kernel to the latest 2.6 available for Etch. Only then, shift your sources.list to Lenny and try a dist-upgrade.
Not sure if this "always start by the kernel" is a general advice, applicable to future Debian releases. Maybe not Lenny to Squeeze. Maybe the point is to look for important milestones that break compatibility between the kernel and libc (user space).

Some minor gotchas still remained to be solved.
First there was some conflict between two packages, both providing the same documentation file (passwd and some PAM-modules), which was evaluated by APT as a no-go and made the dist-upgrade fail. I managed to solve this relatively quickly by an explicit call to dpkg -i --force-overwrite <package> . (Got the full package pathname from the error message printed by apt-get.)

Next, as soon as Perl got upped to Lenny level, apt-get (probably running some Perl scripts) started bitching over and over about some Locale settings being wrong. I first tried setting some things manually, but then I gave up. Googled a bit and found the canonical solution:

apt-get install locales    # perhaps already installed
dpkg-reconfigure locales   # and pick the locales desired
      
After that, Perl and apt-get stopped whining about Locale.

And some really small snippets to finish with:
Between Etch and Lenny, Postfix gets upgraded past v2.4. This brings some minor configuration changes (to master.cf). It seems appropriate to run "postfix upgrade-configuration" manually after the dist-upgrade.
I also lost /usr/sbin/amavisd-new somewhere along the way. Once I realized that, I just ran "apt-get install amavis" and got it back. Also the structure of Amavis config files has changed, so that I had to transmogrify the old config into new config semi-manually (copy and paste individual blocks), mostly because the direct access to clamd doesn't work (had to switch back to clamdscan). Then I realized the new config has prevented .EXE and .ZIP files from being sent, so I removed them from the blacklist.
The new Postfix build from the DEB probably knows SMTP Auth. And, Thunderbird as a client is configured by default to ask for SMTP Auth. If you don't properly configure the Auth in the Postfix, this leads to an authentication failure. There are two ways to solve this: configure SMTP Auth source in the Postfix server, or configure the Thunderbird not to ask for SMTP auth.
An upgrade from Cyrus 2.1 to 2.2 might theoretically require some manual conversion of DB formats for its various indexes, but practically I didn't need pretty much any conversion. I have a limited number of users, and the DB files were mostly plain text -> nothing to convert there. Again the process was fairly well guided.
I also got SpamAssassin derailed somehow along the way, during the upgrade... the remedy was as simple as copying back the old config (or commenting out two lines in SpamAssassin's config file, if memory serves, as the init script would LOUDLY suggest).
Squirrelmail somehow lost the ability to speak our local language. After some inspection, I found out that all that is needed is a
dpkg-reconfigure locales
followed by
apt-get install squirrelmail-locales
and check the $squirrelmail_default_language and $default_charset in /usr/share/squirrelmail/config/config.php. See also aptitude search squirrelmail .

The one thing that I've learned very early and for sure is: when apt-get dist-upgrade fails, don't worry. The packages already updated are not rolled back. The particular package that failed to install may be in a somewhat inconsistent state, but apt-get -f install should take care of that one. After you remove the problem, run apt-get dist-upgrade again and it'll start approximately from the point where it last stopped. Or sometimes, it will start elsewhere - the decision where to start seems to be somewhat random. Only if the install failure affects a key common dependency (single package), apt-get will fail repeatedly right after you start it, until you solve the problem.

Overall I was amazed at how much abuse the APT/DPKG packaging system can tolerate, without bringing the system into a coma. Just consider that you're messing with crucial system packages and even the Glibc library, the process often fails somewhere halfway there, and still the overall outcome is a responsive system, not a hint of some difficulty.

The addition of a new kernel (the old ones remain in /boot, along with their initrd images) is equally seamless. The package addition also entails automagical creation of an initrd, all you have to do is add the respective entry to your bootloader (if you have your own bootloader config, which APT knows about). My existing 2.4 kernel was monolithic, without an initrd, with MD RAID compiled in - I was a little worried if the 2.6 kernel installed via APT would reflect the root on an MD device. Quite to my surprise, it worked perfectly seamlessly. The MD drivers are modules, but were automatically (silently) added to the respective initrd upon the installation of the new kernel.
Amazing, impressive.

Stage 2 : Lenny to Squeeze

After modding /etc/apt/sources.list + "apt-get update" + "apt-get install apt" + "apt-get install aptitude", I first installed the squeeze kernel = 2.6.32-5. This entailed the infamous and somewhat pointless change of HDD device names (hda->sda) - but as I'd installed on /dev/md0,1,2,3 with kernel-space auto-detect (long-ago), I wasn't very afraid... Otherwise I'd have to mod the fstab and maybe the bootloader cfg. That initial step went just fine.

During "apt-get dist-upgrade", there were lots of "warning: missing architecture" from APT (essentially several such lines for every package). Maybe they started pouring after the libc upgrade, which came fairly early on. Or after an APT upgrade. They mentioned some ancient packages still present in /var/lib/dpkg/available.

When asked whether to switch from Bash to Dash, reluctantly I chose "yes".
Hmm: pop-before-smtp is moving to a new TCP port? That's configured in /etc/defaults/pop-before-smtp and /etc/postfix/main.cf. And besides, perhaps I should switch to SASL Auth for Postfix in the first place...

Then I faced a message saying "Unable to migrate to dependency-based boot system". Some useful info followed - essentially a list of some old packages, which kept init scripts around that were missing the newly required LSB tags. Finally, there was a line saying "To reattempt the migration process after the problems have been fixed, run 'dpkg-reconfigure sysv-rc'". Very helpful. And a link to a Debian Wiki entry on dependency-based booting. My offending packages seemd pretty harmless, but I skipped this just to save time. My small single-CPU system with slow drives likely wouldn't benefit from parallel init anyway.

Apache comes with a new default config, ho hum, no problem, let's try to keep the old one for example. Its only task is to serve SquirrelMail, anyway.
Locale broken again, I already know what to do about that one...
Again the same changes to spamassassin, amavis and Cyrus... (no, no, no).
Heh - an update to initramfs-tools, or rather a post-update script, has tried to update an old/unused lilo.conf, and was failing at that. So I just renamed lilo.conf to some other name, and the dist-upgrade finished (this was the very last package).

Ahh well. TLS crypto used by Cyrus. Several messages in /var/log/mail.err:
Fatal error: tls_start_servertls() failed
TLS server engine: No CA file specified. Client side certs may not work
unable to get private key from '/etc/ssl/private/cyrus-global.key
TLS server engine: cannot load cert/key data, may be a cert/key mismatch?
error initializing TLS
Fatal error: tls_init() failed

The one about "No CA file specified" is usually harmless, just a warning. Moreover, it can be zapped by uncommenting the following line in /etc/imapd.conf:
"tls_ca_file: /etc/ssl/certs/cyrus-imapd-ca.pem"
Except that, the filename has a typo, and results in yet another error message, saying "cannot load CA data" - this one pops up in mail.log. So the line in imapd.conf should actually read:
tls_ca_file: /etc/ssl/certs/cyrus-imapd.ca.pem
Well at least I've learned how to properly strace cyrus (I ended up just stracing "/etc/init.d/cyrus2.2 start" explicitly, as the internal hooks in /etc/imapd.conf didn't want to work).

The other messages are more serious. Effectively IMAPs and POP3s are now defunct. It turns out that the key file is okay, but Cyrus cannot read it anymore for some reason, after the upgrade. In /etc/imapd.conf you can see:
tls_key_file: /etc/ssl/private/cyrus-global.key
Looking on disk, you can see that cyrus-global.key is a symlink to ../certs/server.pem (inherited from some old Cyrus install). The trouble is, that /etc/ssl/private (the directory) has ownership root/ssl-cert - i.e. only root and the "ssl-cert" group can enter that directory. And, Cyrus is neither for some reason: neither suid root, nor a member of the ssl-cert group. Some say it is a big no-no to add the "cyrus" user to the ssl-cert group. Rather, the private key file should be put someplace where Cyrus can read it. But voila, the symlinked target directory is readable to anyone and the target file is actually owned by cyrus/mail.
So the simple remedy is: in /etc/imapd.conf, turn the original line into a comment:
#tls_key_file: /etc/ssl/private/cyrus-global.key
and make a copy of that line and redirect it straight to the symlink's target:
tls_key_file: /etc/ssl/certs/server.pem
Voila, the users can log in again.
In my case, the cert and key files are rather stale, I wasn't the one who created them, I have renumbered IP addresses in the meantime... the certs/keys no good anymore, I should generate a new self-signed certificate at the very least. Which implies further trouble: my server has three distinct IP addresses and is serving its SSL-borne services on all three of them (two public uplinks and a private subnet) using a single instance of the Cyrus master. Which is a problem for SSL (designed to run on a single IP addr). Ahh well...

I finished off by dpkg-reconfigure locales and by checking that SquirrelMail can still run in our local language.

With respect to Squirrelmail, I have noticed some glitches in the "vacation message" plugin. Firstly, Proftpd (needed for this plugin) kept failing to start (via inetd), because it doesn't recognize the following option anymore, in /etc/prodftpd/proftpd.conf:
DisplayFirstChdir .message
Just comment that line out.
Secondly, the "auto-response" messages previously stored in iso-8859-2 somehow got displayed in UTF-8 without conversion, leading to garbled national characters. Which entailed quite some fun to get sorted :-) The whole debugging procedure is perhaps worth mentioning, even if it's just a bit of newbie stalking documented. Fun to read, if nothing else :-)

Stage 3 : Squeeze to Wheezy

(... making a backup of the whole /etc/ ... good. )

Let's start with apt-get update, just for a good measure, still with the apt sources pointing to squeeze.
Uh oh: NO_PUBKEY 8B48AD6246925553

Remedy:
gpg --keyserver pgpkeys.mit.edu --recv-key 8B48AD6246925553
gpg -a --export 8B48AD6246925553 | apt-key add -
(Mind the "-" at the end of the line.)
Turns out to be some key for Wheezy. Which would get imported by apt-get update. Which however cannot work, for lack of the key. Ahh well :-)

Good. The next one up is apt-get upgrade. Again, just for a good measure.

Some message about ISC DHCPd whirred past. Ahh well.
Maybe it said that dhcpd was in fact already stopped - that would make sense, it is stopped.
Next, the post-install script for Grub asked, which device is my boot device... fine.

I'll quote this one verbatim:

  cups (1.4.4-7+squeeze2) stable-security; urgency=high

  In order to mitigate a privilege escalation from the lpadmin to root
  (CVE-2012-5519), the /etc/cups/cupsd.conf configuration file is split
  in two configuration files:

  * /etc/cups/cupsd.conf can be edited by members of the lpadmin group
    through the cups web interface;
  * /etc/cups/cups-files.conf can only be edited by root;

  Many sensitive configuration statements can now only be set in
  cups-files.conf. No statements have been moved automatically. Please
  check the respective manpages.
      

Not much has changed, apparently...
Heh, cups-lpd got unhooked from inetd.conf ? Interesting...

Other than that, there were no objections. Good!

Some authors recommend to follow up with further steps (with sources still pointing to squeeze):
apt-get dist-upgrade
dpkg --audit
dpkg --get-selections | grep hold
aptitude ...and press "g".

Top it off by a reboot, just in case.

Next, point the sources to Wheezy - edit /etc/apt/sources.list and make it look e.g. like this:

deb http://ftp.cz.debian.org/debian/ wheezy main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ wheezy main non-free contrib

deb http://ftp.cz.debian.org/debian/ wheezy-updates main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ wheezy-updates main non-free contrib

#deb http://security.debian.org/debian-security wheezy/updates main contrib non-free
#the security stuff may be offline, if you're two Debian versions behind stable
      

Basically, you need to replace any occurrence of "squeeze" with "wheezy" :-)

Let's update the list of packages, from the new sources:
apt-get update
Uh oh... some key missing again... gpg --keyserver pgpkeys.mit.edu --recv-key 7638D0442B90D010
gpg -a --export 7638D0442B90D010 | apt-key add -
Hmm. This is a key for Jessie (= Debian 8). Interesting. Always "one key ahead" seems to be missing :-)

Now for the upgrade.
First step (the easy one):
apt-get upgrade


Second step (possibly the tougher part):
apt-get dist-upgrade

Not much trouble on a lean server, eh? It's the funky special apps that sometimes cause trouble, not the bare system.

Stage 4 : Wheezy to Jessie

(... making a backup of the whole /etc/ ... good. )

Let's start with apt-get update, just for a good measure, still with the apt sources pointing to squeeze.

Good. The next one up is apt-get upgrade. Again, just for a good measure.

Further steps to remove any left-over lint (with sources still pointing to wheezy):
apt-get dist-upgrade
dpkg --audit
...hmm... some packages are missing .md5sums checksums.
aptitude install debsums
debsums -l
cd /var/cache/apt/archives
aptitude download binutils
aptitude reinstall binutils
aptitude remove libdb4.5 old, superseded, unused - be careful
aptitude remove dhcp3-client pre-squeeze version
aptitude remove dhcp3-common pre-squeeze version
dpkg --audit
...good.

dpkg --get-selections | grep hold
aptitude ...and press "g".

Next, point the sources to Jessie - edit /etc/apt/sources.list and make it look e.g. like this:

deb http://ftp.cz.debian.org/debian/ jessie main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ jessie main non-free contrib

deb http://ftp.cz.debian.org/debian/ jessie-updates main non-free contrib
deb-src http://ftp.cz.debian.org/debian/ jessie-updates main non-free contrib

deb http://security.debian.org/debian-security jessie/updates main contrib non-free
      

Basically, you need to replace any occurrence of "wheezy" with "jessie" :-)

Update the list of packages, from the new sources:
apt-get update
No surprises.

Now for the upgrade.
First step (the easy one):
apt-get upgrade


Second step (possibly the tougher part):
apt-get dist-upgrade

On one machine, during the dist-upgrade, the package containing a new udev or something has complained that my running kernel lacks some features required for the new udev (or something) and that if the dist-upgrade proceeds, it will render the machine unbootable.
I scratched my head... I did have a vanilla 3.3.0. installed on the machine, based on squeeze config - and this was already past wheezy, with Wheezy 3.2.0. installed, but not selected as a default in grub. So I tried to change the default boot profile in the debianese grub2. I noticed grub-set-default, noticed its warning to change the recommendation to set GRUB_DEFAULT=saved in /etc/default/grub (I did), and I did "grub-set-default" using the lengthy UUID string (difficult to count the entries in the generated grub.cfg, right?), and i also removed the old /boot/grub/device.map... I rebooted, and ended up with a grub rescue prompt :-(
Boot from installer CD, mount / and /boot, mount --bind /dev /proc /sys, chroot, revert all the silly changes, update-grub2, and also grub-install on the physical drives just for a good measure (grub-install objected against installing onto /dev/md0). Voila, system bootable again. Selected the Debianese 3.2.0 kernel manually, to go ahead with the dist-upgrade. Morale: use an ordinal number to select a default profile in /etc/default/grub. If you stick to the canonical naming of kernel versions in file names under /boot/, the boot profiles are ordered in descending order by kernel version (the latest gets profile No. 0 = first in the queue, top of the stack).

After this intermezzo, apt-get dist-upgrade complained about some messed up dependencies.
Lazy as I am, I just started "aptitude" and pressed "g" twice. It did mention some of the messed up dependencies (that apt-get had complained about) and corrected them.
Back to apt-get dist-upgrade - now it ran fine.

Ehh... LPD not listening on the socket. There's no /var/lib/cups/daemon/cups-lpd. What ho?
Ahh well... cups not installed. aptitude install cups. cups-lpd now back in place.
But, LPD still not listening on the socket! Openbsd-inetd installed, configured...
Heh Samba complains that cups itself does not listen on its socket! What ho?
Cups-daemon is only half-installed. aptitude install cups-daemon. Cups finally running.
But... LPD still not listening! Inetd... Inetd not running! Grr... Openbsd-inetd not really installed!
Aptitude install openbsd-inetd... Inetd finally running, LPD and TFTP finally listening again.
Try to print something... theres NOTHING on the printer! Fumbling in the cups logs... "job-name : bad UTF-8 sequence".
Reported by cups-lpd, generated within cups/ipp.c, ippValidateAttribute().
Hacking this in the source was over my head (decipher the precise call stack, hack it where it hurts, understand all the consequences) - so I followed my decade-old gut feeling and GUTTED THE WHOLE CUPS THING!
Refusing a print job on the grounds of "job name" (doesn't get printed anywhere) not being proper UTF-8, which gets triggered by any print job from Windows which has localized characters in the job name, even though the job name actually gets printed allright in the logs and displayed in a terminal, that's a sin against "be liberal in what you accept". Doesn't look like free software. After all, there's an Apple logo on top of all cups manpages... enough of this.
Back to LPR. Or rather, LPRNG. Didn't have the old recipes backed up, but the printcap generated by cups from its printers.conf was a good start, I merely corrected a thing or two and added some two further tags as a frosting, then I had to learn the hard way about how to tell lpd to listen on a TCP socket and actually accept TCP sessions from outside... (funny thing: you can see lpd listening on TCP:515 in netstat, you can see some SYN and RST packets actually exchanged with clients on port 515, but no jobs accepted and netstat -t doesn't even show any sessions established - it's down to LPD security setup, stupid!) -- and THEN my centralized network printing was finally back to normal :-)
One last touch: change a single word in smb.conf, to tell Samba to print using lprng, rather than cups. All the shared print queues resurfaced in Samba.
Another last touch, actually: finally I could remove the 10-minute cron job, re-enabling paused cups queues.
Yeah. Better than taking a shower :-)
BTW, do you know where LPRNG logs its activity? Job submissions etc? They're not in syslog or /var/log, they're under /var/spool/lpd/<queue name>/ (= among the print queue files) and there are actually *several* log files. And, the log entries only appear several seconds after the job has actually been forwarded to the printer... Two useful commands: 'lpstat all' (shows last job) and 'lpc status all' (shows actual status information you'd expect).
Btw, you may want to add :jobs_done=0: into your printcap (per queue), otherwise the nightly cron job of lprng (checkpc) will complain about 'unremoved' stale jobs in the queue. That's a feature of lprng, it keeps the last job in the directory after it has been successfully processed... also Samba may have a problem interpreting the queue status, unless you specify jobs_done=0.

Side note: after I disabled IPv6 on the box (just to make sure) using a sysctl, Exim4 started complaining that it's unable to bind to ::1 = the IPv6 loopback.
After lots of searching what helped was the following change in /etc/exim4/update-exim4.conf.conf :

#dc_local_interfaces='127.0.0.1 ; ::1'
dc_local_interfaces='127.0.0.1'
      

followed by
/etc/init.d/exim4 force-stop
update-exim4.conf
/etc/init.d/exim4 start
      

Appears to work at last, just as I was about to kick EXIM and install Postfix.
I used force-stop, after I read someplace else that a plain "stop" won't actually stop exim altogether.

Later on, I discovered further subtle changes in the behavior of Samba.

Firstly, I noticed that, on a first logon attempt of a client after the client's reboot or suspend/wakeup, Samba would kick the client (auth failed). If you tried again, Samba would accept the second attempt just fine. For a human user this hurts but is acceptable, for automation scripts (batch files that map drives) this is a showstopper.
I later found other people's reports that if they kill idmapd, the problem appears gone. I considered that a clumsy workaround (especially considering centralized LDAP-based auth) and dug deeper.
Solution:
add the following line to smb.conf:
idmap config * : range = 1000-1999999
I have yet to find exactly what it means / how the ranges are established, but it looks more plausible than killing idmapd.

Secondly, some of my users have notified me that after the upgrade, some .EXE files are no longer executable straight from a Samba share. Files that previously used to be executable.
As I also moved the files to a bigger MD RAID volume with a different FS (EXT3->EXT4), I considered some mischief in privileges correlating to the filesystem, and sure enough there were some old/odd ownerships and privileges across the directory tree, inherited through the ages of upgrades (RedHat, Fedora, Debian) but at a closer look I found out, that Samba indeed changed its behavior too: until Samba version 3.6, it would automatically flag all files as Executable (to Windows clients) - which it no longer does. No longer automatically, unless you specify the following in smb.conf:
acl allow execute always