Faster boot by running init.d scripts in parallel

[Petter Reinholdtsen]

One "hidden" feature of the current Debian boot sustem, is the ability
to run the init.d scripts in parallel. This require dependency based
boot sequencing to be enabled, and the init.d script dependencies to
be complete and correct to work reliably. The feature is hidden and
undocumented, because we plan to drop the CONCURRENCY variable in the
future and make concurrent booting the default when dependency based
boot sequencing is enabled.

If you want to test this feature in testing or unstable, use this
command:

echo CONCURRENCY=makefile >> /etc/default/rcS

It will enable makefile style concurrency, and run N scripts in
parallel during boot, where N is the number of CPUs or cores on the
machine. This only work when dependency based boot sequencing. This
can be enabled using

dpkg-reconfigure insserv sysv-rc

If some services fail to work properly, the problem is most likely
because of bugs in the init.d script headers. See
<URL: http://wiki.debian.org/LSBInitScripts/DependencyBasedBoot > for
clues on how to fix it. Most scripts have correct enough dependency
information, but help is needed to find and fix the remaning few with
bugs. Please make sure to usertag reported bugs to make them show up
on
<URL: http://bugs.debian.org/cgi-bin/pkgreport.cgi?usertag=initscripts-ng-***@lists.alioth.debian.org >

If you want to improve boot speed even more, I recommend installing
the readahead-fedora packages, and boot once with the 'profile' kernel
option to adjust its readahead list to the list of files needed during
boot.

Happy hacking,

[Wouter Verhelst]

Post by Petter Reinholdtsen
If you want to test this feature in testing or unstable, use this
echo CONCURRENCY=makefile >> /etc/default/rcS
It will enable makefile style concurrency, and run N scripts in
parallel during boot, where N is the number of CPUs or cores on the
machine.

That seems suboptimal.

It occurs to me that initscripts are mostly I/O bound (since they need
to start applications that then go waiting in the background), rather
than CPU bound; as such, the number of initscripts that can be ran in
parallel is more dependent on I/O speed than on CPU speed.

It's probably best to make this N be a configurable variable (possibly
defaulting to something based on something we detect from the system
we're running on) rather than have it be something detected and not
configurable.

If it actually is configurable, but you just didn't tell us so, then
please feel free to disregard this mail ;-)

[Olivier Bonvalet]

But combined with "readahead", there is no I/O bound during init.
Most of needed files are preload.

Post by Wouter Verhelst

Post by Petter Reinholdtsen
If you want to test this feature in testing or unstable, use this
echo CONCURRENCY=makefile >> /etc/default/rcS
It will enable makefile style concurrency, and run N scripts in
parallel during boot, where N is the number of CPUs or cores on the
machine.

That seems suboptimal.
It occurs to me that initscripts are mostly I/O bound (since they need
to start applications that then go waiting in the background), rather
than CPU bound; as such, the number of initscripts that can be ran in
parallel is more dependent on I/O speed than on CPU speed.
It's probably best to make this N be a configurable variable (possibly
defaulting to something based on something we detect from the system
we're running on) rather than have it be something detected and not
configurable.
If it actually is configurable, but you just didn't tell us so, then
please feel free to disregard this mail ;-)

[Henrique de Moraes Holschuh]

Post by Olivier Bonvalet
But combined with "readahead", there is no I/O bound during init.
Most of needed files are preload.

Any initscripts that deal with devices will still be I/O bound. And the
current scheduler doesn't help (see current threads in LKML). You should
start no less than two processes per CPU, probably more, if you want the
minimum boot time.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh

[Petter Reinholdtsen]

[Wouter Verhelst]

Post by Wouter Verhelst
That seems suboptimal.

Could be. See the startpar program to see how the scripts are run in
parallel. Note that the boot is mostly CPU bound when readahead is
used, which you should use if you care about boot speed. :)

Post by Wouter Verhelst
If it actually is configurable, but you just didn't tell us so, then
please feel free to disregard this mail ;-)

The startpar program can take arguments to adjust this, but we do not
provide support for this in sysv-rc. Not sure if we want to do so or
not.

Happy hacking,

[Michael Biebl]

Post by Wouter Verhelst

Post by Petter Reinholdtsen
If you want to test this feature in testing or unstable, use this
echo CONCURRENCY=makefile >> /etc/default/rcS
It will enable makefile style concurrency, and run N scripts in
parallel during boot, where N is the number of CPUs or cores on the
machine.

That seems suboptimal.

I think Petter is not right here.
startpar in makefile mode is run with the -p 4 flag in /etc/init.d/rc.

eval "$(startpar -p 4 -t 20 -T 3 -M $1 -P $previous -R $runlevel)"

Post by Wouter Verhelst
startpar is used to run multiple run-level scripts in parallel. The degree of
parallelism on one CPU can be set with the -p option, the default is full
parallelism. An argument to all of the scripts can be provided with the
-a option.

If I read the man page correctly, this means we start up to 4 concurrent
(independent) run level scripts per CPU.



Cheers,
Michael

[Manoj Srivastava]

Post by Wouter Verhelst

Post by Petter Reinholdtsen
If you want to test this feature in testing or unstable, use this
echo CONCURRENCY=makefile >> /etc/default/rcS
It will enable makefile style concurrency, and run N scripts in
parallel during boot, where N is the number of CPUs or cores on the
machine.

That seems suboptimal.
It occurs to me that initscripts are mostly I/O bound (since they need
to start applications that then go waiting in the background), rather
than CPU bound; as such, the number of initscripts that can be ran in
parallel is more dependent on I/O speed than on CPU speed.
It's probably best to make this N be a configurable variable (possibly
defaulting to something based on something we detect from the system
we're running on) rather than have it be something detected and not
configurable.
If it actually is configurable, but you just didn't tell us so, then
please feel free to disregard this mail ;-)

I actually lost 7 seconds, according to bootchard, by setting
that.

manoj

[Michael Biebl]

Post by Manoj Srivastava
I actually lost 7 seconds, according to bootchard, by setting
that.

Would be interesting to have a before and after bootchart so this regression can
be investigated.

Michael

[Petter Reinholdtsen]

[Michael Biebl]

Post by Michael Biebl
Would be interesting to have a before and after bootchart so this
regression can be investigated.

Yes, definitely. Would also be interesting to know what kind of
hardware you got (CPU, harddrive), and if you enabled readahead or
not.

Happy hacking,

[Manoj Srivastava]

Post by Petter Reinholdtsen
[Michael Biebl]

Post by Michael Biebl
Would be interesting to have a before and after bootchart so this
regression can be investigated.

Yes, definitely. Would also be interesting to know what kind of
hardware you got (CPU, harddrive), and if you enabled readahead or
not.

OK. I don't have the older bootchartd.* around, but I can save
my current one, remove the concurrency setting, and grab the new
bootchartd.* stuff for comparison. Give me a couple of days.

manoj

[Manoj Srivastava]

Post by Petter Reinholdtsen
[Michael Biebl]

Post by Michael Biebl
Would be interesting to have a before and after bootchart so this
regression can be investigated.

Yes, definitely. Would also be interesting to know what kind of
hardware you got (CPU, harddrive), and if you enabled readahead or
not.

So, on deeper inspection, I have to recant: setting concurrency
to makefile in /etc/default/rcS does not in fact change the
timing. Which made me wonder why I thought it did, and it might be that
I removed but not purged a package, and that might have made inserv
throw a hissy fit. After purging that packages, reboots with and
without the concurrency bit show absolutely no change.

I can still put up the bootchartd data, but it is pretty boring.

manoj
--
What makes the universe so hard to comprehend is that there's nothing to
compare it with.
Manoj Srivastava <***@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C

[Petter Reinholdtsen]

[Manoj Srivastava]

So, on deeper inspection, I have to recant: setting concurrency to
makefile in /etc/default/rcS does not in fact change the
timing. Which made me wonder why I thought it did, and it might be
that I removed but not purged a package, and that might have made
inserv throw a hissy fit. After purging that packages, reboots with
and without the concurrency bit show absolutely no change.

Right. That sound more sensible. I would expect no change if the
boot is IO bound and not CPU bound, but slower boot with concurrent
booting was a mystery to me. :)

Was this with or without dependency based boot script ordering? The
concurrent booting only work with dependency based boot sequencing.

I can still put up the bootchartd data, but it is pretty boring.

Please do, I find it interesting to learn more how different Debian
systems boot to see if there are new improvements to be found. :)

Happy hacking,

[Manoj Srivastava]

Post by Petter Reinholdtsen
[Manoj Srivastava]

Post by Manoj Srivastava
I can still put up the bootchartd data, but it is pretty boring.

Please do, I find it interesting to learn more how different Debian
systems boot to see if there are new improvements to be found. :)

ok. Late, but better than never:
http://people.debian.org/~srivasta/

bootchart-unopt.png ========= Data from before I started working
on optimizing boot times
bootchart_concurrency.* ========= With CONCURRENCY=Makefile
bootchart.* ========= Normal, serial boot.

manoj
--
Pray: To ask that the laws of the universe be annulled in behalf of a
single petitioner confessedly unworthy.-- Ambrose Bierce
Manoj Srivastava <***@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C

[Petter Reinholdtsen]

[Manoj Srivastava]

Post by Manoj Srivastava
http://people.debian.org/~srivasta/

Thank you. Very interesting to see. :)

Post by Manoj Srivastava
bootchart-unopt.png ========= Data from before I started working
on optimizing boot times

Duration 2 minutes 13 seconds, and as far as I can see using the
legacy boot ordering and sequencial boot.

Post by Manoj Srivastava
bootchart_concurrency.* ========= With CONCURRENCY=Makefile

Duration 1 minute 7 seconds, and using concurrent booting and the new
dependency based boot ordering. Unable to see the start of the
slowest component from the original boot, the opennms, and that is
rather strange. Was opennms started as it should, or is there some
dependency bug in its init.d script?

Post by Manoj Srivastava
bootchart.* ========= Normal, serial boot.

Duration also 1 minute 7 seconds, and as far as I can see this is
using concurrent booting (init.d/rc is starting startpar) too. Seem
to be identical to the other boot. Why did you believe this was using
serial boot? The content of /etc/default/rcS must have had some
CONCURRENCY setting, or there must be a but in sysv-rc. :)

Happy hacking,

[Manoj Srivastava]

Hi,

Well, I have commented it out; I'll try again to reboot after
removing the line.

manoj

#
# /etc/default/rcS
#
# Default settings for the scripts in /etc/rcS.d/
#
# For information about these variables see the rcS(5) manual page.
#
# This file belongs to the "initscripts" package.

TMPTIME=1
SULOGIN=no
DELAYLOGIN=no
UTC=yes
VERBOSE=no
FSCKFIX=yes
RAMRUN=no
RAMLOCK=no
#CONCURRENCY=makefile

[Manoj Srivastava]

Post by Manoj Srivastava
Well, I have commented it out; I'll try again to reboot after
removing the line.

I rebooted, and I still get init.d/rc is starting startpar. I do
not have any line with CONCURRENCY= in /etc/default/rcS, commented or
otherwise. There is definitely a bug somewhere.

manoj
--
The 'A' is for content, the 'minus' is for not typing it. Don't ever do
this to my eyes again. -- Professor Ronald Brady, Philosophy, Ramapo
State College
Manoj Srivastava <***@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C

[Petter Reinholdtsen]

[Manoj Srivastava]

Post by Manoj Srivastava
I rebooted, and I still get init.d/rc is starting startpar. I do
not have any line with CONCURRENCY= in /etc/default/rcS, commented
or otherwise. There is definitely a bug somewhere.

Very strange. This do not happen when I test without the CONCURRENCY
value set. Is the message "Using startpar-style concurrent boot in
runlevel 2" or "Using makefile-style concurrent boot in runlevel 2"
printed during boot, to indicate that concurrent booting is enabled?
It is not printed when I test.

Anyway, probably best to move this discussion to BTS, as a bug against
sysv-rc? No idea how to reproduce, and no clue what the problem is.
Will probably need both to be able to change the behaviour. Of course
if you are happy with concurrent booting, no need to fix it. :)

Happy hacking,

[Felipe Sateler]

Post by Petter Reinholdtsen
[Manoj Srivastava]

Post by Manoj Srivastava
I rebooted, and I still get init.d/rc is starting startpar. I do
not have any line with CONCURRENCY= in /etc/default/rcS, commented
or otherwise. There is definitely a bug somewhere.

Very strange. This do not happen when I test without the CONCURRENCY
value set. Is the message "Using startpar-style concurrent boot in
runlevel 2" or "Using makefile-style concurrent boot in runlevel 2"
printed during boot, to indicate that concurrent booting is enabled?
It is not printed when I test.

Is CONCURRENCY set in /etc/init.d/rc ? I used to have it set there.

[Manoj Srivastava]

Post by Felipe Sateler

Post by Petter Reinholdtsen
[Manoj Srivastava]

Post by Manoj Srivastava
I rebooted, and I still get init.d/rc is starting startpar. I do
not have any line with CONCURRENCY= in /etc/default/rcS, commented
or otherwise. There is definitely a bug somewhere.

Very strange. This do not happen when I test without the CONCURRENCY
value set. Is the message "Using startpar-style concurrent boot in
runlevel 2" or "Using makefile-style concurrent boot in runlevel 2"
printed during boot, to indicate that concurrent booting is enabled?
It is not printed when I test.

Is CONCURRENCY set in /etc/init.d/rc ? I used to have it set there.

/etc/init.d/rc:CONCURRENCY=none

manoj

[Petter Reinholdtsen]

[Manoj Srivastava]

Post by Manoj Srivastava

Post by Felipe Sateler
Is CONCURRENCY set in /etc/init.d/rc ? I used to have it set there.

/etc/init.d/rc:CONCURRENCY=none

Could it be a bootchart problem instead, graphing the wrong file or
something? The version in unstable was broken because some Java
library changed its API. I uploaded a fixed version yesterday.

Happy hacking,

[Manoj Srivastava]

Post by Petter Reinholdtsen
[Manoj Srivastava]

Post by Manoj Srivastava

Post by Felipe Sateler
Is CONCURRENCY set in /etc/init.d/rc ? I used to have it set there.

/etc/init.d/rc:CONCURRENCY=none

Could it be a bootchart problem instead, graphing the wrong file or
something? The version in unstable was broken because some Java
library changed its API. I uploaded a fixed version yesterday.

I have now installed the new bootchart, and I'll see if it makes
any difference to the chart (the next time I reboot -- perhaps over
the weekend, this is killing my uptimes stats)

manoj
--
Just close your eyes, tap your heels together three times, and think to
yourself, `There's no place like home.' -- Glynda the Good
Manoj Srivastava <***@debian.org> <http://www.debian.org/~srivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C

[lkcl]

Post by Petter Reinholdtsen
One "hidden" feature of the current Debian boot sustem, is the ability
to run the init.d scripts in parallel.

some years back, richard lightman wrote depinit. it's a complete
replacement for sysvinit, and it's a parallel initialisation system.

unlike sysvinit, it caught _all_ signals on applications.

i installed it several times, and it showed that startup time could be
reduced from 2 minutes (800mhz PIII) to 25 seconds, and it showed that
shutdown time can be reduced from 1-2 minutes to under 4 seconds.

the reason for the fast shutdown time is that a) shutting down services
typically takes a lot less time than starting them b) depinit would send
increasingly drastic signals to kill a service.

other nice features include:

* monitoring and chasing with extreme aggression anything that looks like
it's out-of-control. applications such as rootkits, which spawn child
processes immediately on startup and on the death of the parent, are good
candidates and were the driving force behind the decision

* script / service chaining (applying the unix pipe philosophy, even to
services). one "service" can output its stdout and/or stderr to _another_
service. richard utilised this e.g. on sshd and other services requiring
authentication. by chaining the output from sshd into another script, he
could monitor attacks against a server _live_ rather than "ohh let's run a
cron job every minute and watch the sshd logs or something oh whoops, too
late". so, the monitoring script could observe three login failures and
*instantly* add a firewall rule.

* automatic re-spawning of services AND termination of dependent services if
re-spawning fails. this is a really important security feature. if the
firewall doesn't come up, or any other particularly critical service (such
as heartbeat monitoring service), do you REALLY want the dependent services
to come up? automatic re-spawning basically does away with the [stupid]
mysql monitoring script, which cannot do a proper job anyway, simply because
the required signals cannot be properly caught [remember: depinit catches
EVERYTHING].

that's not all - it's just everything i can remember.

when i installed depinit, i had to spend considerable time customising
udevscripts, in order to get some speed. the kinds of systems i was
installing it on were waiting for 30 seconds on /sbin/udevsettle, due to the
absolutely ridiculous numbers of pseudo ttys created by the debian linux
kernel. 768 pseudo-ttys!!! come _on_. so, i broke it down into stages.
the first udevsettle "service" created /dev/ttyNN and /dev/pty/NN and the
second one would create and wait for /dev/ttyNNN and /dev/pty/NNN. critical
services would then depend on the first udev service (such as filesystem
mounts, networking, sshd etc) and non-critical services on the second. this
trick was what got most of the boot time down.

[btw it has to be said that udev and the udev scripts are an absolute dog's
dinner. the amount of cpu time wasted by spawning /bin/sh 12 levels deep,
almost a thousand times, is .. um... considerable.]

also one other observation was that many of the scripts these days do _lots_
of "sleep" cycles. putting all of these in serial is a _total_ waste of
time.

also, richard observed that sysvinit [non-parallel] was created when
context-switching was insanely expensive. CPUs now - even low-end ARMs -
are pretty damn good at context switching. it's therefore absolutely
pointless to stack initialisation scripts up in serial, even on a single CPU
system, in the pretense of saving cycles, when you can throw out a hundred
processes in parallel on most CPUs of the past decade and they'll not even
break a sweat.

anyway - i wanted to mention this because depinit remains absolutely streets
ahead of anything else i've encountered,

l.

[Tino Keitel]

On Sun, Sep 20, 2009 at 13:52:59 -0700, lkcl wrote:

[...]

Post by lkcl
some years back, richard lightman wrote depinit. it's a complete
replacement for sysvinit, and it's a parallel initialisation system.
unlike sysvinit, it caught _all_ signals on applications.
i installed it several times, and it showed that startup time could be
reduced from 2 minutes (800mhz PIII) to 25 seconds, and it showed that
shutdown time can be reduced from 1-2 minutes to under 4 seconds.
the reason for the fast shutdown time is that a) shutting down services
typically takes a lot less time than starting them b) depinit would send
increasingly drastic signals to kill a service.

[...]

Post by lkcl
* script / service chaining (applying the unix pipe philosophy, even to
services). one "service" can output its stdout and/or stderr to _another_
service. richard utilised this e.g. on sshd and other services requiring
authentication. by chaining the output from sshd into another script, he
could monitor attacks against a server _live_ rather than "ohh let's run a
cron job every minute and watch the sshd logs or something oh whoops, too
late". so, the monitoring script could observe three login failures and
*instantly* add a firewall rule.
* automatic re-spawning of services AND termination of dependent services if
re-spawning fails. this is a really important security feature. if the
firewall doesn't come up, or any other particularly critical service (such
as heartbeat monitoring service), do you REALLY want the dependent services
to come up? automatic re-spawning basically does away with the [stupid]
mysql monitoring script, which cannot do a proper job anyway, simply because
the required signals cannot be properly caught [remember: depinit catches
EVERYTHING].

Maybe you are interested in the runit init system, as depinit sounds
similar. It doesn't directly support dependencies, but it runs a
service through a pipe and not as a daemon in the background, so that
supervision and direct submission of signals is possible.

Regards,
Tino