Kasun’s Blog

Kasun Indrasiri

  • Kasun Indrasiri

  • Info

    Department of CSE
    University of Moratuwa,
    Sri Lanka

  • Archives

  • Categories

GNU Make parallelization – GNU make with –j Option

Posted by kasun04 on July 27, 2008

The Software Professionals often have deal with massive code bases which comprises of at least 10000 source file or more. If you want build the code base to binaries (or make with GNU auto tools) it would take at lease 10 hrs to build such a code base. (For me it took nearly 8 hours to build the whole source tree on a 8-core CPU server).

And many build processes run for hours with build managers commonly typing ‘make’ or some other third party build tool command and going home for the night (God sake, if the build break in between). GNU Make’s solution to this problem is parallel execution: a simple command-line option causes GNU Make to run jobs in parallel using the dependency in the Makefile to run in the correct order. For that you can use –j option to specify the number of parallel executions that you required.

-j N or –jobs N

To start GNU Make in parallel mode you can to specify either the -j or –jobs option on the command-line
The argument to the option is the maximum number of processes that GNU Make will run in parallel.

For example, typing make –j 4 will allow GNU Make to run up to 4 subprocesses in parallel. This would give a theoretical maximum speed up of 4x cutting build time by a quarter. The theoretical time is,
however, severely limited by restrictions in the Makefile.

Follow these steps before you run in to GNU make parallel build
process.

Step 1:

Verify the number of CPU cores in your build server/machine. You can use following command and count the number of CPU cores (0 – n). (say n)

less /proc/cpuinfo

Step 2:

Give the –j option with the number of CPU cores available (number
of jobs must be less than or equal to the number of CPU cores). For example you
can give,

make –j 8 <other parameters and options>

Caution:

Be careful, if you specify the number of jobs higher than
the number of CPU core, then you may have to build your code base more or less
for a weeks 🙂 owing to context switching overhead.

I hope to discuss the benefits and fit falls of this method in near future.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: