Hello,
I want to make s/k modules with -j 3 . Thus specifying 3 parallel jobs (Hehe I have one of those fancy SMP machines). The problem is this:
make -j 3
.... make[1]: Entering directory `/home/marius/dev/sip-router/modules/avpops' make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. ....
Problem description is here http://lists.samba.org/archive/distcc/2004q1/002160.html(I also checked the gnu make manual and this mail is a rather complete copy-paste of the section in question)
A description of the jobserver implementation in here http://make.paulandlesley.org/jobserver.html.
The code in the Makefile which causes this is :
.PHONY: $(1) $(1): modules.lst @for r in $($(1)) "" ; do \ if [ -n "$$$$r" -a -r "$$$$r/Makefile" ]; then \ $(call oecho, "" ;) \ $(call oecho, "" ;) \ if $(MAKE) -C $$$$r $$(mk_params) || [ ${err_fail} != 1 ] ; then \ :; \ else \ exit 1; \ fi ; \ fi ; \ done; true
Because I thought the problem might be with the for construct(make launches a new shell) I have replaced this with
.PHONY: $(1) $(1): modules.lst @$(foreach r,$($(1)),$(call module_make,$(r),$(mk_params)))
And in Makefile.rules I've added
module_make= if [ -n "$(1)" -a -r "$(1)/Makefile" ]; then \ $(call oecho, "" ;) \ $(call oecho, "" ;) \ ( $$(MAKE) -C $(1) $(2) || [ ${err_fail} != 1 ] ) || exit 1; \ fi ; \
This fixed the warning and multiple make jobs work fine now with make modules.
So. Do I push?!
Marius
On Wednesday 03 February 2010, marius zbihlei wrote:
[..] This fixed the warning and multiple make jobs work fine now with make modules.
So. Do I push?!
Hi marius,
running multiple jobs in parallel would be indeed a nice speedup for the compilation on todays machines. So from my side its ok to do this change, if nobody other know about some potential bad side effects?
Cheers,
Henning
On Feb 03, 2010 at 11:49, marius zbihlei marius.zbihlei@1and1.ro wrote:
Hello,
I want to make s/k modules with -j 3 . Thus specifying 3 parallel jobs (Hehe I have one of those fancy SMP machines). The problem is this:
make -j 3
.... make[1]: Entering directory `/home/marius/dev/sip-router/modules/avpops' make[1]: warning: jobserver unavailable: using -j1. Add `+' to parent make rule. ....
Problem description is here http://lists.samba.org/archive/distcc/2004q1/002160.html(I also checked the gnu make manual and this mail is a rather complete copy-paste of the section in question)
A description of the jobserver implementation in here http://make.paulandlesley.org/jobserver.html.
The code in the Makefile which causes this is :
.PHONY: $(1) $(1): modules.lst @for r in $($(1)) "" ; do \ if [ -n "$$$$r" -a -r "$$$$r/Makefile" ]; then \ $(call oecho, "" ;) \ $(call oecho, "" ;) \ if $(MAKE) -C $$$$r $$(mk_params) || [ ${err_fail} != 1 ] ; then \ :; \ else \ exit 1; \ fi ; \ fi ; \ done; true
Because I thought the problem might be with the for construct(make launches a new shell) I have replaced this with
.PHONY: $(1) $(1): modules.lst @$(foreach r,$($(1)),$(call module_make,$(r),$(mk_params)))
And in Makefile.rules I've added
module_make= if [ -n "$(1)" -a -r "$(1)/Makefile" ]; then \ $(call oecho, "" ;) \ $(call oecho, "" ;) \ ( $$(MAKE) -C $(1) $(2) || [ ${err_fail} != 1 ] ) || exit 1; \ fi ; \
This fixed the warning and multiple make jobs work fine now with make modules.
So. Do I push?!
I like it, but the problem is that it might be slower for -j1. This way you launch one different shell for each module (in fact 2 shells, the () arround $$(MAKE)... launch another subshell). The old way launched everything into the same subshell. We could try 2 different versions, one for no -j or -j1 and one for -jX (looking at the make command line to decide which one to use).
Andrei
I like it, but the problem is that it might be slower for -j1. This way you launch one different shell for each module (in fact 2 shells, the () arround $$(MAKE)... launch another subshell). The old way launched everything into the same subshell. We could try 2 different versions, one for no -j or -j1 and one for -jX (looking at the make command line to decide which one to use).
Andrei
Hello
I have done a strace -f on the make command and for me it looks like a remove a shell invocation. Because in the original code, the @for launch another shell, with the @foreach which is a builtin just expands the text when the Makefile is processed.
[pid 31050] execve("/bin/sh", ["/bin/sh", "-c", "for r in modules/avpops modules"...], [/* 119 vars */]) = 0
If you refere to this line ( $$(MAKE) -C $(1) $(2) || [ ${err_fail} != 1 ] ) || exit 1; \ This is just a shorthand "if" . I could let the "if" as in the original script. (works I've tested it)
Have I missed something ?!
BTW This -j 3 problem also occurs with make modules-doc and make modules-readme.
Cheers Marius
-
Andrei Pelinescu-Onciul -
Henning Westerholt -
marius zbihlei