Re: [PATCH] coccinelle: add pycocci wrapper for multithreaded support

[Luis R. Rodriguez]

On Thu, Apr 10, 2014 at 07:51:29PM +0200, Johannes Berg wrote:
> On Thu, 2014-04-10 at 10:48 -0700, Luis R. Rodriguez wrote:
> 
> > You just pass it a cocci file, a target dir, and in git environments
> > you always want --in-place enabled. Experiments and profiling random
> > cocci files with the Linux kernel show that using just using number of
> > CPUs doesn't scale well given that lots of buckets of files don't require
> > work, as such this uses 10 * number of CPUs for its number of threads.
> > For work that define more general ruler 3 * number of CPUs works better,
> > but for smaller cocci files 3 * number of CPUs performs best right now.
> > To experiment more with what's going on with the multithreading one can enable
> > htop while kicking off a cocci task on the kernel, we want to keep
> > these CPUs busy as much as possible. 
> 
> That's not really a good benchmark, you want to actually check how
> quickly it finishes ... If you have some IO issues then just keeping the
> CPUs busy trying to do IO won't help at all.

I checked the profile results, the reason the jobs finish is some threads
had no work or little work. Hence why I increased the number of threads,
depending on the context (long or short cocci expected, in backports
at least, the long being all cocci files in one, the short being --test-cocci
flag to gentree.py). This wrapper uses the short assumption with 10 * num_cpus

> > Since its just a helper I toss it into the python directory but don't
> > install it. Hope is that we can evolve it there instead of carrying this
> > helper within backports.
> 
> If there's a plan to make coccinelle itself multi-threaded, what's the
> point?

To be clear, Coccinelle *has* a form of multithreaded support but requires manual
spawning of jobs with references to the max count and also the number thread
that this new process you are spawning belongs to. There's plans to consider
reworking things to handle all this internally but as I discussed with Julia
the changes required would require some structural changes, and as such we
need to live with this for a bit longer. I need to use Coccinelle daily now,
so figured I'd punt this out there in case others might make use of it.

  Luis

Attachment: pgpDzRHJJtY6T.pgp
Description: PGP signature