[Python-projects] pyLint: Speeding up pyLint

[Julian Scheid]

Thank you, Juergen and Nicolas, for your replies.

2009/5/29 Nicolas Chauvat <nicolas.chauvat at logilab.fr>:
> Any help in getting pylint to run faster will be greatly
> appreciated. The usual method applies: profile, work on bootle-necks,
> loop.
>
> Running pylint with the --profile=y option will get you some
> statistics to work on.

Yes, I've already had a go at this. I'll see if there are any
low-hanging fruits.

I've also patched pylint today to use the multiprocessing module in
order to parallelize the checker runs. That turned out to be trivial
and while it doesn't provide a linear speed-up (as is to be expected),
it does improve performance significantly on multicore machines. I'll
tidy up the code and send over a patch soon.

Here's another idea that occurred to me. If checkers could be
categorized into those that only work locally on a single file (e.g.
check number of statements per function) and those that need
information from other files (e.g. method not overridden) then for
files that haven't changed between pylint runs (usually the majority
of files) only the latter would have to be run. Unfortunately, from a
quick look at the checkers it looks like local/global checks are often
intermingled in the same source file. Still, this could be one
approach to pursue. Thoughts?

>> The usual answer to this is using continuous integration,
>> e.g. Hudson. Besides making the manual steps involved redundant, it
>> also offers other reports via its web interface and emails interested
>> parties about problems.
>>
>> Python projects have automatic api docs, test & pylint reports, svn
>> change history & last build artifact here. Also building in a
>> non-developer environment means catching some kind of errors, too
>> (dependency ones, for example).
>
> I agree with Jurgen that continuous testing is a good option. At
> Logilab, we do it with http://www.logilab.org/project/apycot
>
> The new apycot version is integrated in a web framework
> <http://www.cubicweb.org/> and you can see it working at the bottom of
> the page http://www.logilab.org/project/logilab-common or on
> http://www.logilab.org/projectapycotconfig/logilab-common
>
> The cubicweb-apycot UI clearly needs a bit of work, bit it is
> functional. We also use apycot to automate the generation of debian
> packages and the updating of our debian repositories, in case that
> would of any interest to you.

Thanks again for these pointers. I am acquainted with CI but I wasn't
aware of Hudson and the pylint integration via the Violations plug-in
yet. I'm playing with it at the moment and it works very well. I'll
check out apycot next week.

Regarding Hudson, If anyone else reading this is new to it, I found
this nice overview of how it all fits together:
http://redsolo.blogspot.com/2007/11/hudson-embraces-python.html

Thanks,

Julian