[Python-projects] pyLint: Speeding up pyLint

[Sylvain Thénault]

On 29 mai 00:47, Julian Scheid wrote:
> Hello,

Hi,
 
> The only problem with this is that pyLint is comparatively slow. In my
> project, the unit tests take around 5s to run, pyLint takes 20s.

you're using the latest version with py 2.5? The _ast based  version of
astng offers some nice speedup.
 
> I understand that pyLint has to do extensive analysis of the source
> code, type inference and checking hundreds of rules so this is fair
> enough. But I am wondering if there is anything I could do - both as a
> pyLint user and as a Python developer - to make pyLint run faster.

much probably :D
 
> I tried playing with the --cache-size parameter but that doesn't seem
> to help. So I wonder if pyLint code could be improved to speed things
> up. For instance, I suspect that pyLint is mostly CPU-bound but runs
> single-threaded, i.e. doesn't make use of multiple CPU cores - is this
> correct?

it is.
 
> Also, I assume that it will parse all third-party libraries used in my
> project (there are quite a few) every time I run it. These libraries
> are only updated infrequently, would it be possible to cache any
> parsing/type inference/etc. pyLint is doing for these libraries?

IMO, caching type inference is really hard since it's context sensitive
(and will be more as it's improved). Caching parsed ast or having an ast
server providing this service has been an idea for a while but we didn't
found time to implement it yet (and probably won't be able to do in a
near future). But first we should check the time taken by parsing in the
analysis.

> My understanding of the pyLint codebase is currently zero, but I'd
> like to offer my help if you don't have the time to look into
> improving pyLint performance.

any help is welcome, thanks. Don't hesitate to ask any question on this 
mailing list if you want some clarification, advice...

-- 
Sylvain Thénault                               LOGILAB, Paris (France)
Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
Développement logiciel sur mesure:       http://www.logilab.fr/services
CubicWeb, the semantic web framework:    http://www.cubicweb.org