Saturday, 17 September 2011

Maven Download time Improvement

While working a bit on wagon (the api used in Apache Maven to download/upload artifacts), I wanted to reduce the http(s) connection creation number (see WAGON-348).
The current embedded http wagon in Maven Core is the lightweight one based on standard java libraries, this means http(s) connection are created for each download requests. As you know this socket creation can be time and resources consuming.
So as we are working on wagon 2.0 version, I have added a connection pooling mechanism in the wagon http (which is now based on Apache Http Client).
To prevent some classloading issues, the wagon jar to used is a shaded one.
You can test that now (with a maven 3.x version) with adding the jar in $M2_HOME/lib/ext/ :

wget -O wagon-http-2.0-SNAPSHOT-shaded.jar "https://repository.apache.org/content/groups/snapshots-group/org/apache/maven/wagon/wagon-http/2.0-SNAPSHOT/wagon-http-2.0-20110917.172345-31-shaded.jar"
&& cp wagon-http-2.0-SNAPSHOT-shaded.jar $M2_HOME/lib/ext/


Test build available here : http://people.apache.org/~olamy/maven/.

So a little improvement, don't be afraid you still will have time to drink one or two coffee when building a maven project :-).
Note download time will be improved if you use more than one repositories and especially https repositories.

Vote here if you want to have this in next maven core release : http://jira.codehaus.org/browse/MNG-5175.

And all feedbacks will be appreciate :-) (even if it's an issue :-) )

Have Fun !
--
Olivier

Apache Maven, Maven, Apache are trademarks of The Apache Software Foundation.

7 comments:

Paulorcf said...

Nice I'll try!

Simo said...

Salut Olivier!
I don't know if it could help, but AFAIK the JFarcand's AsyncHttpClient is faster than the ASF one, if I were you I'd give a try to see if download time could be reduced even more :)

OT: please provide me some doc to understand Maven internals, I'd like to start contributing :)

olamy said...

@Simo this already exists.
But using ASF dogfood is nice !

olamy said...

so committed.
Get a SNAPSHOT build from here : https://builds.apache.org/job/maven-3.0.x/lastBuild/org.apache.maven$apache-maven/

vrm said...

@Simo personally I prefer the hc.a.org API over the nings http client :) since the feature are the same (peristent connection, NIO & pooling I'm pretty sure the download speed will be the sames.

olamy said...

@Simo have a look here: http://wiki.apache.org/HttpComponents/HttpClient3vsHttpClient4vsHttpCore

Olivier said...

La version 2.0 étant sortie, le script devient:
wget -O wagon-http-2.0-shaded.jar "https://repository.apache.org/service/local/artifact/maven/redirect?r=releases&g=org.apache.maven.wagon&a=wagon-http&v=2.0&e=jar&c=shaded" && cp wagon-http-2.0-shaded.jar $M2_HOME/lib/ext