Friday, May 20, 2011

Trick: gather outbound GETs made by Nexus for a $reason

The $reason might differ a lot, I was just curious how to do this in a "lab" environment, to have a list of URLs fetched by Nexus (that were made actually to fulfill client requests). Again, this is a test, not quite usable in production environments -- unless you spice it up maybe.

All I wanted to have a list of URLs (artifacts) that my Nexus fetched during a test. I wanted to check that list, sort it, count the distinct URLs, check for dupes -- if any, etc. This is here just as reference to me in future, or maybe may help somebody else too.

How to do it:

  1. Set up a "clean" Nexus installation, by let's say unzipping the bundle somewhere.

  2. Fire it up, login as "admin" user and set logging to DEBUG level over UI -- Nexus will spit out outgoing HTTP GETs in DEBUG log level like these:
    jvm 1    | 2011-05-20 14:44:58 ... - Invoking HTTP GET method against remote location http://repo1.maven.org/maven2/...
  3. Start some client to fetch against nexus, I did this:
    cstamas@marvin test]$ mvn -s settings-1.xml clean install > b1.txt & mvn -s settings-2.xml clean install > b2.txt & mvn -s settings-3.xml clean install > b3.txt &
    ... and went for a coffee.
  4. process the logs.

Processing the logs

  1. Concat the logs into single file -- if needed. I had to, I ended up with two log files, since DEBUG made wrapper to roll the file based on size I guess.
  2. Filter the logs appropriately, I used combination of tools like grep and awk, to produce my list of URLs

Example session:

$ cp ~/worx/sonatype/nexus/nexus/nexus-distributions/nexus-oss-webapp/target/nexus-oss-webapp-1.9.2-SNAPSHOT-bundle.zip .

$ unzip nexus-oss-webapp-1.9.2-SNAPSHOT-bundle.zip

$ cd nexus-oss-webapp-1.9.2-SNAPSHOT/bin/jsw/macosx-universal-32/

$ ./nexus console

$ cd ../../../logs

$ less wrapper.log

$ less wrapper.log.1

$ cat wrapper.log.1 wrapper.log > remoteFetches.txt

$ less remoteFetches.txt

$ cat remoteFetches.txt | grep "Invoking HTTP GET method against remote location" > remoteFetches-filtered.txt

$ less remoteFetches-filtered.txt

$ awk 'BEGIN{FS=" "}{ printf "%s\n", $18}' remoteFetches-filtered.txt > remoteFetches-urls.txt

$ less remoteFetches-urls.txt

 

It gave me list like this one (unsorted, URLs are ordered as Nexus made them):

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.pom

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-plugins/18/maven-plugins-18.pom

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-plugins/18/maven-plugins-18.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/maven-parent/16/maven-parent-16.pom

http://repo1.maven.org/maven2/org/apache/maven/maven-parent/16/maven-parent-16.pom.sha1

http://repo1.maven.org/maven2/org/apache/apache/7/apache-7.pom

http://repo1.maven.org/maven2/org/apache/apache/7/apache-7.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.jar

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.jar.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.4.3/maven-resources-plugin-2.4.3.pom

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.4.3/maven-resources-plugin-2.4.3.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.4.3/maven-resources-plugin-2.4.3.jar

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.4.3/maven-resources-plugin-2.4.3.jar.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.3.2/maven-compiler-plugin-2.3.2.pom

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.3.2/maven-compiler-plugin-2.3.2.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.3.2/maven-compiler-plugin-2.3.2.jar

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-compiler-plugin/2.3.2/maven-compiler-plugin-2.3.2.jar.sha1

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.7.1/maven-surefire-plugin-2.7.1.pom

http://repo1.maven.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.7.1/maven-surefire-plugin-2.7.1.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire/2.7.1/surefire-2.7.1.pom

http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire/2.7.1/surefire-2.7.1.pom.sha1

http://repo1.maven.org/maven2/org/apache/maven/maven-parent/18/maven-parent-18.pom

http://repo1.maven.org/maven2/org/apache/maven/maven-parent/18/maven-parent-18.pom.sha1

etc...

Have fun!

No comments: