Category: Java

Debugging Maven Tests by Connecting an IDE to the Maven JVM

Posted on by rchapin

In some instances you cannot reproduce a failure or condition running a test in an IDE that manifests itself when you run it on your build server or via maven on the command line.

In that case, it is very helpful to be able to remotely attach your IDE to the running maven process and then step through the code.

To do so you will need to:

Execute maven on  the command line as follows (adding any additional -D args as required by your project):

mvn -Dmaven.surefire.debug test -pl module-in-question

This will run the maven automatically pausing the JVM awaiting for a remote debugger to connect to port 5005.  If you want to have it listen on a different port you can pass it in as follows:

mvn -Dmaven.surefire.debug="-Xdebug  -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8081 -Xnoagent -Djava.compiler=NONE" test -pl module-in-question

Create a debugging run profile in either Eclipse or IntelliJ or your favorite IDE configured to connect to a JVM listening on the specified port.

Then once you have run maven on the command line, simply execute the run configuration in your IDE and debug your application as usual.

If need be, you can run the maven JVM on the cli such that maven does not fork the tests as follows:

mvnDebug -DforkCount=0 test

Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can’t attach to the process [SOLVED]

Posted on by rchapin

If you are attempting to use jmap or another Java memory analysis tool to connect to a running JVM to generate a heap dump, even when running jmap as the same user as that of the running process, and encounter the following error:

Attaching to process ID 2712, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process

Following is the (likely) solution to your problem.

It is likely that the ptrace_scope setting for your system is set to a restrictive mode which will not allow another process access to the memory space of the other process.  See this link for more details.

The fix is to update the setting in /proc/sys/kernel/yama/ptrace_scope.

First, verify the current setting:

 cat /proc/sys/kernel/yama/ptrace_scope

1, indicates a restricted setting.  To update it so that another process with the same UID can attach to that processes memory space do the following (as root)

echo 0 | tee /proc/sys/kernel/yama/ptrace_scope

Now, checking that value should return 0

cat /proc/sys/kernel/yama/ptrace_scope

At this point you should now be able to do a heapdump with jmap on your process with a command similar to the following:

jmap -J-d64 -heap 2712

Setting the Compiler Version for Maven from the Command Line

Posted on by rchapin

By default maven sets the compiler version for you.  Of course, you can always set it in the pom, but there are cases where you cannot modify the pom, and/or you might want to test compilation and tests with different versions of java.

Following are the specific arguments to pass the compiler version to maven from the command line:

mvn clean install -Dmaven.compiler.source=1.7 -Dmaven.compiler.target=1.7

How To Publish Artifacts to the Maven Central Repository

Posted on by rchapin

I have just finished releasing my first project to Maven Central Repository and wanted to capture my notes for the setup of the project and all of the steps required.

Resources

Create account on OSSRH

You will need an account to the sonatype JIRA for OSSRH.  From there, you can request the creation of a new project. See http://central.sonatype.org/pages/ossrh-guide.html for details.

Setup of the Project/pom and Pre-requisites:

PGP keys

http://central.sonatype.org/pages/working-with-pgp-signatures.html

Create a set of PGP keys

gpg2 --gen-key

List the keys

gpg2 --list-keys
/usr/local2/home/rchapin/.gnupg/pubring.gpg
-------------------------------------------
pub   4096R/E5170CE8 2015-03-26 [expires: 2016-03-25]
uid                  Ryan Chapin <rchapin@nbinteractive.com>
sub   4096R/8DAF9AD6 2015-03-26 [expires: 2016-03-25]

The id for the key created is ‘E5170CE8’

Distribute the public key

gpg2 --keyserver hkp://pool.sks-keyservers.net --send-keys E5170CE8

Additions to be added to the pom

Add the <license> tag to your pom. Following are some links regarding available licenses and an example of it’s use in the pom.

I typically use the BSD 3 License, and added the following in the pom:

<project>
  ...
  <licenses>
    <license>
      <name>The BSD 3-Clause License</name>
      <url>http://opensource.org/licenses/BSD-3-Clause</url>
      <distribution>repo</distribution>
    </license>
  </licenses>
  ...
</project>

...

Add the <developers> tag to your pom and add a <developer> entry for yourself:

<project>
  ...
  <developers>
    <developer>
      <id>rchapin</id>
      <name>Ryan Chapin</name>
      <email>rchapin@nbinteractive.com</email>
      <url>http://www.ryanchapin.com</url>
      <roles>
        <role>architect</role>
        <role>developer</role>
      </roles>
      <timezone>America/New_York</timezone>
      <properties>
        <picUrl>http://www.gravatar.com/516f2158d74d134faa9649e9180ef782</picUrl>
      </properties>
    </developer>
  </developers>
  ...
</project>

 Add the <scm> tag to your pom with the details for your repository:

<project>
  ...
  <scm>
    <connection>scm:git:git@github.com:rchapin/hash-generator.git</connection>
    <developerConnection>scm:git:git@github.com:rchapin/hash-generator.git</developerConnection>
    <url>git@github.com:rchapin/hash-generator.git</url>
    <tag>HEAD</tag>
  </scm>
  ...
</project>

Distribution Management and Authentication:  Configure the pom to enable maven to deploy to OSSRH Nexus server with the Nexus Staging Maven plugin.  See below for the release profile configuration which will contain the configs for the nexus-staging-maven-plugin.

<project>
  ...
  <distributionManagement>
    <snapshotRepository>
    <id>ossrh</id>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    </snapshotRepository>
  </distributionManagement>
  ...
</project>

Create a profile to encapsulate the creation of the javadoc and source jar as well as the pgp signing of the artifacts.  In my case, I set the profile to <activeByDefault>true</activeByDefault> to ease release builds and simply invoke a build as such during development of the project to turn off the release profile

mvn package -P\!release

Also, configure the nexus-staging-maven-plugin setting <autoReleaseAfterClose>false</autoReleaseAfterClose> to enable manual inspection of the staging repository BEFORE it is released to central.  To deploy to OSSRH and release to the Central Repository in one step, set autoReleaseAfterClose to true.

<profiles>
    <profile>
      <id>release</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <build>
        <plugins>

          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-javadoc-plugin</artifactId>
            <version>${maven.javadoc.plugin.version}</version>
            <executions>
              <execution>
                <id>attach-javadocs</id>
                <goals>
                  <goal>jar</goal>
                </goals>
              </execution>
            </executions>
          </plugin>

          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-source-plugin</artifactId>
            <version>${maven.source.plugin.version}</version>
            <executions>
              <execution>
                <id>attach-sources</id>
                <goals>
                  <goal>jar</goal>
                </goals>
              </execution>
            </executions>
          </plugin>

          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-gpg-plugin</artifactId>
            <version>1.6</version>
            <executions>
              <execution>
              <id>sign-artifacts</id>
              <phase>verify</phase>
              <goals>
                <goal>sign</goal>
              </goals>
              </execution>
            </executions>
          </plugin>

          <plugin>
            <groupId>org.sonatype.plugins</groupId>
            <artifactId>nexus-staging-maven-plugin</artifactId>
            <version>1.6.5</version>
            <extensions>true</extensions>
            <configuration>
              <serverId>ossrh</serverId>
              <nexusUrl>https://oss.sonatype.org/</nexusUrl>
              <autoReleaseAfterClose>false</autoReleaseAfterClose>
            </configuration>
          </plugin>

        </plugins>
      </build>
    </profile>

</profiles>

Add the maven-release-plugin to the <build> section.  This should include disabling the ‘release’ profile that we added/describe above, and then specify the deploy goal together with the activation of the ‘release’ profile for the deploy goal.

<project>
  ...
  <build>
    ...
    <plugins>
      ...
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-release-plugin</artifactId>
        <version>2.5.1</version>
        <configuration>
          <autoVersionSubmodules>true</autoVersionSubmodules>
          <useReleaseProfile>false</useReleaseProfile>
          <releaseProfiles>release</releaseProfiles>
          <goals>deploy</goals>
        </configuration>
      </plugin>
      ...
    </plugins>
    ...
  </build>
  ...
</project>

Adding credentials for both the distributionManagement and the pgp signing need to be added to the ~/.m2/settings.xml. One entry for your OSSRH login, and another entry for your PGP signing passphrase

OSSRH Login

<setting>
  ...
  <servers>
    <server>
      <id>ossrh</id>
      <username>your-uid</username>
      <password>your-passwd</password>
    </server>
  </servers>
  ...
</settings>

PGP Signing Passphrase

<setting>
  ...
  <profiles>
    <profile>
      <id>ossrh</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <properties>
        <gpg.executable>gpg2</gpg.executable>
        <gpg.passphrase>your-passphrase</gpg.passphrase>
      </properties>
    </profile>
  </profiles>
  ...
</settings>

Performing a SNAPSHOT Deployment

As long as your current pom version still ends in ‘SNAPSHOT’ you can deploy a snapshot version to the OSSRH

mvn clean deploy

To use the SNAPSHOT version in a project, users will need to add the snapshot repo to their Nexus, settings.xml or pom.xml. 

<project>
  ...
  <repositories>
    <repository>
      <id>ossrh-snapshots</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
  </repositories>
  ...
</project>

Release and Deployment

First make sure that all of the code is pushed to the remote repo and that everything is merged into your main branch.  Then make sure that you are ON the main branch on the local machine.

Prepare Release

To do a dry run and insure that pom will be bumped as expected and to check that everything is in order. This will NOT check-in or tag anyting in the scm repository.

mvn release:prepare -DdryRun=true

Check the output and then do the following before running the actual prepare command

mvn release:clean

Execute the release:prepare

mvn release:prepare

 If you encounter any errors

mvn release:prepare -Dresume=false

Alternatively, you can use

mvn release:clean release:prepare

See http://maven.apache.org/maven-release/maven-release-plugin/examples/prepare-release.html for details.

Once the release has been performed the scm repository should be updated with the new tag and the pom should be bumped to the next version for the next iteration of development.

Perform Release

Once the release has been prepared and a tag created and the scm repository has been update, you can deploy to OSSRH. The following will deploy to a staging repository.

mvn release:perform

Inspecting Staging Repo and Releasing to Central

Login to OSSRH via https://oss.sonatype.org/.  Uid and passwd are the same for issues.sonatype.org. See http://central.sonatype.org/pages/releasing-the-deployment.html for details

Releasing to Central

Once you have inspected the staging repo and are ready to release the deployment to Central do the following

cd target/checkout
mvn nexus-staging:release

If this is your first release to central for this project, don’t forget to go back to your project creation request ticket and add a comment that you have promoted your first release so that your promotion can be verified and your artifacts synced with Central.  This step only needs to be done the first time you promote to central with a new OSSRH project.

Debugging MapReduce MRv2 Code in Eclipse

Posted on by rchapin

Following is how to set-up your environment to be able to set breakpoints, step-through, and debug your MapReduce code in Eclipse.

All of the this was done on a machine running Linux, but should work just fine for any *nix machine, and perhaps Windows running Cygwin (assuming that you can get Hadoop and its naitive libraries compiled under Windows).

This also assumes that you are building your project with maven.

Install a pseudo-distributed hadooop cluster on your development box.  (Yes, this calls for another article on exactly how to do that which I will do shortly and link to from here).

Add the following environment variables to .bash_profile to ensure that they will be applied to any login shells (make sure to check the location of the directories for your installed hadoop distribution):

export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native
export HADOOP_HOME=/usr/lib/hadoop

Make sure to include the following dependencies in your pom:

  • hadoop-mapreduce-client-core
  • hadoop-common
  • hadoop-hdfs
  • hadoop-client

After you import your maven project into Eclipse update the Build Path to include the correct path to the Native library shared objects:

  1. Right-click on your project and select ‘Build Path -> Configure Build Path:
  2. Click on ‘Libraries’ tab:
  3. Click the drop-down arrow for the ‘Maven Dependencies’
  4. Click on the drop-down arrow on the ‘hadoop-common’.jar
  5. Select the ‘Native library location’ entry, and click ‘Edit’
  6. Browse to the path of the native directory, in my case it was /usr/lib/hadoop/lib/native.
  7. Click ‘OK’
  8. Click ‘OK’ to close the build path dialogue

Create a run configuration for the Main class in your project:

Make sure that you do not add the /etc/hadoop/conf* dir to the class path.

Add any commandline arguments for input and output directories to the ‘Program arguments’ section of the run configuration, that points to your LOCAL file system and not HDFS.

Afterwhich, you should be able to run your M/R code and debug it through Eclipse.

Unit Testing Private Static Methods With Primitive Array Arguments

Posted on by rchapin

When writing unit tests to cover your entire program you will undoubtedly come across the need to test private methods.  There are arguments that these methods should be tested via integration tests, but there are sometimes when it makes more sense to test all of the permutations in a unit test. This can be achieved using reflection in Java JUnit tests.

What is a little tricky, and was not completely obvious, was how to use reflection to test a private static method that accepted an array of primitives.  Following is a simple example, with explainations in the comments.

Note, this code will not run as it, you would need to transpose it into a valid JUnit test class to bypass the IllegalAccessException.

Class with private method that you want to test:

public class ByteCounter {
    private static int countByteValue(byte[] arr) {
        int retVal = 0;
        for (int i = 0; i < arr.length; i++) {
            retVal += (int) arr[i];
        }
        return retVal;
    }
}

Unit test code:

import java.lang.reflect.Method;

public class StaticArrayReflectionTest {

    public static void main(String[] args) throws Exception {
        byte[] arr = new byte[] { 1, 2, 3, 4 };

        // Get a Class instance of the class to be tested
        Class<ByteCounter> byteCounterClazz = ByteCounter.class;

        /*
         * Get a Method instance for the method to be tested. The part to take
         * note of is how to get a Class instance of a array of primitives.
         */
        Method countByteValueMethod = byteCounterClazz.getDeclaredMethod("countByteValue",
            new Class[] { byte[].class });

        /*
         *
         * Invoke the Method instance passing in the arr argument.
         *
         * Take note that the first argument of invoke is 'null' as there is no object
         * instance on which to invoke the method since the method in question is
         * static. Also notice how the byte array is passed in, wrapped in an Object[]
         */
        countByteValueMethod.invoke(null, new Object[] { arr });
    }
}

Configuring Eclipse to Replace Tabs with Spaces for Indentation

Posted on by rchapin

Following are two basic settings (I believe that there are other language specific, C++ for instance, settings as well).

 For Java:

Window->Preferences->Java->Code Style->Formatter->
Click on ‘New’ to create a new profile and select the profile that you want to copy
Then click ‘Edit’ and select ‘Spaces Only’ from the ‘Tab Policy’ dropdown.

You can further set the indentation and tab size.

For default text editor:

Window->Preferences->General->Editors->Text Editors->Insert spaces for tabs

Excellent Example and Explanation on How to Inject Properties from an External Properties File from Outside of a WAR in a Spring Application

Posted on by rchapin

I am doing some refactoring on a Spring MVC application, pulling out configuration data and login crentials from the spring.xml file.

What I want to do is to consolodate sensitive data into external .properties files that can be read, at runtime by the app and not require recompiling the war to make changes.

Thanks to Ben Northrop and Summa Technologies for such a clear, concise and well written article.

The long and the short of it (copied from the aforementioned article) is to add the following to your spring.xml

<bean id=”propertyPlaceholderConfigurer”  
class=”org.springframework.beans.factory.config.PropertyPlaceholderConfigurer”> 
  <property name=”locations”> 
    <list> 
      <value>classpath:database.properties</value> 
    </list> 
  </property> 
</bean> 


<bean id=”dataSource” class=”com.mchange.v2.c3p0.ComboPooledDataSource”> 
  <property name=”user” value=”${db.user}”/> 
  <property name=”password” value=”${db.password}”/> 
  … 
</bean>

Do Not Use Symlinks in Jetty’s webapps Directory

Posted on by rchapin

I set up a development instance of Jetty on my local machine and have been happily coding, compiling and deploying via a shell script.  The script copies the war from my user’s target directory to the jetty users’s home dir changes the permissions and then moves it to the /webapp dir creating a symlink to the name of the .war that is referenced in a number of config files.

This was working just fine until I did a merge with another developers code and a significant portion of it changed that Jetty had problems determining whether or not is should dump it’s local cache of .class files and configs and got into a very funky state.  Unfortunately, not one in which it was obviously funky, but where, all of the sudden I could not look up defined DataSources in JNDI.

Ultimately, after looking through many other different things, I decided to delete the symlink and, rename the .war in the /webapps dir.

Bingo, worked right away.

© 2005 Ryan Chapin