Welcome to my website. I am always posting links to photo albums, art, technology and other creations. Everything that you will see on my numerous personal sites is powered by the formVistaTM Website Management Engine.

icon.linkedin.jpgicon.twitter.jpg

  • [SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce
    01/14/2016 2:56PM

    I am working on a project and have decided to use Avro for the data serialization format.

    I encountered the following error when trying to set up the unit test to test the mapper implementation through Eclipse:

    java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
        at org.apache.avro.hadoop.io.AvroSerialization.getSerializer(AvroSerialization.java:114)
        at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:82)
        at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67)
        at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:98)
        at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:111)
        at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:676)
        at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:680)
        at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
        at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:130)
        at org.apache.hadoop.mrunit.MapDriverBase.addAll(MapDriverBase.java:141)
        at org.apache.hadoop.mrunit.MapDriverBase.withAll(MapDriverBase.java:247)
        at com.ryanchapin.hadoop.mapreduce.mrunit.UserDataSortTest.testMapper(UserDataSortTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
        at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

    After digging through the source code and finding that method did, infact, exist.  I tried running the same unit test via the maven cli.  It worked just fine.

    After more digging, it turns out that what was happening was that the classpath in Eclipse was using avro-1.7.4 from the hadoop-common and hadoop-mapreduce-client-core jars in my project, and not the 1.7.7 version that I was trying to use.

    To see what the difference between running it via the maven cli and running it in eclipse, I went through the following steps:

    Added the following code to my test code to print out the classpath at runtime:

        // Print out the classpath
        ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader();
        URL[] urls = ((URLClassLoader)sysClassLoader).getURLs();
        System.out.println("---------------------------------------");
        for(int i=0; i< urls.length; i++) {
            System.out.println(urls[i].getFile());
        }
        System.out.println("---------------------------------------");

    Then ran it, in Eclipse and saved off the console output.

    Then, I added a sleep call for 100 seconds in the same place in the code.  This enabled me to run the test again from the terminal and copy the project/target/surefire/ directory which contained the surefirebooter.jar.  Click here to read more about that project.

    After copying that jar to a temporary directory, I unpacked it and then compared the versions of avro between the Eclipse classpath and the classpath from the terminal and noticed that they were different.  Inspecting the dependency tree of my project it was clear that 1.7.4 was part of the hadooop jars I was using.

    Ultimately, I ended up updating my version of avro to 1.7.4 in my pom to eliminate the conflict.

  • Updating all of the pom.xml Version Numbers in a Multi-Module Maven Project
    10/13/2015 5:05PM

    To update the versions of all of the poms in a multiple module project use the versions-maven plugin.

    To update

    $ mvn versions:set -DnewVersion=1.4.0-SNAPSHOT

    Will modify all of the versions of each of the poms to the version specified.  It will create a pom.xml.versionsBackup for each pom file that it modified.  You can then examine each to make sure that it is as you intended.

    If you want, you can revert your change with

    $ mvn versions:revert

    If you are satisfied with the change, you can commit the change with

    $mvn versions:commit

  • Debugging Maven Tests by Connecting an IDE to the Maven JVM
    09/10/2015 1:50PM

    In some instances you cannot reproduce a failure or condition running a test in an IDE that manifests itself when you run it on your build server or via maven on the command line.

    In that case, it is very helpful to be able to remotely attach your IDE to the running maven process and then step through the code.

    To do so you will need to:

    Execute maven on  the command line as follows (adding any additional -D args as required by your project):

    $ mvn -Dmaven.surefire.debug test -pl module-in-question

    This will run the maven automatically pausing the JVM awaiting for a remote debugger to connect to port 5005.  If you want to have it listen on a different port you can pass it in as follows:

    $ mvn -Dmaven.surefire.debug="-Xdebug  -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8081 -Xnoagent -Djava.compiler=NONE" test -pl module-in-question

    Create a debugging run profile in either Eclipse or IntelliJ or your favorite IDE configured to connect to a JVM listening on the specified port.

    Then once you have run maven on the command line, simply execute the run configuration in your IDE and debug your application as usual.

    If need be, you can run the maven JVM on the cli such that maven does not fork the tests as follows:

    $ mvnDebug -DforkCount=0 test


  • Setting the Compiler Version for Maven from the Command Line
    06/18/2015 10:42AM

    By default maven sets the compiler version for you.  Of course, you can always set it in the pom, but there are cases where you cannot modify the pom, and/or you might want to test compilation and tests with different versions of java.

    Following are the specific arguments to pass the compiler version to maven from the command line:

    $ mvn clean install -Dmaven.compiler.source=1.7 -Dmaven.compiler.target=1.7

  • How To Distributed Artifacts in the Maven Central Repository
    04/02/2015 1:27PM

    I have just finished releasing my first project to Maven Central Repository and wanted to capture my notes for the setup of the project and all of the steps required.

    Publishing Artifacts to Maven Central:

    - -------------------------------------
    - Links:

    http://central.sonatype.org/pages/requirements.html
    http://maven.apache.org/guides/mini/guide-central-repository-upload.html
    http://ruleoftech.com/2014/distribute-projects-artifacts-in-maven-central-with-ossrh

    - -------------------------------------
    - Create account on OSSRH: You will need an account to the sonatype JIRA for OSSRH.  From there, you can request the creation of a new project.

       . See http://central.sonatype.org/pages/ossrh-guide.html for details


    - -------------------------------------
    - Setup of the Project/pom and Pre-requisites:

    1.  PGP keys
    http://central.sonatype.org/pages/working-with-pgp-signatures.html

       . Create a set of PGP keys
    $ gpg2 --gen-key

       . List the keys:
    $ gpg2 --list-keys
    /usr/local2/home/rchapin/.gnupg/pubring.gpg
    -------------------------------------------
    pub   4096R/E5170CE8 2015-03-26 [expires: 2016-03-25]
    uid                  Ryan Chapin <rchapin@nbinteractive.com>
    sub   4096R/8DAF9AD6 2015-03-26 [expires: 2016-03-25]

          . The id for the key created is 'E5170CE8'

       . Distribute the public key:
    $ gpg2 --keyserver hkp://pool.sks-keyservers.net --send-keys E5170CE8


    2. Pom additions:
       . License: Add the <license> tag to your pom.

       . Some links regarding available licenses and an example of it's use in the pom.
          . http://choosealicense.com/licenses/
          . http://mojo.codehaus.org/license-maven-plugin/examples/example-license-list.html

       . I ended up using the BSD 3 License, and used the following in the pom:

    <project>
      ...
      <licenses>
        <license>
          <name>The BSD 3-Clause License</name>
          <url>http://opensource.org/licenses/BSD-3-Clause</url>
          <distribution>repo</distribution>
        </license>
      </licenses>
      ...
    </project>


       . Developer:

    <project>
      ...
      <developers>
        <developer>
          <id>rchapin</id>
          <name>Ryan Chapin</name>
          <email>rchapin@nbinteractive.com</email>
          <url>http://www.ryanchapin.com</url>
          <roles>
            <role>architect</role>
            <role>developer</role>
          </roles>
          <timezone>America/New_York</timezone>
          <properties>
            <picUrl>http://www.gravatar.com/516f2158d74d134faa9649e9180ef782</picUrl>
          </properties>
        </developer>
      </developers>
      ...
    </project>


       . Scm:
    <project>
      ...
      <scm>
        <connection>scm:git:git@github.com:rchapin/hash-generator.git</connection>
        <developerConnection>scm:git:git@github.com:rchapin/hash-generator.git</developerConnection>
        <url>git@github.com:rchapin/hash-generator.git</url>
        <tag>HEAD</tag>
      </scm>
      ...
    </project>


       . Distribution Management and Authentication:  Configure the pom to enable maven to deploy to OSSRH Nexus server with the Nexus Staging Maven plugin.  See below for the release profile configuration which will contain the configs for the nexus-staging-maven-plugin.

    <project>
      ...
      <distributionManagement>
        <snapshotRepository>
        <id>ossrh</id>
        <url>https://oss.sonatype.org/content/repositories/snapshots</url>
        </snapshotRepository>
      </distributionManagement>
      ...
    </project>


       . Create a profile to encapsulate the creation of the javadoc and source jar as well as the pgp signing f the artifacts.  In my case, I set the profile to <activeByDefault>true</activeByDefault> true to ease release builds and simply invoke a build as such during development of the project to turn off the release profile:
    $ mvn package -P\!release

          . Additionally, the nexus-staging-maven-plugin is configured <autoReleaseAfterClose>false</autoReleaseAfterClose> to enable manual inspection of the staging repository BEFORE it is released to central.  To deploy to OSSRH and release to the Central Repository in one step, set autoReleaseAfterClose:true.

      <profiles>
        <profile>
          <id>release</id>
          <activation>
            <activeByDefault>true</activeByDefault>
          </activation>
          <build>
            <plugins>

              <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-javadoc-plugin</artifactId>
                <version>${maven.javadoc.plugin.version}</version>
                <executions>
                  <execution>
                    <id>attach-javadocs</id>
                    <goals>
                      <goal>jar</goal>
                    </goals>
                  </execution>
                </executions>
              </plugin>

              <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-source-plugin</artifactId>
                <version>${maven.source.plugin.version}</version>
                <executions>
                  <execution>
                    <id>attach-sources</id>
                    <goals>
                      <goal>jar</goal>
                    </goals>
                  </execution>
                </executions>
              </plugin>

              <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-gpg-plugin</artifactId>
                <version>1.6</version>
                <executions>
                  <execution>
                  <id>sign-artifacts</id>
                  <phase>verify</phase>
                  <goals>
                    <goal>sign</goal>
                  </goals>
                  </execution>
                </executions>
              </plugin>

              <plugin>
                <groupId>org.sonatype.plugins</groupId>
                <artifactId>nexus-staging-maven-plugin</artifactId>
                <version>1.6.5</version>
                <extensions>true</extensions>
                <configuration>
                  <serverId>ossrh</serverId>
                  <nexusUrl>https://oss.sonatype.org/</nexusUrl>
                  <autoReleaseAfterClose>false</autoReleaseAfterClose>
                </configuration>
              </plugin>

            </plugins>
          </build>
        </profile>

      </profiles>


       . Add the maven-release-plugin to the <build> section.  This should include disabling the 'release' profile that we added/describe above, and then specify the deploy goal together with the activation of the 'release' profile for the deploy goal.

    <project>
      ...
      <build>
        ...
        <plugins>
          ...
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-release-plugin</artifactId>
            <version>2.5.1</version>
            <configuration>
              <autoVersionSubmodules>true</autoVersionSubmodules>
              <useReleaseProfile>false</useReleaseProfile>
              <releaseProfiles>release</releaseProfiles>
              <goals>deploy</goals>
            </configuration>
          </plugin>
          ...
        </plugins>
        ...
      </build>
      ...
    </project>

    3.  Adding credentials to the settings.xml:  Credentials for both the distributionManagement and the pgp signing need to be added to the ~/.m2/settings.xml.

       . OSSRH login:

    <setting>
      ...
      <servers>
        <server>
          <id>ossrh</id>
          <username>your-uid</username>
          <password>your-passwd</password>
        </server>
      </servers>
      ...
    </settings>

       . PGP Signing Passphrase:

    <setting>
      ...
      <profiles>
        <profile>
          <id>ossrh</id>
          <activation>
            <activeByDefault>true</activeByDefault>
          </activation>
          <properties>
            <gpg.executable>gpg2</gpg.executable>
            <gpg.passphrase>your-passphrase</gpg.passphrase>
          </properties>
        </profile>
      </profiles>
      ...
    </settings>

    - -------------------------------------
    - Performing a SNAPSHOT deployment:

    1. As long as your current pom version still ends in 'SNAPSHOT' you can deploy a snapshot version to the OSSRH:
    $ mvn clean deploy

       . To use the SNAPSHOT version in a project, users will need to add the snapshot repo to their Nexus, settings.xml or pom.xml. 

    <project>
      ...
      <repositories>
        <repository>
          <id>ossrh-snapshots</id>
          <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
        </repository>
      </repositories>
      ...
    </project>

    - -------------------------------------
    - Release and Deployment:

    1. Push all code to master:  First make sure that all of the code is pushed to the remote repo and that everything is merged into master.  Then make sure that you are ON the master branch on the local machine.


    2. Prepare Release:
       . To do a dry run and insure that pom will be bumped as expected and to check that everything is in order.  Will also NOT checkin or tag anyting in the scm repository
    $ mvn release:prepare -DdryRun=true

       . Check the output and then do the following before running the actual prepare command
    $ mvn release:clean

       . Execute the release:prepare
    $ mvn release:prepare

       . If errors occur:
    $ mvn release:prepare -Dresume=false

       . Alternatively, you can use:
    $ mvn release:clean release:prepare

       . See http://maven.apache.org/maven-release/maven-release-plugin/examples/prepare-release.html for details.

       . Once the release has been performed the scm repository should be updated with the new tag and the pom should be bumped to the next version for the next iteration of development.


    3. Perform Release:  Once the release has been prepared and a tag created and the scm repository has been update, you can deploy to OSSRH:

       . This will deploy to a staging repository.
    $ mvn release:perform


    4. Inspecting Staging Repo and Releasing to Central:
       . Login to OSSRH via https://oss.sonatype.org/.  Uid and passwd are the same for issues.sonatype.org
       . See http://central.sonatype.org/pages/releasing-the-deployment.html for details


    5. Releasing to Central: Once you have inspected the staging repo and are ready to release the deployment to Central do the following:

    $ cd target/checkout
    $ mvn nexus-staging:release


    6. If this is your first release to central, don't forget to go back to your project creation request ticket and add a comment that you have promoted your first release so that your promotion can be verified and your artifacts synced with Central.  This step only needs to be done the first time you promote to central with a new OSSRH project.

Advanced Search

Categories

Archives