Welcome to my website. I am always posting links to photo albums, art, technology and other creations. Everything that you will see on my numerous personal sites is powered by the formVistaTM Website Management Engine.

icon.linkedin.jpgicon.twitter.jpg

  • [SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce
    01/14/2016 2:56PM

    I am working on a project and have decided to use Avro for the data serialization format.

    I encountered the following error when trying to set up the unit test to test the mapper implementation through Eclipse:

    java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
        at org.apache.avro.hadoop.io.AvroSerialization.getSerializer(AvroSerialization.java:114)
        at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:82)
        at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67)
        at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:98)
        at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:111)
        at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:676)
        at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:680)
        at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
        at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:130)
        at org.apache.hadoop.mrunit.MapDriverBase.addAll(MapDriverBase.java:141)
        at org.apache.hadoop.mrunit.MapDriverBase.withAll(MapDriverBase.java:247)
        at com.ryanchapin.hadoop.mapreduce.mrunit.UserDataSortTest.testMapper(UserDataSortTest.java:111)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
        at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
        at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
        at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

    After digging through the source code and finding that method did, infact, exist.  I tried running the same unit test via the maven cli.  It worked just fine.

    After more digging, it turns out that what was happening was that the classpath in Eclipse was using avro-1.7.4 from the hadoop-common and hadoop-mapreduce-client-core jars in my project, and not the 1.7.7 version that I was trying to use.

    To see what the difference between running it via the maven cli and running it in eclipse, I went through the following steps:

    Added the following code to my test code to print out the classpath at runtime:

        // Print out the classpath
        ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader();
        URL[] urls = ((URLClassLoader)sysClassLoader).getURLs();
        System.out.println("---------------------------------------");
        for(int i=0; i< urls.length; i++) {
            System.out.println(urls[i].getFile());
        }
        System.out.println("---------------------------------------");

    Then ran it, in Eclipse and saved off the console output.

    Then, I added a sleep call for 100 seconds in the same place in the code.  This enabled me to run the test again from the terminal and copy the project/target/surefire/ directory which contained the surefirebooter.jar.  Click here to read more about that project.

    After copying that jar to a temporary directory, I unpacked it and then compared the versions of avro between the Eclipse classpath and the classpath from the terminal and noticed that they were different.  Inspecting the dependency tree of my project it was clear that 1.7.4 was part of the hadooop jars I was using.

    Ultimately, I ended up updating my version of avro to 1.7.4 in my pom to eliminate the conflict.

  • Configuring Hidden, Invisible, or Whitespace Characters in The Eclipse Text Editor
    01/08/2016 10:27AM

    The newer (I am currently using Mars, 4.5.0) versions of Eclipse provide very good tools for configuring the visibility of whitespace characters in code.

    To customize your settings go to Window -> Preferences -> General -> Editors -> Text Editors.

    On that page there will be checkbox option next to "Show whitespace characters (configure visibility).

    Clicking on the 'configure visibility' link will allow you to choose what is shown and the opacity of the whitespace characters, which is a really nice touch.

  • Debugging MapReduce MRv2 Code in Eclipse
    03/24/2015 11:36PM

    Following is how to set-up your environment to be able to set breakpoints, step-through, and debug your MapReduce code in Eclipse.

    All of the this was done on a machine running Linux, but should work just fine for any *nix machine, and perhaps Windows running Cygwin (assuming that you can get Hadoop and its naitive libraries compiled under Windows).

    This also assumes that you are building your project with maven.

    Install a pseudo-distributed hadooop cluster on your development box.  (Yes, this calls for another article on exactly how to do that which I will do shortly and link to from here).

    Add the following environment variables to .bash_profile to ensure that they will be applied to any login shells (make sure to check the location of the directories for your installed hadoop distribution):

        export LD_LIBRARY_PATH=/usr/lib/hadoop/lib/native
        export HADOOP_HOME=/usr/lib/hadoop

    Make sure to include the following dependencies in your pom:

    hadoop-mapreduce-client-core
    hadoop-common
    hadoop-hdfs
    hadoop-client


    After you import your maven project into Eclipse update the Build Path to include the correct path to the Native library shared objects:

    Right-click on your project and select 'Build Path -> Configure Build Path:

    Click on 'Libraries' tab:

    Click the drop-down arrow for the 'Maven Dependencies'

    Click on the drop-down arrow on the 'hadoop-common'.jar

    Select the 'Native library location' entry, and click 'Edit'

    Browse to the path of the native directory, in my case it was /usr/lib/hadoop/lib/native.

    Click 'OK'

    Click 'OK' to close the build path dialogue

    Create a run configuration for the Main class in your project:

    Make sure that you do not add the /etc/hadoop/conf* dir to the class path.

    Add any commandline arguments for input and output directories to the 'Program arguments' section of the run configuration, that points to your LOCAL file system and not HDFS.

    Afterwhich, you should be able to run your M/R code and debug it through Eclipse.

  • Configuring Eclipse to Replace Tabs with Spaces for Indentation
    01/23/2015 6:46PM

    Following are two basic settings (I believe that there are other language specific, C++ for instance, settings as well).

     For Java:

    Window->Preferences->Java->Code Style->Formatter->
    Click on 'New' to create a new profile and select the profile that you want to copy
    Then click 'Edit' and select 'Spaces Only' from the 'Tab Policy' dropdown.

    You can further set the indentation and tab size.

    For default text editor:

    Window->Preferences->General->Editors->Text Editors->Insert spaces for tabs

  • Eclipse Android Development Error executing aapt: Cannot run program "/path/to/aapt": error=2, No such file or directory: error=2, No such file or directory
    03/24/2014 2:44PM

    Even though the ADT bundle provides a 64 bit version, the system requirements indicate that "64-bit distributions must be capable of running 32-bit applications."  I failed to see that when installing it under Fedora Core 20 and was getting the following error from Eclipse:

    Error executing aapt: Cannot run program "/home/rchapin/sdks/adt-bundle-linux-x86_64-20131030/sdk/build-tools/android-4.4/aapt": error=2, No such file or directory: error=2, No such file or directory  android_sdk    line 1   Android ADT Problem

    I checked to see if the file was there.  Yep.  I checked to see if was executable.  Yep.

    It was only after finding a blog post about it and doing a $ file command on it that I noticed that it was a 32 bit executable:

    $ file adt-bundle-linux-x86_64-20131030/sdk/build-tools/android-4.4/aapt
    adt-bundle-linux-x86_64-20131030/sdk/build-tools/android-4.4/aapt: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

    All I had to do was install the 32 bit libraries that the binaries are linked against:

    # yum install glibc.i686 zlib.i686 libstdc++.i686 ncurses-libs.i686 libgcc.i686

    Once installed, the error should disappear from Eclipse as it tries to invoke the binaries and do a regular build and/or restarting or cleaning the project should clear the errors.

  • 1 2 3 4 5 >>
Advanced Search

Categories

Archives