Welcome to my website. I am always posting links to photo albums, art, technology and other creations. Everything that you will see on my numerous personal sites is powered by the formVistaTM Website Management Engine.

icon.linkedin.jpgicon.twitter.jpg

  • Subscribe to this RSS Feed
  • Running Dynamically Generated Hive Queries From a Shell Script
    03/26/2014 10:35AM

    If you want to write a HQL hive query and run it mulitple times from a shell script, each time passing it different data for the query, here is a quick example that should get you started.

    The first thing to know is that by specifying n number of -hiveconf key value pairs when invoking hive on the command line will allow you to pass that data into the hive process.

    For example, if you do the following

    $ hive -e 'SELECT * FROM some_table' -hiveconf FOO=blah

    You will have passed in a key of FOO with the value of 'blah' to the hive process.

    A more practical example would be wanting to run the same hive query over multiple data partitions.

    In this example, I've got a hive database that has a 'packets' table partitioned by hours which looks like 2014032601.

    The hive query file (dest_ip_hive.sql) would look like:

    SELECT packets.sourceip FROM packets
    WHERE packets.destip = "${hiveconf:DEST_IP}"
    AND packets.hour = ${hiveconf:HOUR}
    GROUP BY packets.sourceip

    And a shell script that would dynamically set those values for each invocation of hive would look like:

    #!/bin/bash

    #
    # Destination IP that we are using to determine which
    # packets we will examine.
    #
    DEST_IP="10.0.1.10"

    for HOUR in 2014032209 2014032210 2014032211 2014032212
    do

       echo "Running hive query for HOUR $HOUR"

       # Run a hive query from the command line setting variables that will be
       # expaned in the .sql file.
       hive -hiveconf HOUR=$HOUR -hiveconf DEST_IP=$DEST_IP \
       -f dest_ip_hive.sql > ${DEST_IP}-{$HOUR}.out

    done

    For each hour defined in the for loop, we will execute a hive command telling it to run the query contained in the file dest_ip_hive.sql.  The DEST_IP and HOUR variables that will be expanded in the query are passed to hive via the

    -hiveconf HOUR=$HOUR -hiveconf DEST_IP=$DEST_IP

    part of the hive command.  And the output for each query will be written to a different file for each query.



  • Eclipse Android Development Error executing aapt: Cannot run program "/path/to/aapt": error=2, No such file or directory: error=2, No such file or directory
    03/24/2014 2:44PM

    Even though the ADT bundle provides a 64 bit version, the system requirements indicate that "64-bit distributions must be capable of running 32-bit applications."  I failed to see that when installing it under Fedora Core 20 and was getting the following error from Eclipse:

    Error executing aapt: Cannot run program "/home/rchapin/sdks/adt-bundle-linux-x86_64-20131030/sdk/build-tools/android-4.4/aapt": error=2, No such file or directory: error=2, No such file or directory  android_sdk    line 1   Android ADT Problem

    I checked to see if the file was there.  Yep.  I checked to see if was executable.  Yep.

    It was only after finding a blog post about it and doing a $ file command on it that I noticed that it was a 32 bit executable:

    $ file adt-bundle-linux-x86_64-20131030/sdk/build-tools/android-4.4/aapt
    adt-bundle-linux-x86_64-20131030/sdk/build-tools/android-4.4/aapt: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

    All I had to do was install the 32 bit libraries that the binaries are linked against:

    # yum install glibc.i686 zlib.i686 libstdc++.i686 ncurses-libs.i686 libgcc.i686

    Once installed, the error should disappear from Eclipse as it tries to invoke the binaries and do a regular build and/or restarting or cleaning the project should clear the errors.

  • Setting up Android in Eclipse, Unable to resolve target 'android-18'
    03/24/2014 2:22PM

    When setting up Eclipse for Android development, after you have installed the Develeper Tools and NDK Plugins and linked to the ADT-bundle and then imported your project (see othere posts related by searching for 'android') I got the following error:

    [2014-03-24 14:14:53 - android_sdk] Unable to resolve target 'android-18'

    I went and checked to make sure that the Platform.MinPlatformToolsRev property in the ~/path/to/adt-bundle-linux-x86_64-20131030/sdk/tools/source.properties matched that of my project.properties file.

    They were both 18.  Eh?  Why the error?

    So, in Eclipse, I went to the Window > Android SDK Manager and looked at the version of the Android SDK Platform that was installed.  It was:

    Android 4.4.2 (API 19) and the checkbox next to SDK Platform under that sub-heading was checked.

    So, I went back to my project.properties and updated the target property to:

    target=android-19

    which eliminated that error.


  • "Invalid Project Description", project path "overlaps the location of another project" with the same name When Importing Existing Android Project in Eclipse
    03/24/2014 2:06PM

    I've got a new install of Eclipse and am setting up to do some Android development.

    After getting the Develeper Tools and NDK Plugins installed and linking to the ADT-bundle that I downloaded, I was getting an error when attempting to import the existing project by going to File > Import > Android > Existing Android Code Into Workspace.

    The error was:

    Invalid project description /some/path/to/project/dir_name overlaps the location of another project: dir_name

    After a few searches and a couple of tries, using the File > Import > General > Existing Projects into Workspace seemed to work just fine.



  • Restarting XServer in Fedora 20
    03/24/2014 9:49AM

    I was having problems logging in this morning to my laptop.  When I entered my password, it just hung.   I pressed Ctrl-f2 to switch to an alternate tty, logged in as root and checked for errors in /var/log/messages.

    Not seeing anything, I figured, I'd try and restart the Xserver.  Still not being completely familiar with the sysctl paradigm it wasn't obvious how to restart it.

    So, as root, I simply switched the runlevel to 3, and then back to 5 via the following commands, restarting the Xserver and was then able to login.

    # telinit 3

    .... wait for a bit ....

    # telinit 5

    Login, get to work.

Advanced Search

Categories

Archives