Category: Computers

All things computing.

[SOLVED] Delete key not working with x2go server running XFCE

Posted on by rchapin

I am working with a development environment whereby I VPN into an environment with an Ubuntu VirtualBox guest and then initiate an x2go client session to a developement workstation server that lives in the development environment.

I had everythining working exactly as I wanted except that the Delete key did not work in the x2go session.

Previously the xev output when pressing the delete key was:

FocusOut event, serial 36, synthetic NO, window 0x2e00001,
    mode NotifyGrab, detail NotifyAncestor

FocusIn event, serial 36, synthetic NO, window 0x2e00001,
    mode NotifyUngrab, detail NotifyAncestor

KeymapNotify event, serial 36, synthetic NO, window 0x0,
    keys:  0   0   0   0   0   0   0   0   0   0   0   0   0   0 0   0
           0   0   0   0   0   0   0   0   0   0   0   0   0   0 0   0

For the given user/X-session, go to Applications Menu -> Settings -> Keyboard

Then under the ‘Layout’ tab, uncheck the ‘Use system defaults’ and specify the keyboard that you are using.  In my case it was:

Keyboard Model: Generic 105-key (Intl) PC
Keyboard Layout: English (US)

NOTE:  the layout had TWO options that were both ‘English (US)’ and I selected the top-most item.

Log out of the X session, which will kill the x2go session and then log back in to a new session and the delete key works as expected.

xev output when pressing delete key is now:

KeyPress event, serial 36, synthetic NO, window 0x3600001,
    root 0x146, subw 0x0, time 2229006870, (-615,43), root:(879,645),
    state 0x10, keycode 107 (keysym 0xffff, Delete), same_screen YES,
    XLookupString gives 1 bytes: (7f) " 
    XmbLookupString gives 1 bytes: (7f) "
    XFilterEvent returns: False

KeyRelease event, serial 36, synthetic NO, window 0x3600001,
    root 0x146, subw 0x0, time 2229006981, (-612,41), root:(882,643),
    state 0x10, keycode 107 (keysym 0xffff, Delete), same_screen YES,
    XLookupString gives 1 bytes: (7f) "
    XFilterEvent returns: False

Looping Through a List of Files with Spaces in the File Name with Bash

Posted on by rchapin

If you have a list of files that you want to operate on in a loop in bash and some of them have spaces in the file name the default IFS (Internal Field Separator) will match with the space and tokenize the file.

The simple approach is to temporarily set the IFS as follows.  This can be done in a shell script, but the following example is directly on the command line for ‘one-liner’ usage.

OIFS="$IFS"

IFS=$'\n' 

for i in `find ./ -type f -iname '*some_criteria*'`; do "something with $i"; done

IFS="$OIFS"

The previous commands will:

  1. Save the existing IFS
  2. Update the IFS to a newline char
  3. Execute your loop with the results of a find command
  4. Reset the IFS

[SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce

Posted on by rchapin

I am working on a project and have decided to use Avro for the data serialization format.

I encountered the following error when trying to set up the unit test to test the mapper implementation through Eclipse:

java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
    at org.apache.avro.hadoop.io.AvroSerialization.getSerializer(AvroSerialization.java:114)
    at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:82)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:98)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:111)
    at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:676)
    at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:680)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:130)
    at org.apache.hadoop.mrunit.MapDriverBase.addAll(MapDriverBase.java:141)
    at org.apache.hadoop.mrunit.MapDriverBase.withAll(MapDriverBase.java:247)
    at com.ryanchapin.hadoop.mapreduce.mrunit.UserDataSortTest.testMapper(UserDataSortTest.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

After digging through the source code and finding that method did, infact, exist.  I tried running the same unit test via the maven cli.  It worked just fine.

After more digging, it turns out that what was happening was that the classpath in Eclipse was using avro-1.7.4 from the hadoop-common and hadoop-mapreduce-client-core jars in my project, and not the 1.7.7 version that I was trying to use.

To see what the difference between running it via the maven cli and running it in eclipse, I went through the following steps:

Added the following code to my test code to print out the classpath at runtime:

// Print out the classpath
ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader();
URL[] urls = ((URLClassLoader)sysClassLoader).getURLs();
System.out.println("---------------------------------------");
for(int i=0; i< urls.length; i++) {
    System.out.println(urls[i].getFile());
}
System.out.println("---------------------------------------");

Then ran it, in Eclipse and saved off the console output.

Then, I added a sleep call for 100 seconds in the same place in the code.  This enabled me to run the test again from the terminal and copy the project/target/surefire/ directory which contained the surefirebooter.jar.  Click here to read more about that project.

After copying that jar to a temporary directory, I unpacked it and then compared the versions of avro between the Eclipse classpath and the classpath from the terminal and noticed that they were different.  Inspecting the dependency tree of my project it was clear that 1.7.4 was part of the hadooop jars I was using.

Ultimately, I ended up updating my version of avro to 1.7.4 in my pom to eliminate the conflict.

Configuring Hidden, Invisible, or Whitespace Characters in The Eclipse Text Editor

Posted on by rchapin

The newer (I am currently using Mars, 4.5.0) versions of Eclipse provide very good tools for configuring the visibility of whitespace characters in code.

To customize your settings go to Window -> Preferences -> General -> Editors -> Text Editors.

On that page there will be checkbox option next to “Show whitespace characters (configure visibility).

Clicking on the ‘configure visibility’ link will allow you to choose what is shown and the opacity of the whitespace characters, which is a really nice touch.

[SOLVED] Configuring chrooted bind and rndc-confgen Hangs Not Generating a Key

Posted on by rchapin

I am putting together a chrooted installation of named and ran into a problem whereby attempting to generate an rndc.key with rndc-confgen just hangs, never returning and not generating a key.

After doing some searching I discovered that I needed to run the command as follows:

rndc-confgen -a -r /dev/urandom  -t /var/named/chroot

Which outputs the following, generating the key file that I expected.

wrote key file "/etc/rndc.key"
wrote key file "/var/named/chroot/etc/rndc.key"

Using the Eclipse Memory Analyzer (MAT) Remotely on Large Heap Dumps

Posted on by rchapin

Sometime your java applilcation will fail and generate an enormous heap dump.  One that may be too large to be able to transfer to your local machine and to analyze for lack of RAM, time or both.

One solution is to install the MAT tool on the remote server and generate an HTML output of the analysis to download and view locally.  This saves the headache of attempting to get X Windows installed on the remote machine and get all of the ssh tunneling sorted out (which is of course an option as well).

First, download and install the stand-alone Eclipse RCP Application.  Then transfer to your server and unpack.  Then determine how large the heap dump is and, if necessary, modify the MemoryAnalyzer.ini file to instantiate a JVM with enough RAM for your heap dump.

In this example, I have an 11GB heap dump and have modified the last two lines (adding -Xms)

-startup
plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.300.v20150602-1417
-vmargs
-Xmx16g
-Xms16g

Do an initial run to parse the heap dump.  This will generate intermediary data that can be used by subsequent runs to make future analysis faster.

./ParseHeapDump.sh /path/to/heap-dump

After that completes, you can run any of a number of different analysis on the data.  The following is an illustration of how to search for memory leak suspects.

./ParseHeapDump.sh /path/to/heap-dump org.eclipse.mat.api:suspects

Additional reports:

org.eclipse.mat.api:suspects
org.eclipse.mat.api:overview
org.eclipse.mat.api:top_components

To give creadit where it is due, this is basically a copy of a post by Ashwin Jayaprakash, but I wanted to capture it here as well.

Quick-Start and Minimal Cheat Sheet for Chef and Test Kitchen

Posted on by rchapin

Following is a quick-start for getting a Chef and Test Kitchen environment set up to start writing and testing Chef cookbooks.  This is an work in progress, so please if you find something that is incorrect or does not work as expected, send me an e-mail.

Development Environment Set-up:

Unless you are a Ruby developer and already have a complete environmenet set-up the easiest thing to do is to install the Chef Development Kit, https://downloads.chef.io/chef-dk/.

After downloading and installing the package add a CHEF_HOME and RUBY_HOME var pointed to the embedded version of Ruby that comes with the Chef SDK.

  export CHEF_HOME=/opt/chefdk
  export RUBY_HOME=/opt/chefdk/embedded
  PATH=”$CHEF_HOME/bin:$RUBY_HOME/bin:$PATH”

Building Cookbooks:

Generate the cookbook skeleton:

$ chef generate cookbook <name-of-cookbook>

Then cd into that directory and initialize the cookbook:

$ kitchen init –create-gemfile

You will also want to run bundle install, which will require your user to have sudo permissions.

$ bundle install

Run kitchen converge to verify before starting dev.

$ kitchen converge <name-of-vm>

Create a recipe

$ chef generate recipe <name-of-recipe>

[Optional] To use attributes, first create the attributes directory structure

$ chef generate attribute default

Some helpful Tips:

1.  Answer the following questions for each new cookbook you are going to create.  It will help you focus the purpose of the cookbook:

. Name
. Purpose
. Success criteria
. App/Service
. Required steps

2.  Creating template files:

$ chef generate template <name-of-file>

    . Sample variable to be added to a template:
      <%= node[‘hostname’] %>

3.  Available host variables.  You can see all of the variables available to be used in templates by logging in to a chef managed VM and issuing the following command:

$ kitchen login <name-of-vm>

$ ohia | less

4.  Add a IP to a kitchen VM so that additional kitchen VMs can talk to each other.  Add the following to your .kitchen.yml file.

platforms:
  – name: centos-7.1
    driver:
      network:
      – [“private_network”, {type: “dhcp”}]

or

platforms:
  – name: centos-7.1
    driver:
      network:
      – [“private_network”, {ip: “192.168.33.10”}]

When specifying an IP, check the vboxnet0 interface on the host machine to see what network the VirtualBox host is configured and pick an IP in that range.

In my case, I had to make sure that only one machine was configured under ‘platform’, or that each had a network configuration.

Also, I had to have ‘require_chef_omnibus: true’ under the vagrant driver as follows:

driver:
  name: vagrant
  require_chef_omnibus: true

5.  Debugging Attributes:  Using the ‘pp’ module you can format the output of the node.debug(‘<attribute-name>’).

Add the following to a recipe, and run it with kitchen converge:

require ‘pp’

node.default[‘ipaddress’] = ‘1.1.1.1’
pp node.debug_value(‘ipaddress’)

Perl One-Liner for Replacing Multiple Lines of a Text file With Multiple Lines of Text

Posted on by rchapin

When executing ‘search-and-replace’ commands on ASCII under Linux, Unix (or *nix) operating systems, sed works or most cases and makes for reasonably straightforward reading of the script.

If  you want to replace multiple lines of text with multiple lines of text, following is a perl one-liner that does the trick and is much easier to wrangle than trying to do it in sed.

perl -i -pe "BEGIN{undef $/;} s:${EXISTING_LINES}:${REPLACEMENT_TEXT}:smg" file.txt
© 2005 Ryan Chapin