Category: Software

Blacklisting Kernel Modules

Posted on by rchapin

Following is a walkthrough on how to blacklist a kernel module.  The specific example is blacklisting the nouveau driver so that I can install the OEM Nvidia driver.

1. First, blacklist the nouveau driver: Add a line to the textfile /etc/modprobe.d/nouveau-blacklist.conf that contains they keyword ‘blacklist’ and the name of the driver

blacklist nouveau

2. Rebuild the initramfs image file.  First, backup existing initramfs

mv initramfs-3.10.0-327.18.2.el7.x86_64.img initramfs-3.10.0-327.18.2.el7.x86_64.img_2016-06-09.bak

Build new initramfs

dracut -v /boot/initramfs-$(uname -r).img $(uname -r)

3. Reboot the system and confirm that the driver in question is not loaded.

lsmod | fgrep nouveau

How to Configure a User Account in Active Directory So that the Password Never Expires

Posted on by rchapin

Using ADSI Edit, navigate to the user in question.  Right-click on the user and select Properties.

Then, scroll down to the ‘userAccountControl’ property and click the Edit button.

Enter 66048 into the field and then click ‘OK’ and then ‘Apply’.

After closing the edit window, scroll to the right to confirm that the value indicates the following

'0x10200 = (NORMAL_ACCOUNT|DONT_EXPIRE_PASSWORD)'

Using netcat to Mock a RESTful Webservice that Returns JSON

Posted on by rchapin

Let’s say that you are working on a part of a project that needs to consume some JSON data from a forthcoming HTTP service.  That sub-system is being developed by one of your colleagues and is not yet ready for you to stand-up to develop against.

You can use netcat to mock the webservice and return some static JSON data for which you can develop and test against with a simple one-liner.

First, put together your JSON in a file, ‘my.json’ and then run the following command in a terminal:

while true; do echo -e "HTTP/1.1 200 OK\r\n\r\n$(cat my.json)" | nc -l 9998; done

An hitting http://localhost:9998 will return your test JSON data.

How to Use Credentials That Contain Special Characters with curl

Posted on by rchapin

In order to execute curl commands to endpoints with passwords that contain special characters, the cleanest way that I have found to do so is to Base64 encode the authentication string for curl and then pass an Authorization request header along with the request.

In this example the credentials are uid ‘rchapin’ and passwd ‘abc123!@#’.  Normally we would pass this to curl as follows:

$ curl -u rchapin:abc123!@# -X GET https://some-endpoint:443

However, this will not work and the password will need to be sent in some other fashion other than ASCII.

Following are the steps to pass the credentials as Base64:

1. Using your favorite Base64 encoder, generate and ASCII string of the entire ‘username:password’ string that you would normally pass with the -u option for curl

rchapin:abc123!@# converted = cmNoYXBpbjphYmMxMjMhQCM=

2. Now modify the curl command as follows:

$ curl -H "Authorization: Basic cmNoYXBpbjphYmMxMjMhQCM=" -X GET https://some-endpoint:443

How to Configure a Linux Client for Active Directory Authentication

Posted on by rchapin

I am currently working on setting up multiple environments for a new project (DEV, QA, and PROD) and will be integrating all of the servers to an Active Directory Domain Controller for user management.

Following are notes from when I configured a Fedora Core 18 laptop to integrate with an AD server.  It is likely things have changed some from then and I will update this as required for a current version of RHEL7/CentOS 7.

Install the following packages

yum install sssd-tools adcli realmd

sssd will not yet have the /etc/sssd/sssd.conf file installed until you join a domain (see below) so do not try to configure it.

Disable selinux. Perhaps selinux can be re-enabled after joining and configuration?

Search for a realm

realm discover -v
 * Looking up our DHCP domain
 * Discovering for DHCP domain: somedomain.com
 * Searching for kerberos SRV records for domain: _kerberos._udp.somedomain.com
 * Searching for MSDCS SRV records on domain: _kerberos._tcp.dc._msdcs.somedomain.com
 * server1.somedomain.com:88 server2.somedomain.com:88 server3.somedomain.com:88
 * Found kerberos DNS records for: somedomain.com
 * Found AD style DNS records for: somedomain.com
 * Successfully discovered: somedomain.com
somedomain.com
  type: kerberos
  realm-name: SOMEDOMAIN.COM
  domain-name: somedomain.com
  configured: kerberos-member
  server-software: active-directory
  client-software: sssd
  required-package: sssd-tools
  required-package: sssd
  required-package: adcli
  required-package: samba-common
  login-formats: SOMEDOMAIN\%U
  login-policy: allow-any-login

Joined the domain

realm join -U admin --verbose somedomain.com

Allow all users from a given domain to login

realm permit --realm somedomain.com --all

Add the home directory paths

mkdir /home/SOMEDOMAIN

 Start and enable sssd

systemctl enable sssd && systemctl start sssd

Log out from the local user account and then log in as a user in the domain:
    . uid: SOMEDOMAIN\uid
    . This will log you in as that user and create a home account in /home/SOMEDOMAIN/uid

[SOLVED] Delete key not working with x2go server running XFCE

Posted on by rchapin

I am working with a development environment whereby I VPN into an environment with an Ubuntu VirtualBox guest and then initiate an x2go client session to a developement workstation server that lives in the development environment.

I had everythining working exactly as I wanted except that the Delete key did not work in the x2go session.

Previously the xev output when pressing the delete key was:

FocusOut event, serial 36, synthetic NO, window 0x2e00001,
    mode NotifyGrab, detail NotifyAncestor

FocusIn event, serial 36, synthetic NO, window 0x2e00001,
    mode NotifyUngrab, detail NotifyAncestor

KeymapNotify event, serial 36, synthetic NO, window 0x0,
    keys:  0   0   0   0   0   0   0   0   0   0   0   0   0   0 0   0
           0   0   0   0   0   0   0   0   0   0   0   0   0   0 0   0

For the given user/X-session, go to Applications Menu -> Settings -> Keyboard

Then under the ‘Layout’ tab, uncheck the ‘Use system defaults’ and specify the keyboard that you are using.  In my case it was:

Keyboard Model: Generic 105-key (Intl) PC
Keyboard Layout: English (US)

NOTE:  the layout had TWO options that were both ‘English (US)’ and I selected the top-most item.

Log out of the X session, which will kill the x2go session and then log back in to a new session and the delete key works as expected.

xev output when pressing delete key is now:

KeyPress event, serial 36, synthetic NO, window 0x3600001,
    root 0x146, subw 0x0, time 2229006870, (-615,43), root:(879,645),
    state 0x10, keycode 107 (keysym 0xffff, Delete), same_screen YES,
    XLookupString gives 1 bytes: (7f) " 
    XmbLookupString gives 1 bytes: (7f) "
    XFilterEvent returns: False

KeyRelease event, serial 36, synthetic NO, window 0x3600001,
    root 0x146, subw 0x0, time 2229006981, (-612,41), root:(882,643),
    state 0x10, keycode 107 (keysym 0xffff, Delete), same_screen YES,
    XLookupString gives 1 bytes: (7f) "
    XFilterEvent returns: False

Looping Through a List of Files with Spaces in the File Name with Bash

Posted on by rchapin

If you have a list of files that you want to operate on in a loop in bash and some of them have spaces in the file name the default IFS (Internal Field Separator) will match with the space and tokenize the file.

The simple approach is to temporarily set the IFS as follows.  This can be done in a shell script, but the following example is directly on the command line for ‘one-liner’ usage.

OIFS="$IFS"

IFS=$'\n' 

for i in `find ./ -type f -iname '*some_criteria*'`; do "something with $i"; done

IFS="$OIFS"

The previous commands will:

  1. Save the existing IFS
  2. Update the IFS to a newline char
  3. Execute your loop with the results of a find command
  4. Reset the IFS

[SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce

Posted on by rchapin

I am working on a project and have decided to use Avro for the data serialization format.

I encountered the following error when trying to set up the unit test to test the mapper implementation through Eclipse:

java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
    at org.apache.avro.hadoop.io.AvroSerialization.getSerializer(AvroSerialization.java:114)
    at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:82)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:98)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:111)
    at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:676)
    at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:680)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:130)
    at org.apache.hadoop.mrunit.MapDriverBase.addAll(MapDriverBase.java:141)
    at org.apache.hadoop.mrunit.MapDriverBase.withAll(MapDriverBase.java:247)
    at com.ryanchapin.hadoop.mapreduce.mrunit.UserDataSortTest.testMapper(UserDataSortTest.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
    at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
    at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
    at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

After digging through the source code and finding that method did, infact, exist.  I tried running the same unit test via the maven cli.  It worked just fine.

After more digging, it turns out that what was happening was that the classpath in Eclipse was using avro-1.7.4 from the hadoop-common and hadoop-mapreduce-client-core jars in my project, and not the 1.7.7 version that I was trying to use.

To see what the difference between running it via the maven cli and running it in eclipse, I went through the following steps:

Added the following code to my test code to print out the classpath at runtime:

// Print out the classpath
ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader();
URL[] urls = ((URLClassLoader)sysClassLoader).getURLs();
System.out.println("---------------------------------------");
for(int i=0; i< urls.length; i++) {
    System.out.println(urls[i].getFile());
}
System.out.println("---------------------------------------");

Then ran it, in Eclipse and saved off the console output.

Then, I added a sleep call for 100 seconds in the same place in the code.  This enabled me to run the test again from the terminal and copy the project/target/surefire/ directory which contained the surefirebooter.jar.  Click here to read more about that project.

After copying that jar to a temporary directory, I unpacked it and then compared the versions of avro between the Eclipse classpath and the classpath from the terminal and noticed that they were different.  Inspecting the dependency tree of my project it was clear that 1.7.4 was part of the hadooop jars I was using.

Ultimately, I ended up updating my version of avro to 1.7.4 in my pom to eliminate the conflict.

Configuring Hidden, Invisible, or Whitespace Characters in The Eclipse Text Editor

Posted on by rchapin

The newer (I am currently using Mars, 4.5.0) versions of Eclipse provide very good tools for configuring the visibility of whitespace characters in code.

To customize your settings go to Window -> Preferences -> General -> Editors -> Text Editors.

On that page there will be checkbox option next to “Show whitespace characters (configure visibility).

Clicking on the ‘configure visibility’ link will allow you to choose what is shown and the opacity of the whitespace characters, which is a really nice touch.

© 2005 Ryan Chapin