Using netcat to Mock a RESTful Webservice that Returns JSON

Let’s say that you are working on a part of a project that needs to consume some JSON data from a forthcoming HTTP service.  That sub-system is being developed by one of your colleagues and is not yet ready for you to stand-up to develop against.

You can use netcat to mock the webservice and return some static JSON data for which you can develop and test against with a simple one-liner.

First, put together your JSON in a file, → Continue reading “Using netcat to Mock a RESTful Webservice that Returns JSON”

How to Use Credentials That Contain Special Characters with curl

In order to execute curl commands to endpoints with passwords that contain special characters, the cleanest way that I have found to do so is to Base64 encode the authentication string for curl and then pass an Authorization request header along with the request.

In this example the credentials are uid ‘rchapin’ and passwd ‘abc123!@#’.  Normally we would pass this to curl as follows:

$ curl -u rchapin:abc123!@# -X GET https://some-endpoint:443

However, this will not work and the password will → Continue reading “How to Use Credentials That Contain Special Characters with curl”

Solution for MySQL ERROR 1396 (HY000): Operation CREATE USER failed for

This indicates that the user already exists, or did exist but that all of the data for that user has not been deleted.

As the mysql root user:

REVOKE ALL PRIVILEGES, GRANT OPTION FROM ‘user’@’hostname’;
DROP USER ‘user’@’hostname’

Then re-try creating the user. → Continue reading “Solution for MySQL ERROR 1396 (HY000): Operation CREATE USER failed for”

How to Configure a Linux Client for Active Directory Authentication

I am currently working on setting up multiple environments for a new project (DEV, QA, and PROD) and will be integrating all of the servers to an Active Directory Domain Controller for user management.

Following are notes from when I configured a Fedora Core 18 laptop to integrate with an AD server.  It is likely things have changed some from then and I will update this as required for a current version of RHEL7/CentOS 7.

  Run the following commands as → Continue reading “How to Configure a Linux Client for Active Directory Authentication”

[SOLVED] Delete key not working with x2go server running XFCE

I am working with a development environment whereby I VPN into an environment with an Ubuntu VirtualBox guest and then initiate an x2go client session to a developement workstation server that lives in the development environment.

I had everythining working exactly as I wanted except that the Delete key did not work in the x2go session.

Previously the xev output when pressing the delete key was:

FocusOut event, serial 36, synthetic NO, window 0x2e00001,
    mode NotifyGrab, detail NotifyAncestor

FocusIn event, → Continue reading “[SOLVED] Delete key not working with x2go server running XFCE”

My Git Cheat Sheet

Some standard configs:

git config –global user.name “Ryan Chapin”
git config –global user.email “rchapin@nbinteractive.com”
git config –global core.editor vim

Ignore files in your repo:

.gitignore files contain patterns for files or directories to ignore

You can also set up a global .gitignore file for all Git repos via the core.excludesfile setting
$ git config –global core.excludesfile ~/path/to/some/.gitignore

The local .gitignore can be committed to a repo, the global one is only locally visible

Empty dirs need to have some file → Continue reading “My Git Cheat Sheet”

Looping Through a List of Files with Spaces in the File Name with Bash

If you have a list of files that you want to operate on in a loop in bash and some of them have spaces in the file name the default IFS (Internal Field Separator) will match with the space and tokenize the file.

The simple approach is to temporarily set the IFS as follows.  This can be done in a shell script, but the following example is directly on the command line for ‘one-liner’ usage.

$ OIFS=”$IFS”

$ IFS=$’\n’

$ Continue reading “Looping Through a List of Files with Spaces in the File Name with Bash”

How To Remove the Byte Order Mark (BOM) from UTF-8 Encoded Text Files

The easiest way that I have seen so far for doing so is to use tail and simply read everything except the first three bytes (start reading at the 4th byte), as follows:

$ tail –bytes=+4 text_file.txt > text_file-wo-bom.txtContinue reading “How To Remove the Byte Order Mark (BOM) from UTF-8 Encoded Text Files”

[SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce

I am working on a project and have decided to use Avro for the data serialization format.

I encountered the following error when trying to set up the unit test to test the mapper implementation through Eclipse:

java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
    at org.apache.avro.hadoop.io.AvroSerialization.getSerializer(AvroSerialization.java:114)
    at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:82)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:98)
    at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:111)
    at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:676)
    at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:680)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
    at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:130)
    at org.apache.hadoop.mrunit.MapDriverBase.addAll(MapDriverBase.java:141)
    at org.apache.hadoop.mrunit.MapDriverBase.withAll(MapDriverBase.java:247)
    at com.ryanchapin.hadoop.mapreduce.mrunit.UserDataSortTest.testMapper(UserDataSortTest.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
Continue reading “[SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce”

Configuring Hidden, Invisible, or Whitespace Characters in The Eclipse Text Editor

The newer (I am currently using Mars, 4.5.0) versions of Eclipse provide very good tools for configuring the visibility of whitespace characters in code.

To customize your settings go to Window -> Preferences -> General -> Editors -> Text Editors.

On that page there will be checkbox option next to “Show whitespace characters (configure visibility).

Clicking on the ‘configure visibility’ link will allow you to choose what is shown and the opacity of the whitespace characters, which is a really → Continue reading “Configuring Hidden, Invisible, or Whitespace Characters in The Eclipse Text Editor”