If you want to just redirect STDERR to tee, run the following command.
some-command 2> >(tee /var/tmp/output)Category: Programming
Mocking Static Methods That Return void in Java
This is one of those things that I tend to do on a regular basis . . . but unfortunately don’t remember the details each time, so I am adding it for future reference.
Often, developers will want to mock static methods that return void. The Mockito and PowerMockito frameworks provide for this, but the syntax isn’t immediately obvious.
Following is an example.
public class SomeClass {
public static void doSomething(String arg1, int arg2) {
// Method that does something...
}
}import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.Mockito;
import org.powermock.api.mockito.PowerMockito;
import org.powermock.core.classloader.annotations.PowerMockIgnore;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;
/*
* The RunWith and PrepareForTest annotations are following annotations are
* necessary to mock the static methods in the SomeClass class. The RunWith
* enables the class to be run via PowerMock, and the PrepareForTest is an array
* of the classes with static members that we want to mock.
*
* The PowerMockIgnore annotation tells PowerMock to defer the loading of
* classes with the names supplied to the system classloader. This will vary
* depending on the dependency tree that you are using/testing. It is also
* not necessary, but here for example purposes.
*/
@RunWith(PowerMockRunner.class)
@PowerMockIgnore({
"javax.management.*",
"javax.net.ssl.*",
"org.apache.log4j.*"
})
@PrepareForTest({ SomeClass.class })
public class SomeTestClass {
@Test
public void shouldDoSomethingExpected() throws Exception {
// Set up the SomeClass's static members for mocking
PowerMockito.mockStatic(SomeClass.class);
// Configure the mock for the method in question.
// The following syntax is what is key here
PowerMockito.doNothing()
.when(
SomeClass.class,
"doSomething",
Mockito.anyString(),
Mockito.anyInt());
}
}
Solution for Executing Native Process from Java that Requires sudo
If you are building a Java program that requires the ability to execute native commands on the machine which require sudo it requires some additional considerations other than just writing the Java code.
The problem is that sudo, by default, requires a tty for executing sudo such that a password can entered. Even if you configure sudoers to grant NOPASSWD access to a specific command you will still get the following error
sudo: sorry, you must have a tty to run sudoIn my case, I was writing a set of integration tests in Java that needed to be able to start and stop a service to run a test.
I settled on adding an additional sudoers config file in /etc/sudoers.d. This ended up be the cleanest and most encapsulated change that did not then require any special considerations in the Java code.
The change simply involved adding a file with the following contents to /etc/sudoers.d which indicates that running sudo for the rchapin user does NOT require a tty and then grants access to the specific commands.
Defaults:rchapin !requiretty
rchapin ALL=(root) NOPASSWD: /bin/systemctl stop rabbitmq-server.service
rchapin ALL=(root) NOPASSWD: /bin/systemctl start rabbitmq-server.serviceDynamically Instantiating Classes in Python
A common design pattern is to define a class or list of classeses in configuration such that at runtime the classes can be dynamically instantiated.
I’ve done this before in many other languages and had need to do so today in Python (2.7.11)
It seems as though the clean way to do so is by using the Pyton importlib module. By using it, it enables you to cleanly dynamically import sub modules
Following is an example:
import importlib
klass_1_module = 'some.module.name'
klass_1_classname = 'KlassNameOne'
klass_2_module = 'klass2module'
klass_2_classname = 'KlassNameOne'
Klass1 = getattr(importlib.import_module(klass_1_module), klass_1_classname)
klass1_instance = Klass1()
Klass2 = getattr(importlib.import_module(klass_2_module), klass_2_classname)
klass2_instance = Klass1('some_arg')Java PowerMock Could not reconfigure JMX java.lang.LinkageError Solution
If you are using Mockito and PowerMock to build mocks for your Java tests and you run into the following error:
2016-05-05 17:31:20,204 main ERROR Could not reconfigure JMX java.lang.LinkageError: loader constraint violation: loader (instance of org/powermock/core/classloader/MockClassLoader) previously initiated loading for a different type with name "javax/management/MBeanServer"
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at org.powermock.core.classloader.MockClassLoader.loadUnmockedClass(MockClassLoader.java:238)
at org.powermock.core.classloader.MockClassLoader.loadModifiedClass(MockClassLoader.java:182)
at org.powermock.core.classloader.DeferSupportingClassLoader.loadClass(DeferSupportingClassLoader.java:70)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)Simply add the following as a class level annotation:
@PowerMockIgnore( {"javax.management.*"} )Looping Through a List of Files with Spaces in the File Name with Bash
If you have a list of files that you want to operate on in a loop in bash and some of them have spaces in the file name the default IFS (Internal Field Separator) will match with the space and tokenize the file.
The simple approach is to temporarily set the IFS as follows. This can be done in a shell script, but the following example is directly on the command line for ‘one-liner’ usage.
OIFS="$IFS"
IFS=$'\n'
for i in `find ./ -type f -iname '*some_criteria*'`; do "something with $i"; done
IFS="$OIFS"The previous commands will:
- Save the existing IFS
- Update the IFS to a newline char
- Execute your loop with the results of a find command
- Reset the IFS
How To Remove the Byte Order Mark (BOM) from UTF-8 Encoded Text Files
The easiest way that I have seen so far for doing so is to use tail and simply read everything except the first three bytes (start reading at the 4th byte), as follows:
tail --bytes=+4 text_file.txt > text_file-wo-bom.txt[SOLVED] java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter When Using Avro Data with MapReduce
I am working on a project and have decided to use Avro for the data serialization format.
I encountered the following error when trying to set up the unit test to test the mapper implementation through Eclipse:
java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter;
at org.apache.avro.hadoop.io.AvroSerialization.getSerializer(AvroSerialization.java:114)
at org.apache.hadoop.io.serializer.SerializationFactory.getSerializer(SerializationFactory.java:82)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:67)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:98)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:111)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:676)
at org.apache.hadoop.mrunit.TestDriver.copyPair(TestDriver.java:680)
at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:120)
at org.apache.hadoop.mrunit.MapDriverBase.addInput(MapDriverBase.java:130)
at org.apache.hadoop.mrunit.MapDriverBase.addAll(MapDriverBase.java:141)
at org.apache.hadoop.mrunit.MapDriverBase.withAll(MapDriverBase.java:247)
at com.ryanchapin.hadoop.mapreduce.mrunit.UserDataSortTest.testMapper(UserDataSortTest.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)After digging through the source code and finding that method did, infact, exist. I tried running the same unit test via the maven cli. It worked just fine.
After more digging, it turns out that what was happening was that the classpath in Eclipse was using avro-1.7.4 from the hadoop-common and hadoop-mapreduce-client-core jars in my project, and not the 1.7.7 version that I was trying to use.
To see what the difference between running it via the maven cli and running it in eclipse, I went through the following steps:
Added the following code to my test code to print out the classpath at runtime:
// Print out the classpath
ClassLoader sysClassLoader = ClassLoader.getSystemClassLoader();
URL[] urls = ((URLClassLoader)sysClassLoader).getURLs();
System.out.println("---------------------------------------");
for(int i=0; i< urls.length; i++) {
System.out.println(urls[i].getFile());
}
System.out.println("---------------------------------------");Then ran it, in Eclipse and saved off the console output.
Then, I added a sleep call for 100 seconds in the same place in the code. This enabled me to run the test again from the terminal and copy the project/target/surefire/ directory which contained the surefirebooter.jar. Click here to read more about that project.
After copying that jar to a temporary directory, I unpacked it and then compared the versions of avro between the Eclipse classpath and the classpath from the terminal and noticed that they were different. Inspecting the dependency tree of my project it was clear that 1.7.4 was part of the hadooop jars I was using.
Ultimately, I ended up updating my version of avro to 1.7.4 in my pom to eliminate the conflict.
Configuring Hidden, Invisible, or Whitespace Characters in The Eclipse Text Editor
The newer (I am currently using Mars, 4.5.0) versions of Eclipse provide very good tools for configuring the visibility of whitespace characters in code.
To customize your settings go to Window -> Preferences -> General -> Editors -> Text Editors.
On that page there will be checkbox option next to “Show whitespace characters (configure visibility).
Clicking on the ‘configure visibility’ link will allow you to choose what is shown and the opacity of the whitespace characters, which is a really nice touch.
Using the Eclipse Memory Analyzer (MAT) Remotely on Large Heap Dumps
Sometime your java applilcation will fail and generate an enormous heap dump. One that may be too large to be able to transfer to your local machine and to analyze for lack of RAM, time or both.
One solution is to install the MAT tool on the remote server and generate an HTML output of the analysis to download and view locally. This saves the headache of attempting to get X Windows installed on the remote machine and get all of the ssh tunneling sorted out (which is of course an option as well).
First, download and install the stand-alone Eclipse RCP Application. Then transfer to your server and unpack. Then determine how large the heap dump is and, if necessary, modify the MemoryAnalyzer.ini file to instantiate a JVM with enough RAM for your heap dump.
In this example, I have an 11GB heap dump and have modified the last two lines (adding -Xms)
-startup
plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.300.v20150602-1417
-vmargs
-Xmx16g
-Xms16gDo an initial run to parse the heap dump. This will generate intermediary data that can be used by subsequent runs to make future analysis faster.
./ParseHeapDump.sh /path/to/heap-dumpAfter that completes, you can run any of a number of different analysis on the data. The following is an illustration of how to search for memory leak suspects.
./ParseHeapDump.sh /path/to/heap-dump org.eclipse.mat.api:suspectsAdditional reports:
org.eclipse.mat.api:suspects
org.eclipse.mat.api:overview
org.eclipse.mat.api:top_componentsTo give creadit where it is due, this is basically a copy of a post by Ashwin Jayaprakash, but I wanted to capture it here as well.