Computers – Page 4 – Ryan Chapin's Website

Diffing the output of two commands

Posted on January 26, 2023January 26, 2023 by rchapin

The GNU diff command on most Linux and UNIX systems will diff the contents of two files. With Bash, you can, using process substitution, take the output of any arbitrary command and process its input, or output, as a file descriptor. In this way, you can then use diff against the output of two commands as follows

diff <(cmd1) <(cmd2)

Both cmd1 and cmd2 will appear as a file name/file descriptor. The < character indicates that the file descriptor should be read to obtain the output.

Creating Custom Vagrant Boxes for Integration Testing

Posted on December 30, 2022March 17, 2023 by rchapin

For some things, a docker container will not work. For example, I am working on a Python project to automate the deployment and configuration of VMs and bare-metal boxes. It involves installing packages and configuring the OS and to properly test it requires a full fledged VM and not just a container.

Use packer to define the box you want to build

Install packer. The description straight from apt show packer is “HashiCorp Packer – A tool for creating identical machine images for multiple platforms from a single source configuration” Then create your pkr.hcl file (I am using the newer HCL DSL). The following is for a box that extends Debian 11.

variable "version" {
  type    = string
  default = ""
}

locals { timestamp = regex_replace(timestamp(), "[- TZ:]", "") }

source "vagrant" "xfce4_debian_11" {
  add_force    = true
  communicator = "ssh"
  provider     = "virtualbox"
  source_path  = "debian/bullseye64"
}

build {
  sources = ["source.vagrant.xfce4_debian_11"]

  provisioner "shell" {
    execute_command = "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'"
    script          = "provision.sh"
  }
}

Build the box

packer build xfce4_debian_11.hcl

This will create an output-xfce4_debian_11 directory and build the box writing it into that directory. The contents of which, when finished will be

output-xfce4_debian_11/
|-- package.box
`-- Vagrantfile

Create a Vagrant file that uses the pre-built box

For your integration testing you will be spinning up an “instance” of the box you just built.

Create a directory for your instance and cd into it: mkdir test-instance && cd test-instance
Initialize the vagrant instance and then add the box that you built; vagrant init test-instance && vagrant box add test-instance <path-to-dir>/output-xfce4_debian_11/package.box
Update the Vagrant file with any additional configs that you require and then run vagrant up to start it.

Using Environment Variables in a Vagrant File

Posted on December 30, 2022 by rchapin

An easy way to parameterize a Vagrant file is to use environment variables. You simply need to export a variable and then refer to it in the Vagrant file with the following syntax.

Export the variable:

export SSH_PORT=22222

Sample Vagrant file that reads the exported variable

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "debian/bullseye64"
  config.vm.box_version = "11.20221219.1"

  config.vm.network "forwarded_port", guest: 22, host: ENV['SSH_PORT']
end

Declaring, Exporting, and Reading Dynamic Variables in Bash

Posted on December 24, 2022December 25, 2022 by rchapin

If you want to dynamically define and export variable names in Bash here is the TLDR;

# Define the name of the variable
key="my-dynamic-var-name"

# Declare it and export it
declare -gx "$key"="some-value"

To then access that value via a dynamically generated variable name

# Create a variable that contains the variable name
var_to_access="my-dynamic-var-name"

# Read the value
my_var_value=${!var_to_access}

Read the man page for declare for more details and read this article for a really good explanation and further examples.

To read the declare man page

help declare

[SOLVED] ‘Virtualized Intel VT-x/EPT is not supported on this platform. Continue without virtualized Intel VT-x/EPT’ on Windows 11 host

Posted on November 5, 2022November 26, 2022 by rchapin

I was trying to run a Debian, Linux, guest on a Windows 11 Enterprise host with virtualization enabled for the VMWare Linux guest so that I could install minikube. Minikube requires running a VM on the host on which minikube is running, so essentially, a VM within a VM.

After enabling the “Virtualize Intel VT-x/EPT or AMD-V/RVI” setting and attempting to boot the VM it indicated that this configuration was not supported on this platform.

After quite a bit of digging I figured out the settings required for the Windows 11 host.

Disable Hyper-V
1. Open the Control Panel and search for “program”
2. Click on Turn Windows features on or off which will open another modal
3. Ensure that Hyper-V and all of its child check-boxes are de-selected
4. Ensure that Virtual Machine Platform is deselected
Disable Core Isolation/Memory integrity
- Open Settings and click on **Privacy & Security** in the left-hand nav
- Click on Windows Security
- Click on Device Security
- Click on Core isolation details under the Core isolation heading
- Toggle the Memory integrity setting to Off and restart your machine

How to check if a file is sourced in Bash

Posted on November 5, 2022 by rchapin

Sometimes you will want to ensure that a file is sourced instead of executed. This ensures, among other things, that any environment variables that the script defines remain in your current shell after the script completes.

To do so, use the following to check whether the file was sourced or run in a sub-shell

(return 0 2>/dev/null) && sourced=1 || sourced=0
echo "sourced=$sourced"

Bash allows return statements only from functions and in a scripts top level scope IF it is sourced. If you try to use a return statement outside of a function in a non-sourced context it returns an error.

Pruning directories from find

Posted on November 5, 2022 by rchapin

I have no idea why, but for some reason I always have a hard time remembering the exact syntax for find when I want to prune some list of directories from a search.

Let’s say that you want to execute a find in a directory where there are a lot of .git directories and you don’t want to search through the guts of the repo directories. With the following command we specify the prune predicate ahead of the search for any file that has ‘*.json’ in the file name.

find ./ -type f -iwholename '*.git' -prune -o -name '*.json' -print

Another way to do it is to exclude specific directories from a search. With the following command we first specify a set of directories to exclude from the search, by specific path and name, and then execute a search for the specific files.

find ./ -type d \( -path ./grpc-java -o -path ./go-in-mem-datastore \) -prune -o -name '*.json' -print

Using cut with a delimiter of any amount of whitespace

Posted on November 2, 2022November 5, 2022 by rchapin

The TLDR; is to first use tr to replace all occurrences of any horizontal whitespace character with a single space, and then squeeze down any number of spaces to a single space and then define the delimiter for cut as a single space. The following example assumes that you want to see from the 5th column to the end of the line.

<do-something-to-generate-input> | tr '[:blank:]' ' ' | tr -s ' ' | cut -d ' ' -f5-

The previous command will, after using -s to squeeze repeated spaces into one and then cut from the 5th field to the end of the line.

Running GUI apps locally as root in a non-root session

Posted on June 23, 2022June 23, 2022 by rchapin

There are instances when you need to run an X Window application. For me this is often running a terminator instance as root so that I can create tabs and split the window as still be root in each of those terminals.

In order for the root user to be able to connect to the X server you need to provide it with “credentials”. In this case it is on the same box and not over the network so the use of cookie authentication is acceptable.

As the user that is already authenticated to the X server run the following command to get the cookie used to connect to the the current $DISPLAY

rchapin@leviathan:~$ xauth list $DISPLAY
leviathan/unix:0  MIT-MAGIC-COOKIE-1  5fb2c0e68f4618ee4fa2202e1e4ae937

su to root and then run the following to add that cookie to roots authorization file

root@leviathan:~# xauth add leviathan/unix:0 MIT-MAGIC-COOKIE-1 5fb2c0e68f4618ee4fa2202e1e4ae937

You should now be able to run X windows applications from that root terminal.

VS Code “Test result not found for:” When Running Tests for a Python Project [SOLVED]

Posted on June 9, 2022 by rchapin

I finally was able to get Visual Studio Code set-up correctly to run and debug unit and integration tests for a Python 3.8 project that I am working on (I’ll add a link to that post here once it is up).

After making some changes to the code and adding a test I got the following error when trying to debug the test:

Test result not found for: ./mylibs/integration_tests/myclient_integration_test.py::MyClientIntegrationTest::test_happy_path

? An odd error message, to be sure.

After a little while I figured out that when this happens it is ultimately the result of some syntax, interpretation error that occurs at runtime that the IDE may not flag as a problem for you.

Check the Output panel and click on the drop-down and select Python Test Log to see the stack trace of the error to see where you have a typo.