Category: Bash

Use printf to join an array in Bash

Posted on by rchapin

If you would like to join an array of elements with a defined delimiter in Bash there is an easy way to go about it by using printf. Following is an example

#!/bin/bash

declare -a arr=()

for i in `seq 1 5`
do
  arr=("${arr[@]}" $i)
done

# Generate a single string joined by a comma.  The printf string can contain
# any arbitrary delimiter.
printf -v joined '%s,' "${arr[@]}"

# Print out the string minus the trailing comma
echo "${joined%,}"

Diffing the output of two commands

Posted on by rchapin

The GNU diff command on most Linux and UNIX systems will diff the contents of two files. With Bash, you can, using process substitution, take the output of any arbitrary command and process its input, or output, as a file descriptor. In this way, you can then use diff against the output of two commands as follows

diff <(cmd1) <(cmd2)

Both cmd1 and cmd2 will appear as a file name/file descriptor. The < character indicates that the file descriptor should be read to obtain the output.

Declaring, Exporting, and Reading Dynamic Variables in Bash

Posted on by rchapin

If you want to dynamically define and export variable names in Bash here is the TLDR;

# Define the name of the variable
key="my-dynamic-var-name"

# Declare it and export it
declare -gx "$key"="some-value"

To then access that value via a dynamically generated variable name

# Create a variable that contains the variable name
var_to_access="my-dynamic-var-name"

# Read the value
my_var_value=${!var_to_access}

Read the man page for declare for more details and read this article for a really good explanation and further examples.

To read the declare man page

help declare

How to check if a file is sourced in Bash

Posted on by rchapin

Sometimes you will want to ensure that a file is sourced instead of executed. This ensures, among other things, that any environment variables that the script defines remain in your current shell after the script completes.

To do so, use the following to check whether the file was sourced or run in a sub-shell

(return 0 2>/dev/null) && sourced=1 || sourced=0
echo "sourced=$sourced"

Bash allows return statements only from functions and in a scripts top level scope IF it is sourced. If you try to use a return statement outside of a function in a non-sourced context it returns an error.

Pruning directories from find

Posted on by rchapin

I have no idea why, but for some reason I always have a hard time remembering the exact syntax for find when I want to prune some list of directories from a search.

Let’s say that you want to execute a find in a directory where there are a lot of .git directories and you don’t want to search through the guts of the repo directories. With the following command we specify the prune predicate ahead of the search for any file that has ‘*.json’ in the file name.

find ./ -type f -iwholename '*.git' -prune -o -name '*.json' -print

Another way to do it is to exclude specific directories from a search. With the following command we first specify a set of directories to exclude from the search, by specific path and name, and then execute a search for the specific files.

find ./ -type d \( -path ./grpc-java -o -path ./go-in-mem-datastore \) -prune -o -name '*.json' -print

Using cut with a delimiter of any amount of whitespace

Posted on by rchapin

The TLDR; is to first use tr to replace all occurrences of any horizontal whitespace character with a single space, and then squeeze down any number of spaces to a single space and then define the delimiter for cut as a single space. The following example assumes that you want to see from the 5th column to the end of the line.

<do-something-to-generate-input> | tr '[:blank:]' ' ' | tr -s ' ' | cut -d ' ' -f5-

The previous command will, after using -s to squeeze repeated spaces into one and then cut from the 5th field to the end of the line.

Using fc to Edit and Re-execute Bash Commands

Posted on by rchapin

I recently learned about the Bash built-in fc. It is a great tool that enables you to edit and re-execute commands from your bash history.

Oftentimes there is a command in your history that instead of just grepping through the history and then re-executing as-is you’ll want to make a modification or two. With fc you can first edit it in your favorite editor and then when closing the editor fc will execute the command.

For me, vim is my editor of choice. Add the following to your .bashrc and fc will automatically open vim for you.

export FCEDIT=vim

Then, simply run fc passing it the id of the command in your history that you want to edit and then execute.

fc 1234

Flush Commands to BASH History Immediately

Posted on by rchapin

I cannot take credit for figuring this one out. Original post is here.

TLDR; is to add the following to your ~/.bashrc

export PROMPT_COMMAND='history -a'

Following are the history configs that I use

######################################################################
shopt -s histappend
HISTSIZE=-1
HISTFILESIZE=-1
HISTCONTROL=ignoreboth
HISTTIMEFORMAT="[%F %T] "
export PROMPT_COMMAND='history -a'
######################################################################

Configuring rsyslog to rotate log files from log messages streamed to it from a Systemd service

Posted on by rchapin

In general, I have moved to writing all of my applications to write their log output to STDOUT. This makes running them on the command line, in an IDE, on a bare metal box, VM, or in a container completely decoupled from how you store and view the logs. No more having multiple logging configs for each flavor of deployment.

In this particular case, I am running an application in a container (but it isn’t necessary that it is in a container) controlled by systemd and using rsyslog to forward all of the log messages to a specific output file. A requirement of writing log files to a local disk is that you must be able to rotate and truncate them by size so that you don’t fill up your disk; in either normal operation or some error condition that ends up inadvertently generating a large amount of log messages in a short period of time.

For the following example, we will us the service identifier my_program_identifier. You will update this to define something relevant to your deployment.

To configure your service in this manner you first need to add the appropriate options to the [Service] section of your unit file.

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=my_program_identifier

Then you define an rsyslog.d config file as follows for my_program.conf

$outchannel my_program_log_rotation,/var/log/my_program/my_program.log, 1073741824, /etc/my_program/log-rotate.sh

if $programname == "my_program_identifier" then :omfile:$my_program_log_rotation
& stop

In the rsyslog conf file we define an Output Channel. The TLDR; is that an output channel enables you to define the file name to which you want to write, the max size (in bytes) and a command (or path to a script or program) to run when the file reaches the limit.

In the previous example, we declare an output channel with the $outchannel directive. We then give it the identifier my_program_log_rotation. Then define the path of the log file, the max size, and a shell script that will run to rotate the file for us.

The next line defines how to act upon each of the log messages with the "my_program_identifier" that we defined in the unit file.

Following is a working sample of the log-rotate.sh script.

#!/bin/bash
  
LOG_DIR=/var/log/my_program
FILE_NAME=my_program.log
MAX_NUM_FILES=10

for i in `ls -1 $LOG_DIR/${FILE_NAME}.* | sort --field-separator=. -k3 -nr`
do
  # Grab the number (last token) for all of the numbered files
  log_num=$(echo "$i" | awk -F\. '{print $NF}')

  # If it is equal to or greater than our max number of files
  # just delete it.
  if [ "$log_num" -ge $MAX_NUM_FILES ]
  then
    rm $i
    continue
  fi

  target_num=$((log_num + 1))
  target_file_name="$LOG_DIR/${FILE_NAME}.${target_num}"
  mv -f $i $target_file_name
done

mv -f $LOG_DIR/$FILE_NAME $LOG_DIR/${FILE_NAME}.1

Deploy your updated unit file, your rsyslog.d conf file, and the shell script and you should have it up and running.

© 2005 Ryan Chapin