Category: Linux

Pruning directories from find

Posted on by rchapin

I have no idea why, but for some reason I always have a hard time remembering the exact syntax for find when I want to prune some list of directories from a search.

Let’s say that you want to execute a find in a directory where there are a lot of .git directories and you don’t want to search through the guts of the repo directories. With the following command we specify the prune predicate ahead of the search for any file that has ‘*.json’ in the file name.

find ./ -type f -iwholename '*.git' -prune -o -name '*.json' -print

Another way to do it is to exclude specific directories from a search. With the following command we first specify a set of directories to exclude from the search, by specific path and name, and then execute a search for the specific files.

find ./ -type d \( -path ./grpc-java -o -path ./go-in-mem-datastore \) -prune -o -name '*.json' -print

Using cut with a delimiter of any amount of whitespace

Posted on by rchapin

The TLDR; is to first use tr to replace all occurrences of any horizontal whitespace character with a single space, and then squeeze down any number of spaces to a single space and then define the delimiter for cut as a single space. The following example assumes that you want to see from the 5th column to the end of the line.

<do-something-to-generate-input> | tr '[:blank:]' ' ' | tr -s ' ' | cut -d ' ' -f5-

The previous command will, after using -s to squeeze repeated spaces into one and then cut from the 5th field to the end of the line.

Running GUI apps locally as root in a non-root session

Posted on by rchapin

There are instances when you need to run an X Window application. For me this is often running a terminator instance as root so that I can create tabs and split the window as still be root in each of those terminals.

In order for the root user to be able to connect to the X server you need to provide it with “credentials”. In this case it is on the same box and not over the network so the use of cookie authentication is acceptable.

As the user that is already authenticated to the X server run the following command to get the cookie used to connect to the the current $DISPLAY

rchapin@leviathan:~$ xauth list $DISPLAY
leviathan/unix:0  MIT-MAGIC-COOKIE-1  5fb2c0e68f4618ee4fa2202e1e4ae937

su to root and then run the following to add that cookie to roots authorization file

root@leviathan:~# xauth add leviathan/unix:0 MIT-MAGIC-COOKIE-1 5fb2c0e68f4618ee4fa2202e1e4ae937

You should now be able to run X windows applications from that root terminal.

Using fc to Edit and Re-execute Bash Commands

Posted on by rchapin

I recently learned about the Bash built-in fc. It is a great tool that enables you to edit and re-execute commands from your bash history.

Oftentimes there is a command in your history that instead of just grepping through the history and then re-executing as-is you’ll want to make a modification or two. With fc you can first edit it in your favorite editor and then when closing the editor fc will execute the command.

For me, vim is my editor of choice. Add the following to your .bashrc and fc will automatically open vim for you.

export FCEDIT=vim

Then, simply run fc passing it the id of the command in your history that you want to edit and then execute.

fc 1234

Configuring rsyslog to rotate log files from log messages streamed to it from a Systemd service

Posted on by rchapin

In general, I have moved to writing all of my applications to write their log output to STDOUT. This makes running them on the command line, in an IDE, on a bare metal box, VM, or in a container completely decoupled from how you store and view the logs. No more having multiple logging configs for each flavor of deployment.

In this particular case, I am running an application in a container (but it isn’t necessary that it is in a container) controlled by systemd and using rsyslog to forward all of the log messages to a specific output file. A requirement of writing log files to a local disk is that you must be able to rotate and truncate them by size so that you don’t fill up your disk; in either normal operation or some error condition that ends up inadvertently generating a large amount of log messages in a short period of time.

For the following example, we will us the service identifier my_program_identifier. You will update this to define something relevant to your deployment.

To configure your service in this manner you first need to add the appropriate options to the [Service] section of your unit file.

StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=my_program_identifier

Then you define an rsyslog.d config file as follows for my_program.conf

$outchannel my_program_log_rotation,/var/log/my_program/my_program.log, 1073741824, /etc/my_program/log-rotate.sh

if $programname == "my_program_identifier" then :omfile:$my_program_log_rotation
& stop

In the rsyslog conf file we define an Output Channel. The TLDR; is that an output channel enables you to define the file name to which you want to write, the max size (in bytes) and a command (or path to a script or program) to run when the file reaches the limit.

In the previous example, we declare an output channel with the $outchannel directive. We then give it the identifier my_program_log_rotation. Then define the path of the log file, the max size, and a shell script that will run to rotate the file for us.

The next line defines how to act upon each of the log messages with the "my_program_identifier" that we defined in the unit file.

Following is a working sample of the log-rotate.sh script.

#!/bin/bash
  
LOG_DIR=/var/log/my_program
FILE_NAME=my_program.log
MAX_NUM_FILES=10

for i in `ls -1 $LOG_DIR/${FILE_NAME}.* | sort --field-separator=. -k3 -nr`
do
  # Grab the number (last token) for all of the numbered files
  log_num=$(echo "$i" | awk -F\. '{print $NF}')

  # If it is equal to or greater than our max number of files
  # just delete it.
  if [ "$log_num" -ge $MAX_NUM_FILES ]
  then
    rm $i
    continue
  fi

  target_num=$((log_num + 1))
  target_file_name="$LOG_DIR/${FILE_NAME}.${target_num}"
  mv -f $i $target_file_name
done

mv -f $LOG_DIR/$FILE_NAME $LOG_DIR/${FILE_NAME}.1

Deploy your updated unit file, your rsyslog.d conf file, and the shell script and you should have it up and running.

How To Compile and Install New SELinux Plicy Modules

Posted on by rchapin

Following is a quick how-to on compiling and adding addition SELinux modules.

When configuring and deploying new and/or custom services on systems that are enforcing SELinux you will likely have to compile addition SELinux modules.

This how-to includes how to go through each step of compiling a new module one-by-one; similar to the model of breaking down the compilation of C and C++ into it’s composite steps.

Step 1:  Gather the audit.log entries

You will need to determine which action(s) that SELinux is blocking.  To do so, you can tail the /var/log/audit/audit.log file.  You will see something similar to the following

type=AVC msg=audit(1517605342.101:88032): avc:  denied  { write } for  pid=7236 comm="check_zookeeper" path="/tmp/sh-thd-1517587323" dev="dm-0" ino=308042 scontext=system_u:system_r:nrpe_t:s0 tcontext=system_u:object_r:tmp_t:s0 tclass=file
type=SYSCALL msg=audit(1517605342.101:88032): arch=c000003e syscall=2 success=no exit=-13 a0=1e2df10 a1=2c1 a2=180 a3=0 items=0 ppid=7232 pid=7236 auid=4294967295 uid=997 gid=994 euid=997 suid=997 fsuid=997 egid=994 sgid=994 fsgid=994 tty=(none) ses=4294967295 comm="check_zookeeper" exe="/usr/bin/bash" subj=system_u:system_r:nrpe_t:s0 key=(null)
type=PROCTITLE msg=audit(1517605342.101:88032): proctitle=2F62696E2F62617368002F7573722F6C6F63616C2F6E6167696F732F706C7567696E732F636865636B5F7A6F6F6B65657065722E7368002D2D73746174

Take that output and save it into a file.

Step 2: Generate the Type Enforcement (te) File From the Log Output

audit2allow -m new-module > new-module.te < audit-log-output

Step 3:  Check and Compile the SELinux Security Policy Module (mod) File From the .te File

checkmodule -M -m -o new-module.mod new-module.te

Step 4:  Create the SELinux Policy Module Packet (pp) File From the .mod File

semodule_package -o new-module.pp -m new-module.mod

Step 5:  Install the SELinux Policy Module

semodule -i new-module.pp

Adding a New Disk to a Linux Server and Creating an LVM Partition

Posted on by rchapin

There are a number of tutorials online for adding a new disk to a machine and then extending an existing LVM partition to use the new device.

This particular tutorial covers the use case of adding a new disk to a Linux server and then creating a NEW LVM partition on it without modifying the existing devices and LVM partitions.

The first thing you will need to do is add the physical device to the server (or VM).

Then, you need to confirm that the OS can ‘see’ the device.  The following command will show you the list of avaiable disk devices.

# fdisk -l

Disk /dev/sdb: 80.5 GB, 80530636800 bytes, 157286400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Here, we see that the OS can ‘see’ the /dev/sdb device.  For the rest of this tutorial, we will assume that your new device is /dev/sdb.

Using fdisk, create a primary partition on the new device

# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xc78ce5fd.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-157286399, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-157286399, default 157286399):
Using default value 157286399
Partition 1 of type Linux and of size 75 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

After partitioning re-run fdisk to list the partitions

# fdisk -l

Disk /dev/sdb: 80.5 GB, 80530636800 bytes, 157286400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xc78ce5fd

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   157286399    78642176   83  Linux

Now, create an LVM Physical Volume (PV)

# pvcreate /dev/sdb1
  Physical volume “/dev/sdb1” successfully created.

Create the LVM Volume Group (VG)

# vgcreate centos_repos /dev/sdb1
  Volume group “centos_repos” successfully created

Execute the vgdisplay command to list all of the Volume Groups

# vgdisplay

  — Volume group —
  VG Name               centos_repos
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               75.00 GiB
  PE Size               4.00 MiB
  Total PE              19199
  Alloc PE / Size       0 / 0   
  Free  PE / Size       19199 / 75.00 GiB
  VG UUID               FDgd3y-keqV-riq6-vb46-C2F5-JJa2-Ew2DW4

Create a LVM Logical Volume (LV).  In this case I am going to use the entire drive

# lvcreate -n repos –size 74.9G centos_repos
  Rounding up size to full physical extent 74.90 GiB
  Logical volume “repos” created.

lvdisplay will list all of the existing Logical Volumes

# lvdisplay
…
  — Logical volume —
  LV Path                /dev/centos_repos/repos
  LV Name                repos
  VG Name                centos_repos
  LV UUID                pvNLX4-3wTf-2eMY-RebF-WnFU-8y9F-BRidMn
  LV Write Access        read/write
  LV Creation host, time nebula, 2017-10-20 17:36:38 +0000
  LV Status              available
  # open                 0
  LV Size                74.90 GiB
  Current LE             19175
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:4

Now we need to format the LV.  In this case we will use ext4, you may choose another filesystem format.  Be sure to use the LV Path returned by lvdisplay.

# mkfs.ext4 /dev/centos_repos/repos
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4915200 inodes, 19635200 blocks
981760 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2168455168
600 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   

Now you can mount the file system as usual and/or add it to /etc/fstab.

© 2005 Ryan Chapin