If you have a list of files that you want to operate on in a loop in bash and some of them have spaces in the file name the default IFS (Internal Field Separator) will match with the space and tokenize the file.
The simple approach is to temporarily set the IFS as follows. This can be done in a shell script, but the following example is directly on the command line for ‘one-liner’ usage.
for i in
→ Continue reading “Looping Through a List of Files with Spaces in the File Name with Bash”
The easiest way that I have seen so far for doing so is to use tail and simply read everything except the first three bytes (start reading at the 4th byte), as follows:
tail --bytes=+4 text_file.txt text_file-wo-bom.txt
→ Continue reading “How To Remove the Byte Order Mark (BOM) from UTF-8 Encoded Text Files”
Following is *nix a command that you can use to add the contents of a source text file to the start of another text file (the source file).
echo -e '0r <source_file_name\nw' | ed -s <target_file_name
→ Continue reading “Adding the Contents of a Source File to the Beginning of a Target File”
If you want to extract from the nth token to the end of the line, following is how you can do that with awk:
Given a source file with the following:
line1 -- 01 0011 1
line2 -- 01 0011 2
line3 -- 01 0011 3
line4 -- 01 0011 4
line5 -- 01 0011 5
line6 -- 01 0011 6
line7 -- 01 0011 7
line8 -- 01 0011 8
line9 -- 01 0011 9
line10 -- 01 0011
→ Continue reading “Use awk to Print from nth element to the End of the Line”
The following is a one-liner for generating a random string of a fixed size in bash, where the possible characters to use in the string are any digit, letter, and a newline.
By adding the newline, you are fairly sure to prevent getting one long line of text.
< /dev/urandom tr -dc "[:digit:][:alpha:][\n]" | head -c1000 file.out
→ Continue reading “Generate a Random String of a Specified Size with a Shell Script”
Let’s say you have some program that is generating output to STDOUT and you want to write a script to read that output from STDIN and use it as a variable in your script.
To do so:
echo "SOME_VAR = $SOME_VAR"
→ Continue reading “Writing a BASH Script to Read from STDIN to a Variable”
If you have text files created under DOS/Windows and need to convert the CRLF (carriage return and line feed) characters to LF (line feed) character, here is a quick one-liner.
cat file.txt | perl -ne 's/\x0D\x0A/\x0A/g; print' file.txt.mod
You can also use dos2unix, however, especially under Cygwin I have seen dos2unix fail without giving any meaningful information about why it was unable to complete the task. In that case, you can just do it by hand. → Continue reading “One-Liner for Converting CRLF to LF in Text Files”
When writing utility scripts in Bash it is tempting to simply pass positional arguments, use $1, $2, etc. and be done with it. However if you want to either share this utility with other members of your team and/or incorporate it into your system, it makes sense to implement your command line argument parsing in a more flexible and maintainable manner.
Using getopt you can very easily pass a variety of command line options and arguments.
Following is a link → Continue reading “Parsing Command Line Arguments with getopt in Bash”
If you want to pass an array of items to a bash function, the simple answer is that you need to pass the expanded values. That means that you can pass the data as a quoted value, assuming that the elements are whitespace delimited, or you can pass it as a string and then split it using an updated IFS (Internal Field Separator) inside the function.
Following is an example of taking the output of a Hive query (a single → Continue reading “Passing an Array as an Argument to a Bash Function”
Let’s say that you have some number of files for which you want to create a containing directory that is named with all but the last token of the file name, and you want to remove just the last token to create the name of the directory.
Much easier to explain with an example. Given this list of files:
You want to create a directory for each → Continue reading “Removing the Last Token From a String in Bash with awk”