recentpopularlog in

kme : shellscripting   318

« earlier  
Programmable Completion (Bash Reference Manual)
Finally, any prefix and suffix specified with the -P and -S options are added to each member of the completion list, and the result is returned to the Readline completion code as the list of possible completions.

bash  completion  tabcompletion  programmablecompletion  shellscripting  solution 
8 days ago by kme
An introduction to bash completion: part 2
I created a script and I use bash_completion, but I cannot figure out how to allow the completion of "--name=value1 --name=value2" (the "=" sign in the middle stop any other completion, and I tried to play with suffixes/prefixes without any success :s

This works:

<code class="language-bash">case "$cur" in
COMPREPLY=( $(compgen -W "${_tables[*]}" -P "-t" \
-- "$cur" ) )
return 0
COMPREPLY=( $(compgen -W "${_tables[*]}" -P "--table=" \
-- "$cur" ) )
COMPREPLY=( $(compgen -W "${_opts[*]}" -- "$cur") )
bash  programmablecompletion  tabcompletion  shellscripting  fuckina  solution 
8 days ago by kme
bash - Wildcard expansion (globbing) in a string composed of quoted and unquoted parts - Stack Overflow
I had a variable that I was doing a regex comparison on within a Bash '[[ ]]' conditional that could /sometimes/ have the value of '*'; quoting didn't seem to help, but 'set -f' did.
bash  shellscripting  globbing  fileglobs  globs  workaround  solution 
9 days ago by kme
stdout - How to make output of any shell command unbuffered? - Stack Overflow

Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:
<code class="language-bash">stdbuf -i0 -o0 -e0 command</code>

This totally works for something like this:
<code class="language-bash">stdbuf -i0 -o0 -e0 find . -name 'core*' -printf '%f\t%s\n' | head</code>
bash  shellscripting  pipes  unix  buffering  solution 
6 weeks ago by kme
Bash test: get the directory of a script
Dude made the mistake of assuming the whole world was Linux.
whereami  bash  shellscripting  tipsandtricks  tests  interesting 
8 weeks ago by kme
Get the source directory of a Bash script from within the script itself - Stack Overflow
Lots of conflicting opinions here, but here are two solid options:
<code class="language-bash">
DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )
DIR=$( dirname "$(readlink -f "$0")" )
bash  shellscripting  newbie  whereami  tipsandtricks 
8 weeks ago by kme
linux - Bash arrays and negative subscripts, yes or no? - Stack Overflow
This is *way* easier than stuff I was trying to do before. The reason this doesn't show up in the help for "Arrays" in the bash manual is because it's a type of "Shell Parameter Expansion" that just happens to behave the way it does when the subscripted thing is an array.
If you just want the last element
<code class="language-bash">$ echo ${muh[*]: -1}

If you want next to last element
<code class="language-bash">$ echo ${muh[*]: -2:1}
bash  shellscripting  arrays  subscripting  syntax  fuckina  solution 
8 weeks ago by kme
linux - bashdb startup error: bashdb/lib/ line 91: /dev/pts/2: Permission denied - Stack Overflow -
Got this error message in a CentOS 7 Docker image, added user to the 'tty' group with 'vigr', then also had to 'chmod g+r' /dev/pts/0.
bash  shellscripting  bashdb  errormessage  maybesolution 
9 weeks ago by kme
bash - unshift args after calling shift 1 - Unix & Linux Stack Exchange
You never need to use shift 1 in the first place. Just use the positional arguments and slice around their indices to pass the arguments.
<code class="language-bash">first_arg="$1"</code>

Once you do this, the rest of the arguments can be accessed as "${@:2}". The notation is a way to represent from positional argument 2 to till the end of the list.
bash  shellscripting  positionalarguments  parameters  solution 
9 weeks ago by kme
Force flushing of output to a file while bash script is still running - Stack Overflow
I found a solution to this here. Using the OP's example you basically run
<code class="language-bash">stdbuf -oL /homedir/MyScript &> some_log.log</code>

and then the buffer gets flushed after each line of output. I often combine this with nohup to run long jobs on a remote machine.
<code class="language-bash">stdbuf -oL nohup /homedir/MyScript &> some_log.log</code>

This way your process doesn't get cancelled when you log out.
bash  shellscripting  buffering  annoyance  workaround  solution 
11 weeks ago by kme
Arithmetic expressions [Bash Hackers Wiki]
Should note that return value is always decimal, and that you can use 'printf' to format the result in another base, e.g.,
<code class="language-bash">printf "%o\n" $(( 0777 & 0066 ))
# result: 66</code>
bash  expansion  arithmeticexpansion  math  needshelp  shellscripting 
11 weeks ago by kme
bash - REOPEN: Perform arithmetic expansion inside parameter expansion? - Unix & Linux Stack Exchange
Again with the 'set' trick for testing script arguments interactively:
<code class="language-bash">set 6 7; echo $(($1*$2))</code>
bash  parameterexpansion  arithmeticexpansion  shellscripting  parameters  positionalarguments  idiom  tipsandtricks 
11 weeks ago by kme
linux - bash + arithmetic calculation with bash - Unix & Linux Stack Exchange
Since arithmetic expansion only does integer division, the proposed answer uses AWK, passing in values as variables, with '/dev/null' as the input file, and the math in a 'BEGIN' block.
bash  awk  shellscripting  floatingpoint  math  tipsandtricks  solution 
11 weeks ago by kme
Builtin Bash any base to decimal conversion – Phoxis
Bash has an interesting builtin feature to convert from any base to decimal, which is a part of bash's arithmetic evaluation features. In this post i will quickly introduce you with this feature. Arithmetic Expansion To allow arithmetic evaluation of an expression and substitution of the result bash has the following construct. For example The…
bash  shellscripting  math  calculator  numericconversion  base64  base8  base10  tipsandricks 
11 weeks ago by kme
Abundant Scripting Is Good | UNIX Scripting Tips and Ideas - Kimball Hawkins
I doubt there is any System Administrator out there who doesn't realize that scripting is good, but I've found there are few who actually use scripting as I feel it should be used -- as a way to programatically improve, simplify and automate many of the complex and tedious tasks of system monitoring and maintenance.…

My first rule is, if I have to do any task more than once, I’ll script it. This applies especially if errors in the command can have serious consequences.

My second rule is, if I’m going to script it, I’m going to make it robust enough to be useful in all applicable situations.

My third rule is I never hard-code specific information in a script if I can help it.

Fourth is: If a script can figure something out, don’t require the user to enter it, or choose it or find it.

Fifth is: Document it!
sysadmin  shellscripting  unix  linux  bash 
12 weeks ago by kme
yest download |
Needs to be modified to support 'now', or 'today' as starting time/dates.
Download yest for free. This is a command line date/time manipulation and formatting program, very useful in scripts. You can easily add or subtract days, hours and/or minutes from a specified date.
timeanddate  unix  c  shellscripting  commandline  essential  movein  needshelp 
12 weeks ago by kme
Bash scripting cheatsheet
Has links to the Bash Hackers wiki, which are helpful.

Has a really good reference for array / associative array syntax, too!
bash  shellscripting  cli  unix  linux  cheatsheet  fuckina 
12 weeks ago by kme
bash - LINES and COLUMNS environmental variables lost in a script - Stack Overflow
<code class="language-bash">lines=$(tput lines)
columns=$(tput cols)</code>

More context:
Often, on modern systems, the $COLUMNS and $LINES variables are not environment variables. The shell sets these values dynamically after each command and we usually cannot access them from non-interactive scripts. Some programs respect these values if we export them, but this behavior isn't standardized or universally supported.

Instead, the stty and tput utilities provide portable means to determine the terminal size from a script (the commands described below are currently undergoing standardization for POSIX)
bash  shellscripting  posix  tput  terminal  solution 
may 2019 by kme
bash - Pad one-digit, two-digit and three-digit numbers with zeros with sed - Stack Overflow
<code class="language-bash">$ sed -E 's/([[:digit:]]+)/000&/g;s/0+([[:digit:]]{4})/\1/g' file.txt</code>
sed  textprocessing  shellscripting  solution 
may 2019 by kme
Bash Reference Manual |
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
bash  shell  shellscripting  ansicquoting  quoting  dammitbrain  reference 
may 2019 by kme
command line - sed weirdness, unmatched { - Super User |
The 'e' GNU extension (execute process) also falls into this category: it must have a newline right after it; try using the ANSI-C quoting ($'string') feature of Bash to substitute in a literal newline.
sed  shellscripting  solution  syntax 
may 2019 by kme
How to setup trap in bash functions? - Unix & Linux Stack Exchange
Accepted answer suggests making function asynchronous (put it in the background?) and using 'wait'. I do not understand this.
Protip: Send the foreground process a QUIT with Ctrl-\ when Ctrl-C, -Z, -D aren't working.

Clear traps at the end of execution with trap - list of signals here

Fun on a non-production system: Try setting a trap for all sensible signals (e.g., not KILL). kill -l (not that's a lower-case L not a number 1 or pipe |)
bash  shellscripting  signals  processmanagement  tipsandtricks  reference 
april 2019 by kme
interrupt handling - How to trap ERR when using 'set -e' in Bash - Stack Overflow |
This wasn't my original question, which was why was "trap 'cleanup; exit 1' 1 2 15" not working, but the answer is you have to trap 'ERR', too.
If you want to combine set -e (same as: set -o errexit) with an ERR trap, also use set -o errtrace (same as: set -E).

In short: use set -eE in lieu of just set -e:
<code class="language-bash">

set -eE # same as: `set -o errexit -o errtrace`
trap "echo BOO!" ERR

function func(){
ls /root/

bash  signals  trap  shellscripting  solution  reference 
april 2019 by kme
Sed command returning "invalid command code" - Stack Overflow
I had this error on macOS and the problem was that '-i' expects a (zero-length) string argument. But this is a weird-ass way of telling you that.
macos  bsd  sed  shellscripting  annoyance  erromessage  solution 
march 2019 by kme
use of alternation "|" in sed's regex - Super User |
<code style="language-bash">echo "blia blib bou blf" | sed 's/bl\(ia\|f\)//g'</code>
For anyone else confused by this answer \| only works in gnu sed (gsed on os x) not vanilla sed (sed on os x). – Andrew Hancox Apr 4 '12 at 14:54
sed  shellscripting  textprocessing  syntax  newbie  dammitbrain  linuxonly  solution 
march 2019 by kme
linux - How to use sed to remove the last n lines of a file - Stack Overflow |
Yeah, it's possible in 'sed', but ugly.

<code class="language-bash">head -n -2 myfile.txt</code>
bash  linux  sed  shellscripting  textprocessing  solution 
march 2019 by kme
history - Why is the xargs -i option deprecated? - Unix & Linux Stack Exchange |
This makes me feel old, because I remember when 'xargs' *only* had '-i' and '-l', and this change feels completely arbitrary to me.
The -l and -i options appear in the 1997 version of the POSIX standard, but do not appear in the 2004 version of the standard. Therefore you should use -L and -I instead, respectively.
xargs  unix  shellscripting  posix  explained  solution 
march 2019 by kme
linux - Why redirect stdin inside a while read loop in bash? - Stack Overflow |
The clear intent here is to prevent do_something from reading from the sample.text stream, by ensuring that its stdin is coming from elsewhere. If you're not seeing differences in behavior with or without the redirection, that's because do_something isn't actually reading from stdin in your tests.


By the way, I would write this more as follows:
<code class="language-bash">exec 3</dev/tty || exec 3<&0 ## make FD 3 point to the TTY or stdin (as fallback)

while read -a args; do ## |- loop over lines read from FD 0
do_something "${args[@]}" <&3 ## |- run do_something with its stdin copied from FD 3
done <sample.text ## \-> ...while the loop is run with sample.txt on FD 0

exec 3<&- ## close FD 3 when done.</code>

It's a little more verbose, needing to explicitly close FD 3, but it means that our code is no longer broken if we're run with stdout attached to the write-only side of a FIFO (or any other write-only interface) rather than directly to a TTY.
bash  shellscripting  stdin  filedescriptors  epiphany  tipsandtricks  solution 
march 2019 by kme
macos - Why CURL return and error (23) Failed writing body? - Stack Overflow |

(For completeness and future searches) It 'a matter of how CURL manages the buffer, the buffer disables the output stream with the -N option.
<code class="language-bash">
curl -s -N "URL" | grep -q Welcome
unix  shellscripting  curl  pipes  errormessage  annoyance  solution 
march 2019 by kme
posix - How to check if stdin is from the terminal or a pipe in a shell script? - Stack Overflow |
<code class="language-bash">#!/bin/sh
if [ -t 0 ]; then
echo running interactivelly
while read -r line ; do
echo $line
bash  shellscripting  stdin  dammitbrain  solution 
march 2019 by kme
bash - How to add a carriage return with sed? - Stack Overflow |
In this case, this guy was right.
sed is for simple subsitutions on individual lines, that is all. For anything else you should be using awk:
shellscripting  sed  awk  textprocessing  whitespace  sortof  solution 
march 2019 by kme
How can I join elements of an array in Bash? - Stack Overflow |
Other, more complicated solutions (multiple-letter separators) require 'printf' and have their own problems.

<code class="language-bash">
IFS=, eval 'joined="${foo[*]}"'
bash  arrays  shellscripting  syntax  solution  likepython 
march 2019 by kme
The #! magic, details about the shebang/hash-bang mechanism |
what's special about #!

#! was a great hack to make scripts look and feel like real executable binaries.

But, as a little summary, what's special about #!? (list mostly courtesy of David Korn)

- the interpretername must not contain blanks
- the length of the #! is much smaller than the maximum path length
- $PATH is not searched for the interpreter
- (apart from an absolute path, the #! line also accepts a relative path,
- and #!interpreter is equivalent to #!./interpreter,
- however, it's not of any practical use)
- the interpreter usually must no be a #! script again
- the handling of arguments in the #! line itself is varying
- the setuid mechanism may or may not be available for the script
- there's no way to express #!$SHELL

And why shebang? In music, '#' means sharp. So just shorten #! to sharp-bang. Or it might be derived from "shell bang". All this probably under the influence of the american slang idiom "the whole shebang" (everything, the works, everything involved in what is under consideration). See also the wiktionary, jargon dictionary or Merriam-Websters. Sometimes it's also called hash-bang, pound-bang, sha-bang/shabang, hash-exclam, or hash-pling (british, isn't it?).

According to Dennis M. Ritchie (email answer to Alex North-Keys) it seems it had no name originally.
And Doug McIllroy mentioned (TUHS mailing list), that the slang for # at Bell Labs most probably was "sharp" at the time.
bourne  bash  posix  shell  shebang  unix  linux  shellscripting  history  butwhy  explained 
february 2019 by kme
The Unix Shell: Instructor Notes |
Many people have questioned whether we should still teach the shell. After all, anyone who wants to rename several thousand data files can easily do so interactively in the Python interpreter, and anyone who’s doing serious data analysis is probably going to do most of their work inside the IPython Notebook or R Studio. So why teach the shell?

The first answer is, “Because so much else depends on it.” Installing software, configuring your default editor, and controlling remote machines frequently assume a basic familiarity with the shell, and with related ideas like standard input and output. Many tools also use its terminology (for example, the %ls and %cd magic commands in IPython).

The second answer is, “Because it’s an easy way to introduce some fundamental ideas about how to use computers.” As we teach people how to use the Unix shell, we teach them that they should get the computer to repeat things (via tab completion, ! followed by a command number, and for loops) rather than repeating things themselves. We also teach them to take things they’ve discovered they do frequently and save them for later re-use (via shell scripts), to give things sensible names, and to write a little bit of documentation (like comment at the top of shell scripts) to make their future selves’ lives better.

The third answer is, “Because it enables use of many domain-specific tools and compute resources researchers cannot access otherwise.” Familiarity with the shell is very useful for remote accessing machines, using high-performance computing infrastructure, and running new specialist tools in many disciplines. We do not teach HPC or domain-specific skills here but lay the groundwork for further development of these skills. In particular, understanding the syntax of commands, flags, and help systems is useful for domain specific tools and understanding the file system (and how to navigate it) is useful for remote access.

Finally, and perhaps most importantly, teaching people the shell lets us teach them to think about programming in terms of function composition. In the case of the shell, this takes the form of pipelines rather than nested function calls, but the core idea of “small pieces, loosely joined” is the same.

Installing Bash and a reasonable set of Unix commands on Windows always involves some fiddling and frustration. Please see the latest set of installation guidelines for advice, and try it out yourself before teaching a class.

Tab completion sounds like a small thing: it isn’t. Re-running old commands using !123 or !wc isn’t a small thing either, and neither are wildcard expansion and for loops. Each one is an opportunity to repeat one of the big ideas of Software Carpentry: if the computer can repeat it, some programmer somewhere will almost certainly have built some way for the computer to repeat it.

Building up a pipeline with four or five stages, then putting it in a shell script for re-use and calling that script inside a for loop, is a great opportunity to show how “seven plus or minus two” connects to programming. Once we have figured out how to do something moderately complicated, we make it re-usable and give it a name so that it only takes up one slot in working memory rather than several. It is also a good opportunity to talk about exploratory programming: rather than designing a program up front, we can do a few useful things and then retroactively decide which are worth encapsulating for future re-use.
shellscripting  unix  linux  shell  bash  butwhy  programming  sevenplusorminustwo  teaching  workshop  reference  advice  bestpractices 
february 2019 by kme
rgaiacs/swc-shell-split-window: Script to split the shell using tmux |
Script to split the shell using tmux. Contribute to rgaiacs/swc-shell-split-window development by creating an account on GitHub.
teaching  unix  shell  shellscripting  workshop  utility  software  tmux 
february 2019 by kme
Hide curl output - Unix & Linux Stack Exchange |
<code class="language-bash">
curl --write-out '%{http_code}' --silent --output /dev/null
webdevel  webmaster  curl  http  statuscode  shellscripting  solution 
february 2019 by kme
GNU Parallel 20110522 ('Pakistan') - YouTube |
Replacing 'for' loops with GNU Parallel:

<code class="language-bash">
# generate reduced size, thumbnail at three different JPEG qualities
# replaces three nested 'for' loops
parallel convert -resize {1} -quality {2} {3} {3.}_{1}_q{2}.jpg ::: 800 30 ::: 25 50 95 ::: *.png
gnuparallel  parallelism  shellscripting  sysadmin  imageprocessing  video  tutorial 
february 2019 by kme
shell - Executing Python multi-line statements in the one-line command-line - Stack Overflow |
you could do
<code class="language-bash">
echo -e "import sys\nfor r in range(10): print 'rob'" | python

or w/out pipes:
<code class="language-bash">
python -c "exec(\"import sys\nfor r in range(10): print 'rob'\")"

<code class="language-bash">
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
python  shellscripting  oneliner  solution 
february 2019 by kme
bash - How can I prevent unsupported 'shopt' options from causing errors in my .bashrc? - Unix & Linux Stack Exchange |
But if you really dislike redirecting the error away, you can use the completion mechanism to perform introspection. This assumes that you don't have antiquated machines with bash ≤ 2.03 that didn't have programmable completion.
<code class="language-bash">
shopt_exists () {
compgen -A shopt -X \!"$1" "$1" >/dev/null
if shopt_exists direxpand; then
shopt -s direxpand

This method avoids forking, which is slow on some environments such as Cygwin. So does the straightforward 2>/dev/null, I don't think you can beat that on performance.
bash  configfile  shellscripting  shopt  programmablecompletion  tabcompletion  schooling  solution 
february 2019 by kme
Bash Builtins (Bash Reference Manual) |
Here's how you pretty-print arrays in Bash:
The -p option will display the attributes and values of each name. When -p is used with name arguments, additional options, other than -f and -F, are ignored.
bash  shellscripting  prettyprinting  arrays  solution 
february 2019 by kme
bats-core/bats-core: Bash Automated Testing System |
Bash Automated Testing System. Contribute to bats-core/bats-core development by creating an account on GitHub.
shellscripting  bash  testing  framework  unittesting  automation  devel  alternative  to  ts  bats 
january 2019 by kme
In AWK, is it possible to specify "ranges" of fields? - Stack Overflow |
The answer is "no"; AWK does not support ranges of subscripts like Perl or Python slices.
unix  shellscripting  awk  syntax  newbie  workaround  solution 
january 2019 by kme
« earlier      
per page:    204080120160

Copy this bookmark:

to read