recentpopularlog in

kme : unix   472

« earlier  
hard drive - Disk usage per user in Linux / Unix - Server Fault
<code class="language-bash">find . -type f -printf "%u %s\n" \
| awk '{user[$1]+=$2}; END{for(i in user) print i,user[i]}'</code>
diskusage  sysadmin  linux  unix  solution 
2 days ago by kme
find disk usage per user on a filesystem
<code class="language-bash">find / -printf "%u %s\n" \
| awk '{user[$1]+=$2}; END{ for( i in user) print i " " user[i]}'</code>
bash  unix  sysadmin  diskusage  quota  storage  solution 
2 days ago by kme
macos - Add a locale in Mac OSX - Stack Overflow
Looking into this found that, as of Mac OS X 10.10.3, collation is still broken for Spanish and most European languages. Collation definitions for these locales are linked to an ASCII definition. This ends up breaking things such as ORDER BY clauses on PostgreSQL.


Also, WTF is does 'la_LN' mean anyway?
macos  elcapitan  annoyance  sorting  brokenness  collation  lc_all  dateandtime  unix  maybesolution 
7 weeks ago by kme
Th Most Handy du (Disk Usage) Commands in Linux - Make Tech Easier
Honestly did not know about the '--threshold' or exclusion options.
sysadmin  diskspace  unix  linux  du  tipsandtricks 
7 weeks ago by kme
newlines - What's the point in adding a new line to the end of a file? - Unix & Linux Stack Exchange
Example: the output of GNU sort always ends with a newline. So if the file foo is missing its final newline, you'll find that sort foo | wc -c reports one more character than cat foo | wc -c.

Not necessarily the reason, but a practical consequence of files not ending with a new line:

Consider what would happen if you wanted to process several files using cat. For instance, if you wanted to find the word foo at the start of the line across 3 files:
<code class="language-bash">cat file1 file2 file3 | grep -e '^foo'</code>
newlineterminator  unix  textfiles  textprocessing  explained 
8 weeks ago by kme
stdout - How to make output of any shell command unbuffered? - Stack Overflow


Try stdbuf, included in GNU coreutils and thus virtually any Linux distro. This sets the buffer length for input, output and error to zero:
<code class="language-bash">stdbuf -i0 -o0 -e0 command</code>


This totally works for something like this:
<code class="language-bash">stdbuf -i0 -o0 -e0 find . -name 'core*' -printf '%f\t%s\n' | head</code>
bash  shellscripting  pipes  unix  buffering  solution 
9 weeks ago by kme
windows - How to fix PuTTY showing garbled characters? - Server Fault
The analogous fix on CentOS (7) is to use 'localectl', maybe, except that requires DBus to be running, which it isn't for a fresh-out-of-the-box Docker container. ¯\_(ツ)_/¯
If the locale returns something like POSIX, issue
<code class="language-bash">update-locale LANG=en_US.utf8</code>
at the command line - see thomas-krenn.com/de/wiki/Locales_unter_Ubuntu_konfigurieren – koppor Dec 19 '15 at 11:05
docker  utf8  characterencoding  locale  mojibake  terminal  unix  shell  ubuntu  solution  centos  sortof 
june 2019 by kme
umask - cmdline Unix Permissions bits calculator - Unix & Linux Stack Exchange
Useful for this tidbit alone:
<code class="language-bash">printf '%o\n' "$((value ^ 511))"</code>
unix  sysadmin  permissions  permissionbits  posixbits  bashscripting  calculator 
june 2019 by kme
Abundant Scripting Is Good | UNIX Scripting Tips and Ideas - Kimball Hawkins
I doubt there is any System Administrator out there who doesn't realize that scripting is good, but I've found there are few who actually use scripting as I feel it should be used -- as a way to programatically improve, simplify and automate many of the complex and tedious tasks of system monitoring and maintenance.…

My first rule is, if I have to do any task more than once, I’ll script it. This applies especially if errors in the command can have serious consequences.

My second rule is, if I’m going to script it, I’m going to make it robust enough to be useful in all applicable situations.

My third rule is I never hard-code specific information in a script if I can help it.

Fourth is: If a script can figure something out, don’t require the user to enter it, or choose it or find it.

Fifth is: Document it!
sysadmin  shellscripting  unix  linux  bash 
june 2019 by kme
yest download | SourceForge.net
Needs to be modified to support 'now', or 'today' as starting time/dates.
Download yest for free. This is a command line date/time manipulation and formatting program, very useful in scripts. You can easily add or subtract days, hours and/or minutes from a specified date.
timeanddate  unix  c  shellscripting  commandline  essential  movein  needshelp 
june 2019 by kme
Bash scripting cheatsheet
Has links to the Bash Hackers wiki, which are helpful.

Has a really good reference for array / associative array syntax, too!
bash  shellscripting  cli  unix  linux  cheatsheet  fuckina 
may 2019 by kme
unix - tar – extract discarding directory structure - Super User | https://superuser.com/
If you just want to remove a few path segments, then --strip-components=n or --strip=n will often do:
<code class="language-bash">
tar xvzf tgz --strip=1
</code>
unix  linux  sysadmin  tar  archive  backupandrecovery  solution 
may 2019 by kme
The Computer Chronicles - UNIX (1985) - YouTube | https://www.youtube.com/

Stewart Cheifet: ... Why is there sudden excitement about UNIX?

Gary Kildall: Well it shouldn't be a sudden excitement--UNIX itself has been around since the late 60s. The problem is that micros haven't had the power to support... they haven't had the large amount of main memory, the hard disk, the fast processor, and so forth, but nowadays micros do have that power, and so UNIX becomes a serious contender for an operating system standard.

OK, in fact many people are saying UNIX become *the* standard operating system of the future, but there are *many* uses of UNIX going on right now. We have a report....
unix  history  computing  video 
may 2019 by kme
domain - Create Unix Named Socket from the Command Line - Server Fault | https://serverfault.com/
For supervisord, I thought I had to create the Unix domain socket, and I also thought that it was the same as a FIFO (named pipe). Nope. To both.
fifo  namedpipe  socket  unixdomainsocket  unix  solution 
may 2019 by kme
Intrepid command line directory traversal - BrettTerpstra.com | https://brettterpstra.com/
I use Terminal (well, iTerm 2) for file management on my Mac more often than I use Finder. Typing out long path names is often more tedious than drilling through Finder folders, though. I have enough
unix  shell  bash  commandline  filesystemnavigation  tipsandtricks 
april 2019 by kme
Example for renaming foo? does not work as described · karelzak/util-linux@ed21c47 | https://github.com/
Was so scratching my head about this, turned out it was fixed in util-linux 2.33.1, which my Linux distro just hadn't upgraded to yet.
manpage  unix  linux  utility  software  sysadmin  solution 
march 2019 by kme
permissions - How to chmod without /usr/bin/chmod? - Unix & Linux Stack Exchange | https://unix.stackexchange.com/
You can run the loader directly, and pass it the command you want to run:
<code class="language-bash">
/lib/ld-linux.so /bin/chmod +x /bin/chmod
</code>

Your path to the loader might vary. On a 64-bit system you need to choose the right one based on how chmod was compiled; the 64-bit version is named something like /lib64/ld-linux-x86-64.so.2

The chmod utility relies on the chmod() system call (see man 2 chmod). So you could do this with a few lines of C, or just about any other language that has a wrapper around it (which would be most of them). Very few *nix systems are going to lack a C compiler and a perl interpreter; most linux distros require the later to work.
<code class="language-bash">
perl -e 'chmod 0755, "foobar.file"'
</code>
linux  unix  sysadmin  chmod  tipsandtricks  disasterrecovery 
march 2019 by kme
history - Why is the xargs -i option deprecated? - Unix & Linux Stack Exchange | https://unix.stackexchange.com/
This makes me feel old, because I remember when 'xargs' *only* had '-i' and '-l', and this change feels completely arbitrary to me.
The -l and -i options appear in the 1997 version of the POSIX standard, but do not appear in the 2004 version of the standard. Therefore you should use -L and -I instead, respectively.
xargs  unix  shellscripting  posix  explained  solution 
march 2019 by kme
pipe - cryptsetup: Attaching loopback device failed - Unix & Linux Stack Exchange | https://unix.stackexchange.com/
It seems that cryptsetup requires the LUKS header to be either regular file or device. If you need to provide the LUKS header as an output from a process/stream, you can easily circumvent the restriction by sending it to /dev/ram
<code class="language-bash">
cat LUKS-HEADER > /dev/ram0
</code>

(provided that your kernel supports ramdisk)

Then you can then simply use your cryptsetup command as:
<code class="language-bash">
cryptsetup luksOpen /dev/sdb1 sdb1 --header /dev/ram0
</code>

Keep in mind, the LUKS header will stay in /dev/ram0 disk until you free up the space. To free up the memory, you can use the blockdev command:
<code class="language-bash">
blockdev -v --flushbufs /dev/ram0
</code>
luks  crypto  blockdevices  linux  unix  sysadmin  tipsandtricks 
march 2019 by kme
linux - how to kill the tty in unix - Stack Overflow | https://stackoverflow.com/
This was essentially what I needed
<code class="language-bash">ps -ft pts/6 -t pts/9 -t pts/10</code>

And my actual command line looked like this:
<code class="language-bash">
# probably could've done this with just 'who -al' instead of who + ps
who | grep userid \
| awk '{print $2}' \
| parallel 'ps -ft {} \
| tail -n +2' \
| awk '{print $2}'
| xargs kill -HUP</code>

And with some refinement, I was able to do something like this:
<code class="language-bash">
who -al | grep old \
| awk '{print $7}' \
| sudo xargs kill -HUP</code>

Also:
I had the same question as you but I wanted to kill the gnome terminal which I was in. I read the manual on "who" and found that you can list all of the sessions logged into your computer with the '-a' option and then the '-l' option prints the system login processes.
<code class="language-bash">who -la</code>

Also:
<code class="language-bash">pkill -9 -t pts/0</code>
unix  linux  sysadmin  pseudoterminal  processmanagement  solution 
march 2019 by kme
macos - Why CURL return and error (23) Failed writing body? - Stack Overflow | https://stackoverflow.com/

(For completeness and future searches) It 'a matter of how CURL manages the buffer, the buffer disables the output stream with the -N option.
<code class="language-bash">
curl -s -N "URL" | grep -q Welcome
</code>
unix  shellscripting  curl  pipes  errormessage  annoyance  solution 
march 2019 by kme
unix - Use GNU screen as login "shell" - Super User
<code class="language-bash">
# if $STY is not set...
if [ -z "$STY" ]; then
exec screen -ARR
fi
</code>
unix  linux  sysadmin  screen  login  loginshell  solution 
march 2019 by kme
The #! magic, details about the shebang/hash-bang mechanism | https://www.in-ulm.de/
what's special about #!

#! was a great hack to make scripts look and feel like real executable binaries.

But, as a little summary, what's special about #!? (list mostly courtesy of David Korn)

- the interpretername must not contain blanks
- the length of the #! is much smaller than the maximum path length
- $PATH is not searched for the interpreter
- (apart from an absolute path, the #! line also accepts a relative path,
- and #!interpreter is equivalent to #!./interpreter,
- however, it's not of any practical use)
- the interpreter usually must no be a #! script again
- the handling of arguments in the #! line itself is varying
- the setuid mechanism may or may not be available for the script
- there's no way to express #!$SHELL

And why shebang? In music, '#' means sharp. So just shorten #! to sharp-bang. Or it might be derived from "shell bang". All this probably under the influence of the american slang idiom "the whole shebang" (everything, the works, everything involved in what is under consideration). See also the wiktionary, jargon dictionary or Merriam-Websters. Sometimes it's also called hash-bang, pound-bang, sha-bang/shabang, hash-exclam, or hash-pling (british, isn't it?).

According to Dennis M. Ritchie (email answer to Alex North-Keys) it seems it had no name originally.
And Doug McIllroy mentioned (TUHS mailing list), that the slang for # at Bell Labs most probably was "sharp" at the time.
bourne  bash  posix  shell  shebang  unix  linux  shellscripting  history  butwhy  explained 
february 2019 by kme
swcarpentry/windows-installer: Software Carpentry installer for Windows. | https://github.com/
Software Carpentry installer for Windows. Contribute to swcarpentry/windows-installer development by creating an account on GitHub.
workshop  windows  unix  msysgit  commandline  bash  installation  installer  script  python 
february 2019 by kme
Man pages for Git Bash on Windows 7 - Super User | https://superuser.com/
Yeah, if you want man pages by default, it's actually more prudent to do a default install of Cygwin.

You /could/ install Python, then 'pip install tldr', but that's not a good solution for a workshop.
windows  bash  git  shell  unix  workshop  howto 
february 2019 by kme
The Unix Shell: Instructor Notes | https://swcarpentry.github.io/
Many people have questioned whether we should still teach the shell. After all, anyone who wants to rename several thousand data files can easily do so interactively in the Python interpreter, and anyone who’s doing serious data analysis is probably going to do most of their work inside the IPython Notebook or R Studio. So why teach the shell?

The first answer is, “Because so much else depends on it.” Installing software, configuring your default editor, and controlling remote machines frequently assume a basic familiarity with the shell, and with related ideas like standard input and output. Many tools also use its terminology (for example, the %ls and %cd magic commands in IPython).

The second answer is, “Because it’s an easy way to introduce some fundamental ideas about how to use computers.” As we teach people how to use the Unix shell, we teach them that they should get the computer to repeat things (via tab completion, ! followed by a command number, and for loops) rather than repeating things themselves. We also teach them to take things they’ve discovered they do frequently and save them for later re-use (via shell scripts), to give things sensible names, and to write a little bit of documentation (like comment at the top of shell scripts) to make their future selves’ lives better.

The third answer is, “Because it enables use of many domain-specific tools and compute resources researchers cannot access otherwise.” Familiarity with the shell is very useful for remote accessing machines, using high-performance computing infrastructure, and running new specialist tools in many disciplines. We do not teach HPC or domain-specific skills here but lay the groundwork for further development of these skills. In particular, understanding the syntax of commands, flags, and help systems is useful for domain specific tools and understanding the file system (and how to navigate it) is useful for remote access.

Finally, and perhaps most importantly, teaching people the shell lets us teach them to think about programming in terms of function composition. In the case of the shell, this takes the form of pipelines rather than nested function calls, but the core idea of “small pieces, loosely joined” is the same.

Installing Bash and a reasonable set of Unix commands on Windows always involves some fiddling and frustration. Please see the latest set of installation guidelines for advice, and try it out yourself before teaching a class.

Tab completion sounds like a small thing: it isn’t. Re-running old commands using !123 or !wc isn’t a small thing either, and neither are wildcard expansion and for loops. Each one is an opportunity to repeat one of the big ideas of Software Carpentry: if the computer can repeat it, some programmer somewhere will almost certainly have built some way for the computer to repeat it.

Building up a pipeline with four or five stages, then putting it in a shell script for re-use and calling that script inside a for loop, is a great opportunity to show how “seven plus or minus two” connects to programming. Once we have figured out how to do something moderately complicated, we make it re-usable and give it a name so that it only takes up one slot in working memory rather than several. It is also a good opportunity to talk about exploratory programming: rather than designing a program up front, we can do a few useful things and then retroactively decide which are worth encapsulating for future re-use.
shellscripting  unix  linux  shell  bash  butwhy  programming  sevenplusorminustwo  teaching  workshop  reference  advice  bestpractices 
february 2019 by kme
rgaiacs/swc-shell-split-window: Script to split the shell using tmux | https://github.com/
Script to split the shell using tmux. Contribute to rgaiacs/swc-shell-split-window development by creating an account on GitHub.
teaching  unix  shell  shellscripting  workshop  utility  software  tmux 
february 2019 by kme
GNU Parallel 20110205 - The FOSDEM Release - YouTube | https://www.youtube.com/
Using '--pipe'

Each block of 3 is sorted, but whole output is not sorted:
<code class="language-bash">
# when used with --pipe, '-N' is the number of records to read

seq 1 10 | shuf | parallel --pipe -N 3 sort -n
# result:
# 1
# 3
# 10
# 2
# 4
# 6
#⋮
</code>

Sort each group of records individually, output to files w/ '--files':
<code class="language-bash">
seq 1 10 \
| shuf \
| parallel --pipe --files -N 3 sort -n
# result:
# /tmp/sNGZtP6Kr8.par
# /tmp/q44fQdincg.par
# /tmp/CSVAPt5Ybe.par
</code>

Pipe through another parallel to merge pre-sorted files & clean up temps using '-mj1' is basically like 'xargs' (all arguments on one command line); with recent versions, may want to use '-X' instead.
<code class="language-bash">
seq 1 10 \
| shuf \
| parallel --pipe -N 3 sort -n \
| parallel -mj1 sort -nm {} ';' rm {}
</code>
gnuparallel  parallelism  sysadmin  unix  textprocessing  tipsandtricks  video 
february 2019 by kme
« earlier      
per page:    204080120160

Copy this bookmark:





to read