recentpopularlog in

mechazoidal : unix   261

« earlier  
How are Unix pipes implemented?
"This important property, along with the POSIX requirement that writing up to PIPE_BUF (at least 512) bytes into a pipe must be atomic, allows processes to communicate with each other over a pipe in a way that regular files (which have no such guarantees) could not support. With a regular file, a process could write all its output and hand it over to the next process, or they could operate in lock-step by using an external signalling mechanism (like a semaphore) to let each other know when they were done reading or writing. Pipes eliminate all this complexity."
osdev  piperesearch  unix  history  programming 
5 days ago by mechazoidal
CS50 Programmer's Manual
Manual pages for the C standard library (and the CS50 Library), with student-friendly annotations.
unix  c  linux  programming  reference 
5 weeks ago by mechazoidal
pledge() and unveil() in SerenityOS – Andreas Kling – I like computers!
More for a nice simple explanation of how pledge() and unveil() work.

pmz: Note that these are unix-specific, in a plan9 system these would both be handled by custom namespaces
unix  osdev  pmz  openbsd 
9 weeks ago by mechazoidal
FGA: Readiness protocol problems with Unix daemons
Discussing how daemon processes can signify readiness to an overall system, and how a lot of systems get it wrong AND bad habits from the 80's.
unix  linux  systemd  osdev  reference 
10 weeks ago by mechazoidal
Unix, Plan 9, and Lurking Smalltalk
good lobsters discussion:

"meta-levelfacilities appear less susceptible to design-levelchurn than base-level language features, because they are one step closer to a relatively small set of recurring concepts (whose recurrence though Smalltalk, Unix and Plan 9 I have been documenting). These abstractions appear to sit comfortably as the "waist in the hourglass", supporting diverse surface forms for languages above,and running on diverse hardware-supported "big objects" below"

"Far from being replaced by an all-conquering programming language, one interpretation of the role of operating systems is as an infrastructure providing the mechanisms that allow languages to comeand go, while maximising the composability of the software written using them.Languages themselves may then become "views" onto a space of objects that is managed by the operating system, rather than by a per-language VM"
smalltalk  plan9  pmz  piperesearch  unix  osdev  pdf 
10 weeks ago by mechazoidal
kitty - the fast, featureful, GPU based terminal emulator — kitty 0.15.1 documentation
"The code in kitty is designed to be simple, modular and hackable. It is written in a mix of C (for performance sensitive parts) and Python (for easy hackability of the UI). It does not depend on any large and complex UI toolkit, using only OpenGL for rendering everything.

Finally, kitty is designed from the ground up to support all modern terminal features, such as unicode, true color, bold/italic fonts, text formatting, etc. It even extends existing text formatting escape codes, to add support for features not available elsewhere, such as colored and styled (curly) underlines. One of the design goals of kitty is to be easily extensible so that new features can be added in the future with relatively little effort."
terminal  tools  unix 
11 weeks ago by mechazoidal
Mitigations | Is OpenBSD secure?
as usual, with a grain of salt, etc.
pmz: some libraries/fixes are neat(arc4random, otto allocator, pledge, signify), albeit not all useful outside of a unix system
openbsd  unix  security  pmz  2019  osdev  programming 
december 2019 by mechazoidal
GitHub - geoffmeyers/interceptty
A github repo for the original Scott Gifford version:
"probably one of the most indispensable tools for dealing with serial data since we can easily replay or monitor transactions."
terminal  serial  modem  tools  security  unix  repo:github 
december 2019 by mechazoidal
How do you organize your $HOME directory? | Lobsters
"Instead of a t function, I have a ~/tmp folder. Because ~/tmp is persistent, I end up with stuff in there that I’m afraid to delete, so I made a ~/tmp/tmp for stuff that’s actually temporary."
unix  lobsters  discussion 
december 2019 by mechazoidal
PureDarwin | Moving the Darwin Community in the Right Direction!
The unofficial successor to OpenDarwin.
It uses XFree86, WindowMaker, and the XNU kernel: but all you really get is a bare-bones Unix. Without Cocoa or the other Apple proprietary stuff it's kinda :shrug:
darwin  osx  unix  os  apple 
november 2019 by mechazoidal
GitHub - mvdan/sh: A shell parser, formatter, and interpreter (sh/bash/mksh)
formatting/indenting/style tool for *sh. Use ShellCheck if you need a linter!
tools  bash  unix  shell  golang  repo:github 
november 2019 by mechazoidal
Creating sockets — Idea of the day
Some parts of this are Linux-specific(systemd, strace), but the AF_UNIX family is widly available(even on windows now!). Even some niche tips like "try calling connect(AF_UNSPEC) on a connected socket for faster reset"
linux  networking  reference  programming  unix 
november 2019 by mechazoidal
Pipecut Project Home
"Pipecut tries to facilitate the development of pipelines by letting you see your data and your shell commands at the same time, eliminating the back and forth editing cycle of entering and quitting more(or less), then recalling and editing the command line. Since pipecut has an AST view of the Unix command line, it can provide shortcuts, optimizations, and do code generation that would not be possible otherwise."
last updated 2015, see for a modern (accidental) reinvention
piperesearch  unix  pipes  tools  visualization  shell 
october 2019 by mechazoidal
poll vs select vs event-based
"I failed to find a thorough comparison page on poll() vs select() so I wrote my own. [...] Differences between poll() and select() and to the end some mentions about the more modern event-driven alternatives such as epoll(), kqueue and more. I recommend a library such as libev or libevent."
events  programming  concurrency  networking  reference  unix 
september 2019 by mechazoidal
Dream Deploys: Atomic, Zero-Downtime Deployments
Using unix tricks: calling accept(2) instead of bind or listen, and using the atomic operation of rename(2) to change symlinks
2015  webdev  sysadmin  hosting  unix 
september 2019 by mechazoidal
Kerberized NFSv4 (in a mixed-OS environment)
The various tribal knowledge for using Kerberos between unixes and windows. Lobsters comment notes that he's using wireguard at the low level:
networking  filesystems  sysadmin  devops  windows  unix  reference  2018  kerberos 
september 2019 by mechazoidal
sortix / Sortix · GitLab
"a small self-hosting operating-system aiming to be a clean and modern POSIX implementation. It is a hobbyist operating system written from scratch with its own base system, including kernel and standard library, as well as ports of third party software. It has a straightforward installer and can be developed under itself. Releases come with the source code in /src, ready for tinkering. "
pmz  unix  repo:gitlab  osdev  os 
september 2019 by mechazoidal
linux - How to make ffmpeg write its output to a named pipe - Stack Overflow
mkfifo <name>
use the "pipe:1" to output to stdout, and use "-f <format>"(else ffmpeg will not be able to guess the output format)
stackoverflow  tips  ffmpeg  unix 
august 2019 by mechazoidal
learn/simple-dlna-browser at master · javier-lopez/learn · GitHub
Simple example of browsing upnp/dlna AV media servers in a shell. Uses socat and curl for network traffic.
dlna  upnp  unix  shell  repo:github  reference 
august 2019 by mechazoidal
libxo - A Library for Generating Text, XML, JSON, and HTML Output — libxo 0.8.4 documentation
Contrast with nushell: "the library aims to minimize the cost of migrating code to libxo".
PR: maybe a lib to ease emitting PR messages(cbor/yang, structured text, etc)?
freebsd  library  piperesearch  unix 
august 2019 by mechazoidal
Phil Hagelberg / orb · GitLab
"an operating system designed for embedding in a game in order to facilitate learning programming and unix skills."
(now wondering if this is the "OS" he referred to about having multiple ones talking to polywell)
pmz: is it possible to do something like this for plan9/inferno systems, using lua? as a simulator and other things
lua  osdev  repo:gitlab  gamedev  pmz  os  unix 
august 2019 by mechazoidal
"It provides a lightweight audio & MIDI server and a fully documented user-space API to access either the server or directly the hardware in a uniform way. Sndio is designed to work for desktop applications, but pays special attention to synchronization mechanisms and reliability required by music applications."

HN comment: "'it doesn't look like sndio targets the application->application audio routing use-case, nor does it appear to prefer a lower latency audio pull-API. So, it does not look like by design it can be a jack replacement."
midi  audio  unix  openbsd  linux  API  pmz 
august 2019 by mechazoidal
GitHub - tomnomnom/gron: Make JSON greppable!
msn@lobsters: "turns tree-structured json into the line structured data that unix tools expect and allows you to search and edit it using familiar tools like sed, rather than having to learn jq’s mini-language (that I have to look up every time I need it)."
Golang  json  tools  unix  repo:github  jq 
july 2019 by mechazoidal
Slava Pestov on Twitter: "Emacs: text editor = programming language + operating systemSmalltalk: programming language = text editor + operating systemUnix: operating system = text editor + programming language…"
@slava_pestov: @fogus
Emacs: text editor = programming language + operating system
Smalltalk: programming language = text editor + operating system
Unix: operating system = text editor + programming language
osdev  unix  tweet  smalltalk  emacs 
july 2019 by mechazoidal
Live Free or Dichotomize - Using AWK and R to parse 25tb
Every generation, these techniques are rediscovered. Note how Apache Spark choked out!
datascience  aws  unix  awk  r_language 
june 2019 by mechazoidal
Rethinking files
"Arguably, an operating system lives and dies by the quality of its nexuses. The quality of Unix's filesystem as an integration and maintenance nexus makes it a pleasing system to work with, and has doubtless earned it much favour.

How do we square this with the view that “everything is a file” is perhaps a mistake? First, let's observe what actual utility the “everything is a file” concept gave us. It seems clear that what it gave us was a common set of verbs with which to operate on files."
("Nexuses" was the previous article)

InPermutation@lobsters: "The piece at the bottom about array support is the other important half of PowerShell. It supports full reflection on objects, which means you never have to grovel around with awk or tr just to get the data out of a column of output. Arrays, hashtables, and objects are all built in to the language."
piperesearch  pmz  plan9  unix  powershell  multipart 
june 2019 by mechazoidal
Afio archiver home page
"Afio makes cpio-format archives. It deals somewhat gracefully with input data corruption, supports multi-volume archives during interactive operation, and can make compressed archives that are much safer than compressed tar or cpio archives. Afio is best used as an `archive engine' in a backup script. "

Note that it's 30 years old, and in maintenance mode. Much of its esoteric/major uses are obsolete, but it may still be useful with some highly-specific needs. Note that you can perform the "tar pipe" trick with it for recursive copy of directory trees across a network.
cpio  archiving  unix  tools  reference  backup 
may 2019 by mechazoidal
Finn O'leary / zhmenu · GitLab
"an adaptable selective menu for the X Window System. The main utility of it is to provide a pinyin input method, however the database may be swapped out for other databases, and the run script altered to serve as a dmenu-like program."
x11  pmz  repo:gitlab  i18n  tools  unix 
may 2019 by mechazoidal
xdotool - fake keyboard/mouse input, window management, and more - semicomplete
"This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11’s XTEST extension and other Xlib functions.

Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops."
x11  tools  unix  linux 
may 2019 by mechazoidal
"a growing collection of the unix tools that nobody thought to write long ago when unix was young."
Note that it's a tad Linux-centric, and also includes a "parallel" tool which MAY conflict with GNU parallel in some packaging systems.
unix  tools  text  shell  linux 
april 2019 by mechazoidal
A fork() in the road | Lobsters
Good historical discussion. PR: note windows optimized for threads?
osdev  lobsters  discussion  unix  piperesearch  history 
april 2019 by mechazoidal
rdist(1) - when Ansible is too much — OpenBSD Amsterdam
"we wanted to show how we wrapped doas(1) around it."
Note that drist might be slightly better for your needs: "The reason people stopped using rdist and rsync et al took over was that it used to not support SSH or diffs. It appears it does now, but I last looked a long time ago."
sysadmin  openbsd  unix 
march 2019 by mechazoidal
Communication Files: Interprocess IO before Pipes (2017)
Noting that DTSS had a full file-*ish* IPC/comm interface way before plan 9(which gets a mention, noting that people copied the shallow parts but not the core idea)
He mentions dgsh but not CMS pipelines.
A note is that standard unix pipes don't have standardized ways of passing open IO connections outside of stdin/stdout(again, like CMS Pipelines?). Pipes and the pipe combinator were always linked on unix edict made them easier to learn.
paper  pdf  history  unix  plan9  pipes  ipc  piperesearch 
march 2019 by mechazoidal
Solene's percent % : Using haproxy for TLS layer
"This article explains how to use haproxy to add a TLS layer to any TCP protocol. This includes http or gopher. The following example explains the minimal setup required in order to make it work, haproxy has a lot of options and I won’t use them.

The idea is to let haproxy manage the TLS part and let your http server (or any daemon listening on TCP) replying within the wrapped connection."
tls  security  tips  haproxy  unix  sysadmin 
march 2019 by mechazoidal
Fixing Unix/Linux/POSIX Filenames: Control Characters (such as Newline), Leading Dashes, and Other Problems
"In general, kernels should emphasize mechanism not policy. The problem is that currently there’s no mechanism for enforcing *any* policy[on filename characters]. Yet it’s often easy for someone to create filenames that trigger file-processing errors in others’ programs (including system programs), leading to foul-ups and exploits. Let administrators determine policies like which bytes must never occur in filenames, which bytes must not be prefixes, which bytes must not be suffixes, and whether or not to enforce UTF-8. "

Also note the top on how to 'rm' weird files: "rm ./-r" or "rm ./*"
PR: note section on "no leading/trailing space", for internal names or mapping to a filesystem
unix  linux  filesystems  shell  piperesearch  osdev  reference  posix 
march 2019 by mechazoidal — Improving your workflow since 2019
Tracking new CLI tools in the spirit of Inconsolation
blog  cli  unix  openbsd  piperesearch  pmz  productivity 
march 2019 by mechazoidal
Files - sbase - suckless unix tools
FRIGN: "We kept them very minimal and must note that one motivation for it was the horrible complexity of and lacking UTF-8-support by the GNU coreutils. Probably due to its easy portability, sbase is, among other things, used in Google Fuchsia."
Unportable Linux tools are in "ubase", and combining both of them results in a busybox-like system.
unix  tools  osdev  suckless  repo:suckless  programming 
january 2019 by mechazoidal
Decoded: GNU coreutils – MaiZure's Projects
"This is a long-term project to decode all of the GNU coreutils in version 8.3. This resource is for novice programmers exploring the design of command-line utilities. It is best used as an accompaniment providing useful background while reading the source code of the utility you may be interested in. This is not a user guide -- Please see applicable man pagesfor instructions on using these utilities."
unix  reference  gnu  posix 
january 2019 by mechazoidal
[Toybox] More than you really wanted to know about patch.
Rob goes over the messy history of patch, including git improvements
diff  patch  history  reference  unix  version_control 
january 2019 by mechazoidal
"a tiny window manager that I created as an exercise in minimalism. It is also maybe helpful in learning some of the very basics of creating a window manager. It is only around 50 lines of C. "
programming  x11  wm  unix  c 
january 2019 by mechazoidal
Dynamic Linking
"Given that degree of sharing, the low cost of RAM, and the increase in OS complexity, slowness and insecurity in the implementations of dynamic libraries that I've seen, I don't see a need for dynamic libraries. (Remember that the real impetus for adding them to Unix was X11 and its big and badly-factored libraries, which most of us aren't blessed with.)"
osdev  plan9  programming  unix  history 
january 2019 by mechazoidal
BRLTTY - Official Home
"a background process (daemon) which provides access to the Linux/Unix console (when in text mode) for a blind person using a refreshable braille display. It drives the braille display, and provides complete screen review functionality. Some speech capability has also been incorporated."
Also note BrlAPI , which can provide an API for programs to output the text that can then be fed into BRLTTY OR speech synthesizers
a11y  unix  text  terminal  braille 
december 2018 by mechazoidal
OpenBSD manual pages
If you don't have access to a working OpenBSD system and need some reference
reference  unix  openbsd 
december 2018 by mechazoidal
Restricting Processes
Covers more modern stuff like cgroups, runlevels, and namespaces
unix  osdev  posix  reference  security 
december 2018 by mechazoidal
Jehanne Operating System - Simplicity awakes
Lots of good stuff: a rebuttal/analysis of the Unix/C "worse is better" approach, a note on how the various concurrent Unix system calls will trip each other up, how the Plan 9 design lessened but did not fully mitigate it, and how he's fixing it in the Jehanne P9 derivative. Also some notes from Italian artists on simplicity philosophy.
plan9  osdev  concurrency  2018  unix  pmz 
november 2018 by mechazoidal
Xft but for XCB
On using the font-for-xcb library to make it a little easier
x11  facme  rendering  fonts  unix 
september 2018 by mechazoidal
Integrated development window manager
Duplicating plan9's plumber with a 100-line shell script
plan9  unix  x11 
may 2018 by mechazoidal
Spacenav web site
Open source driver for 3dconnexion SpaceNavigator devices. Only Linux and FreeBSD at this point
(Can the driver work with other BSDs?)
usb  unix  x11  linux  hardware  drivers 
april 2018 by mechazoidal
su/sudo from root to another user allows TTY hijacking and arbitrary code execution
the "TIOCSTI" trick: "The problem with su and sudo is that although they change the UID of the executed process to the non-root user, the (pseudo) terminal is still that of the root user and thus accessible to the non-privileged user. The current terminal of a program is always accessible through /dev/tty. By opening this device as non-root user and then using the TIOCSTI ioctl(2) to fake user input on that terminal, the user can inject commands in the terminal. "
2016  security  unix  shell  sudo  exploit 
april 2018 by mechazoidal
Shell, Awk, and Make Should Be Combined
(could this be a project, modulo what oil's already done)
shell  awk  make  unix  makefile  pmz  piperesearch  oil_shell  2016 
december 2017 by mechazoidal
NCurses Disk Usage
"a disk usage analyzer with an ncurses interface. It is designed to find space hogs on a remote server where you don't have an entire graphical setup available, but it is a useful tool even on regular desktop systems. Ncdu aims to be fast, simple and easy to use, and should be able to run in any minimal POSIX-like environment with ncurses installed."
tools  unix  devops  cli  sysadmin 
november 2017 by mechazoidal
The Xinu Page
May be easier than xv6 to port to other arches?
os  programming  embedded  learning  unix  osdev 
november 2017 by mechazoidal
Terminal Recorders: A Comprehensive Guide
"Recording a terminal session and converting into a nice animated GIF to embed on a website. Sounds pretty simple, right? I’ve occasionally wanted to embed terminal recorders into blog posts, but it wasn’t until recently that I decided to actually look into some of the tools available to do it. It turns out that there are a lot of them. That alone isn’t necessarily a bad thing, but it also unfortunately turns out that most of them have some pretty serious issues."
- "If you care about fonts and color schemes, then use a generic screen recorder like gifine. "
- "If you don’t mind using the Terminus font, then use ttystudio."
- "If you want a web-based player–and you don’t care about mobile users or relying on an external service– then use Asciinema."
terminal  unix  tips  tools  recording 
november 2017 by mechazoidal
"a set of test suites for POSIX operating systems designed to make it easy to compare differences between operating systems and to find operating system bugs. It consists of test suites that focus on different operating system areas. This page visualizes the results for the free software POSIX operating systems that are relevant today."
unix  osdev  testing  posix 
october 2017 by mechazoidal
dotfiles/whither at master · garybernhardt/dotfiles · GitHub
"Run this to find out where your disk space is going. Pass the paths that you're curious about as arguments."
tools  unix  repo:github 
october 2017 by mechazoidal
the basic ontology of a c program
"i wanted to understand the system-level abstractions that were involved in the simplest c programs - ideally, to a point where i could recognize and understand all components in a compiled binary for a simple program. hopefully, even if you are on other configurations, this will give you some resources to start reasoning about the souls of your programs and the universes they live in."
c  unix  programming  elf  reverse_engineering  openbsd 
october 2017 by mechazoidal
A Little Story About the `yes` Unix Command | Matthias Endler
"The trivial program yes turns out not to be so trivial after all. It uses output buffering and memory alignment to improve performance. Re-implementing Unix tools is fun and makes me appreciate the nifty tricks, which make our computers fast."
c  unix  programming  optimization  rust 
october 2017 by mechazoidal
ntpd won't save you from one particular rogue bit
"ntpd... didn't care. ntpdate... also didn't care. Why should they? As far as they can tell, the lower 32 bits of time is just fine, so they think nothing is wrong. That's all they care about. [...] Given this, you'd assume that an offset of hundreds of years would generate a ridiculous offset as it dragged the clock backwards, and, well, you'd usually be right, except for this special situation where you hide all of the "extra time" behind that one bit."
ntp  unix  sysadmin 
september 2017 by mechazoidal
Parallel processing with unix tools
some minor tips, including that parallel(1) isn't suited to working on large single files(try presplitting with "split -n")
unix  tips  dsec  shell 
september 2017 by mechazoidal
« earlier      
per page:    204080120160

Copy this bookmark:

to read