Activity & memory

Alt-M or Command → Activity monitor opens the process monitor. It comes from the bundled System plugin, which is on by default and can be turned off in Options → Plugins. The process list takes the active pane, and the other pane shows a memory overview of the selected process. Enter or a double-click opens that process in a window over the monitor, with its memory map on one side and a preview of the selected region’s bytes on the other. Esc closes it, and Tab switches between the sides. Alt-W or the [+] button maximizes the focused pane, [=] restores the split, and [-] hides the monitor. The monitor samples once a second while it is open and stops when you close it. Sampling pauses while a memory inspection runs and resumes when the result arrives.

The list shows CPU, resident memory, memory growth, and disk read and write rates. CPU 100% means one fully busy core. Resident pages shared between processes count toward each of them, so adding up the RAM of all processes does not give the system total. CPU and disk rates appear from the second sample on.

CPU is an average of the last few seconds in which older use fades out, as in top, so a list sorted by CPU holds still while a process works in short bursts. The overview and the system CPU at the top of the monitor are averaged the same way. H hides the processes that used no CPU in that time, and the filter row says how many are hidden. A process that goes quiet stays listed until its average falls to 0.0. In the tree, the processes a busy one descends from stay listed with it. Press H again to show them all.

Key	Action
Enter	Open the process’s memory map / move to the region preview
/	Filter by name or PID; ↑↓ still move, Enter keeps the filter, Esc clears it
C / M / G / N / P	Sort by CPU / RAM / growth / name / PID; press again to reverse
I / O	Sort by read / write rate
T	Show or hide the process tree
H	Hide or show the processes that used no CPU in the last few seconds
Left / Right	In the tree, fold the selected process’s branch or unfold it
K	Send a signal or change the priority, also from the process window
E	Scan every process for high entropy
Alt-N	Show the selected process’s traffic on the network screen
Alt-O	Open the folder of the selected program beside the list
Space	Pause or resume sampling
R	Refresh the current view
Esc	Go back; in the process list, close the monitor or return to the network screen

Alt-O opens the folder of the selected process’s program in the other pane, with the cursor on the program, and the files take the keys. The list keeps its filter, its order and the process you selected. Tab moves the keys between the list and the files, and Alt-O on another row moves the files there. In those files you can view, edit, delete, compress, extract, and make folders. Copies and moves need the other panel, so they wait until the monitor closes. Esc in the list, or Alt-M, brings the overview back, and Alt-M again closes the monitor. In the open files and the memory map, Alt-O shows the selected open file or the file the region maps. When the program was deleted while it runs, its folder opens with a notice. A notice also tells you when the file is not in its folder on this computer, as happens for a program that runs inside a container. Alerts, startup items and the Docker view work the same way.

Every screen keeps its filter when you close it and shows it again when it opens. To clear a filter, press / and then Esc.

Click a column heading to sort, and double-click a process to open its memory map. The overview shows current resident memory, the highest value seen, the change since monitoring started, CPU, virtual address space, and a chart of resident memory over the last 60 seconds. Footprint, swapped, private, and shared totals appear once the memory map has been read, if the system reports them. Enter in the memory pane opens the memory map. The memory map is read when you ask for it; press R to read it again after the process changes its mappings.

Below the chart, Process tree lists the processes the selected one descends from, the process itself, and everything it started, each indented under the one that started it and shown with its resident memory. When they do not all fit, the last line says how many are left out.

In the tree that T shows, each process is listed under the one that started it, and ▾ marks a process with others under it. Left folds the selected process’s branch; on a process with nothing under it, Left folds the branch it belongs to and selects the process that started it. Right unfolds a folded branch, and on an open one steps to the first process in it. A folded process shows ▸ and how many processes it hides. A branch stays folded until its process ends. While a filter applies, the list is flat.

Signals and priority

K opens Signal or priority for the selected process, in the list and in its process window with the memory map or open files. Press a choice’s letter, or move to it and press Enter. Esc closes the dialog and sends nothing.

Key	Choice	What it does
T	Terminate	Sends TERM, which asks the process to quit
I	Interrupt	Sends INT, the same as Ctrl-C in its terminal
H	Hang up	Sends HUP; many services read their settings again
K	Kill	Sends KILL, which ends the process at once without letting it clean up
S	Stop	Sends STOP, which pauses the process
C	Continue	Sends CONT, which resumes a stopped process
L	Lower priority	Raises its nice value by 5, up to 19, so it waits longer for the CPU when the system is busy
R	Raise priority	Lowers its nice value by 5, down to -20

Right before sending, Borg checks that the PID still belongs to the process you chose, started at the same time, so a new process that reused the PID is left alone. The status line confirms what was sent, and the list samples again at once. The overview says when a process is stopped, and shows its Nice value when it is not 0.

You can signal your own processes and lower their priority. The system refuses the rest: another user’s process, and raising a priority, which only an administrator may do. Borg then offers A Administrator, which sends the same choice through the helper described under Administrator access. Borg sends no signals to itself; F10 quits it.

Memory map

The map has three columns:

• Footprint is the memory charged to this process alone: pages it has written, plus pages swapped out or compressed. It is the number to watch for leaks, and on macOS it matches the Memory column in Activity Monitor.
• Resident is the pages in RAM right now, including pages shared with other processes.
• Mapped is address space, including reservations. It says nothing about RAM use.

A process can map gigabytes, mostly system libraries shared with every other process, while its footprint is only a few megabytes. Regions start out grouped by purpose: heap and allocations, thread stacks, the program’s own code and data, shared libraries, mapped files, shared memory, system and other allocations, and reserved ranges and guard pages. Groups and the regions in them are ordered by footprint, with reservations last. Libraries and mapped files get one row per file; expand a file to see its regions. Group names are worked out from the labels the operating system gives. Selecting a group or file explains what it is and shows its share of the footprint and its largest members. Enter or Right expands a group or file; Left collapses it, or the one that contains the selected row. G switches between the grouped and the raw list.

Open files

1 Memory and 2 Open files switch between the inspection tabs; you can also click them. The open files list shows each descriptor, its access mode, its type, and its path or network endpoint; the other pane shows full details, the file size, and the current offset when they are available. On macOS it also includes entries such as cwd (current directory) and txt (mapped executable). R refreshes the list. Each tab keeps its position in the list when you switch. When access is restricted, Borg offers to try again as administrator. For a regular file, the details also preview its contents, as the quick view does: Tab moves into the preview so you can scroll it, and F3 opens the file in the viewer. Borg opens the file by its path with your own permissions and leaves the process’s descriptor alone.

Region preview

When you select a readable region, Borg captures up to 64 KiB of it in the other pane. If the start of the region holds only zeros, the capture skips ahead to the first page with data, and the line above the bytes says where in the region the window is. Each row starts with its offset from the start of the region, and a wide pane shows more bytes on a row. When you move quickly, Borg waits a moment before reading; selecting another region clears the old preview and cancels its pending read. Regions that cannot be read show a message instead. Press Tab or Enter to scroll the preview. Esc there goes back to the list, and Esc in the map goes back to the process list with its cursor and filter as they were.

Runs of zero rows collapse into a single row, such as “· 60K zero bytes ·”. ] and [ jump to the next and previous block of data, and > and < capture the next and previous 64 KiB of the region. H switches between hex and text, R captures the shown window again, and D highlights what changed since the previous capture of the same process and address. Text view shows printable ASCII and replaces other bytes with dots. Captures are taken from the live process without pausing it, so a capture is not a consistent image of the whole process. They stay in memory unless you export them. E saves the raw bytes to a new file in the active file panel’s folder, readable only by you. Existing files are never overwritten.

M switches the preview between bytes and a page map of the region: one cell per 4 KiB page, or per run of pages in a large region, marked by what it holds: pointers into the process’s own memory, text, UTF-16 text, code, compressed or encrypted data, other data, filler, zeros, and pages not in memory. A legend shows each kind’s share of the region. When the map is focused, the arrow keys, PgUp, PgDn, Home, and End move a cursor, and Enter opens the page under it as bytes. Regions larger than 16,384 pages are sampled evenly. On Linux, the map asks the system which pages are in memory and does not read the rest, so it neither brings swapped pages back nor touches new ones; on macOS, pages that were never touched read as zeros. Kinds are guessed from the bytes alone, so treat them as a hint.

Pages whose entropy reaches 7.2 bits per byte are drawn in bold red, or bold alone where red would be hard to read, whatever their kind. Packed or encrypted code stands out that way in an executable region, where Borg counts every page that is not zeros or filler as code. The legend gives their share as high entropy, the cursor’s line adds the entropy of the page under it, and the same line ends with the highest entropy in the region. A note under the legend says what high and low entropy usually mean. Compressed caches, images, archives and keys are high too, so look at what the region holds before you suspect it.

Memory search

Press / in the memory pane, or Ctrl-F in either pane, to search memory. Choose a mode, whether case must match, and whether to search the process’s readable memory or the current capture. The form opens with what it held when you last closed it. Text searches UTF-8, UTF-16 (both byte orders), and UTF-32 (both byte orders). Unicode also normalizes compatibility characters and combining sequences: for example, 𝐁𝐨𝐫𝐠 matches Borg. It does not treat different letters as equal just because they look alike. Hex takes byte pairs such as 42 6F 72 67.

Strings and Indicators only read memory the process filled in itself: memory not backed by a file and that the process may write to; on macOS, regions it never wrote to are skipped too. Strings lists each run of at least six printable ASCII or UTF-16LE characters once, with the number of copies in memory, and skips filler such as repeated characters or punctuation; if you give a query, it keeps only the strings that contain it. Indicators keeps the strings that look like a URL, an IP address, an email address, a file path, a command line, a secret (a private key, an access token, or a password assignment), or a long run of base64, and labels each one.

Borg also reads the strings in the files the process has mapped, up to 1 GiB, and marks with • the strings found only in memory: text the process built, received, or decoded, which none of its files contain. Indicators come first, then strings found only in memory, then those its files also hold, and strings that look like words come before the rest. The list keeps the best 2,000. Where other matches show their region, each row shows the kind and the number of copies, and the preview shows the string in place; C steps through its copies. Treat these results as leads worth a closer look. Ordinary programs hold plenty of URLs, paths, and keys too.

Matches take the list pane and show their address, encoding, and region. Selecting a match previews the bytes around it in the other pane, with the match highlighted. Enter moves into the preview, and Esc goes back to the matches. Esc in the matches returns to the memory map or process list you came from. Searching a saved capture uses its bytes without reading the running process. Process searches read memory in chunks, report bytes they could not read, and stop after 2,000 matches or 60 seconds. N continues from where the search stopped, and Ctrl-C stops it and keeps the results. The coverage report separates searched bytes from unreadable ones. A process can change its memory while it is being searched.

Code integrity check

I, in the memory map or among search matches, checks whether the process is running the code its files contain. It compares the code in memory with the files it was mapped from and reports pages that differ, searches memory not backed by a file for the headers of executable images (ELF, Mach-O, and PE), and lists executable memory not backed by a file, memory that is both writable and executable, and code from deleted, nameless, or temporary files. It also checks the process as a whole: libraries injected through LD_PRELOAD, LD_AUDIT, DYLD_INSERT_LIBRARIES, or /etc/ld.so.preload, library search paths, a tracer on Linux, a program whose file has been deleted or replaced, and a process name that differs from its program’s.

Findings take the list pane, with alerts first, then warnings, then notices. Selecting one previews the bytes it is about, with the evidence highlighted, and the rows below the list say what else could explain it. Programs that compile code while they run, such as browsers and Java and JavaScript engines, always hold code that no file backs, so for them those findings are only notices. R checks again, Ctrl-C stops a check and keeps what it found so far, and Esc goes back. A check reads at most 512 MiB of code and 1 GiB of other memory, for up to 45 seconds, and says if it stopped early.

On Linux, the page tables show which code pages a process has changed, so only those are read; on macOS, regions without modified pages are not read either. Code in the macOS shared cache has no file of its own and is not compared. Memory a check cannot read is counted, and Borg offers A Administrator. Treat findings as leads, because debuggers, profilers, updaters, and security tools leave the same traces.

Scanning memory with rules

Y, in the memory map or among search matches, scans the memory the process filled in itself with the same rules as a file scan: Borg’s and those in your rules folder (see Scanning files with rules). Like Strings, it reads memory not backed by a file and memory the process may write to; on macOS, regions it never wrote to are skipped too. A region with no page in memory or in swap is not read, and on Linux the page tables show which pages the process never touched, so those are not read either. Swapped pages are read, since a payload can wait there. Memory is read in pieces of 16 MiB that overlap by 4 KiB, so a match that crosses from one piece into the next is still found. Each rule is listed once per region, at its first match.

Matches take the list pane, with alerts first. Each one names the rule and where it comes from, Borg’s rules or your file, then the bytes it matched and what the rule looks for, and the rows below the list say what else could explain it. Selecting a match previews the bytes around it with the match highlighted. Rule files left out come last, with the compiler’s message. R scans again, Ctrl-C stops the scan and keeps what it found, and Esc goes back. A scan reads at most 1 GiB, for up to 45 seconds, lists up to 2,000 matches, and says if it stopped early. Memory it cannot read is counted, and Borg offers A Administrator, which runs the scan in the helper with your rules folder.

A rule scan is triage: a quick look at one process or folder for patterns someone has described. It does not watch the system, block anything or clean up, which is what endpoint protection is for. Some things stay out of its reach:

• macOS keeps the memory of its own programs and of programs built with the Hardened Runtime from every other program, administrators included (see Administrator access).
• On Linux, kernel.yama.ptrace_scope decides whose memory you may read. At 1, as on Ubuntu, you may read only your own child processes, so Borg offers A Administrator for the rest. At 3, nobody may read another process, administrators included.
• A kernel rootkit can hide memory, files and processes from every program, Borg included, so a clean scan proves little.
• A packed or encrypted program shows its real code only while it runs. Scanning its memory then sees what a scan of its file cannot.

Borg reads memory without attaching as a debugger, with process_vm_readv on Linux and mach_vm_read on macOS, so the process sees no tracer, which is what most anti-debugging checks look for. Monitors that run as administrator, such as auditd, eBPF tools and macOS Endpoint Security clients, can still log the reads. A file scan opens every file it reads, which inotify and fanotify report. On Linux, it leaves the access time of the files you own as it was.

Entropy scan of every process

E in the process list opens Entropy scan, which reads sampled pages of every process for bytes as dense as packed code. Press a choice’s letter, or move to it and press Enter. Esc closes the dialog.

Key	Choice	What it reads
C	Code of all processes	The executable regions, where packed or encrypted code shows
M	All memory of all processes	Every readable region, so data such as keys and compressed files too
L	Last results	Nothing new: it shows the hits of the last scan again

Borg reads up to 1,024 pages of 4 KiB from a region and 32,768 from a process, spread evenly. A process with many regions gets fewer from each, but never fewer than 16. A region with at least one page of 7.2 bits per byte or more is a hit. Hits take the list pane: code first, then memory the process wrote, then files mapped as they are on disk, and within each the region with the larger share of high pages first. Code with 70% or more of its pages high stands out in bold, by the same rule as the search for packed executables. Each row gives the kind (code, written or file), that share with the highest entropy, and the process name, PID and region. The other pane shows the region’s addresses, size and access, how many sampled pages are high, where the first of them starts, and what such bytes usually are. The 2,000 strongest hits are kept, and the summary counts the weaker ones left out.

Enter on a hit opens the process’s memory map at that region, with the page map on and its cursor on the first dense page sampled, so you can look at the bytes around it. Esc goes back to the map, and again to the hits.

Borg reads 4 processes at once. A region with no page in memory or in swap is not read, and on Linux the page tables show which pages are in memory, so the scan neither touches new pages nor brings swapped ones back. Caches every process maps alike are left out: the dyld shared cache on macOS, and the pages the kernel maps into every process on Linux. A file mapped unchanged, such as a shared library, is read in the first process that maps it only.

A scan stops after 90 seconds and leaves the rest for N, which goes on with the processes it did not reach. Ctrl-C stops it sooner and keeps the hits, and R scans again from the start. Above the hits, one line counts the processes scanned and the hits, and the next the processes left out: unreadable, protected and not reached. On macOS, vmmap takes a few seconds to list a process’s regions, so a scan of every process can take a few rounds of N.

Processes whose memory macOS protects are counted and skipped, since nobody can read them. For the processes Borg could not read, it offers A Administrator, which scans them in the helper for at most 60 seconds, so the helper finishes within its 2-minute limit even after a slow password prompt. Dense bytes are common: compressed caches, images, fonts and keys are high by nature, so treat hits as leads and read the details before you suspect a process.

Administrator access

When reading a memory map or a capture is restricted, a search could not read part of the memory, an entropy scan could not read some processes, or the system refuses a signal or priority change, Borg offers A Administrator. On macOS this opens the system’s authorization prompt; on Linux it uses pkexec and needs an authorization agent to be running. Only a short-lived helper runs with elevated rights. Borg itself keeps running normally, and the helper only returns map data, a limited number of bytes or the hits of a scan, or sends the one signal or priority you chose.

macOS shields the memory of its own programs (System Integrity Protection) and of programs built with the Hardened Runtime from every other program, administrators included. Borg reads this from the program’s signature: the process details say Memory protected, a refused read explains why, and A Administrator is not offered, since it cannot help. Their memory maps and open files are still shown. It is not offered either when Borg already runs as administrator. On Linux, security policy can still deny access; Borg shows the error and the process monitor keeps working as usual.

A process is identified together with its start time, so a new process that reuses a PID does not inherit another process’s history. Memory maps, inspection results, captures, and history all have size limits. The Borg Commander name stays visible in the bottom-right corner everywhere, including the console, viewer, monitor, dialogs, and screensaver.

Alerts

Borg can watch what starts on a machine and tell you when something looks wrong. Turn on Watch: unusual process starts in F9 → Options → Plugins. The watcher lives in the background service that all your Borgs on the machine share, so every Borg reads the same alerts. That service runs while a Borg is open. To keep watching with no terminal open, start the service at boot as Named sessions and the background service describes.

Alerts stay on that machine until you name a place to send them, and the page says so while they do (see Alerts off the machine).

Alt-L or Command → Alerts opens the page. The list takes the active pane, newest alert first, and the other pane shows the selected alert in full. Each row shows the time, the rule that fired, the program, the parent that started it, the user it ran as, and what has become of the process. The line above the list counts the alerts and how many nobody has read yet.

The details say in one sentence why the rule fired, then the command, the program, the parent, the user, what the standard streams led to, and where the process stands now. Under Process tree each parent appears above its child, so you can read the path from a service down to the command that worries you. Connections lists the sockets the process held at that moment, Program hash is the SHA-256 of its program, and Rules shows what Borg’s rules and yours matched in that program (see Scanning files with rules).

A command longer than 2 KiB keeps its start and its end, 2 KiB together, with … between them. The commands of its parents and the values of its preload variables are cut the same way. That keeps the newest 200 alerts inside the log however long their commands run, and a long first argument cannot push the end of a command out of view. Ctrl-C copies the command as the log keeps it.

Key	Action
Enter	Show the process in the activity monitor
C	Let a frozen process go on running
K	End the process, after a question
A	Allow starts like this one from now on
Alt-O	Open the folder of the alert’s program beside the list
/	Filter the rows; ↑↓ still move, Enter keeps the filter, Esc clears it
R	Read the log again
Ctrl-C	Copy the command the process ran
Alt-W	Maximize the list
Esc	Close the page, which keeps its filter

The filter keeps the alerts that contain every word you type, in the rule, its short name, the reason, the program, the parent, the user, the command and what Borg did. Put ! before a word to leave out what it matches: shell-on-socket !root keeps the remote shells of every other user.

Alerts live in alerts.json in the state folder, owner-only, the newest 200 of them. An alert about a process that is still frozen stays beyond that, so you can always let it go on or kill it. Borg keeps the file within 8 MiB, the most it reads back, and leaves out the oldest alerts when they would not fit. The Borg’s files window counts them under State (see Borg’s files). Opening the page marks what it holds as read, and the count of unread alerts follows the log while Borg is open, whether or not the page is.

When Borg finds an alerts.json it cannot read, such as one an older version let grow past 8 MiB, it renames the file to alerts-unreadable.json before it writes a new log, so the alerts in it stay on disk. A later one gets a numbered name such as alerts-unreadable (2).json. The Alerts page names the file below the list for as long as it is there, so delete it once you have looked through it.

What the watcher looks for

Each rule names one thing that is odd about a process that has just started. The level says how much it means: alert is seldom innocent, warning often has an ordinary reason, notice is there so the picture is complete.

Rule	What fires it	Level
Shell on a network socket	A shell whose input, output or errors go to a network socket, which is what a remote shell looks like. A socket that stays on the machine, such as the one systemd gives a service’s output to the journal, does not count	alert
Remote shell on a terminal	A shell on a terminal that a program opened for it, where that program’s own input, output or errors go to a network socket, or the program is a relay such as socat or nc. That is a remote shell turned into a full terminal, as pty.spawn in Python makes one	alert
Program deleted while it runs	The program’s file is gone, so nothing on disk says what the process is	alert
Program with no file	Borg can find no program file for the process at all	alert
Library forced into a program	LD_PRELOAD, LD_AUDIT or DYLD_INSERT_LIBRARIES is set in its environment	alert
Shell from a service	A web server, application runtime or database started a shell	warning
Shell without a terminal	A shell with no terminal on either side, so nobody typed it	warning
Shell on a program’s terminal	A shell on a terminal that a program opened for itself and nobody logged in to, such as one that expect or docker exec opens	warning
Looking around the machine	Four different commands such as whoami, id, uname and ss from one program within an hour, with nobody logged in to the terminal they run on. Commands it runs through one shell after another count together, so a web shell that starts sh -c for each command is caught too	warning
Program in a temporary folder	The program lies under /tmp, /var/tmp, /dev/shm or /run/shm	warning
Name differs from the program	The process calls itself one thing and runs another	warning
Parent never started it before	This parent has never started this program on this machine	notice

For each process Borg reads the program it runs and whether its file was deleted, the command, the user, the terminal it belongs to and who opened it, what its standard input, output and errors lead to, the preload variables in its environment, and the chain of parents above it. A socket counts as a network socket when the process’s network lists it as TCP or UDP.

Whether somebody logged in to a terminal is decided the same way as for freezing (see Freezing a process): over SSH, at the console, in tmux or screen, through sudo or su, or in a terminal window. A terminal that a program opened for itself does not count, whether an editor’s remote server, docker exec or a reverse shell opened it.

Borg learns the machine before it acts. For the first 7 days it records which parent starts which program here and freezes nothing, and the Response column says Would freeze where it would have. After that, a pair it has seen before stays quiet for the rules that can be learned: the three shell rules, the name that differs, and the parent that never started it. The run of commands is learned by the program it comes from: it stays quiet when that program ran each of its commands during the learning week, so a monitoring agent that runs df and ps all day stays quiet, while a web server that never ran id does not. The five alert-level rules fire every time, because a shell on a socket or a forced library never becomes normal by happening often. Whatever runs during that week counts as normal, so turn the watcher on while you trust the machine. Starts that a rule learning cannot quiet fired on stay out of what Borg learns.

A on an alert allows starts like it from then on, by its program and its parent. Those allowances go in watch-allowed.json in the state folder, which the watcher reads and takes in within a minute. The [watch] section of the configuration holds the same thing by hand (see Configuration). Borg writes down at most 20 processes a minute and freezes at most 5, and the line above the list says how many findings it left out when more fired at once. A process Borg freezes is always written down, however many alerts came before it, so you can let it go on. Notices are counted apart, so a run of them never takes the room of a warning, and up to 20 processes a minute that Borg would have frozen past its limit of 5 are written down too.

Freezing a process

Once the learning period is over, Borg freezes the process an alert-level or warning-level rule fired on. Freezing sends a stop signal to the process and everything below it. A frozen process has not exited, so its service keeps running, systemd and Docker have nothing to restart, and whatever it was about to do waits where it is until you look. The Response column says Frozen, and C lets it go on while K ends it. Killing is final, so Borg asks first.

Borg starts freezing once you have seen every alert from the learning week that says Would freeze, or 3 days after the week ends, whichever comes first. Opening the Alerts page marks them read, and the watcher notices within a minute. That way you can check what the week caught and allow a harmless program with A before anything stops, and a machine nobody looks at is still protected. Until then, the page’s footer says how long freezing waits, and the details of a new alert say that it is waiting.

Borg never freezes the system’s first process, itself or anything it started, a program on the list it never freezes (init, systemd, sshd, sudo, su, login and the like, plus any name you add under keep-running), a process on a terminal somebody logged in to, or a start you have allowed. A notice never stops a program. The details say which of these kept a process running.

A program on the built-in list is spared only where the system keeps its programs, such as /usr and /bin, so a copy named sshd in another folder is frozen like anything else. Names you add under keep-running match wherever the program lies. A terminal counts when somebody logged in to it: over SSH, at the console, in tmux or screen, through sudo or su, or in a terminal window. A terminal multiplexer counts only for root and for people’s accounts, numbered from UID_MIN in /etc/login.defs (1000 on most systems), because a service account can start one too. A terminal that a program opened for itself does not count, which keeps a remote shell from hiding behind one. Terminals opened by an editor’s remote server or by docker exec do not count either, so if Borg freezes something you ran there, C lets it go on and A allows it.

Borg checks a process’s identity, its number together with the moment it started, right before it signals it, so a number another process has taken in the meantime is left alone.

The watcher as root

Without more rights, Borg hears only about its own user’s processes, by looking at the process list four times a second, and the page says so under the list. A short command can be gone before the next look.

To hear about every process the moment it starts, install the part that runs as root:

borg service watch install

This asks for your password once and installs a separate unit, borg-commander-watch.service, which runs borg --watch-root. That part loads Borg’s own small programs into the kernel, which read each program start before the new program runs: the program, the command line exactly as it was passed, the process that started it and the user. A command that ends at once is still reported, and so is the command line of a process that rewrites it afterwards. The root part reads /proc for the rest, and stops, continues or ends a process when Borg asks. It does nothing else: the unit drops every capability it does not need, and Borg’s own interface keeps running as you. The two sides talk over a socket in the state folder that only its owner can reach, Borg accepts a connection only from a process running as root, and the root part connects only to a socket that belongs to the user it was installed for. borg service watch uninstall removes it, and borg service watch status prints where the events come from and what has been found. The install and the status both warn while alerts have no place to go (see Alerts off the machine).

The programs need Linux 5.8 or later. On an older kernel, or where the root part may not load them, it hears the kernel’s process events and reads everything from /proc, and the page says so under the list. After updating Borg, run borg service watch install again, so the unit gets any rights the new version needs.

Alerts from either side are the same alerts, in the same log. A process another user owns can be frozen only by the root part, so resuming or killing one from the page goes through the watcher in the service. When nothing can reach it, the page says only the owner or an administrator may do that.

The watcher is a Linux feature for now. On macOS the page opens and reads its log, and the source says nothing is watching.

Alerts off the machine

The Alerts page lives on the machine it watches, so whoever takes that machine over can delete the log. Send each alert somewhere else as it happens: a copy that has arrived is out of their reach, and a heartbeat tells you when the watcher goes quiet. Add [watch.notify] to the configuration on the server, with any of these places or several. Every address uses https://, or http:// to the server itself:

[watch.notify]
name = "web-01"             # how messages name this machine, its host name otherwise
level = "warning"           # the least an alert needs to be sent: alert, warning or notice

[watch.notify.telegram]
token = "123456:ABC-your-bot-token"
chat = "123456789"

[watch.notify.ntfy]
url = "https://ntfy.sh/a-long-random-topic"
token = "tk_your_token"     # where the topic needs one

[watch.notify.webhook]
url = "https://example.com/borg"
secret = "a long random string"

[watch.notify.heartbeat]
url = "https://hc-ping.com/your-check-id"
minutes = 5

Then run borg service watch notify. It sends a test message to each place and prints what each one answered.

• Telegram: create a bot with @BotFather, which gives you the token. Send the bot a message, then open https://api.telegram.org/bot<token>/getUpdates and take chat.id from the answer.
• ntfy: subscribe in the ntfy app to a topic with a long name nobody can guess, or to one on your own ntfy server.
• Webhook: Borg posts every message to the address as JSON. With secret, each request carries X-Borg-Signature: t=<time>,v1=<signature>: the HMAC-SHA256, under the secret, of the time, a dot and the body, so your receiver can tell Borg’s messages from forged ones. X-Borg-Event says what the message is: alert, evidence (the program’s SHA-256 and rule matches, which follow an alert once Borg has read the program), started, armed or test.
• Heartbeat: Borg calls the address every minutes while the watcher runs and adds /fail when something is wrong: the thread that hashes programs stopped, a place has not taken a message for 30 minutes, Borg gave up on one, or Borg could not read its outbox. A service such as healthchecks.io tells you when the calls stop, which is what happens when someone kills the watcher.

While no place takes alerts, they stay in the log on the machine, and Borg says so wherever you meet the watcher. The Alerts page shows Alerts stay on this machine. above the details of the selected alert, or in the last row of the list when the list is maximized. You get the same line when you turn the watcher on in Options → Plugins, from borg service watch install and borg service watch status, and in the service’s output when the watcher starts. A heartbeat alone does not end the warning. It tells you that the watcher runs and carries no alerts, so add Telegram, ntfy or a webhook as well.

The warning gets stronger when Borg is close to acting. From a day before the learning period ends it reads Borg is about to freeze processes here, and nobody will be told., and once freezing has started, Borg freezes processes here and nobody is told. Borg still watches, learns and freezes without a place to send alerts, so set one up before the learning period ends on a machine you do not look at every day.

The service sends each alert within seconds of writing it down, and also says when the watcher starts and when freezing begins. Notices stay on the page unless level is notice. A message a place refuses waits in watch-outbox.json in the state folder and is tried again, 5 seconds later at first and every 30 minutes at most. Borg keeps up to 500 waiting messages, fewer when long ones would take the file past 16 MiB, and tries each for 48 hours. The next message says how many it gave up on.

A message carries at most 1,000 bytes of the command line, the program’s path and the reason for the alert. A longer one keeps its start and its end with an ellipsis between them, so padding the start of a command cannot hide what it does.

When Borg finds a watch-outbox.json it cannot read, it renames the file to watch-outbox-unreadable.json and starts a new outbox. A later one gets a numbered name such as watch-outbox-unreadable (2).json. The messages that were waiting stay in that file and are never sent, so the heartbeat fails once and says why, and each place’s numbering starts again at 1. Look through the file, then delete it.

Each place numbers its messages from 1, and every message’s JSON carries the SHA-256 of the one before it in previous. A gap in the numbers, or a previous that does not match, means a message went missing or was changed on the way. The numbering happens on the server, so it cannot catch someone who controls the server.

The tokens sit in the configuration, which anyone running as your user on the server can read. With them, someone can send messages that look like Borg’s and carry the numbering on without a gap, and a Telegram bot can delete what it sent in the last 48 hours. Whoever takes over the server can also delete the messages still waiting to leave. What arrived before that stays as it was, and a webhook to a machine you control keeps it as it arrived. When the watcher goes quiet, the heartbeat is what tells you.

What the watcher cannot see

The watcher judges each program as it starts, which leaves some things out of its sight. Know them before you rely on it.

• Code that runs inside a program that is already running starts nothing new, so no rule looks at it. A web application whose own interpreter is taken over keeps the name and parent it always had.
• Processes running before the watcher started are never examined, and whatever runs during the learning week counts as normal.
• Without the root watcher, Borg sees only your own processes, and a command that ends within a quarter of a second can be gone before Borg looks. Services that run as another user, such as www-data or postgres, need the root watcher. When the root watcher stops, Borg goes back to watching your own processes until it runs again.
• Before Linux 5.8, or where the root part cannot load its programs, Borg reads each new process in /proc after the kernel reports it. A command that ends at once can be gone by then, and a process can change its command line first.
• Some kernels send no process events, as on some virtual machines and container hosts. Borg running as root and the root watcher then look at every process four times a second, and a short command can be gone before they look.
• A burst of new processes can fill the queue between the kernel and Borg, and the kernel then drops events, so some starts are never looked at.
• A command that ended before Borg could read /proc carries only what the programs read: its program, command line, parent and user. Its terminal, standard streams and preload variables ended with it, so the rules about them cannot fire. Program deleted while it runs fires on it only when its file was already gone as it started, as for a program run from memory.
• The programs keep the first 4 KiB of a command line, and Borg takes a longer one from /proc while the process is still there. They find the command line and the program’s file through the type information the kernel publishes at /sys/kernel/btf/vmlinux. On a kernel built without it, the command line comes from /proc as well. A command that ended first is then named by the path it was started by, which can be relative to its working folder or lead through a symbolic link, so Program in a temporary folder can miss it.
• Anything running as the user the watcher runs as can stop the watcher, change its files in the state folder, or let a frozen process go on with kill -CONT. Freezing buys you time to look, and the process’s owner can still undo it.
• A process on a terminal somebody logged in to is never frozen, even when the login used a stolen password or key. The rules still fire and write alerts.
• Looking around the machine counts the commands of one program within an hour. Someone who waits longer between commands, or runs each one from a different program, stays below it.
• A process that detaches from its parent before Borg freezes the parent keeps running.
• Without [watch.notify], alerts reach you only when you open the page. With it, someone who takes over the machine in the seconds after an alert can still stop the message before it leaves, and then the heartbeat’s silence is what tells you.