1<html><head><title>toybox news</title> 2<!--#include file="header.html" --> 3 4<p>Toybox combines the most common Linux command line utilities together 5into a single BSD-licensed executable that's simple, small, fast, 6reasonably standards-compliant, and powerful enough to turn Android into 7a development environment. See the links on the left for details.</p> 8 9<h2>News</h2> 10 11<!-- new entry template 12<a name="DD-MM-YYYY" /><a href="#01-01-2001"><hr><h2><b>Jan 1, 2001</b></h2></a> 13<blockquote> 14<p> 15Don't Panic 16</p> - The Hitchhiker's Guide to the Galaxy</p> 17</blockquote> 18 19<p><a href=downloads/toybox-0.0.1.tar.gz>Toybox 0.0.1</a> 20(<a href=https://github.com/landley/toybox/releases/tag/0.0.1>git commit</a>) 21is out, with prebuilt <a href=downloads/binaries/0.0.1>static binaries</a> and 22<a href=downloads/binaries/mkroot/0.0.1>mkroot images</a> 23bootable under QEMU (built using a <a href=https://github.com/landley/linux/tree/mkroot-0.1>lightly patched</a> linux-0.1).</p> 24 25<p>New commands</p> 26 27<u>Features</u>: 28<u>Bugfixes</u>: 29<u>Library</u>: 30<u>Mkroot</u>: 31<u>Pending</u>: 32<u>Cleanup</u>: 33<u>Portability</u>: 34<u>Documentation</u>: 35<u>Plumbing</u>: 36<u>Test suite</u>: 37<u>Build</u>: 38--> 39 40<a name="08-04-2024" /><a href="#08-04-2024"><hr><h2><b>April 8, 2024</b></h2></a> 41<blockquote> 42<p>Another thing that got forgotten was the fact that against all probability a 43sperm whale had suddenly been called into existence several miles above the 44surface of an alien planet. And since this is not a naturally tenable position 45for a whale, this poor innocent creature had very little time to come to terms 46with its identity as a whale before it then had to come to terms with not being 47a whale any more.</p> 48<p>- The Hitchhiker's Guide to the Galaxy</p> 49</blockquote> 50 51<p><a href=downloads/toybox-0.8.11.tar.gz>Toybox 0.8.11</a> 52(<a href=https://github.com/landley/toybox/releases/tag/0.8.11>git commit</a>) 53is out, with prebuilt <a href=/bin/toybox/0.8.11>static binaries</a> and 54<a href=/bin/mkroot/0.8.11>mkroot images</a> 55bootable under QEMU (built using a <a href=/bin/mkroot/0.8.11/linux-patches>lightly patched</a> linux-6.8).</p> 56 57<p>Highlights include a new <a href=quickstart.html>quickstart</a> 58web page, shorter <a href=/bin>/bin</a> links to prebuilt binaries, 59lots of work on mkroot, and a new "canned" toybox build to reduce 60dependencies.</p> 61 62<p><u>New commands</u>: 63Rob added <b>tsort</b>, and promoted <b>fold</b> and <b>getopt</b>. 64Oliver Webb added <b>ts</b> and <b>csplit</b>, and 65Elliott added <b>memeater</b>.</p> 66 67<p>And one command got temporarily de-promoted: <b>passwd</b> is "default n" 68(disabled in defconfig) this release due to the ongoing lib/passwd.c 69infrastructure rewrite that isn't quite done yet. (See the Library section.)</p> 70 71<u>Features</u>: 72The new "<a href=https://github.com/landley/toybox/commit/d1acc6e88be5>canned</a>" 73toybox build in <b>scripts/prereq/build.sh</b> lets toybox provide its own 74build prerequisites by compiling a minimal toybox against saved headers (ala 75"cc -I scripts/prereq/generated file1.c file2.c..."). This provides 76most of the commands toybox needs to configure and build itself (except make, 77bash, and the compiler toolchain). This may help <a href=https://github.com/landley/toybox/commit/3bbc31c78b41>bootstrap</a> toybox on systems 78that don't provide a modern Linux command line out of the box: install 79toybox-prereq at the start of the $PATH, add 80a .config file, and run <b>scripts/make.sh</b>.</p> 81 82<p>New <b>mv -x</b> option to atomically swap two files, 83<b>env -e FILE</b> executes a different command than argv[0] on the command 84line, the <b>reset</b> command now puts raw terminals back into "cooked" mode, 85negative offsets in <b>sort -k</b> count right to left, 86<b>setsid</b> grabs the tty more forcefully (removing the need for getty) 87and the -cc option will try to steal it, 88<b>mount</b> now shows file= when displaying loopback mounts, 89<b>wc -L</b> shows longest line length, 90<b>tar -h</b> now detects a lot more hardlinks, 91<b>cp</b> now copies xattrs for directories (not just files).</p> 92 93<p>Elliott taught <b>file</b> to recognize wasm binary modules and 94display JPEG width/height, added the LOWER_UP, DORMANT, and ECHO 95flags to <b>ifconfig</b>, added PAC and BTI decoding to <b>readelf</b>, 96cache size querying to <b>sysconf</b> (supported by glibc and bionic but 97not musl-libc), and added a "paste" menu to <b>microcom</b>. Kalesh Singh 98taught <b>readelf</b> to decode a new note type (NT_ANDROID_TYPE_PAD_SEGMENT). 99Christopher Ferris improved <b>readelf</b> section flags handling, 100with test. Oliver Webb suggested <b>count -l</b>. 101Aditya submitted <b>netcat -z</b>.</p> 102 103<u>Bugfixes</u>: 104Fixed <b>printf \0</b> and <b>grep -w</b> (which never quite worked right), 105<b>ls -k</b> now switches off --block-size, 106<b>install -dm 02750</b> works (sticky bit support for -d), 107<b>tar</b> is better at following symlinks and extracting records with broken 108metadata, <b>gzip</b> now knows 109knows how to rename between "tgz/tar" extensions, 110<b>mkpasswd -m sha*</b> salt length can now range from 8-16, not just 16, 111<b>ps</b> can now handle an session ID of 0 (which is what PID 1 starts with 112before the first call to setsid()), 113<b>pwgen</b> can now use ~ in output, 114<b>xargs</b> now stops when a child exits with 255, 115<b>iconv</b> now exits with error if any characters failed to convert with -c, 116things like <b>truncate -s 1e2</b> were saying "too large" and now correctly 117say "not integer" (it's not scientific notation, it's exabyte suffix with 118trailing garbage), 119<b>timeout</b> now takes its exit code from SIGCHLD instead of wait() 120to fix a race condition (SIGCHLD can happen before calling wait and the 121kernel won't deliver status twice, so you can't wait AND have the 122signal handler). The usual round of gcc warnings got tracked down, confirmed 123to be useless, and removed, but despite gcc constantly warning "variable is 124not actually used uninitialized" there were some uninitialized variables in 125<b>dd</b> it DIDN'T warn about: found and fixed.</p> 126 127<p>Elliott added an error message to catch <b>xxd -r</b> receiving -p format 128input without -p, 129fixed <b>readelf -n</b> for x86-64 ibt/shstk notes 130(<a href=https://github.com/landley/toybox/commit/6c23ff0168c1>whatever those 131are</a>), 132and fixed <b>blkid</b> not to check filesystem signatures that would continue 133past the end of the loaded buffer. 134Michael Shavit pointed out that <b>devmem</b> command line number parsing 135wasn't unsigned (although fixing that means it no longer parses the kmg unit 136suffixes). 137Peter Collingbourne fixed <b>inotifyd</b> skipping the first mask character. 138Taisuke Matsushiro fixed a fencepost error preventing <b>stty</b> from 139setting serial baud rate to 57600. 140Askar Safin fixed a typo in mkroot/packages/busybox.</p> 141 142<u>Library</u>: 143The in-progress rewrite of <b>lib/password.c</b> no longer uses shadow.h 144(which lets it build against bionic), 145and a new <b>lib/hash.c</b> where toybox implements its own crypt() using 146the md5/sha1/sha3 etc hash functions we've already got (to work around glibc's 147ongoing deprecation of posix features), but neither are quite finished yet. 148New <b>lib/elf.c</b> with plumbing shared by <b>file</b> and <b>readelf</b>, with 149additional arm and riscv flag decoding, 150the <b>peek()</b> family of functions now return long long instead of 151int64_t so the base types are consistent on 32/64 bit, 152<b>lib/args.c</b> now handles the high bit octal escape sequences in trailing 153[-abc] blocks, so you can have punctuation participate in option groups, 154switching off an option via grouping now sets it back to its default 155value (not to zero), and numeric and date parsing report out of range numeric 156or millisecond values.</p> 157 158<u>Mkroot</u>: 159There's a <b>README</b> in the mkroot directory now. (Microsoft Github was 160picking up the README.root file used by the tar-for-web.sh script 161and showing that instead.) 162Extended <b>run-qemu.sh</b> so it can be run from other directories 163(prepending $(dirname $0)/ to linux-kernel and initramfs.cpio.gz when 164loading them into qemu). 165The initramfs now has <b>/etc/os-release</b> with the toybox $VERSION. 166The big if/else staircase of linux kernel target configurations 167moved into its own <b>get_target_config()</b> shell function, with 168a big comment block right before it explaining what all the variables 169it sets mean, and the generic kconfig symbol list got moved out to 170a <b>GENERIC_KCONF</b> variable assignment right after that function. 171The <b>microconfig</b> format got compressed slightly more 172by use of bash brace expansion, decoded to comma separated values by 173the be2csv function ala KCONF="$(be2csv SPI{,_BITBANG} IOMMU_SUPPORT)" . 174The init script no longer calls <b>oneit</b>, instead doing the work in the 175init script. (Redirect stdin/out/err to the "real" console device on 176the last line of /sys/class/tty/console/active, call setsid, trap SIGCHLD, 177reboot when the shell exits. This (and the setsid tty stealing upgrades) also 178fixed <b>ps</b> not showing any processes in mkroot because stdin/stdout weren't 179properly claimed as the controlling tty. 180The base kernel config now uses <b>CONFIG_PANIC_TIMEOUT=1</b> so 181we don't have to say panic=1 on the kernel command line anymore. 182The baseconfig now automatically adds CONFIG_MODULES and 183CONFIG_MODULES_UNLOAD when any <b>$MODULES</b> are listed. 184Moved the kernel modules archive into the "docs" directory. 185New <b>$NOTOYBOX</b> variable skips the toybox build if set. 186The <b>$DTB</b> files no longer have a path, instead using "find" 187to install them. 188The build now creates the externally loadable cpio.gz file even when 189building initramfs into the kernel, but sticks it in the docs/ subdirectory 190when it's not used.</p> 191 192<p>Several new <b>mcm-buildall.sh</b> targets (or1k, riscv32, riscv64, sh4eb) 193taking advantage of upstream musl-cross-make updating itself to use 194musl-1.2.5. The target list in mcm-buildall.sh moved into a <b>TARGETS=()</b> 195array with a big comment before it explaining the format, and the script 196is better about building host toolchains 197to build other cross compilers with: it tries to autodetect the host 198type (moving the relevant one to the front of the build list), and 199will also build a missing host toolchain when given targets on the 200command line. Added microblaze and or1k kernel configs, neither of which 201know how to exit qemu yet. (If Linux has got a reboot mechanism it knows 202how to poke that qemu is listening for on either virtual board, I haven't 203found it yet.) Also added sh4eb with FDPIC support (see kernel patch, which 204allows qemu-system-sh4eb to run the sh2eb nommu root filesystem), and armv4l 205(the 486 of arm).</p> 206 207<u>Pending</u>: 208Elliott fixed and improved <b>strace</b>, translating more system calls. 209Rob enabled command recursion in <b>toysh</b> (so it can call non-builtins as 210builtins), fixed calling "exec" on NOFORK commands (which MUST run 211in the shell's process context), redid the code to handle trailing 212backslashes gluing lines together, and it no longer leaks script 213filehandles into child processes. 214Oliver Webb noticed that OLDTOY(MAYFORK) aliases didn't become shell 215builtins, added <b>tr -t</b>, in <b>vi</b> added line gotos in ex mode, 216'g', 'v', 'j' commands, and backwards search, cleaned up stuff in 217<b>xzcat</b>, <b>getopt</b>, <b>bc</b>, and removed a bunch of useless autogenerated 218<b>bc</b> tests. Jarno Mäkipää added the list of supported commands to <b>vi</b>'s 219help text. 220Mathieu Anquetin fixed <b>ip link set</b> parsing, and NUL byte handling 221in <b>hexdump</b>. 222General cleanup passes on <b>expr</b> (which now uses 64 bit comparisons), 223<b>tr</b>, and <b>dhcpd</b>.</p> 224 225<u>Cleanup</u>: 226pass over <b>lsattr</b>, <b>getconf</b>, <b>acpi</b>, and <b>blkid</b>. 227Use FLAG() macros in <b>ifconfig</b> and <b>restorecon</b>. Removed TODO block 228from <b>nbd-client</b>. 229More error_msg("%s", str) converted to error_msg_raw(str). 230Trimmed the GLOBALS() sizes (using <b>scripts/probes/GLOBALS</b>): 231removed 1k of global data from <b>cksum</b> and switched a 2k global 232in <b>grep</b> to a malloc. Simplified scripts/probes/bloatcheck and 233mkroot/record-commands.</p> 234 235<u>Portability</u>: 236Fixed <b>truncate -s 8g</b> on 32 bit hosts, 237updated <b>mcm-buildall.sh</b> to autodetect host type and automatically 238select newer package versions, 239and moved the <b>linux32</b> command to its own file because BSD can't 240build it. 241MacOS had a use-after-free with the getpwuid() return value being recycled 242a more aggressively than Linux. 243Elliott fixed diff.test to work with mksh, and posix_fallocate() 244to work with MacOS. 245More commands enabled in <b>make bsd_defconfig</b> and <b>make 246macos_defconfig</b>. 247Added some (void) typecasts to netstat, oneit, and watchdog to try to 248shut up compilers that insist on responding to uncorrectable failures. 249(Such as fgets() reading known /proc data from the kernel, writes to 250the watchdog timer that will reboot the system if they fail, or PID 1 calling 251dup() on stdin to create stdout and stderr after closing the old ones. 252If any of that goes wrong, what are we supposed to do about it?)</p> 253 254<u>Documentation</u>: 255The website now has a <a href=quick.html>quickstart</a> page, 256and the site now defaults to the about page (updated index.html symlink). 257The "current release" is now in the common header displayed by most pages. 258The mkroot dir has a 259<a href=https://github.com/landley/toybox/blob/0.8.11/mkroot/README>README</a>, and its <a href=faq.html#mkroot>faq entry</a> 260has been updated. 261New option to compress help text with <b>gzip</b>, and help text size now shows 262up in <b>make bloatcheck</b>. 263Fix <b>sed --help</b> to show full help, <b>patch -F</b> is now mentioned 264in the help text, tweaks to help text in <b>cut</b>, <b>grep</b>, 265<b>blkdiscard</b>. 266Elliott did a cleanup pass over "usage:" lines and alphabetized more help text 267options. 268Christian Koestlin fixed a typo in the <b>dmesg</b> docs. 269The <a href=release.txt>release 270procedure checklist</a> has been updated, and the README generated 271by <b>mkroot/tar-for-web.sh</b> got updated. 272New faq entry explaining why toybox doesn't have (or need) cttyhack, 273and the architecture list got updated to match mcm-buildall.sh. 274The usual roadmap updates.</p> 275 276<p>New <b>scripts/probes/cmd2dpkg</b> script shows what 277debian package (if any) each toybox command lives in, but only checks 278installed packages (dpkg-query -S "$(which $COMMAND)") and debian breaks 279up source packages strangely, so did not used it to redo the roadmap section 280listing packages (created using Linux From Scratch's list of commands 281installed by each source package).</p> 282 283<u>Plumbing</u>: 284The <b>yes</b> command now uses writev() to go way faster, mostly to 285prove it can. 286The default <b>stdout</b> buffer type switched to block buffering to make 287Elliott happy, which is a rich source of bugs (many of which Elliott 288has already whack-a-moled) and means among other things 289that mixing write() and fwrite() calls now requires manual flushing. 290Meanwhile xflush() became xferror() because it usually 291wasn't flushing anyway, and now that's more noticeable. 292Removed <b>TOYFLAG_LOCALE</b> and just set utf8 locale for all commands. 293The dirtree code now sets <b>DIRTREE_SYMFOLLOW</b> in "again" when we followed 294a symlink to get to this node, callers checking <b>DIRTREE_COMEAGAIN</b> need 295to mask out the bit they're interested in. 296Improved <b>scripts/probes/findglobals</b> output to be more legible.</p> 297 298<u>Test suite</u>: 299Added <b>make test_help test_install</b> targets. (The standalone 300command targets conflict with the help/install make targets, but you 301can still build them via scripts/single.sh and there's no reason not 302to be able to individually test them.) 303New <b>utf8locale</b> shell function attempts to enable a UTF8 locale for 304TEST_HOST, which was used to fix <b>wc.test</b>. 305Colin Cross fixed <b>pidof.test</b> when stdout is a socket. 306Something called "ecryptfs" has a maximum filename length of 143 bytes, 307so our <b>tar.test</b> for long filenames support needs to be skipped there 308because the filesystem can't handle it. Similarly, our <b>truncate.test</b> 309was failing because storing nothing on ecryptfs wasts more space than expected. 310The <b>tar</b> tests now fetch user/group names with "stat" because 311between Linux, Android, FreeBSD, and MacOS, there aren't really consistent 312user and group names for any existing files. Added nbd-client.test 313Thiébaud Weksteen fixed the <b>getfattr</b>/<b>setfattr</b> tests on 314filesystems using selinux. Oliver Webb added <b>sha3sum</b> tests. 315Elliott stabilized another slightly racy <b>ifconfig</b> test (big test 316farm, hits the weird corner cases).</p> 317 318<u>Build</u>: 319Lots of changes to <b>scripts/make.sh</b> and friends to speed up and parallelize 320header generation. The build now figures out which toys/*/*.c files 321to compile by searching for unindented USE() macros wrapping NEW(commandname), 322which among other things fixes enabling the builtins when selecting just 323CONFIG_SH in menuconfig (instead of doing a "make sh" single command build). 324It also probes for optional shared libraries in parallel. 325New <b>scripts/probes</b> directory to collect scripts that aren't actually 326used by the build, with a README in it. 327Updated .gitignore so it doesn't complain about spurious name collisions in 328subdirectories. Elliott shut up a 329<a href=https://github.com/landley/toybox/commit/2c36636377e1>truly sad</a> 330warning where despite sizeof(ptrdiff_t) always being sizeof(long), gcc 331nevertheless warns if you printf("%ld", ptr-ptr) and wants a magic invented 332"%td" type on 32 bit systems (but is just fine on 64 bit). (Or we could add a 333literally NOP typecast to long.) And silenced a spurious gcc 13.2 warning 334in date.c. And Rob shut up <a href=https://github.com/landley/toybox/commit/a17fee778851>more broken gcc warnings</a>. 335Updated .gitignore so it doesn't complain about spurious name collisions in 336subdirectories.</p> 337 338<a name="30-07-2023" /><a href="#30-07-2023"><hr><h2><b>July 30, 2023</b></h2></a> 339<blockquote> 340<p>The way it functioned was very interesting. When the Drink button was 341pressed it made an instant but highly detailed examination of the subject's 342taste buds, a spectroscopic analysis of the subject's metabolism and then 343sent tiny experimental signals down the neural pathways to the taste centers 344of the subject's brain to see what was likely to go down well. However, no 345one knew quite why it did this because it invariably delivered a cupful of 346liquid that was almost, but not quite, entirely unlike tea.</p> 347<p>- The Hitchhiker's Guide to the Galaxy</p> 348</blockquote> 349 350<p><a href=downloads/toybox-0.8.10.tar.gz>Toybox 0.8.10</a> 351(<a href=https://github.com/landley/toybox/releases/tag/0.8.10>git commit</a>) 352is out, with prebuilt <a href=downloads/binaries/0.8.10>static binaries</a> and 353<a href=downloads/binaries/mkroot/0.8.10>mkroot images</a> 354bootable under QEMU (built using a <a href=https://github.com/landley/linux/tree/mkroot-6.4>lightly patched</a> linux-6.4).</p> 355 356<p>New command <b>shuf</b>, and Elliott added <b>i2ctransfer</b>. 357Finished cleaning up <b>dd</b> a promoted it out of pending.</p> 358 359<p>Lots of changes to mkroot, <a href=http://landley.net/talks/mkroot-2023.txt>working</a> 360on updated documentation but the release is already overdue...</p> 361 362<p><u>Features</u>: 363Added a bunch of options to <b>tar</b> (-s --ignore-case --wildcards 364--anchored --wildcards-match-slash --one-file-system), <b>ls --sort</b> 365(which takes a comma separated list of fallback sorts, and new 366sort types to sort by extension, case insensitive, and directory first), 367<b>cpio -R</b> to set user/group of all files and <b>cpio -L</b> to 368follow symlinks, <b>gzip -t</b> to test file integrity, <b>find -readable</b>, 369<b>cut -DFR</b> now has --longopts aliases (as suggested by the coreutils list). 370Taught <b>blkid</b> to recognize iso (cdrom) image files, <b>oneit</b> now 371autodetects console from sysfs with no -c argument, added "next address" 372support to <b>i2cget</b> and enforced bus limit, 373added posix 1x2x3 support to <b>dd</b> (with overflow detection, and 0x 374at the start still means hexadecimal).</p> 375 376<p>Jarno Mäkipää taught <b>i2cget</b> to read without passing a command byte. 377Elliott added <b>xxd -e</b> (little endian), <b>lspci -x</b> (hex dump of 378pci config space), <b>ls --block-size</b>, widened the default 379VSIZE field width in Android's default <b>ps</b> display (they're up to 10 gigabytes of 380ASLR padding for hello world), and increased the <b>vi</b>/<b>top</b> 381output buffer size to reduce flicker redrawing the screen.</p> 382 383<p><u>Bugfixes</u>: 384Taught <b>switch_root</b> to move mountpoints into the new root. 385The <b>head</b> command now uses lseek to restore unused data from seekable 386inputs (tested with <b>read i; echo =$i; head -n 1; done < input</b>). 387In <b>lspci</b> handle PCI bridges and don't show "revision 0". 388Missing /proc entries no longer cause <b>vmstat</b> errors, the output 389fits into 80 columns more often, and adds reclaimable to the cached field. 390Fix </b>help -u -au</b>, and the "see othercommand" generation. 391Adding fuzz support subtly broke <b>patch</b> in a couple places: fixed -R and 392matching EOF. Trying to create a directory that already exists is no longer an 393error for <b>cpio -i</b>, and <b>-d</b> just warns about directory creation 394failure instead of exiting. Adding <b>--wildcards</b> logic to <b>tar</b> 395broke the full string 396matching (it checked after every / but not at the start of the string 397when there weren't any), and <b>tar</b> now waits for the compression program 398to exit and passes on any error result instead of killing it (which could 399truncate output). Fixed more <b>realpath/readlink</b> corner cases, with tests. 400Fix a <b>sed</b> bug when s//&/ (the copy-matched-text operator) was the 401first change made to the string. Fix error message for <b>find</b> -commands 402that need an argument but were at the end of the command line (it said 403"unknown command" instead of "command needs argument"). Fixed <b>killall5</b> 404parsing of command names with spaces in them. Fixed <b>grep -m -A</b> used 405together (-m stops new matches but -A still shows lines after match). 406Fixed <b>mv</b> to --preserve=all when copying between filesystems. 407Moved the EPIPE skip from perror_exit() to verror_msg() so we never 408output a "hey, our output pipe unexpected closed!" error message when 409we're not the last command in a pipeline.</p> 410 411<p>Franke Busse reported that <b>bunzip2</b> was deleting its input file 412on error, and <b>date -s +abc</b> was segfaulting. 413Jooyung Han fixed <b>nsenter</b> using -C and -t together. 414More fixes from Eric Roshan-Eisner's toysh fuzzing, and he also sent a fix 415for an error message in the <b>patch</b> command. 416Elliott fixed stdin handling in <b>log</b> and <b>logger</b> to read a line at 417a time and log each one separately, fixed an off by one bounds check 418in <b>readelf</b> (and factored out the bounds checking logic so it's 419common between sections, and improved the error messages), fixed 420a <b>gpiod</b> crash.</p> 421 422<p><u>Library</u>: 423Implemented DIRTREE_BREADTH to do breadth first processing of directories 424(among other things, allowing entries to be sorted and processed in a 425consistent order). The dirtree plumbing now has macros for the mode bits 426instead of hardwired numbers, 427better adler32 calculation in deflate.c, 428and <b>fs_type_name()</b> can now convert most of the linux/magic.h constants 429into "mount -t TYPE" names.</p> 430 431<p><u>Mkroot</u>: 432Moved mkroot into its own directory: you now run <b>mkroot/mkroot.sh</b> instead 433of scripts/mkroot.sh, and all the mkroot plumbing should be under there now. 434What was scripts/root is now <b>mkroot/packages</b>, and I added a "busybox" 435package to download and build that (useful for testing command 436implementations side by side).</p> 437 438<p>Reorganized the contents of each root/$ARCH build directory so only 439the files necessary to run are at the top level, and everything else 440(kernel configs) is in a docs/ subdirectory. Added linux-microconfig 441with the three CSV lines actually used by the build. 442New <b>mkroot/tar-for-web.sh</b> script that generates tarballs for 443uploading to the web page and adds two README files (one at the top level 444and one to each tarball's docs directory). The generated tarballs don't 445contain the fs/ dir, instead the README explains how to extract 446and recreate the cpio.gz archive.</p> 447 448<p>Taught KCONF=value,value,value... CSV format to understand NAME=VALUE 449entries, and merged most KERNEL_CONFIG lines into KCONF. 450(Note that while it can handle NAME="this that" you need to escape it as 451NAME=\"this\ that\" to work right.) Added the legacy NLS 452dependencies the VFAT driver needs (even though we selected UTF8) to the 453base config. The init script now tests if stdin is open via 454"2>/dev/null <0 || blah" so only does the exec redirect for /dev/console 455when necessary. 456Fixed the .config checker to replace the toybox config when CONFIG_SH=y 457isn't set, so "make defconfig; mkroot/mkroot.sh" should work now. 458Switched initramfs packaging to use more portable <b>cpio -R 0:0</b> 459instead of --no-preserve-owner. Moved the -m 256 before $@ in run-qemu.sh 460so setting memory size on the command line overrides the default. 461The <b>mkroot/record-commands</b> setup script will now use logpath 462out of the $PATH if it exists, rather than having to compile it each time. 463Bo Svangård fixed a typo in the cpio command line that's ignored by toybox but 464breaks with the host tools.</p> 465 466<p><u>Pending</u>: 467Moritz Weber put in more work on <b>git</b> (still very unfinished). 468Eli Lipsitz fixed bugs in init and login. Elliott added risc-v support 469to strace, and fixed :w in vi (it didn't 470takes a filename argument and would crash if you started with no filename) 471and improved error reporting. Jarno Mäkipää taught vi about backspace 472and improved empty file handling. Rob fixed vi -s. Various cleanups to 473<b>tcpsvd</b>.</p> 474 475<p>More toysh work: command line option parsing stops at first nonoption argument, 476all piped segments are now implicitly subshelled so variable 477lifetimes match bash's and loops don't have to exit before proceedinng to the 478next one (ala "while true; do echo hello; done | while read i; do echo $i; done" 479hanging on a full output pipe). Several changes to HERE document processing 480(variable expansions can cause line continuations in a HERE document, etc), 481and better line continuation handling in general, added set -u, 482fixed if/while return codes, wired up the "local" builtin, fixed an escape 483passthrough bug reported by Mingliang Hu, support undelimited redirects 484(ala cat<file with no spaces), fixed ${X::} with no arguments.</p> 485 486<p><u>Cleanup</u>: 487Converted <b>chgrp</b> to FLAG() macros and did some minor cleanup on <b>bootchartd</b>, 488(which needs tests), <b>basename</b>, <b>bzcat</b>, <b>test</b>, 489<b>i2cdetect</b>. Elliott switched <b>flock</b> 490to FLAG() macros, switched nohup to use octal permissions insted of macros, 491widened the default VSIZE field width in Android's default <b>ps</b> display 492(which is now inserting 10 gigabytes of ASLR padding into "hello world"), and increased 493the vi/top output buffer size to reduce flicker redrawing the screen. Multiple cleanups to 494<b>cpio</b>. Added requested --gratuitous-longopt synonyms to <b>ls</b>. 495Consistently capitalize TODO comments in the tree so they're easier to search for. 496Silence yet more perennial "warning: variable is never used uninitialized" gcc false 497positives.</p> 498 499<p><u>Portability</u>: 500utf-8 support still isn't automatic in libc (you have to set an appropriate locale to 501enable it), and locales aren't entirely standardized: try C.UTF-8 (which MacOS hasn't got) 502and fall back to en_US.UTF-8 (which Gentoo hasn't got). 503Removed itoa() from seq.c because cygwin pollutes the namespace with 504DOS debris.</p> 505 506<p>Dan Bryant renamed xrunread()'s "stdin" argument to "to_stdin" to work around 507broken headers that #define stdin as a macro instead of a global variable. 508Elliott enabled copy_file_range() on Android.</p> 509 510<p><u>Documentation</u>: 511Calling <b>command --help</b> and <b>toybox --help command</b> shows 512the toybox URL banner, but "help command" doesn't. 513Reorder the <b>find</b> help text so match filters that take an argument are mostly 514in the left column. Minor help text tweaks in sleep, find, xxd, cpio. 515The README now points to the same binary cross compilers as the FAQ. 516Update <b>sed</b> help text now that -e and -f get processed in command 517line order when used together.</p> 518 519<p><u>Plumbing</u>: 520The FLAG() macros now always return 1 or 0, so we can CONSTANT*FLAG(X) without needing 521a !! in there. The help plumbing now has HELP_FLAGS. 522Each command's option string in NEWTOY() may now use octal escapes with the 523high bit set to use control characters as option flags, with the resulting 524FLAG_X## macro using the 2 digit hex value (without the high bit set). For 525example if you needed "command -:" then ":" is ascii 58 (hex 3A, 526toybox's <b>ascii</b> command is useful for this), +128 is 186, which in octal 527becomes "\272" and tested with if (FLAG(X3A)) in the code.</p> 528 529<p><u>Test suite</u>: 530Each command.test now runs in an empty subdirectory, and working files like 531expected/actual go elsewhere (only "input" is created for you in there, and only 532when you provide input in the current test's 4th argument), so the directory 533listing doesn't contain anything the test didn't put there. (This simplified 534several tests.) 535The realpath test broke on mksh (which doesn't export $PWD by default). 536Elliott fixed the ifconfig tests on android. General tidying and adding 537tests to sed, ls, modinfo. The tar tests broke on github because it 538switched to a filesystem that can't do sparse files with 4k granularity, 539added a check to skip those tests (and also skip them on MacOS, which 540fails for a different reason.) 541When a test's name field was blank (meaning the actual test command line 542should be displayed as the name) the test plumbing was adding the command 543name to the display twice. 544The <b>make tests</b> target now defaults to building toybox with ASAN when 545not testing an existing binary. (You still have to specify ASAN=1 manually 546for individual command tests.)</p> 547 548<p>Fixed <b>demo_utf8towc</b> which had the range check backwards (0x10ffff is 549the biggest _output_, not input). The range check was an attempt to avoid 550spending a long time going through all 4 billion entries, but wasn't testing 551everything it needed to, so now it advances the counter with a bit shift 552based on the number of bytes consumed by the match (avoiding large swaths 553of redundant checks but still theoretically testing all meaningfully different 5544 byte inputs).</p> 555 556<p><u>Build</u>: 557Fix return code of verbose mode in scripts/build.sh so <b>make V=1</b> succeeds. 558Speed up the MacOS build: their gsed is slow to launch so avoid one repeated call in a 559loop, and reimplement the wait $PID codepath because their bash is 17 years old. 560The llvm+bionic toolchain warns about printf(" "+FLAG(m)) unless you add an 561extra ,0 argument, but gcc+glibc warns if you DO add the extra argument... so 562I used the write() system call to avoid the compiler "helping" in a way I 563can't make it STOP. Similarly, now that FLAG(u) is always 1 or 0, 564<b>size = FLAG(u) ? : sizeof(toybuf)</b> 565is either 1 or 4096, which is what I _want_, and the compiler warning about it 566is counterproductive. LLVM needs a new -Wno-invalid-source-encoding flag 567or else it complains about octal escapes in string constants. (When I want 568to put utf-8 in there I can just do so, why would I octal escape it? 569Stop "helping".)</p> 570 571<a name="10-01-2023" /><a href="#10-01-2023"><hr><h2><b>January 10, 2023</b></h2></a> 572<blockquote> 573<p>"Why," Arthur said, "is there a sofa in that field?"</br> 574"I told you!" shouted Ford, leaping to his feet. "Eddies in the space-time 575continuum!"</br> 576"And this is his sofa, is it?"</p> 577<p>- The Hitchhiker's Guide to the Galaxy</p> 578</blockquote> 579 580<p><a href=downloads/toybox-0.8.9.tar.gz>Toybox 0.8.9</a> 581(<a href=https://github.com/landley/toybox/releases/tag/0.8.9>git commit</a>) 582is out, with prebuilt <a href=downloads/binaries/0.8.9>static binaries</a> and 583<a href=downloads/binaries/mkroot/0.8.9>mkroot images</a> 584bootable under QEMU (built using a <a href=https://github.com/landley/linux/tree/mkroot-6.1>lightly patched</a> linux-6.1).</p> 585 586<p>The new <b>nbd-server</b> command interoperates with nbd-client to serve 587network block devices (using the v1 protocol), and Moritz Weber contributed a read-only <b>git</b> 588implementation to pending.</p> 589 590<p><u>Features</u>: New <b>grep</b> fast path for fixed or simple patterns that 591don't need the full regex engine (I.E. "^", ".", "$" but not "*" or "[]") 592most noticeable when searching for many patterns at once. 593Improved <b>tar --xform</b> support parsing flags= and trailing s/// scope 594flags (but using --xform now requires toybox sed in the $PATH).</p> 595 596<p>Added <b>sort -C</b>, <b>lsusb -i</b>, <b>netcat -n</b>, 597<b>swapoff -a -v</b>, <b>httpd -v</b>, <b>nbd-client -b</b>, 598a <b>uname -p</b> stub to mollify package builds, 599<b>sleep</b> accepts multiple arguments, 600<b>sed</b> now parses interlaced -e and -f arguments in order, 601several new options in <b>readlink</b> and <b>realpath</b> (and better 602handling of relative paths), and UDP mode in netcat is more useful now: 603<b>netcat -u -s 127.0.0.1 -p 9876 -l</b> can type at <b>netcat -u 127.0.0.1 9876</b> 604in both directions. (Figuring out when the other side hangs up is an unsolved 605problem, but that's UDP for you.) 606Elliott added <b>ls -N</b>.</p> 607 608<p><u>Bugfixes</u>: Fixed off by one error in <b>sort -c</b> output, 609fixed <b>sed -z</b> and other sed cases where Linux has never obeyed 610Posix (N at EOF does a default print, l escapes \n), 611<b>patch --dry-run</b> should no longer create or delete files, 612<b>gzip</b>/<b>zcat</b> couldn't handle concatenated archives, 613the ./configure of gmake 4.3 depends on <b>test</b> treating one argument -a as a synonym for -e, 614autodetectiong compression types in <b>tar</b> should be more reliable now 615and sanitize weird path corner cases (like .. past /) better, 616<b>scripts/make.sh</b> (and thus "make toybox") should no longer truncate log 617files stderr is redirected to, 618<b>httpd</b> now handles ? and # in URLs, 619<b>xxd</b> disables columns for -c 0 and groups for -g 0 (so you can get 620a long interrupted string of hex digits out of it), two fixes to 621<b>mountpoint</b> (the conversion to same_file() was inappropriate because 622the logic isn't quite the same, and -q should also quiet "not found" errors), 623fixed <b>httpd</b> not always displaying index.html files instead of directory 624contents, fixed <b>comm -</b> not recognizing it as stdin, and multiple fixes to 625<b>timeout</b> which now kills process group and thus child processes, 626isn't suspended by SIGTTIN, and recursive commands it calls don't 627inherit an inappropriate SIGCHLD handler.</p> 628 629<p>Yi-Yo Chiang fixed loopback <b>mount</b> (the recent switch to xrunread() 630kept the newline from losetup's output) and added <b>tail +123</b> 631(old-style synonym for tail -n +123). 632Daniel Mentz added scmversion to the <b>modinfo</b> tag list (Android uses 633it for external modules). Alexander Holler fixed <b>su</b> to not require 634/etc/shadow when run as root, and nomas2000 reported an inverted test in 635<b>date</b>'s check for trailing rubbish. Daniel Mentz fixed an off by 636one in <b>grep -f</b> that discarded the last character of the pattern file's 637contents. Kelvin Zhang made <b>modinfo</b> better at handling symlinks. 638Antoni Villalonga fixed <b>fmt</b> on 64 bit big endian systems (like s390x and 639powerpc). Li Cheng fixed <b>mount</b>'s type detection not autodetecting 640the need to bind mount files when type "none" was specified in fstab. 641Tomasz Sterna reported that <b>modprobe</b> resolving dependencies shouldn't 642feed a NULL pointer to the syscall's options arguments (the kernel goes -EWTF), 643and Vincent Donnefort made modprobe work when /proc/modules isn't available.</p> 644 645<p><u>Library</u>: 646The TOYFLAG_ARGFAIL() exit value is now the default <b>error_msg()</b> sets, 647<b>sendfile()</b> now falls back to the read/write loop for any error (not just 648EINVAL, the kernel sends ENOSYS and EXDEV and who knows what else), 649<b>xgetrandom()</b> can now return arbitrary amounts of data (looping internally 650as necessary), 651<b>xwaitpid()</b> returns status 127 for cases (like bad PID) which don't 652return a status for the PID, 653several fixes to <b>xabspath()</b> as part of the readlink/realpath work, 654new <b>octal_deslash()</b> to remove octal escapes common in kernel strings 655(ala /proc/mounts), 656the <b>FLAG_x</b> macros always use a 64 bit type now (so you don't have to 657worry about toys.optflags &= ~FLAG_x; blanking the top 32 bits), 658replaced memcmp() with new library function <b>smemcmp()</b> to placate ASAN 659when returning first difference in known differing arguments of unequal length. 660Elliott added three more filesystem types to <b>fs_type_name()</b> which is 661used by <b>stat -f</b> and friends.</p> 662 663<p><u>Mkroot</u>: 664New <b>scripts/test_mkroot.sh</b> runs each target under qemu to confirm 6651) it boots, 2) the block device works, 3) networking works, and 4) the clock is set 666reasonably. It runs them in parallel, with an httpd instance on the host to 667fetch a file from, and a timeout to detect hangs. 668New <b>scripts/root/overlay</b> package copies $OVERLAY directory into 669target filesystem, ala <b>scripts/mkroot.sh overlay OVERLAY=$PWD/blah</b> 670The init script now mounts any /dev/?da on /mnt and when /mnt/init exists 671sets $HANDOFF to that instead of running a shell prompt, so you can provide 672automated control images that run code in the emulator automatically. (The 673emulator process still when whatever $HANDOFF called exits.)</p> 674 675<p>Some kernel changes finally made it upstream so we can drop workarounds 676for them, such as kernel commit 677<a href=https://github.com/torvalds/linux/commit/f8f0d06438e5>f8f0d06438e5</a> 678fixing the <a href=https://landley.net/notes-2014.html#13-08-2014>longstanding</a> 679"allnoconfig has =y" issue so miniconfig doesn't need to add a magic #comment 680line forcing a symbol OFF anymore. The remaining kernel patches (including 681several maintained locally but hadn't merged into mkroot) moved to a 682<a href=https://github.com/landley/linux/tree/mkroot-6.1>seperate repository</a> 683so mkroot.sh doesn't have to call sed on the build snapshot. (They're are also 684included in the mkroot binary release directory, and were all submitted to lkml 685months if not years ago.)</p> 686 687<p>The host airlock setup (creating the temporary restricted environment 688mkroot builds packages in both so they don't pick up strange dependencies from 689the host, and to prove the build can use the toybox commands) 690removed a bunch of commands (dd, diff, vi, xzcat, ar, nm) no longer 691needed by the linux-6.1 build. The kernel patches also remove the need 692for gcc (it can use the "cc" symlink all modern distros install, and 693autodetects whether that points to gcc or clang) and bc (which is just 694<a href=https://landley.net/notes-2013.html#18-05-2013>generally obsolete</a>). 695Left those two in for one more release, but NEXT time building a vanilla 696kernel without the <a href=https://landley.net/toybox/downloads/binaries/mkroot/0.8.9/linux-patches/0001-try-generic-compiler-name-cc-before-falling-back-to-.patch>gcc removal patch</a> 697and <a href=https://landley.net/toybox/downloads/binaries/mkroot/0.8.9/linux-patches/0004-Replace-timeconst.bc-with-mktimeconst.c.patch>bc removal patch</a> 698may require you to add <b>HOST_EXTRA="gcc bc"</b> to the mkroot command line 699for the airlock build to add extra symlinks to root/build/airlock.</p> 700 701<p>Added a kernel build for powerpc64 big endian, 702m68k and powerpc now support "run-qemu.sh -hda file.img", and m68k only 703has one /dev/?db (disabled Apple Desktop Bus in the config). 704Added basic module support (the kernel build saves modules in modules.cpio.gz 705so it can extract them when you rebuild the root filesystem but not the kernel) 706and added <b>scripts/root/tests</b> which (among other things) builds two 707innocuous modules that depend on each other (fscache and cachefiles) for 708modprobe/insmod testing that hopefully won't break anything else. (If 709there was a scripts/root/pending/ tests would be in it: it does not actually 710run the test suite under mkroot yet, because toysh needs work.)</p> 711 712<p><u>Pending</u>: Elliott made <b>strace</b> build on 32-bit x86. 713James Farrell added <b>diff -f</b>. 714Alexander Holler fixed two parameter substitution bugs in <b>toysh</b>. 715Rob fixed assignment suppression, && and || parsing, math 716priority, a number of ASAN test failures, untangled and commented the 717brace expansion logic, and other ongoing shell work.</p> 718 719<p>A large rewrite of diff happened but couldn't be checked in unfinished 720because if people are using stuff out of pending, how do you do a pending for 721pending? (Meta-pending?)</p> 722 723<p><u>Cleanup</u>: 724<b>cksum</b>, <b>od</b>, <b>file</b>, <b>getopt</b>, <b>gpiod</b>, 725<b>nbd-client</b>, better output messages for <b>host</b>, 726and make <b>base64</b> use line buffering. 727Moved -Wno-string-plus-int into portability.sh and had it only apply to 728clang. Moved num_cache() out of lib into netstat (its only user).</p> 729 730<p>In the test suite plumbing, $SKIPNEXT was replaced by <b>$SKIP</b> holding 731a count that counts down with each test (so you can easily skip a block of 732tests). Fixed <b>txpect</b> to actually listen to error code (oops) and to not 733complain about multiple X. The old dochroot() and mkchroot() shell functions were removed from 734scripts/runtest.sh because they required root access to use (which mkroot 735does not). More conversion to use <b>testcmd</b> in individual command.tests 736(both to remove workarounds for shell builtins, and to be more concise). 737Remove redundant cleanup from several tests (which run as proper child processes 738now, and the work directory is deleted and recreated between each, so 739individual tests need less cleanup).</p> 740 741<p><u>Portability</u>: More tests now pass on MacOS and FreeBSD 742(weird filesystems, root vs wheel group name, dealing with headers that 743define stdin as a macro and don't define PATH_MAX, the ancient bash in MacOS 744doesn't understand &>>, won't let normal users use "chmod +s", 745symlinks don't have all permission bits set, no O_PATH and the Linux definition 746is used for something else, and so on), at least half of which Elliott did. 747Not sure we've quite got Mac and BSD 748doing "gmake test_singlecommand" right yet, but it's a lot closer. 749Added all the commands to <b>make bsd_defconfig</b> that compile on FreeBSD 13 750(which is no guarantee they WORK). 751ASAN=1 should work in more places now, including <b>make_test_command</b>. 752The build uses $CC insted of cc in more places, and doesn't apply $ASAN 753for $HOSTCC (the legacy kconfig/ directory stands no chance).</p> 754 755Fedora inexplicably sticks \x09 style hex escapes into the dmesg output and 756expects them to be parsed. 757Android is still running the test suite under mksh which doesn't 758understand bashisms like <(command). LLVM insisted that "ptr - 1" could 759never be NULL (demonstrably untrue) and needed some (unsigned long) typecasts 760to force it to actually do the math. 761Khem Raj aliased timer_settime to timer_settime64 when the first isn't 762available. Antoni Villalonga debianized some install paths and added 763a bunch of --longopt synonyms for existing options that package builds 764use.</p> 765 766<p><u>Documentation</u>: The <a href=help.html>help page</a> 767generation logic (toybox help -av) now filter out invisible aliases and 768outputs "See command" for visible aliases, plus fixes for broken corner 769cases like <b>nohup --help</b>. 770The <a href=https://landley.net/toybox/git>local git repo</a> 771now has a simple index.html generated by a 772<a href=https://github.com/landley/toybox/blob/0.8.9/scripts/git-static-index.sh>small shell script</a>, 773linking to the <b>git format-patch</b> files for 774each commit, applicable via <b>git am</b>.</p> 775 776<p>There was a 777<a href=https://github.com/landley/toybox/pull/244>bug report</a> that specifying the cross compiler prefix twice 778includes the prefix twice (ala CROSS_COMPILE=prefix- CC=prefix-cc results in 779prefix-prefix-cc) which is what happens when you ask it to do that, yes. It's now 780<a href=faq.html#cross>better documented</a>. (See also 781<a href=https://lkml.iu.edu/hypermail/linux/kernel/2202.0/01505.html>this kernel patch</a>.)</p> 782 783<p>Yi-Yo Chiang's losetup fix spawned an email thread where the 784android developers <a href=http://lists.landley.net/pipermail/toybox-landley.net/2022-August/029139.html>explained</a> why Android doesn't (and can't) 785use devtmpfs. Rob forwarded a link to linux weekly news to see if he could get 786any attention from the kernel guys, but they didn't reply.</p> 787 788<a name="12-08-2022" /><a href="#12-08-2022"><hr><h2><b>August 12, 2022</b></h2></a> 789<blockquote><p> 790Huge as office blocks, silent as birds. They hung in the air exactly the same 791way that bricks don't.</p> 792<p>- The Hitchhiker's Guide to the Galaxy</p> 793</p></blockquote> 794 795<p><a href=downloads/toybox-0.8.8.tar.gz>Toybox 0.8.8</a> 796(<a href=https://github.com/landley/toybox/releases/tag/0.8.8>git commit</a>) 797is out, with prebuilt <a href=downloads/binaries/0.8.8>static binaries</a> and 798<a href=downloads/binaries/mkroot/0.8.8>mkroot images</a> 799bootable under QEMU (built using vanilla linux-5.19).</p> 800 801<p>No new or promoted commands this time. (Time based releases mean NOT 802holding the release to wait for things like that.)</p> 803 804<p>Thanks to Google for sponsoring the project's maintainer to focus on toybox 805for a while. (They want a hermetic build, I want Android to build under 806Android. Working on it...)</p> 807 808<p><u>Features</u>: 809The new <b>timeout -i</b> option kills a command for inactivity (output 810to stdout resets the timer). 811Basic support for <b>tar --xform</b> (but not yet the special tar-only s///x 812options). 813--long options now support unambigous abbrevations by default, so you can 814go "ls --col" and it'll figure out you mean --color. 815Added <b>blkid -o</b> understanding "full", "value", and "export" output 816formats. Added <b>nsenter -C</b> (cgroup namespace) and <b>-a</b> (enable all 817supported namespaces). 818Added <b>mount -R</b> and switched the bind mount default to be recursive 819(it still autodetects --bind and --loop mounts based on the source/target type, 820all --bind or --rbind do is switch the recursive flag on and off).</p> 821 822<p>Elliott implemented <b>tar --null</b>, 823and taught <b>file</b> to recognize some Linux kernel images, and 824binaries for Linux's <a href=https://github.com/torvalds/linux/commit/fa96b57c1490>newly merged</a> 825arch/loongarch (a <a href=https://linleygroup.com/newsletters/newsletter_detail.php?num=4623>mips64 variant</a> 826from <a href=https://phoronix.com/news/LoongArch-MIPS-Copy-Kernel>china</a>).</p> 827 828<p><u>Bugfixes</u>: 829Several cleanups to taskset, mostly due to reviewing it before making 830an explainer video, but the review found an inappropriate use of 831TOYFLAG_STAYROOT that got fixed. 832Argument parsing for <b>unshare</b> and <b>nsenter</b> should now match 833debian's (I.E. "unshare -npu" works the same as "unshare -n -p -u" not 834"unshare --net=pu"). 835<b>kill 1</b> no longer says "unknown pid 1: not permitted" (it 836says "bad pid" like everywhere else). 837<b>test -x</b> and friends now test live access, not permission bits. 838(Root ignores a lot of permissions, chmod u-x blocks you from executing your 839own file even if g+x and o+x should let you access it, things like selinux can 840further break the unix permission model, etc). 841 842<p>In lib/ a fencepost error in <b>chomp()</b> could read one byte past the start of the string, and 843<b>xpclose_both()</b> no longer closes the parent's stdin and stdout 844when they're explicitly passed through to the child. 845In scripts/runtest.sh, moving tests into subshells broke failure reporting with VERBOSE=all. 846While "make install" was creating directories when $PREFIX pointed to a 847nonexistent directory, "make toybox" didn't even though it tried to place the 848output file there.</p> 849 850<p>James Farrell switched the test suite from detecting root via $UID to 851$(id -u). (It's not a security issue, this just controls which tests we attempt. 852It's another workaround for missing features in mksh.) 853Taolaw spotted that httpd could segfault when xabspath() returns NULL 854(for error conditions such as infinitely looping symlinks 855or "." not resolving when the current directory was deleted).</p> 856 857<p><u>Documentation</u>: 858We <a href=design.html>recorded</a> the 859move from C99 to the C11 standard, which was actually made a while ago: 860the typecast constants we've 861been using for a while are a 2011 feature not available in the 1999 edition 862of the C standard. (This came up when we had to move to C11's "_Noreturn" to work around a compiler bug in llvm. 863Yes __attribute__((__noreturn__)) should be identical, that's the bug part.)</p> 864 865<p>The website's nav bar got updated (among other things, linking to 866our <a href=https://www.youtube.com/channel/UC4VFy3wc1nzq5tUHhiti6fw>youtube 867channel</a>), and the FAQ entry on cross compiling got 868expanded. 869The toys/example commands moved from "make list" to "make list_example" 870(since they're not enabled in defconfig either). 871The cross compiling example command line in "make help" now sets 872LDFLAGS=--static instead of putting that in CFLAGS.</p> 873 874<p>The web archive is back, and since the missing messages were 875never delivered (the mailserver trying to deliver them timed out and bounced 876them back after a couple days) this means unlike previous failures the web 877archive isn't out of sync with reality: the hole was the list actually being 878down until Dreamhost tech support figured out what they broke this time.</p> 879 880<p><u>Pending</u>: 881More work on toysh: case/esac was confused by nested flow control blocks, 882support for [[ < > =~ ]] tests, implemented the rest of the $((math)) 883operators and support for ((math)) tests and "for ((math;math;math))" loops, 884and so on. (The next toysh goal is to get the test suite running under mkroot, but 885other stuff keeps coming up.) 886James Farrell upgraded diff, adding three line format options and support 887for diffing FIFOs. 888Rob did several rounds of dd cleanup, then got distracted by diff cleanup 889which turned into a complete rewrite of diff. 890Neither was ready to promote in time for this release though (although the release 891got delayed almost a week before Rob admitted that).</p> 892 893<p><u>Library</u>: 894The <b>lib/args.c</b> command line parser now has two optional modifiers 895(" " and ";") for each argument type, providing 4 combinations (both unset, 896both set, one of each set), and all 4 now have defined semantics and 897tests/skeleton.test exercises them. (This was motivated by <b>unshare</b> 898needing a new combination to match the debian behavior.) 899<b>lib/args.c</b> also added "~" to collate bare longopts (so multiple bare 900longopts can be synonyms setting the same flag or argument variable).</p> 901 902<p>New function <b>xrunread()</b> which launches a child function, writes a 903string to its stdin, and returns everything the child wrote to stdout as another 904string (or NULL if it couldn't run or exited with an error), leaving 905stderr alone (so it goes to the same place the parent's goes).</p> 906 907<p>New functions <b>same_file()</b> and <b>same_def_ino()</b> to simplify 908file metadata comparisons. (If you're wondering why something so simple should 909have a function encapsulating the logic, this release also has at least 3 910different bugfix commits for thinkos from switching all the commands over 911to use them instead of doing the test ourselves. All missing/extra ! or 912&& vs || level stuff.)</p> 913 914<p>The <b>exit_signal()</b> handler now blocks signals so <b>sigatexit()</b> won't 915re-enter the list when it receives two different killer signals. (Since 916<b>xexit()</b> capture uses siglongjmp() this should clean itself up 917when not actually exiting.)</p> 918 919<p><u>Plumbing</u>: 920Several warning suppressions due to things like llvm's fortify finding 921false positives, gcc's "int and long are identical on 32 bits!" warnings 922(yes... they are), the MacOS build being crotchety about vfork()... All verified 923non-issues at present.</p> 924 925<p>The build plumbing now puts more effort into resolving dependencies and figuring 926out when stuff needs to be rebuilt. This includes noticing when the 927toolchain we're building with changes flags, and outputting a warning 928when it detects the need to run "make oldconfig" again. (At the moment 929calling kconfig/ from scripts/ would be a layering violation: I need to 930rewrite kconfig from scratch before it can do that automatically.).</p> 931 932<p>Removed most compile time probes from <b>scripts/genconfig.sh</b>, 933replaced with things like #ifdef (_NR_syscall) and 934#if __has_include(<utmpx.h>), or acknowledging when a probe is a bug 935workaround for a specific libc version and testing for that instead, 936plus simply deleting some config options due to the 7 year horizon expiring 937so we can trust it to be there now.</p> 938 939<p>Usual batch of musl workarounds for behavior that 940<a href=https://github.com/landley/toybox/commit/9a5d7a063c8c>doesn't match</a> 941glibc and bionic or where they've been 942<a href=https://www.openwall.com/lists/musl/2022/07/27/5>arguing about</a> 943stuff for years without actually fixing it. 944Ed Maste of FreeBSD added some FreeBSD checks/cases to 945lib/portability.[hc], mostly in places we already supported Apple or 946OpenBSD. 947The top level "configure" is now re-entrant (shouldn't append stuff 948to enviornment varibles like $CFLAGS twice when read twice). 949The MacOS defconfig now includes a lot more commands.</p> 950 951<p><u>Test suite</u>: 952The diff for failing tests is now output after the FAIL line instead 953of before, "make tests" now runs tar.test by default, 954a new skeleton.test uses toys/example/skeleton.c to exercise the 955lib/args.c command line parsing plumbing, 956scripts/runtest.sh now uses do_pass and do_fail consistently, 957and Rob keeps accidentally checking in bashisms and Elliott keeps submitting 958patches to remove them because android runs the tests under mksh and 959toysh isn't ready yet.</p> 960 961<p>Work in progress to make TEST_HOST=1 pass on newer debian, which is a moving 962target. (The tests didn't change, but the behavior of the gnu utilities 963is not consistent. Or standardized. Or coherent. Or predictable. Or sane.) 964Some minor changes to commands like <b>xxd</b> and <b>cmp</b> to 965make their output look more like what debian's currently producing.</p> 966 967<p>Making the tests pass on MacOS is harder: that calls group 0 "wheel" 968instead of "root" (so things like "tar.test" don't get the same result), 969stuff like "du" or "ls" often cares what filesystem you're mounted on 970(and MacOS hasn't even got the same ones _available_), 971and we even found a mac kernel bug where extending a sparse 972file sparsely allocates the previous length as actual zeroed disk space 973(ala "truncate -s 3m file; truncate -s 5m file; du file" says it's using 9743 megabytes of disk space). There's probably more but 975that's where we stopped looking for now: since Rob doesn't have a mac 976to test on and the github tests don't reliably run when new commits are 977pushed (skipped when the servers are busy), fixing this hasn't 978been a priority. Lots of little red X in the github history because of this, 979though...</p> 980 981<p><u>Mkroot</u>: 982The output file names are now consistent across architectures: each 983root/$TARGET directory now contains an <b>fs</b> subdirectory with the 984expanded root filesystem, packaged into <b>initramfs.cpio.gz</b>, 985a <b>linux-kernel</b> built from <b>linux-fullconfig</b> which is built 986from <b>linux-miniconfig</b>, and a <b>run-qemu.sh</b> script to launch 987it under qemu. No more architecture name in the filenames (that's what 988the directory name is for), and no more "bzImage" or "vmlinux" depending 989on what filename the kernel build produced.</p> 990 991<p>The fs/init script has better chroot support (not warning about /dev/shm and 992/dev/pts already existing). 993The airlock build no longer includes ftpd (you can use 994httpd and wget, scp out of dropbear, mount a network filesystem from the 995host, or if you really want to still use ftp can match ftpget and ftpput). 996When mkroot is downloading extra packages (for example in the dropbear build), 997it falls back to the host wget if it can't fetch with toybox's version. 998(This is because toybox hasn't got built-in https support yet, it's on the 999todo list.) 1000Updated the dropbear build to the current version and fixed a compiler 1001path issue. 1002Fixed version skew in the <a href=http://lkml.iu.edu/hypermail/linux/kernel/2110.3/00278.html>workaround</a> 1003so the Linux kernel build doesn't require a third ELF package (but only 1004for x86, not for any other architecture).</p> 1005 1006<a name="23-07-2022" /><a href="#23-07-2022"><hr><h2><b>July 23, 2022</b></h2></a> 1007<p>Sorry the mailing list has been down since the 19th, Dreamhost 1008<a href=https://landley.net/notes-2022.html#22-07-2022>happened 1009again</a>. (First the web archive was empty, then it was down, then it was 1010restored from a backup taken before the "upgrade" but was not updating with 1011new messages, which they denied was an actual problem but now timeout bounce 1012messages from all the attempts to send to it since it broke have started 1013coming back saying a path doesn't exist on their server...)</p> 1014 1015<p>Dreamhost tech support has been informed of the issue a half-dozen times, 1016and did at least replace the "Currently there are no messages" page with a 1017backup from before they broke it on the first day. Unfortunately they have yet 1018to acknowledge anything else is wrong with it since then, and don't seem to 1019have support staff working over the weekend. This used to be an 1020<a href=https://landley.net/dreamhost.txt>annual</a> 1021<a href=https://landley.net/dreamhost2.txt>event</a> for them, 1022but they hadn't done a truly major breakage in 1023<a href=#12-21-2015>a while</a>. I _think_ the 11 month gap that last link 1024mentions is still missing, but the server's down again (or at least timing 1025out) so I can't check.</p> 1026</span> 1027 1028<a name="06-05-2022" /><a href="#06-05-2022"><hr><h2><b>May 6, 2022</b></h2></a> 1029<blockquote><p> 1030The mighty ships tore across the empty wastes of space and 1031finally dived screaming on to the first planet they came across - which 1032happened to be the Earth - where due to a terrible miscalculation of scale 1033the entire battle fleet was accidentally swallowed by a small dog.</p> 1034<p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 1035 1036<p><a href=downloads/toybox-0.8.7.tar.gz>Toybox 0.8.7</a> 1037(<a href=https://github.com/landley/toybox/releases/tag/0.8.7>git commit</a>) 1038is out, with prebuilt <a href=downloads/binaries/0.8.7>static binaries</a> and 1039<a href=downloads/binaries/mkroot/0.8.7>mkroot images</a> 1040bootable under QEMU (using vanilla linux-5.17).</p> 1041 1042<p>Rob also <a href=downloads/binaries/toolchains/latest>uploaded</a> 1043the prebuilt binary cross and native compilers used to build those 1044mkroot images (there's a <a href=downloads/binaries/toolchains/latest/README>README</a> 1045in there), and started producing 1046<a href=https://www.youtube.com/channel/UC4VFy3wc1nzq5tUHhiti6fw>tutorial videos</a> explaining the 1047toybox commands, their implementation, toybox's infrastructure...</p> 1048 1049<p>The commands <b>host</b>, <b>wget</b>, <b>openvt</b>, and 1050<b>deallocvt</b> were cleaned up and promoted. 1051Elliott contributed the new commands <b>uclampset</b>, <b>gpiodetect</b>, 1052<b>gpioinfo</b>, <b>gpioiget</b>, <b>gpiofind</b>, and <b>gpioset</b>, 1053and Rob wrote a simple <b>httpd</b> (largely to give wget.test something to 1054talk to).</p> 1055 1056<p>One command was removed: <b>catv</b> didn't really serve a purpose 1057(everybody just uses <b>cat -v</b>).</p> 1058 1059 1060 1061 1062<p><u>Features</u>: 1063<b>top</b> can now move the list with the LEFT/RIGHT cursor keys (changing the 1064sort field is now SHIFT LEFT/RIGHT). Added <b>find -samefile</b>, 1065<b>cmp -n</b> (and wired up skip1 skip2), 1066<b>tar --strip components</b> (and tar --owner and --group can now specify :UID and :GID making tar.test more portable), 1067<b>lsusb</b> and </b>lspci</b> now read /etc/{usb,pci}.ids[.gz] (when 1068available) to provide human readable device descriptions, 1069<b>ifconfig</b> can now rename interfaces. 1070Moritz Weber added <b>wget</b> post support for form data. 1071In toysh, <b>cd</b> got updated ("cd -" jumps to $OLDPWD, 1072and cd can now handle deleted directories).</p> 1073 1074<p><u>Bugfixes</u>: 1075<b>cp --preserve xattr,context</b> work together now, and 1076Dario Freni fixed <b>cp --xattr</b> to avoid copying selinux context. 1077The <b>sort -u</b> option now implies -s, <b>wget -O -</b> now writes to stdout, 1078<b>pwget -B</b> removes more characters other implementations don't emit, 1079</b>time</b>'s realtime output was not properly zero padding the fractional 1080part, and <b>sed</b> now works with multiple w to the same file. 1081Fixed <b>nl -E</b>, <b>realpath -e</b>, <b>find</b> -newerXY and @time.nanosecond parsing, 1082Various <b>sh</b> (it handles ${X: -1:1} properly now, 1083which is not the same as ${X:-1:1} because :- has special meaning). 1084The <b>hello</b> command in the examples directory now wait for a keypress before 1085exiting when run as PID 1 (because calling wait() returned "No child process" 1086immediately.) 1087The multiplexer's <b>toybox command --help</b> once again shows the command's 1088help (not toybox --help). 1089Elliott fixed a strip bug in the MacOS build, taught <b>file</b> not to be confused 1090by ELF BSS sections but to otherwise detect more invalid ELF files, 1091and fixed an <b>xargs</b> bug where stdin was inappropriately O_CLOEXEC. 1092Github user hg42 reported a bug where <b>tar</b> was looping trying to remove ".." 1093at the start of names.</p> 1094 1095<p><u>Library</u>: 1096Taught flags.h to #define its own CLEANUP macros so switching FLAG() 1097contexts requires fewer steps. 1098When args.c is just enforcing max/min argument count but collecting no 1099options, imply "?" so "command -123" isn't an unknown option. 1100Moved lib/help.c into main.c (nothing else in lib/ had to rebuild 1101when the config changes which commands are enabled). 1102The remains of linestack.c were renamed utf8.c, and 1103toys.h now includes sys/syscall.h so commands don't have to.</p> 1104 1105<p>Yank things with only one remaining user: moved get_line() 1106to patch.c, struct str_len to bc.c, and struct ptr_len to ps.c 1107Removed tty_esc() and tty_jump() wrappers. 1108Various cleanups to lib/password.c and lib/tty.c. 1109Removed the PEDANTIC_ARGS config option which just set a macro that was never 1110used, xsignal_all_killers() now takes SIG_DFL as the reset value instead 1111of explicitly checking for NULL, 1112chomp() now removes any number of trailing \r\n, and it's now safe to 1113llist_pop(0) just like dlist_pop(0). 1114LongPing Wei improved support for copy_file_range().</p> 1115 1116<p><u>Build</u>: 1117If you don't set $PREFIX, <b>make install</b> now defaults to $PWD/install 1118(which does not require root access to write to, and you can tar up your 1119pile of symlinks from there). 1120More command sub-options removed (NETCAT_LISTEN and CAT_V).</p> 1121 1122<p>Some of the uglier scripts/make.sh plumbing moved into scripts/portability.sh 1123(which should now be safe to source twice), 1124and moved almost all config varible definitions to "configure" now. The 1125new $GENDIR variable theoretically lets the "generated" directory live 1126somewhere else (not very tested yet). 1127Various tweaks to scripts/make.sh to try to avoid unnecessary rebuilds 1128and library probes when toolchain and config haven't changed. 1129Added -Wno-pointer-sign to $CFLAGS because passing an unsigned char * to a 1130function that expects char * is not a problem. 1131 1132<p>The airlock install in scripts/install.sh had the $PENDING items sorted 1133into "started" and "not started". Add git and bash to $PENDING (used 1134out of $PATH by --version and the powerpc64le build, respectively).</p> 1135 1136<p><u>Cleanup</u>: 1137Removed the lib/linestack.c plumbing (which was never finished and hadn't 1138been touched in years). 1139Merged lspci.c into lsusb.c (so they can share config file reading 1140infrastructure), and merged chvt/deallocvt into openvt.c. 1141Minor cleanups to ping, fsync, ionice, pmap, truncate, timeout, tty, 1142factor, mount. 1143Went through and replaced \033 with \e in strings (since clang supports 1144it and \033 is just awkward). Added LL to constant 0 in the FLAG macros 1145to prevent gcc from warning that 0<<32 might produce 0. 1146Moved llvm's -Wno-string-plus-int into configure instead of probing 1147for it, since gcc no longer dies when asked to suppress an unknown warning.</p> 1148 1149<p><u>mkroot</u>: 1150the ANSI escape to undo QEMU's wordwrap bug is now in both the init script 1151and launch wrapper script (so wordwrap isn't broken while running the 1152emulated system, nor if the emulated system fails to run). 1153If devtmpfs was already mounted on /dev, we still need to create dev/pts. 1154The init script now includes the linux kernel bug workaround to enable ping 1155support. The armv7 target enabled CONFIG_ARM_LPAE in the kernel to work around 1156around <a href=https://bugs.launchpad.net/qemu/+bug/1790975>a QEMU bug</a> 1157that's gone unfixed for 4 years. Enable network support in m68k target. 1158The mcm-buildall.sh script got some minor updates, and the resulting 1159cross and native compiler binaries are now downloadable so you don't have 1160to run the build if you don't want to.</p> 1161 1162<p><u>Docs</u>: 1163"toybox --help" no longer says "see toybox --help". 1164There's an ANSI escape cheat sheet at the start of lib/tty.c. 1165Reworded some command directory READMEs, and tweaked menuconfig display. 1166Converted www/doc/mount.html and www/doc/ext2.html from text to html. 1167Freshened up design.html a little (and fixed some typos pointed out by 1168Peter Ennis). 1169Replaced the git internals video URL in roadmap.html with a different 1170performance of the same talk that hasn't been made private. 1171Added the kconfig-language.txt from Linux-2.6.12 to the kconfig/ directory.</p> 1172 1173<a name="30-11-2021" /><a href="#30-11-2021"><hr><h2><b>November 30, 2021</b></h2></a> 1174<blockquote><p> 1175Busy? We'll I've just got this bulldozer to lie in front of or otherwise it'll 1176knock my house down but otherwise no, not especially, why?</p> 1177<p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 1178 1179<p><a href=downloads/toybox-0.8.6.tar.gz>Toybox 0.8.6</a> 1180(<a href=https://github.com/landley/toybox/releases/tag/0.8.6>git commit</a>) 1181is out, with prebuilt <a href=downloads/binaries/0.8.6>static binaries</a> and 1182<a href=downloads/binaries/mkroot/0.8.6>mkroot images</a> 1183bootable under QEMU (using vanilla linux-5.15).</p> 1184 1185<p><u>Make root</u>: The <a href=faq.html#mkroot>system builder</a> got upgraded, with better 1186logging and layout of the output directory, plus <a href=https://github.com/landley/toybox/blob/0.8.6/scripts/mkroot.sh>scripts/mkroot.sh</a> 1187should be easier to read and understand now. 1188(Also added basic support for the <a href=http://lists.landley.net/pipermail/toybox-landley.net/2021-August/012497.html>qualcomm hexagon</a>, 1189although it just builds a chroot filesystem, you can't boot a hexagon kernel under qemu 1190<a href=https://lists.nongnu.org/archive/html/qemu-devel/2021-07/msg06430.html>quite yet</a>.)</p> 1191</p> 1192 1193<p><u>New commands</u>: Rob added <b>linux32</b>. 1194Dan Brown added <b>sha256sum</b> (and the related 224/384/512 variants). 1195In pending, Elliott contributed <b>strace</b> and 1196Moritz Röhrich contributed <b>hexdump</b>.</p> 1197 1198<p><u>Features</u>: 1199Elliott added <b>date -s</b>, <b>pmap -p</b>, <b>tail -F -s</b>, <b>kill -0</b>, 1200<b>reboot/halt/poweroff -d</b>, taught <b>file</b> to 1201recognize Mach-o universal binaries, made Android's <b>log</b> 1202take input from stdin when it has no arguments, added <b>tail --bytes --lines</b> 1203(the --long synonyms for -c and -n), 1204added <b>i2cdetect -q</b>, taught <b>i2cdump</b> to write XX for 1205read failures, made <b>timeout</b> use the monotonic clock, 1206and modernized the output of <b>uptime</b>.</p> 1207 1208<p>Rob added <b>find -quit -lname -ilname</b> and <b>-d</b> as a synonym for -depth, 1209taught <b>cut -d $'\n'</b> 1210to cut by line and added posix <b>cut -nb</b> (which wraps to the start of -c 1211utf8 characters), added <b>cpio --ignore-devno --renumber-inodes</b>, 1212taught <b>cal</b> to use 3 arguments (day month year), added <b>tar --selinux</b> 1213support and a "had errors" message at the end, made <b>uname -o</b> 1214say "Toybox", and fixed the <b>grep -Fw</b> combination (used by the 64 bit 1215powerpc kernel build).</p> 1216 1217<p>Ella-0 added <b>split -n</b>. Peter Collingbourne added <b>grep -L</b>.</p> 1218 1219<p><u>Outside world</u>: The 0BSD license is <a href=https://lwn.net/Articles/868882/>now part</a> of the 1220<a href=https://standards.iso.org/ittf/PubliclyAvailableStandards/>ISO-5962</a> 1221standard. Of the currently <a href=https://github.com/search?q=license%3A0bsd&type=Repositories&ref=advsearch>38k 0BSD licensed projects on github</a> toybox 1222isn't even the most popular, it's fourth.</p> 1223 1224<p>Elliott asked that toybox's <b>cut -DF</b> feature (a cheaper 1225alternative to <b>awk '{print $3}'</b> be supported by other packages so 1226it might someday make it into posix, so Rob 1227<a href=http://lists.busybox.net/pipermail/busybox/2021-June/088886.html>submitted it to busybox</a> 1228where it was <a href=https://git.busybox.net/busybox/commit/?id=0068ce2fa0e3>eventually merged</a>.</p> 1229 1230<p>Added a local <a href=https://landley.net/toybox/git>git mirror</a> for 1231those reluctant to trust Microsoft Github.</p> 1232 1233<p>In the web nav bar, our <a href=https://isfreenodedeadyet.com/>freenode</a> 1234channel moved to libera.chat 1235(like <a href=https://boingboing.net/2021/05/19/freenode-irc-staff-quit-after-new-owner-seizes-control.html>everyone else's</a>; 1236<a href=https://lwn.net/Articles/856543/>there</a> 1237<a href=https://hackaday.com/2021/05/20/freenode-debacle-prompts-staff-exodus-new-network/>was</a> 1238<a href=https://twitter.com/marcan42/status/1396280769778098185>a</a> 1239<a href=https://twitter.com/mjg59/status/1396237336988446722>certain</a> 1240<a href=https://www.devever.net/~hl/freenode_abuse>amount</a> 1241<a href=https://twitter.com/TwitchiH/status/1395756163527028739>of</a> 1242<a href=https://twitter.com/joepie91/status/1395364883303182339>drama</a> 1243<a href=https://twitter.com/RichFelker/status/1404816846734385164>involved</a>).</p> 1244 1245<p><u>Pending</u>: lots more work on <b>toysh</b>: it has basic $((math)) support 1246now, += assignment, the "declare" command and corresponding variable 1247types (declare -i, declare -u, etc), several bugfixes, greatly 1248improved support for background processes on NOMMU systems, and a bunch 1249more sh.test entries.</p> 1250 1251<p>Elliott fixed <b>modprobe</b>'s parsing of short lines. Rob fixed a 1252division by zero error when <b>vi</b> tries to edit a zero length file. 1253Anqi Dong made sure that syslogd messages sent to sockets end with a 1254newline. Eric Molitor significantly upgrade <b>wget</b> with http 1.1, 1255chunked encoding, and (external library) https (TLS 1.2) support, 1256plus several bugfixes. Michael Hope fixed <b>tftpd</b>'s option parsing.</p> 1257 1258<p><u>Documentation</u>: The roadmap.html and status.html pages 1259got noticeable updates. (status.html is now directly generated by 1260scripts/mkstatus.py, the old status.html wrapper that server-side-included 1261status.gen has been removed). 1262The <a href=design.html>design</a> 1263page provides better links to the LP64 documentation, and the 1264<a href=code.html>code</a> page explains a corner case where the filename 1265and first command name expect to match (and how to work around it if they don't). 1266Some website files weren't in the repository: two jpegs linked from html 1267files, and the www/.htaccess file that triggers the server side includes 1268in the navigation menu is now checked in. Johannes Schmitz fixed a 1269typo in about.html.</p> 1270 1271<p>The <b>shred</b> usage line now includes all options. 1272The <b>logwrapper</b> command now warns when it isn't built standalone. 1273The <b>help</b> command no longer shows a header line about toybox unless 1274it's called with no arguments. (It still shows it for command --help.) 1275The --help option no longer needs to be the first option, and is 1276checked before the "Not root" error message for restricted commands.</p> 1277 1278<p><u>Bugfixes</u>: 1279Elliott made <b>ps</b> exit with status 1 if no processes were shown (for LTP), 1280fixed an off-by-one in <b>dmesg</b> and a bounds check in <b>netstat</b>, 1281fixed <b>pidof -s</b>, and taught <b>rmmod</b> to handle multiple arguments. 1282Denys Vlasenko pointed out that <b>free -t</b> is "total" not "terabytes". 1283Samanta Navarro pointed out a leftover 32 bit limitation in argument groups. 1284Mikhail Kashkarov fixed mkdir's error handling for existing directories 1285without -p. 1286Rob fixed the 32 bit <b>readelf</b> build on Android, fixed a race condition 1287in <b>tar</b> leading 1288to the occasional spurious archiver warning, 1289<a href=https://github.com/landley/toybox/commit/127cff38809e>changed cp</a> 1290to work around a posix violation in the kernel, made xparsedate() handle more 1291whitespace and ignore trailing + or - (a NOP timezone), made <b>su</b> log the 1292right username, switched getuid() checks to geteuid() in <b>df</b> and <b>ping</b>, 1293fixed a place in the test suite where do_test() 1294could get confused by symlinks in the $PATH (calling realpath too early 1295and losing the name of the command being run), 1296made <b>env -i</b> use the old $PATH to run its command, 1297and made <b>grep --color</b> highlight all matches.</p> 1298 1299<p>Worked around a bug in glibc+gcc 9.3 where linking against librt.a 1300(for things like timer_create()) pulls in libgcc_eh.a which uses 1301threads, meaning you can't statically link against librt.a from 1302a non-threaded app. (I hadn't noticed because it works fine with the musl 1303and llvm/bionic toolchains, only fails statically linking on e.g. debian.) 1304The <a href=https://github.com/torvalds/linux/commit/d18760560593>kernel fix</a> for the cpio encrypted filesystem issue worked around in 0.8.5 made 1305it in. Similarly, the 1306FS_IOC_{GET,SET}{VERSION,FLAGS} ioctl family are <a href=https://www.spinics.net/lists/linux-api/msg49101.html>historically broken</a> 1307(taking an unsigned int argument while advertising signed long), 1308which <a href=https://github.com/landley/toybox/commit/dcfac5c983ad>we fixed</a> and reported upstream to a long 1309round of bikeshedding (sadly 1310the result seems to have been the man page maintainer 1311<a href=https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/commit/man2?id=3de87d46840d>giving up</a> 1312on documenting ioctls at all.</p> 1313 1314<p><u>Cleanups</u>: 1315<b>cut</b> switched to FLAG() macros and no longer allows -f with -F, 1316<b>eject</b> uses FLAG() macros and the linux/eject.h header, 1317prettycool@protonmail added tabs to <b>time</b>'s output, and Rob removed 1318the floating point dependency from time. Minor cleanups 1319to lsattr, lspci, nsenter, xxd, file, insmod, modprobe, arp, bootchartd, pmap, 1320comm, lsof, blkid, crond, init, userdel, makedevs, and scripts/make.sh. 1321The "int fd = fd;" 1322initialization hack to shut up gcc's "may be used uninitalized, but isn't" false 1323positive warning has been replaced by a QUIET macro that resolves 1324to nothing on llvm (which doesn't have the gcc bug). 1325Ryan Pritchard reported a bug where <b>alias ls="ls --color"</b> would prevent 1326--help or --version from being the first argument to the command, which was 1327a pain to fix without breaking things like <b>echo -- --version</b> and 1328<b>basename -s --help file--help</b>. 1329(Both toy_singleinit() in main.c _and_ get_optflags() in lib/args.c now 1330check for those.)</p> 1331 1332<p><u>Infrastructure</u>: 1333xabspath() now has a more granular (flag based) control interface.</p> 1334 1335<p><b>scripts/make.sh</b> now uses wait -n (available since bash 4.3 in February 2014) 1336to keep the processors busier during a parallel build. 1337Tests are now running in a subshell, which means functions and local 1338variables defined in one don't pollute others during "make tests". 1339 1340<p>Guo Chuang reported that lsattr won't build on an 8 year old version of 1341<a href=https://github.com/landley/toybox/pull/292#issuecomment-899895909>centos</a>, 1342which is still supported for 2 more years because Red Hat can 1343be paid to have a 10 year support horizon, longer than toybox's 7. Rob 1344fixed it (and merged a similarly ancient compilation failure fix when 1345building pending/ip.c against a 3.10 kernel), 1346but the <a href=http://lists.landley.net/pipermail/toybox-landley.net/2021-August/012537.html>philosophal argument</a> 1347remains <a href=http://lists.landley.net/pipermail/toybox-landley.net/2021-September/012603.html>unresolved</a>.</p> 1348 1349<p>Standalone builds can now override the .config file location with 1350KCONFIG_CONFIG same as the multiplexer build. scripts/single.sh can 1351now cope with a missing .config file (it uses the defconfig values 1352for the CONFIG_TOYBOX_* global symbols). 1353"make install_airlock" no longer needs host versions of sha256sum and sha512sum.</p> 1354 1355<p>In the test suite, <b>VERBOSE=spam</b> now shows expands successful tests 1356like it would for errors.</p> 1357 1358<p><b>scripts/record-commands</b> got updated and logwrapper got renamed <b>logpath</b>.</p> 1359 1360<a name="19-05-2021" /><a href="#19-05-2021"><hr><h2><b>May 19, 2021</b></h2></a> 1361<blockquote><p>It was of course a descendant of these eccentric poets 1362who invented the spurious tales of impending doom which enabled the people of 1363Golgafrincham to rid themselves of the useless third of their population. 1364The other two-thirds stayed firmly at home and lived full, rich and happy lives 1365until they were all suddenly wiped out by a virulent disease contracted from 1366a dirty telephone.</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 1367 1368<p><a href=downloads/toybox-0.8.5.tar.gz>Toybox 0.8.5</a> 1369(<a href=https://github.com/landley/toybox/releases/tag/0.8.5>git commit</a>) 1370is out, with prebuilt <a href=downloads/binaries/0.8.5>static binaries</a> and 1371<a href=downloads/binaries/mkroot/0.8.5>mkroot images</a> 1372bootable under QEMU (using vanilla linux-5.12, except that s390 needed another 1373<a href=http://lkml.iu.edu/hypermail/linux/kernel/2105.2/02904.html>perl 1374removal patch</a>).</p> 1375 1376<p>This development cycle had thirty commits to sh.c fixing bugs and adding 1377features to <b>toysh</b>. Shell functions and local variables are now implemented, 1378along with the "set" builtin and trace support, and job control is 1379about halfway in. The shell as a whole is <a href=https://en.wikipedia.org/wiki/Ninety%E2%80%93ninety_rule>maybe 80% done</a>, but still 1380has some obvious gaps like $((math)) and interactive command line 1381history/editing, plus a bunch of testing.</p> 1382 1383<p><u>New commands</u>: 1384Moritz Röhrich added <b>pwgen</b> and <b>base32</b>, and 1385Elliott Hughes added <b>unicode</b>. The 1386<b>readelf</b> command got promoted out of pending, and 1387<b>sha3sum</b> is now switched on in defconfig.</p> 1388 1389<p>Michael Christensen added <b>chsh</b> to pending, which hasn't been promoted 1390yet because it's part of a group of commands (useradd, userdel, groupadd, 1391groupdel) that should all be promoted together at some point 1392(and depend on lib/passwd.c, which could use some more work).</p> 1393 1394<p><u>Upgrades</u>: 1395<b>cpio</b> now supports extracting multiple concatenated archives. 1396<b>file</b> now recognizes android's new XML binary format. 1397<b>devmem</b> now works on nommu, can handle 0x prefxes on the address, 1398and prints 1399hex or decimal output to match the address type. 1400<b>df</b> -a now shows overmounted filesystems. 1401<b>test</b> -k checks the sticky bit. 1402Added -t to <b>install</b>, <b>cp</b>, and <b>mv</b>, and -u to <b>cp</b> 1403and <b>cpio</b>. 1404<b>find</b> -executable checks that _we_ can execute them (u+x could be owned 1405by a different user or have selinux weirdness).</p> 1406 1407<p>Elliott added several features to <b>date</b> (--iso, %:z output format, and 1408--utc as a synonym for -u), added a <b>sed</b> s///x option, 1409added units to <b>ulimit</b> output, and 1410made several UI changes to <b>hexedit</b> including a color 1411mode (which is now the default, press x to toggle between old and new 1412display), masked out the file type in <b>chmod</b> error messages and 1413taught it to handle complex modes (like u+s+s), 1414hardened <b>file</b> and <b>readelf</b> against invalid input, 1415added a portability.h workaround for macos statvfs for <b>df</b>, 1416enabled line buffering in <b>echo</b> and <b>yes</b>, 1417and significatly sped up <b>tr</b>. Rob similarly optimized <b>seq</b> 1418and <b>count</b>.</p> 1419 1420<p><b>make tests</b> now defaults to VERBOSE=fail (and now has VERBOSE=all to 1421continue after failure, and VERBOSE=quiet to show FAIL: lines without 1422diff -u output). 1423Cleanups to tee, netstat, file, and df. Added "static" annotations in a 1424lot of commands, made more things use FLAG() macros. Ethan Sommer pointed out 1425an unnecessary wrapper function in chgrp. 1426In pending, Elliott did some cleanups and bugfixes to gettty, telnet, telnetd, 1427tftpd, init, modprobe, and dd, and added more features to vi. 1428Rob did some cleanup on modprobe.</p> 1429 1430<p><u>Bugfixes</u>: 1431Yi-Yo Chiang fixed a <b>rm</b> bug where -f could get confused by read 1432only filesystems and produce an error trying to unlink a nonexistent 1433file, pointed out <b>install</b> -D was broken when the target file 1434already exists, pointed out a <b>cpio</b> interaction with certain encrypted 1435filesystems where readlink() wouldn't write as much payload as the header 1436promised (corrupting the archive) and fixed another cpio bug where -t test 1437mode would chown() files, and fixed quoting in mount.test. 1438Elliott fixed a sequencing issue in <b>setsid</b>'s system calls, 1439fixed some <b>sed</b> tests (sed's lie-to-autoconf --version 1440output was preventing toyonly tests from triggering) and 1441fixed <b>tar</b>'s base 256 output for very large files. 1442Nir Lichtman pointed out that <b>rm</b> -p shouldn't try to rmdir / at the end, 1443Luke Frankart pointed out a typo that broke <b>test</b> checking file modes, 1444Firas Khalil Khana fixed a duplicate entry in the roadmap, 1445Ethan Sommer fixed <b>install</b> -Dt, 1446Andy Hu requested that <b>mount</b> handle mixed options like 1447-tnomsdos,smbfs. 1448Denys Nykula reported a <b>chmod</b> regression, 1449asm89 pointed out a <b>netcat</b> bug writing transactions larger than 14504k, 1451Daniel Santana fixed <b>df</b> combining -P and -k, 1452Robin Hsu fixed <b>grep</b> getting confused by libc setting errno when 1453a library function call succeeded. 1454Puck Meerburg taught the <b>gunzip</b> code not to be confused by FEXTRA entries 1455(as described in RFC 1952) and added a test suite entry for that.</p> 1456 1457<p><b>find</b> handles empty arguments properly now (with test), <b>date</b> 1458-I with no arguments no longer segfaults, <b>grep</b> -f /dev/null no 1459longer adds a "" regex matching everything, -z no longer splits patterns 1460on \n, and -o can print zero length matches, 1461<b>cp</b> -t works with one argument and with --parents. 1462<b>readelf</b> displays properly in 80 columns and doesn't reject .bss 1463off end of file, and <b>sed</b> s///2 now works when there isn't more than one 1464match. 1465<b>tar</b> now removes empty directories where it wants to put a non-directory. 1466<b>microcom</b> sets serial device terminal attributes correctly.</p> 1467 1468<p>The prebuilt x86-64 binaries shouldn't segfault on older x86-64 processors 1469anymore (workaround for gcc bug). 1470Cross compiling was sometimes confusing install.sh because it created 1471a hyphenated "toybox-$TARGET" and then tried to make symlinks to just "toybox".</p> 1472 1473<p><u>Build</u>: 1474CONFIG_TOYBOX_I18N went away: toybox builds always support utf8 now. 1475Ella-0 added OpenBSD support (but not a separate defconfig, try 1476"make freebsd_defconfig").</p> 1477 1478<p><u>Library</u>: 1479utf8towc() is now unsigned instead of wchar_t, matching wctoutf8(). 1480xclose() tests if fd is -1 so the caller doesn't have to. 1481txpect (the test suite's built-in "expect", implemented in shell) can 1482now do regex matches, and sh.test uses this.</p> 1483 1484<p>Elliott added the failing regex to the xregcomp() error message, and 1485made multiple portability tweaks (header tweaks to build in more 1486environments, running tests on Raspberry Pi OS and under mksh, 1487fix utf8 locale selection on macos).</p> 1488 1489<p><u>Docs</u>: 1490<b>chmod</b>'s help text explains why o+s doesn't set +t (and has a 1491test for it). The <b>echo</b> help text is more compact. 1492Checked in a better conversion of <a href=https://landley.net/toybox/toycans.png>toycans.png</a>.</p> 1493 1494<a name="24-10-2020" /><a href="#24-10-2020"><hr><h2><b>October 24, 2020</b></h2></a> 1495<blockquote><p> 1496"We are now cruising at a level of two to the power of twenty-five thousand to 1497one against and falling, and we will be restoring normality just as soon as we 1498are sure what is normal anyway."</p> 1499<p>- The Hitchhiker's Guide to the Galaxy</p> 1500</blockquote> 1501 1502<p>After a longer and slightly more 1503<a href=https://github.com/landley/toybox/commit/07a896862ddf>turbulent</a> 1504development cycle than some, 1505<a href=downloads/toybox-0.8.4.tar.gz>Toybox 0.8.4</a> 1506(<a href=https://github.com/landley/toybox/releases/tag/0.8.4>git commit</a>) 1507is out with new commands <b>sha3sum</b> and <b><a href=https://github.com/landley/toybox/commit/6b4c32ae3986>watchdog</a></b>.</p> 1508 1509<p>There are <a href=downloads/binaries/mkroot/0.8.4>prebuilt mkroot binaries</a> 1510now, tiny toybox linux systems for a dozen architectures, all bootable under 1511qemu. (The vmlinux in each tarball is a vanilla linux-5.9 kernel built 1512from the included config file.)</p> 1513 1514<p>The FAQ got <a href=faq.html>noticeably larger</a>, and the README has 1515more links. New command features include the <b>sed -s</b> flag, 1516<b>cpio --no-preserve-owner</b> now affects archive creation, 1517Elliott added <b>tar -I</b> and multi-type 1518<b>find -type a,b,c</b> support, Mark Salyzyn added <b>xargs -P</b> to 1519run parallel jobs, the <b>ps</b> and <b>top</b> commands 1520now autodetect pid length, and <b>top</b> adjusts units for memory display based on 1521system size.</p> 1522 1523<p><u>Toysh and toyroot</u>: 1524lots of new work on <b>toysh</b>: 29 commits to sh.c since last release, adding 1500 1525lines and deleting 700, plus a bunch of sh.tests entries). Since last release 1526we implemented wildcards, case/esac and select, brace expansion sequences 1527(ala {1..10..2} and {a..z}), 1528the remaining variable slice types ${a#y} ${a%y} ${a^y} ${a,y} 1529${a/search/replace}, the "<b>source</b>" shell builtin, 1530the start of job control, and several bugfixes. 1531Plus the standalone "<b>make sh</b>" build understands MAYFORK now. It's 1532still missing features like functions() and $((math)) but it's getting 1533close to usable now.</p> 1534 1535<p><u>Documentation</u>: 1536Some of the README contents moved to the FAQ, and the README's "presentations" 1537section got some new links. Several new <a href=faq.html>FAQ</a> entries explaining things like 1538mkroot ("how do I build a working Linux system 1539with toybox"), cross compiling (how to get/setup the 2 compilers and 15403 libc we regression test against), and "where does toybox fit into the 1541linux/android ecosystem". 1542The "toybox --help" output now lists the project's web page (<a href=https://github.com/landley/toybox/issues/50>by request</a>). 1543Elliott removed 1544getevent (an android board bringup/hardware debugging tool built by running 1545a python script against kernel headers, not really in scope for toybox), 1546fixed xargs help formatting, and taught the toybox multiplexer's 1547command list output (and "kill -l") to measure the current the terminal width 1548when wordwrapping (previously hardwired to 80 columns). 1549If you're curious, I checked in my <a href=www/release.txt>release procedure 1550checklist</a>, and 1551fixed a stale link in the nav bar on the left ("statistics" changed domains). 1552Firas Khalil Khana fixed some issues in the roadmap, and Rob 1553tweaked the roadmap so status.html is slightly more current.</p> 1554 1555<p><u>Tests</u>: 1556The "make tests" target now fails if any of the tests it ran failed. 1557Lots of TEST_HOST tests got fixed (checking for specific error messages exposes 1558TEST_HOST to version skew, and scripts/test.sh will now skip TEST_HOST 1559commands that aren't installed) and added toyonly annotations as necessary 1560(replacing most uses of SKIP_HOST). 1561Eric Molitor added automated github tests on MacOS and Ubuntu using their 1562"workflows" thing. 1563The clang asan plumbing slows some code down more than 10 times, so some testing timeouts were expanded. 1564 1565<p><u>Pending</u>: 1566Eric Molitor did a bunch of work on route: moved it to sbin, added xsend and 1567xrecv, taught display_routes() and setroute to use the rtnetlink API and 1568do hostname lookups, added support for mss, win, and irtt, merged ipv6 and ipv4 1569codepaths with autodetection of address type, implemented RTA_CACHEINFO 1570support, switched exit paths to perror_exit(), and removed unused code. 1571Ethan Sommer fixed warnings in dhcpd, removed a bunch of unnecessary 1572; after GLOBALS() blocks and an unnecessary return in df. 1573Erik Moqvist fixed dns setting in the dhcp client. 1574Rob did some cleanup on bootchartd, traceroute, getty... 1575Elliott fixed getty to reliably update utmp. 1576Chris Sarra added ipv6 support to wget. 1577Ariadne Conill submitted several small fixes from testing Alpine Linux with 1578toybox, and Antoni Villalonga i Noceras fixed typos in error messages.</p> 1579 1580<p><u>Bugfixes</u>: 1581Elliott avoided sign extension in devmem, and Ethan Sommer switched it from 1582getpagesize() to posix sysconf(_SC_PAGESIZE). 1583Elliott fixed chmod -R 1584ignoring dangling symlinks, fixed stty <a href=https://github.com/landley/toybox/issues/251>misparsing c_iflags</a>, fixed the chattr f2fs test with 1585compression enabled, taught blkid not to show empty tags, taught xparsedate() 1586to read date's default output format, fixed a recent echo -e \0 regression, 1587changed cpio -p parsing to match a <a href=http://lists.landley.net/pipermail/toybox-landley.net/2020-August/011955.html>bug</a> in the gnu version 1588which an existing script <a href=https://github.com/landley/toybox/commit/fa1af3b085cc>depended on</a>, 1589and switched hwclock back to looking 1590only at /dev/rtc0 for <a href=https://github.com/landley/toybox/commit/70e2232ce61c>reasons</a> involving kernel version skew and vendor bug reports.</p> 1591 1592<p>David Legault pointed out that unescape2() (and thus echo -e) wasn't handling 1593\0 right. 1594Khem Raj reported that mips glibc doesn't have SIGSTKFLT, leading to 1595a build break if we assume glibc behaves consistently, which led to a bunch of 1596macos signal portability cleanups from Elliott. 1597William Djupström fixed tar extracting long file paths, adding hardlinks 1598to an archive, and reported that --exclude wasn't working. 1599Peter McConalogue pointed out that cp/mv -i prompt should 1600default N, and mv should only prompt when stdin is a tty. 1601Patrick Oppenlander made rtcwake and hwclock default to UTC if /dev/adjtime 1602isn't available, suggested watchdog catch SIGINT, and caught an 1603uninitialized offset in blkdiscard which gcc apparently didn't. 1604Ryan Pritchard reported that <b>file</b> wasn't getting gif heights right, 1605and Elliott added gif version output so TEST_HOST provided similar output. 1606Martin Stjernholm fixed <b>cp -P</b> to not follow symlinks and updated 1607the help text to say it's not the default. 1608Antoni Villalonga added more --long asiases to <b>chgrp/chown/rmdir</b>. 1609 1610<p>The symlink indirection code subtly broke xexec() so it would still try to call 1611a builtin when given a path to a command (fixed now). 1612The code to trip \n off xgetline() was using the length of the allocation 1613instead of the length of the read. 1614Netcat no longer leaks sockfd into child processes. 1615Patch "fuzz" support was outputting context lines from the hunk, not the 1616file. Chrisrfq fixed i2cdetect parameter reading. 1617Chris Sarra taught dd not to throw an error trying to truncate device files, 1618added a misshing fflush() to lib/password.c, fixed a parse error in logger 1619preventing local facility logging from working right, taught init 1620to reload inittab when it receives SIGHUP, and fixed a memory leak in tar.</p> 1621 1622<p><u>Library</u>: 1623New xvdaemon() function daemonizing on nommu systems, to wean commands 1624off daemon() which requires TOYBOX_FORK. 1625Teach sendfile_len() to use copy_file_range() when available, with compile 1626time probe. Fix xsignal_all_killers() to install the correct handler. 1627Rob switched dirtree_path() to a non-recursive implementation, and 1628taught dirtree that a top level entry named "" is equivalent to 1629"." but should not show up in dirtree_path(), 1630read_password() now reads from tty device rather than stdin, and handles ctrl-c and ctrl-d, and 1631human_readable() now has HR_NODOT so it can print single digit numbers without going "0.0". 1632Petri Gynther increased the vmstat column sizes so it didn't fit 1633in 80 columns anymore. 1634Elliott added UTC offset support to xparsedate().</p> 1635 1636<p><u>Infrastructure</u>: 1637mkroot now only passes --no-preserve-owner to cpio (so initramfs files 1638belong to root rather than whichever user ran the build) when using toybox's 1639version, because the other one is brain-damaged and errors out not 1640understanding it. (It understands it for extract, but not create. Yes really.) 1641New PENDING variable you can set to enable more than just sh and route out 1642of pending. 1643The "make distclean" no longer deletes root_download (where mkroot's "extra plumbing" 1644file saves source code downloads it tarballs, such as dropbear).</p> 1645 1646<p><u>Cleanup</u>: 1647Cleanup openvt (35 lines added, 78 lines removed) 1648Cleanup blkdiscard so lib/args.c parses the -o and -l numbers, which means 164932 bit systems are limited to 2 gigabyte sizes which is a TODO item systemwide. 1650oneit now uses flag macros and mentions -rn in the help. demo-utf8towc 1651now has main.c call setlocale for it. 1652Some cleanup on stty.</p> 1653 1654<a name="11-05-2020" /><a href="#11-05-2020"><hr><h2><b>May 11, 2020</b></h2></a> 1655<blockquote> 1656<p>Ford: Ah. A 1657<a href=https://www.youtube.com/watch?v=WSsR419HBpQ>towel</a>. 1658Keep this and guard it with your life.</p> 1659<p>Arthur: Huh?</p> 1660<p>Ford: Listen, it's a tough universe. There's all sorts of people and things 1661trying to do you, kill you, rip you off, everything. If you're going to 1662survive out there, you've really got to know where your 1663<a href=towel.jpg>towel</a> is.</p> 1664<p>- The Hitchhiker's Guide to the Galaxy</p> 1665</blockquote> 1666 1667<p>Despite everything <a href=downloads/toybox-0.8.3.tar.gz>Toybox 0.8.3</a> 1668(<a href=https://github.com/landley/toybox/releases/tag/0.8.3>git commit</a>) 1669is finally out, with new commands <b>rtcwake</b> from Elliott Hughes and 1670<b>blkdiscard</b> from Patrick Oppenlander. 1671The big news is "<b>make root</b>" now boots to a shell prompt, 1672with toysh making it all the way through toyroot's init script. 1673(Some people were looking forward to <b>patch --fuzz</b> support too.)</p> 1674 1675<p><u>Toyroot</u>: <b>make root</b> now does what it says on the tin, it 1676builds a bootable toybox-based Linux system using two source 1677packages (toybox and linux). The trivial version is "make root && sudo chroot 1678root/host/fs /init". Here's 1679a <a href=http://lists.landley.net/pipermail/toybox-landley.net/2020-April/011667.html>post with instructions</a> if you want to know how to build the 1680cross compilers for testing the various architectures. The self-contained 1681<a href=https://github.com/landley/toybox/blob/0.8.3/scripts/mkroot.sh>script 1682that builds a simple bootable Linux system</a> is 250 lines long, and 1683should be easy to read if you want to know how it works.</p> 1684 1685<p>It also has basic module support, meaning arguments that aren't X=Y variable 1686assignments are the names of scripts to run to build more 1687packages at the end of the build. I checked in an <a href=https://github.com/landley/toybox/blob/master/scripts/root/dropbear>example package</a>, 1688and there's generic "download and extract source tarballs" 1689<a href=https://github.com/landley/toybox/blob/master/scripts/root/plumbing>plumbing</a> available to such modules. 1690(The Makefile doesn't know how to pass module names through to the script, 1691so instead of "make root" you have to 1692call the script directly, ala "scripts/mkroot.sh CROSS=sh4 LINUX=~/linux dropbear".)</p> 1693 1694<p>The resulting root filesystem now uses /root as the home 1695dir for UID 0, and creates /dev/fd and /dev/shm in devtmpfs. The build works 1696around a kernel 1697build bug where "make distclean" doesn't work in a "cp -sfR" symlink 1698directory. (The bug meant if you pointed LINUX= at unclean source, it was 1699unhappy, so now it distcleans the source directory before copying it. This 1700modifies said source directory, which is not ideal, but as usual the kernel guys 1701<a href=http://lkml.iu.edu/hypermail/linux/kernel/2002.2/00083.html>ignored 1702the bug report</a>, so a workaround was required.) 1703The CROSS=all build announces each target in the title bar, puts 1704its logs into root/log, and has better trap handling to stop with one 1705ctrl-c. 1706Since last release it uses a more concise config format for the various kernel 1707arch targets (which shrank the script by about 50%), and merged 1708the old (now removed) scripts/cross.sh into the main script so 1709"make CROSS=armv7l LINUX=~/linux" is 1710literally just a call to "scripts/mkroot.sh CROSS=armv7l LINUX=~/linux" now.</p> 1711 1712<p>As for <b>scripts/mcm-buildall.sh</b> building cross compilers, 1713of COURSE gcc 8.3 requires a different configuration to build the same 1714toolchain as previous versions, it's gcc. It now builds a proper 1715nommu libc for sh2eb without a broken fork() call in it that can never 1716be used but prevents compile-time probes from detecting nommu, 1717and checks that the cross compiler completed before trying to build 1718the native compiler.</p> 1719 1720<p>This was all tested with a recent 1721<a href=https://github.com/richfelker/musl-cross-make/commit/5086175f2902>version</a> 1722of musl-cross-make with the top level Makefile 1723edited to use BINUTILS_VER = 2.32 because the newer one 1724<a href=https://www.spinics.net/lists/linux-sh/msg56844.html>breaks the kernel 1725build</a> 1726and LINUX_VER = 4.19.90 because the default musl-cross-make config uses an 1727out-of-tree headers package for some reason (those who forget history are 1728<a href=https://lkml.org/lkml/2006/4/28/194>doomed to repeat it</a>) which 1729breaks m68k and s390x. I won't post binaries of the resulting toolchains 1730because they're GPLv3, but <a href=https://musl.cc/>Thalheim might</a>. 1731(I've also <a href=http://lists.landley.net/pipermail/toybox-landley.net/2020-May/011673.html>test built</a> 1732with the Android NDK, but bionic's startup code currently segfaults before 1733calling main() if it can't open /dev/null, and the kernel guys 1734<a href=https://lkml.org/lkml/2017/9/13/651>ignored my patches</a> to 1735make CONFIG_DEVTMPFS_MOUNT work in initramfs.)</p> 1736 1737<p>At the moment toyroot cheats and uses two commands out of pending: toysh 1738is about 80% of the way to being useful but still missing some 1739obvious features like function support, job and terminal control, command 1740history, $((math)), wildcard expansion... plus a lot of bash features like array 1741variables, so it isn't out of pending yet. And route needs to be redone to use the 1742netlink interface that can handle multiple routing tables. For the moment 1743scripts/mkroot.sh adds both of them to defconfig if you haven't already 1744got a .config when you run it. (If you build and can't boot, your .config 1745probably hasn't got CONFIG_SH switched on. Fix it in menuconfig, or 1746rm .config and try gain.)</p> 1747 1748<p><u>New toybox features</u>: 1749Elliott taught <b>patch</b> to understand [FILE [PATCH]] arguments, made 1750<b>cal</b> highlight the current day, added -T to <b>cp</b>/<b>mv</b>, 1751did a lot of work on <b>lsattr</b>/<b>chattr</b> (including adding -p 1752and "chattr ="), added tar --absolute-names, taught <b>id</b> to 1753support numeric lookup and handle unknown groups, made -G show all 1754groups, and removed context= from -Z. 1755Rob added <b>patch -F</b> (fuzz factor) support and <b>help -u</b> (usage only), 1756taught <b>echo -e</b> about bash extensions like \uXXXX unicode escapes, 1757<b>netcat -L</b> no longer automatically includes stderr (new -E 1758option does that), 1759<b>setsid</b> now uses -c (like the man page says) instead of -t, and added 1760-w (wait) and -d (detach from tty). 1761Andrew Ilijic added <b>ls -w</b>, removed trailing whitespace on output, 1762and added tests for -C and -x.</p> 1763 1764<p><u>Library</u>: 1765another fix to 32 bit option parsing of long long optflag values, 1766remove getdirname() and just use the libc function (which modifies 1767its argument, but we don't have to free a malloc), 1768dlist_terminate() can now be called repeatedly on the same list, 1769new relative_path() function finds path from one directory to another, 1770and readfd() works like readfile() but on an already open fd. 1771Use MPATH macros in mkpathat(). Elliott added macOS versions of 1772dev_minor()/dev_major()/makedev() to portability.c, 1773implemented posix_fallocate() for macOS, removed a 1774leftover uClibc workaround in fallocate that was breaking macOS, 1775and moved the table of ELF architectures from file to lib/lib.c (so 1776readelf can share). 1777Park Ju Hyung pointed out the fast path of fdlength() was commented out, 1778and we switched it to the 64 bit API while we were there (and then Elliott 1779added macOS support). 1780Joeky taught file to recognize 7z archives.</p> 1781 1782<p><u>Pending</u>: 1783The shell got a bunch of work: standalone "make sh" now includes the 1784multiplexer for builtin commands like "exit" and "cd". Added MAYFORK annotation 1785for commands that exist in the $PATH but can also be run within the shell 1786process (and sometimes have different behavior within the shell): 1787currently applied to help, echo, false, kill, printf, pwd, test, time, and 1788true.</p> 1789 1790<p>Elliott added new commands <b>getopt</b> and <b>readelf</b>. 1791Jarno Mäkipää taught wget how to follow http 301 and 302 redirects, 1792and did lots of work on vi. (Elliott also did work on vi.) 1793Gavin Howard fixed a comparison bug in bc. 1794Ethan Sommer fixed numerous small issues (including several build 1795warnings and FLAG() macro conversions).</p> 1796 1797<p><u>Bugfixes</u>: 1798<b>tar</b> extract now deletes files and symlinks where it's making a directory, 1799<b>find -L -type -l</b> now finds dangling symlinks, extra #ifdefs in 1800portability.h prevent old gcc versions from barfing on __has_include(), 1801xgetline() now returns NULL at EOF, tee with no arguments was 1802writing to stdout twice, setsid works on nommu (I.E. vfork) now, 1803<b>netcat -L</b> no longer accumulates zombie processes, 1804<b>sntp</b> now uses adjtimex instead of adjtime (to build on bionic). 1805xcreate_stdio() was checking WARN_ONLY in the wrong field (and tar was 1806passing it in the wrong field, so it worked there). 1807Several people wrestled with the <b>xargs</b> "argument too long" problem. 1808Alessio Balsini fixed memory leaks in loopback_setup() and "wayling" 1809added a missing continue to losetup. 1810Elliott fixed xargs -E, various things in modinfo, 1811added an error check to gzip when using zlib (which copies non-gzip data to 1812the output verbatim for some reason), 1813found an case where dirtree could use 1814uninitialized data and silenced "Invalid argument" warnings (triggered by 1815Android's selinux policies making stat() and readlink() fail), 1816fixed locale support in macOS (both in iconv 1817and in main.c), taught stat to show filesystem time on macOS, 1818fixed a 32 bit truncation in sntp, fixed a memory access one byte outside 1819of its array in patch.c, removed the cpio --trailer option, 1820and widened the pid display fields in ps to 18216 digits. Rich Felker fixed find.c assuming 1822time_t is a long (y2038 issue on 32 bit). 1823Greg Kaiser fixed a thinko in get_block_device_size(). 1824Jarno Mäkipää fixed utf8 support in cut -C, and cp --parents. 1825David Legault complained that dir/.* tells rm to delete dir/.. and we'd do it. 1826JakeSFR pointed out a bug in file's identification of broken symlinks. 1827William Haddon fixed cp to treat a directory with a trailing slash 1828teh same as one without. 1829Denys Nykula fixed rm -i not to prompt for an empty "" argument. 1830SebiderSushi reported that chmod g+s wasn't working. 1831The linux kernel doesn't let an O_PATH fd work with fgetxattr(), so 1832Elliott switched <b>ls</b> to use the path-based functions now (which is racy, 1833it means you can stat() one inode and get the xattrs for a different one, 1834but nobody in kernel land seems to use xattrs much so they're not fully 1835supported by the API, and who's crazy enough to use xattrs for security 1836anyway). Derick Pallas pointed out an xclearenv() bug. 1837Atatsulo (or possibly luolongjuna) did a lot of work on the <b>ping</b> 1838command: pointed out min/range/max were out of order, 1839that we shouldn't print a summary unless we received at least one 1840reply packet, and implemented ttl support.</p> 1841 1842<p><u>Build plumbing</u>: 1843The "make root CROSS=all" build announces each target in the title bar, puts 1844its logs into root/log, and has better trap handling to stop with one 1845ctrl-c.</p> 1846 1847<p>Fixed scripts/single.sh to work when PREFIX has no trailing slash, 1848make silentoldconfig no longer feeds "y" to kconfig (which puts menus in a 1849loop) and instead just feeds in newlines to accept whatever the default is. 1850The non-git version number I keep forgetting to update each release moved 1851from main.c to toys.h. 1852Fixed a couple different errors in mkflags.c (one of which caused ls --color 1853to set all the other flags).</p> 1854 1855<p>Elliott added more macos support and tests, and added 1856fallocate, cp, mktemp, and mv to the macOS defconfig.</p> 1857 1858<p><u>Cleanup</u>: 1859Rob did some cleanup on xargs, ls, md5sum, and sort, tidied up main.c a bit, 1860made cp, base64, dmesg, and free use FLAG macros, 1861and switched fallocate to new style global names. 1862Elliott Hughes switched rfkill from the old byte at a time get_line() 1863to libc getline(), made du use FLAG() macros, and mad chattr use 1864standard toybox argument parsing for -v and -p. 1865Several commands (help, cp) had sub-options removed from menuconfig. 1866Merged realpath into readlink.c, and use xabspath() instead of libc realpath().</p> 1867 1868<p><u>Documentation</u>: 1869New roadmap section about <a href=roadmap.html#packages>other packages</a> 1870that toybox can (eventually) replace.</p> 1871 1872<p>Update roadmap to note that posix-2008 moved to a different URL (content 1873at the old URL undergoes random changes), and link to the IETF RFCs. 1874design.html explains more of the history of environment sizes on links 1875and has a #bits anchor tag.</p> 1876 1877<p>Update the <a href=design.html#bits>LP64 section</a> of design.html to 1878fish the documents out of archive.org now that unix.org is gone, 1879and show the actual size table locally.</p> 1880 1881<p>Elliott improved the help of <b>date</b> and <b>chattr</b>, and 1882made the usage: lines in the posix directory more consistent.</p> 1883 1884<p>Shrank the sed help from 150 lines to 90-ish.</p> 1885 1886<p><u>Tests</u>: 1887New "txpect" performs interactive tests, running through a sequence of 1888writes to a command's stdin and reads from stdout and stderr, when failing it 1889reports the first non-matching step. (This for example lets sh.test check 1890the shell line continuation logic prompts with $PS1 and $PS2 appropriately 1891with various unfinished input lines, and that "echo hello; if" doesn't 1892output hello before prompting for the next line of input.) Added 1893VERBOSE=xpect to print out each read and write successfully performed by txpect.</p> 1894 1895<p>runtest.sh only creates an "input" file when the input argument isn't empty, 1896EVAL doesn't supply -- (you have to provide your own if you want that).</p> 1897 1898<p>Rob added basic <b>stat</b>, <b>patch</b>, and <b>tee</b> tests. 1899Jarno added a bunch of <b>vi</b> tests. 1900Elliott fixed tests for <b>ifconfig</b>, <b>lsattr</b>, <b>chattr</b>, 1901and <b>date</b>, added tests for find, id, xargs, and made 1902the id, iconv, env, file, printf and cat tests work on macOS (and skipped 1903the du tests there). Rob cleaned up chmod tests.</p> 1904 1905<p><u>Sheer pedantry</u>: 1906true and false now have usage: lines (which you have to "help false" to see 1907because they (intentionally!) don't support --help. 1908Renamed get_chunk()/dump_chunk() to read_chunk()/write_chunk() in tail.c 1909Elliott fixed some typos.</p> 1910 1911<a name="18-10-2019" /><a href="#18-10-2019"><hr><h2><b>October 18, 2019</b></h2></a> 1912<blockquote> 1913<p>"In those days spirits were brave, the stakes were high, men were real 1914men, women were real women and small furry creatures from Alpha Centauri 1915were real small furry creatures from Alpha Centauri." 1916</p> <p>- The Hitchhiker's Guide to the Galaxy</p> 1917</blockquote> 1918 1919<p><a href=downloads/toybox-0.8.2.tar.gz>Toybox 0.8.2</a> 1920(<a href=https://github.com/landley/toybox/releases/tag/0.8.2>git commit</a>) 1921is finally out (only two months late).</p> 1922 1923<p>The new commands this time are minor variants of existing ones 1924(<b>dnsdomainname</b> is hostname -d, <b>arch</b> is uname -m), but about 19251/3 of toysh got implemented in pending, <a href=https://github.com/landley/mkroot>mkroot</a> got merged as a "make root" target (see Build), 1926and Android's <a href=http://lists.landley.net/pipermail/toybox-landley.net/2019-August/010845.html>hermetic build</a> 1927work continued resulting in a lot of fixes.</p> 1928 1929<p>The android-specific <b>getprop</b>, <b>setprop</b>, <b>start</b>, and <b>stop</b> 1930commands were removed from toybox because they grew dependencies on android 1931libraries, and since toybox tries to avoid mandatory external dependencies 1932they moved to another package.</p> 1933 1934<p><u>Build</u>: 1935The new "make root" target builds a simple toybox root filesystem, 1936using scripts/mkroot.sh which is a stripped 1937down version of the external <a href=https://github.com/landley/mkroot>mkroot</a> 1938project merged into toybox. (Alas, at the moment it wants sh and route out 1939of pending, which aren't quite load bearing yet.)</p> 1940 1941<p>The new "scripts/mcm-buildall.sh" convenience script builds cross 1942compilers based on gcc+musl multiple targets (arm, x86, mips, m68k, s390, 1943microblaze...) 1944using <a href=https://github.com/richfelker/musl-cross-make>musl-cross-make</a>, 1945and a new scripts/cross.sh to easily cross compile for one or more targets 1946using the resulting compilers. (This works with "make root", "make 1947toybox", and other builds understanding the CROSS_COMPILE environment variable.)</p> 1948 1949<p>Setting the environment variable ASAN=1 enables the 1950llvm address sanitizer. (You may need to 1951export ASAN_SYMBOLIZER_PATH on debian to use it, they add the version number to 1952the filename in /usr/bin which confuses llvm.)</p> 1953 1954<p>Patrick Oppenlander fixed make -j (our make is a wrapper around 1955scripts/make.sh which was already doing a parallel build, but if you called 1956make with -j it wouldn't wait for make.sh to finish and the build broke).</b> 1957 1958<p>Lots of fixes for the MacOS build.</p> 1959 1960<p><u>New options</u>: 1961We upgraded <b>tar</b> to extract some older tarballs, <b>file</b> added -b and -s 1962options and can also recognize older tarballs, 1963some xz archives, .otf fonts, perf/simpleperf data files, 1964android boot images, and dtb files. Upgraded <b>file</b> to show the device type 1965for block and char devices, the target for symlinks, and it says _why_ it can't 1966open a file.</p> 1967 1968<p>Several commands ignore some options rather than erroring on them: 1969<b>patch</b> ignores the -f -g# and --no-backup-if-mismatch options 1970(all of which we were already doing by default anyway), and <b>xargs</b> 1971ignores -P# (which requests parallel execution, maybe add it later but 1972single threading works), and Denys Nykula made <b>wget</b> ignore 1973--no-check-ceritificate and <b>gzip</b> ignore -n (again, we weren't doing 1974either anyway) to satisfy pkgsrc.</p> 1975 1976<p>William Haddon taught <b>diff</b> and <b>patch</b> to support special characters in path 1977names (via quoting) and more timestamp formats. 1978Eric Molitor implemented <b>nl</b> -v (with negative and zero starting values), 1979and fixed a MacOS compatibility issue in the build scripts. 1980Tom Cherry added <b>netcat</b> -U (UNIX domain socket support). 1981Denys Nykula taught <b>wget</b> to call ftpget for ftp:// URLs. 1982Ethan Sommer added ln -t and rmdir --ignore-fail-on-non-empty. 1983Elliott Hughes added <b>grep</b> -R, <b>killall</b> -w, tar --mode, 1984 fixed <b>ls</b> -Z and spacing in -lZ and --full-time, 1985added <b>xargs</b> --max-args and removed xargs -I (which we never actually 1986supported) and tweaked the size calculation to more closely match other 1987versions, 1988added xargs -o and made -p read from /dev/tty, added <b>blkid</b> -s, SEC_TYPE for ext3, 1989msdos, and ntfs, f2fs and ntfs LABEL, fixed vfat/ntfs UUID, and switched 1990the endianness for UUID of msdos/ntfs/vfat, simplified the octal mode 1991display in <b>lsof</b>, added <b>pidof</b> -x (thus making the default behavior _not_ 1992be -x).</p> 1993 1994<p>When readdir() succeeds but stat() fails, <b>ls</b> now prints "??? ? ? filename" 1995entries instead of "cannot access" errors. (Still trying to make ls / on 1996android work more gracefully with the SELinux weirdness they added in 9.)</p> 1997 1998<p>Added %C to <b>stat</b> and made it show device type in its default output, 1999<b>cmp</b> now works 2000with 1 argument (implicit - as second argument reading from stdin). 2001<b>sed</b> added exit codes to the q command (q123), added Q, and fixed a 2002bad error message with "!". 2003Removed TAIL_SEEK config option (it just always does that now), 2004<b>find</b> grew several new options (%Z, -true, -false, and -newerXY), <b>ln</b> added -T. 2005The argument to <b>mktmp</b> --tmpdir is now optional.</p> 2006 2007<p><u>Bugfixes</u>: 2008The unshare probe bit-rotted so it was impossible to build <b>unshare</b> 2009and <b>nsenter</b> on some distros. 2010Commit 771e94e2a08 broke toybox's ability to say "unknown command" when 2011you pointed a symlink at it that it that didn't correspond to a command, 2012pointing a symlink at another symlink 2013to make toybox run a command under a name it doesn't recognize 2014(ln -s toybox sh; ln -s sh ash) had an off by one error, 2015<b>ifconfig</b> no longer error_exits if the "read mac address" ioctl fails (Android N 2016and later block that for non-root users to prevent device tracking). 2017When the libcrypto library was disabled, md5sum was building commands it 2018hasn't got built-in support for yet (like sha3sum) which acted like redundant 2019copies of md5sum, 2020<b>tar</b> wasn't correctly reading or writing sparse files with a hole at 2021the end, and tar tzf blah.tar.xz was calling xz when it meant xzcat, 2022<b>xargs</b> would spuriously fail on large memory systems (due to a missing 2023"unsigned" in a typecast), 2024<b>basename</b> can now remove suffixes starting with a dash, 2025<b>cp</b> -r unlinks() an existing symlink before trying to 2026recreate the symlink (otherwise cp -r into an existing directory will 2027try to follow the existing symlink and create another symlink at its 2028destination, failing if a file already exists there, which 2029was confusing gentoo's package manager), <b>timeout</b> now produces the 2030right exit code when the timed command intercepts the exit signal, 2031and <b>hostname</b> -d no longer segfaults on machines with no domain name.</p> 2032 2033<p>Denys Nykula fixed the <b>ftpget</b> freeze (it was sending REST instead 2034of RETR), made <b>mv</b> work with a trailing slash in the source, 2035fixed a <b>find</b> segfault, 2036and made <b>env</b> always exec rather than recursing to a builtin 2037(on the theory #!/usr/bin/env is expected to search the $PATH).</p> 2038 2039<p>Elliott improved <b>killall</b>'s handling of long filenames, made <b>kill</b> work 2040when killall5 is disabled (missing FORCE_FLAGS), fixed error reporting in 2041<b>hostname</b>, fixed several things in <b>xargs</b>, 2042made <b>printf</b> \c and \0 octal escapes work, fixed a <b>find</b> -name corner 2043case (find src/*.c -name file.c) and find dangling symlink behavior, 2044made some <b>rm</b> error messages consistent, fixed \x behavior in echo and 2045printf, fixed linestack.c's buffer length for utf8 U+XXXX escape printing, 2046fixed a race in <b>losetup</b>, fixed <b>grep</b> -F with multiple matches on the same 2047line and an -I failure in the existing grep tests, made <b>readlink</b> notice 2048when it has more than one argument, and made various 2049fixes to <b>modinfo</b>.</p> 2050 2051<p>Ethan Sommer reported that <b>stat</b> of an suid/sticky file output 5 digits of 2052octal stat date instead of 4. 2053Rob fixed an unaligned access in <b>grep</b>. Alessio Balsini removed an incorrect 2054null termination in <b>losetup</b>. 2055Jarno Mäkipää fixed <b>hexedit</b> 2056scrolling up and down on older terminals, and fixed <b>ls</b> segfaulting with a broken 2057(non-UTF8) locale (didn't expect wcrtomb to return -1). 2058Andrew Ilijic made ls put 2 spaces between columns like other implementations 2059(which helps when terminals get confused and apply UTF8 combining characters 2060to the space after a filename). 2061</p> 2062 2063<p><u>Library</u>: 2064Added dlist_lpop() to use a dlist as a stack (it removes the last entry 2065instead of the first), and dlist_pop() now works on a dlist_terminate()d list. 2066DIRTREE_STATLESS returns entries we couldn't stat() (with a zeroed ->st 2067field, and ->again |= 2), but filling in the file type from readdir(). 2068Coversions between signal names and numbers now include all the Linux signals, 2069not just posix ones. The functions sig_to_num() and num_to_sig() moved into 2070lib/portability.h, because MacOS has different signals. 2071TOYFLAG_MAYFORK allows commands to run in toysh's process without forking, 2072but also makes them accessable from the toybox multiplexer or standalone. 2073A MAYFORK command has to clean up after itself (even in error paths) and 2074can't discard anything we need to keep (such as closing stdout).</p> 2075 2076<p>Elliott did a big xbind/xconnect cleanup (adding xbindany and xconnectany), 2077taught xregcomp that an empty regex matches the whole line 2078(because FreeBSD, and thus MacOS, doesn't already do that), and 2079replaced several uses of get_line() (which does single byte reads) 2080with getline() (which doesn't).</p> 2081 2082<p><u>Plumbing</u>: 2083"make clean" doesn't produce a screenful of unnecessary output now, 2084rm, ln, cp, kill, and netcat use the FLAG() macros now, makedevs uses the new 2085GLOBALS naming format, timeout now uses xwaitpid(), and we fixed an 2086off by one error in xwaitpid().</p> 2087 2088<p>Elliott removed a workaround for old NDK versions from getconf and 2089moved sort off of get_rawline() (which let us remove it). 2090Rob undid some loop unrolling in md5sum/sha1sum (we have libcrypto if you want 2091an assembly optimized version instead of an understandable version).</p> 2092 2093<p>Android ndk-r20 doesn't define the __ANDROID_NDK__ symbol, so we switched 2094to using clang's __has_include() (which is an undefined macro and thus a NOP 2095resolving to false on gcc).</p> 2096 2097<p><u>Pending</u>: 2098Rob did a lot of work on toysh: continuation support works (it knows when to ask 2099for another line of text and when to run what it's got), as do 2100nested if/else/elif statements, and while and for loops (supporting "for i" 2101and "for i in", but not for ((;;)) yet). 2102It parses pipes and redirections but doesn't perform them yet. 2103The next big todo chunk is environment variable resolution. 2104</p> 2105 2106<p>Gavin Howard fixed some bugs in bc, such as not returning error when raising 2107zero to a negative power. 2108Jarno Mäkipää continues to extend vi, adding yank and push, support for tabs, 2109rewriting delete, changing the drawing code, etc. Elliott also added several 2110fixes and features to vi, 2111fixed host to cope with large DNS responses, switched crontab to 2112use getline(), did work on man, fixed fixed several things in dd, 2113implemented diff --strip-trailing-cr and made diff not need /tmp. 2114Mike Bennett added extended attribute support to ip.c's route change.</p> 2115 2116<p><u>Test suite</u>: The killall and pgrep tests were having occasional 2117spurious failures due to a race condition (the test script would fork() 2118to call sleep and for just a moment there were two instances of the test 2119script, reporting an extra process if killall or pgrep ran then). 2120 2121<p>Elliott Hughes did a lot of work on the test suite: add kill -l tests, 2122make losetup.test notice (and skip) if loopback 2123devices area already in use, blkid.test can TEST_HOST e2fsprogs blkid now, 2124modinfo.test notices (and skips) if there's no /proc/modules or modules 2125directory, ifconfig.test skips if "dummy0 up" fails and disabled the 2126pointtopoint tests entirely (the kernel never supported it?), 2127truncate.test is less 2128confused by SELinux, hostname.test can run some tests as non-root, 2129addressed a race condition in killall.test on slow systems, 2130made chgrp.test, chown.test, and tar.test handle an empty /etc/passwd or /etc/group 2131(we were checking whether they were _there_ but some android devics have 2132empty ones), 2133fixed a couple things in env.test, extended killall.test to explicitly cover 2134long and short names (and not assume the shell is /bin/sh), taught split.test not to rely on bash process substitution, 2135added missing TZ=utc to touch.test, fixed cp.test to pass for root 2136or with a restrictive umask, and fixed the FAILCOUNT arithmetic to use more portable 2137$(()). Elliott also made some tests more flexible (find.test can handle any 2138error message about symlink loops, etc), and disabled others (such as du) 2139when SELinux is enabled.</p> 2140 2141<p>Rob taught testcmd not to use the shell builtin for TEST_HOST unless there 2142isn't one in the $PATH, work if $PWD has a space in it, 2143and make kill.test stop testing the shell builtin 2144insead of the command, quieted tests that were producing output 2145other than their PASS/FAIL line, and made NOSPACE tests ignore the 2146presence/absence of leading/trailing whitespace as well as differences 2147in the amount of space.</p> 2148 2149<p><u>Documentation</u>: 2150The <a href=roadmap.html>roadmap</a> got freshened up a bit, with updates 2151for android from Elliott. 2152In <a href=design.html>the design page</a> the Features section starts with 21534 new paragraphs about the scope of toybox, and a new License section talks 2154about 0BSD. 2155The "./toybox --help" output now explains how to do install_flat by hand.</p> 2156 2157<p><u>Sheer pedantry</u>: 2158Elliott fixed a double close in loopfiles_lines() so strace doesn't 2159have an EBADF in it, and added a missing "static" in lib/net.c.</p> 2160 2161<a name="21-05-2019" /><a href="#21-05-2019"><hr><h2><b>May 21, 2019</b></h2></a> 2162<blockquote> 2163<p>"I like the cover," he said. "'Don't Panic.' It's the first helpful or intelligible thing anybody's said to me all day."</p> <p>- The Hitchhiker's Guide to the Galaxy</p> 2164</blockquote> 2165 2166<p><a href=downloads/toybox-0.8.1.tar.gz>Toybox 0.8.1</a> 2167(<a href=https://github.com/landley/toybox/releases/tag/0.8.1>git commit</a>) 2168is out.</p> 2169 2170<p>Android is now using toybox as part of a <a href=http://lists.landley.net/pipermail/toybox-landley.net/2018-November/009824.html>"hermetic" build</a> 2171(meaning Android is building under toybox, which is halfway to Android 2172building under Android). See Android's <a href=https://android.googlesource.com/platform/prebuilts/build-tools/+log>build-tools</a> and <a href=https://android.googlesource.com/platform/build/soong/+log>soong</a> repositories to follow along with development, or check the list for the <a href=http://lists.landley.net/pipermail/toybox-landley.net/2019-May/010485.html>current status</a>.</p> 2173 2174<p><u>New commands</u>: Isaac Dunham added <b>mcookie</b>, which is used by 2175X11. Elliott Hughes added <b>devmem</b>, which replaces toolbox's "r".</p> 2176 2177<p><u>Promoted</u>: <b>tar</b>, <b>gunzip</b>, and <b>zcat</b> were cleaned 2178up and moved out of pending. 2179Tar is a near-rewrite with support for --owner --group and --sparse, 2180capable of autodetecting "wget blah.tar.bz2 -O - | tar xv", a --restrict option 2181that only allows files to extract into a single subdirectory, and so on.</p> 2182 2183<p><u>Added to pending</u>: 2184Jarno Mäkipää contributed a new <b>vi</b> implementation. Makepost added 2185<b>man</b>.</p> 2186 2187<p><u>Updates</u>: Elliott added <b>find</b> -wholename/-iwholename, -printf, 2188and -context, <b>grep</b> --exclude-dir, and <b>echo</b> -E. 2189Rob added <b>blkid</b> -U and -L, <b>mount</b> now has UUID= support via blkid -U, 2190you can now specify TZ="blah" in the <b>date</b> -d input string (although 2191why the double quotes are required is an open question), and <b>sed</b> now 2192understands ,+N relative ranges. Makepost added <b>rm</b> -v. 2193The human_readable fields in <b>ps</b>/<b>top</b>/<b>iotop</b> now use all 2194available space to show more precision, <b>iotop</b> displays swap bandwidth 2195properly (it was showing a signed field as unsigned), and 2196<b>touch</b> -t and -d autodetect the time format.</p> 2197 2198<p><u>Cleanups</u>: In pending Rob did some bc cleanup, 2199Colin Davidson did some arp cleanup, 2200Uros Prestor fixed route argument parsing, and 2201in bc Daniel Rosenberg fixed bc_vec_concat()'s allocation and 2202Gavin Howard fixed an overflow bug in bc_num_ulong().</p> 2203 2204<p>Elliott made scan_key() detect cursor keys and such with several more 2205terminal types, made more better about directories and non-tty output, 2206cleaned up inconsistent uses of ARRAY_LEN(), taught file 2207to recognize a c-sky ELF machine, and fixed timezone and daylight 2208savings time issues in date and its tests. 2209Rob simplified netcat and ping using new lib/net.c infrastructure. 2210When run as non-PID 1 (for a chroot) oneit's exit code now indicates 2211which signal it's exiting due to. 2212Hadrian Węgrzynowski #defined constants for mkpathat. 2213Date now uses the current time for unspecified fields (instead of zeroing 2214them), and should set the weekday properly. 2215Sort's -k2.3,4.5 syntax is now documented and behaves more closely to 2216other implementations (silly as that behavior is). 2217Significant speedup to sed s/a/b/g on a megabyte long line (it was 2218O(n^2) and now it's quite snappy).</p> 2219 2220<p>Several cases of warning about stdout failing to write were silenced 2221because things like piping the output to head means stdout is a special 2222case: its output is often discarded and that's fine. And xprintf() 2223and friends only checks stdout for errors so it can exit early ("yes | head" 2224should not spin endlessly) without actually flushing anything. 2225There's an ongoing design debate about line buffering stdin and stdout 2226that'll have to be resolved next dev cycle.</p> 2227 2228<p><u>Bugs</u>: 2229Elliott sent in a lot of fixes for building AOSP with toybox: 2230support for more date formats, sed substitution of empty capture groups 2231and an endless loop with -EPIPE, fixed find -path/-ipath, made loggit() 2232only feed LOG_AUTH to syslog when we have a tty, gave grep a bunch 2233of --longname synonyms for short options, added getfattr --only-values, 2234and timeout --foreground --preserve-status and --signal.</p> 2235 2236<p>Top got a lot of tweaks: it no longer collates thread's CPU usage into the 2237parent thread (in -H mode), hides the cursor in interactive mode, updates the 2238display properly in batch mode, and accepts ENTER 2239as a synonym for SPACE when refreshing the display. The header lines 2240now abbreviate long numbers with units (via human_readable()), 2241more reliably spans the terminal width, and says "Thread" instead 2242of "Task" when it's showing threads. The task/thread count now adds up 2243(understanding more kernel states and treating unknown ones as "sleeping"). 2244Top now treats -o "" as an error (previously it displayed an empty screen).</p> 2245 2246<p>William Djupström pointed out login needs to chown() the tty, and made it 2247clear the environment when -p is absent. 2248Vidar Holen fixed inverted find -exec exit status. 2249Rob made chgrp() stop relying on libc's printf("%s", NULL) printing "(null)" 2250because a <a href=https://github.com/varnishcache/varnish-cache/issues/2879>newly added gcc bug</a> turns that into a build break, and 2251fixed an outright embarassing xstrndup() bug. 2252Liwugang fixed a segfault when stat's format string had a trailing %. 2253A missing else in grep meant -x was sometimes ignored.</p> 2254 2255<p><u>Infrastructure</u>: 2256in the build scripts/portability.sh now tests that the compiler it's trying 2257to use exists up front. We no longer link against librt because on glibc that's 2258pulling in libpthread for some unknown reason.</p> 2259 2260<p>TOYFLAG_ARGFAIL() lets commands return custom exit values when option parsing 2261fails (because AOSP build scripts cared), so timeout --unknown returns 125 2262and so on: chroot, diff, cmp, env, grep, nohup, and sort also use it now with 2263various values.</p> 2264 2265<p>New FLAG() macros: FLAG(x) expands to (toys.optflags&FLAG_x) and returns 2266zero if the command line argument's bit isn't set and nonzero if it is 2267(meaning -x was or wasn't encountered on the command line this time by 2268lib/args.c). Lots of commands got converted to use them.</p> 2269 2270<p>New lib/env.c with reset_env(), xsetenv(), and xunsetenv() which frees 2271the old environment variable when it wasn't inherited from exec(). This 2272lets long-running loops reset environment variables without memory leaks.</p> 2273 2274<p>New xnotify() plumbing in lib/portability.c with Linux and Mac versions 2275(which should also work on BSD).</p> 2276 2277<p>New xparsedate() and xvali_date() functions allow date, tar --mtime, and 2278touch to understand the same date formats, 2279xpopen_both() now knows how to use existing filehandles for the child's 2280stdin/stdout, and no longer redirects stderr (which hides errors), 2281new nanomove(), nanodiff(), union socksaddr, xrecvwait(), 2282and human_readable_long() to show more than 3 digits of output.</p> 2283 2284<p>Taught xgetaddrinfo() to return a wildcard address for a NULL host, 2285xconnbind() sets SO_REUSEADDR by default, 2286fileunderdir() returns the abspath to the file now (since we had to work 2287it out anyway), and 2288xabspath() uses O_PATH now (because Android's SELinux rules got more aggressive).</p> 2289 2290<p>Several new xsendfile() variants, including sendfile_len() to copy 2291a specific amount of data and xsendfile_pad() to append zeroes to the 2292output when there isn't enough input.</p> 2293 2294<p>regexec0() is now using libc's REG_STARTEND supported by every libc out 2295there _except_ musl. (It originated in BSD and was adopted by glibc, bionic, 2296and uClibc 15 years ago, and is also supported in macosx. Musl's maintainer 2297<a href=https://www.openwall.com/lists/musl/2013/01/15/26>turned down</a> 2298the feature request as an invalid use case he didn't think they should 2299be trying to do.)</p> 2300 2301<p><u>Docs</u>: 2302the <a href=license.html>license page</a> now has an SPDX identifier, and links to more 2303history/context. The actual LICENSE file had its title line removed 2304so <a href=https://github.com/github/choosealicense.com/pull/643#issuecomment-463746027>github could recognize it</a> as 0BSD. (We're still waiting for 2305"<a href=https://github.com/github/choosealicense.com/pull/643#issuecomment-464380789>a while</a>" to pass before github actually does, though.)</p> 2306 2307<p>A new www/doc/mount.txt file describes how mount works under the covers.</p> 2308 2309<p>Rob consistently misspelled "canonical" and "millisecond", mountpoint 2310has a synopsis now, 2311Kevin van der Kamp fixed a typo in netstat's help text, and Elliott 2312Hughes fixed typos everywhere and made the tense, capitalization, and 2313punctuation of help text more consistent.</p> 2314 2315<p>New paragraph in the code style part of design.html explaining that 2316"char* a, b;" is not how C works, and since you wind up with "char *a, b;" 2317anyway you might as well say that.</p> 2318 2319<p>Given dreamhost's tendency to repeatedly delete the mailing list 2320archive, the nav bar on the left links to a backup web archive now too.</p> 2321 2322<p>The FAQ now says why we do time based releases.</p> 2323 2324<p><u>Testing</u>: Added VERBOSE=nopass to not show passing tests. 2325The test suite now does an rm -rf on testdir between each command so 2326debris files don't accumulate. 2327New skipnot function skips the next test if a command line fails, and 2328toyonly function only runs command when we're testing the toybox version 2329of command. (This isn't the same as TEST_HOST, the host version could be 2330toybox.) 2331The find -newer test calls "sleep .1" to avoid intermittently failing with an 2332identical timestamp. 2333Elliott added the start of a file.test rogues gallery. 2334No longer call lsattr in "make tests" because the behavior differs by 2335filesystem and it needs fixing somehow.</p> 2336 2337<a name="08-02-2019" /><a href="#08-02-2019"><hr><h2><b>February 8, 2019</b></h2></a> 2338<blockquote> 2339<p>"Most readers get as far as the Future Semiconditionally Modified 2340Subinverted Plagal Past Subjunctive Intentional before giving up; and 2341in fact in later editions of the book all pages beyond this point have 2342been left blank to save on printing costs."</p> 2343<p>- The Hitchhiker's Guide to the Galaxy</p> 2344</blockquote> 2345 2346<p><a href=downloads/toybox-0.8.0.tar.gz>Toybox 0.8.0</a> 2347(<a href=https://github.com/landley/toybox/releases/tag/0.8.0>git commit</a>) 2348is out.</p> 2349 2350<p>Toybox now builds on MacOS and FreeBSD, thanks to the efforts of Elliott 2351Hughes and Ed Maste respectively. Use the "make macos_defconfig" and "make 2352freebsd_defconfig" targets to enable the set of commands that compile on 2353each so far.</p> 2354 2355<p><u>New commands</u>: Added an <b>sntp</b> client/server (RFC 4330 Simple Network Time 2356Protocol, a compatible subset of ntp). The <b>test</b> command was rewritten 2357and promoted out of pending.</p> 2358 2359<p><u>New options</u>: <b>grep</b> now has --color, supports embedded NUL bytes in its input, 2360recognizes binary files, and passes all of grep.test (in case you 2361needed the middle numerical field of -bB, etc). 2362Josh Gao added ipv6 and UDP support to <b>netcat</b>. 2363Volodymyr Medvid reported that install -d doesn't honor -o or -g. 2364Elliott Hughes did a lot of work to support hermetic Android/AOSP builds: 2365<b>mkdir</b> accepts both --parent and --parents as synonyms for -p, <b>touch</b> ignores -f, 2366<b>basename</b> added -s to remove a trailing suffix, <b>dirname</b> now supports multiple 2367arguments, <b>cmp</b> accepts --quiet and --silent as synonyms for -s, <b>hostname</b> 2368added -sfd, <b>head</b> added --bytes as a synonym for -c and --lines as a synonym 2369for -n, <b>mktemp</b> added -t and fixed -u, <b>sed</b> added -z and -iEXT to keep backup files, 2370<b>md5sum</b> and sha1sum added --status and --check as synonyms -s and -c, 2371<b>readlink</b> added --canonicalize as a synonym for -f, <b>sort</b> grew -V, 2372<b>patch</b> added -s its synonym --quiet, <b>stat</b> added --format as 2373a synonym for -c, <b>xargs</b> added -p -t -r, 2374Eduardas Meile asked 2375that <b>umount</b> ignore -c. Reverend Homer added a small optimization to file.c, 2376and Elliott taught <b>file</b> to recognize riscv ELF binaries. Peter Collingbourne 2377taught <b>ls</b> -t to use the nanoseconds field. 2378<b>patch</b> has better support for patching a file with a tab in the name.</p> 2379 2380<p><u>Bugfixes</u>: 2381<b>cp</b> --preserve was segfaulting when you didn't specify _what_ to 2382preserve (it now correctly defaults to "mot") and didn't get the permissions 2383right when copying a symlink's contents as a regular file, 2384<b>sort</b> -x didn't work when 2385attached to a key, <b>host</b> didn't allocate a big enough buffer for worst 2386case ipv6 address size, <b>sed</b> needed a ; between b and } when other 2387implementations don't (we're already well past what posix says but a script 2388out in the field broke...), 2389and several fixes to <b>hostname</b> in a container, <b>ps</b>/<b>top</b> were 2390misreading the VIRT and SHM fields.</p> 2391 2392<p><u>Pending</u>: Gavin Howard updated <b>bc</b> and added more tests, 2393Marius Adaskevicius pointed out <b>mdev</b> -s should follow symlinks, 2394Yangchun Fu reported a <b>dhcp</b> checksum bug, <b>modprobe</b> needed errno reset 2395to avoid reporting spurious errors in verbose mode and no longer 2396exits with status 1 if it can't find /etc/modprobe.conf, the <b>more</b> command 2397wasn't always flushing stdout when it exited.</p> 2398 2399<p><u>Build</u>: 2400<b>chrt</b> no longer #warns about the musl sched_get_priority_min() bug, but 2401instead works around it. 2402We were mixing setjmp/siglongjmp (harmless in some contexts but it kind 2403of annoyed FreeBSD), and 2404make install_airlock now adds sha256sum (because the Linux 4.20 build now 2405needs that for the s390x target).</p> 2406 2407<p><u>Coding style</u>: 2408Rob converted the rest of the option GLOBALS() to the new single letter 2409coding style, and the new FLAG(x) macro is a slightly tidier way to say 2410"toys.optflags&FLAG_x". 2411Removed CFG_SORT_BIG (the sort command always 2412has the full functionality now. The general future direction or toybox 2413is to either have a command or not have it; multiple versions of the 2414same command aren't worth the complexity in testing, documentation, 2415or system adminstration).</p> 2416 2417<p><u>Library code</u>: 2418The mkflags plumbing can now support arbitrary punctuation as option 2419names via an escape syntax (because mkfs.vfat specifies an offset with -@), 2420and lib/args.c now only sets FLAGS_NODASH when the first argument 2421didn't have a dash (allowing "ps ax" and "ps -ax" to behave differently).</p> 2422 2423<p>Added an xrename() function, 2424xchdir() has better error reporting, xconnect()/xbind() had their 2425implementations merged, xsendto() moved from ping to lib, xpoll() now 2426measures elapsed time and only waits for the remaining period when restarted, 2427and Eduardas Meile converted several error_exit() to error_exit_raw() (because 2428yocto 2.6 buils with -Werror=format-security by default).</p> 2429 2430<p>There's a scripts/portability.sh now which lets the build use gsed and gmake 2431(when available) in more places to avoid broken host versions on MacOS and FreeBSD. 2432The syntax of the "noreturn" attribute 2433changed slightly, some calls to strnstr() were replaced with strcasestr(), 2434we work around MacOS' lack of features.h, portability.h can now do Apple's 2435endianness macros, and so on.</p> 2436 2437<p>Following the <a href=faq.html#support_horizon>seven year rule</a> 2438Elliott removed support or glibc 2.10, and also 2439reformatted a lot of --help text for consistency (removing trailing 2440periods and such).</p> 2441 2442<p><u>Test suite</u>: 2443The test suite now has its own version of "pending", specifically "make tests" 2444skips test files without the executable bit set (unless you export $TEST_ALL). 2445This eliminates "expected failures" (I.E. files with unresolved todo items) 2446from the global regression test. Additional tests can be added to the global 2447regression test with chmod +x as the todo items they document get resolved.</p> 2448 2449<p>Several places the test suite got fluffed out, including guards to skip 2450root-only tests when run as a normal user. The testcmd function prints the 2451short name instead of the full path to the command. 2452The tests for the test command 2453("test.test") no longer test the shell builtin but correctly test the 2454toybox command. (Now say that ten times fast.)</p> 2455 2456<p>Bash version skew required replacing "continue" with "return" to stop 2457tests early in bash 4.4 (for VERBOSE=fail).</p> 2458 2459 2460<a name="31-10-2018" /><a href="#31-10-2018"><hr><h2><b>October 31, 2018</b></h2></a> 2461<blockquote><p> 2462"In the beginning the Universe was created. 2463This has made a lot of people very angry and been widely regarded as a bad move."</p><p>- The Hitchhiker's Guide to the Galaxy 2464</p> 2465</blockquote> 2466 2467<p><a href=downloads/toybox-0.7.8.tar.gz>Toybox 0.7.8</a> 2468(<a href=https://github.com/landley/toybox/releases/tag/0.7.8>git commit</a>) 2469is out (a month late).</p> 2470 2471<p>Toybox <a href=http://lists.landley.net/pipermail/toybox-landley.net/2018-October/009769.html>now builds</a> with the Android NDK (r18 release), almost out of the box. 2472You have to add an "llvm-cc" symlink to "clang", then use CROSS_COMPILE=llvm- 2473with the appropriate $PATH (or absolute path in CROSS_COMPILE) for the 2474target you want to build for. (To run it on a non-android host, you probably 2475want "LDFLAGS=--static" too.)</p> 2476 2477<p><u>New commands</u>: <b>i2cdetect</b>, <b>i2cdump</b>, <b>i2cget</b>, and <b>i2cset</b> 2478courtesy of Elliott/Android. The <b>watch</b> command got a complete (tty-aware) 2479rewrite, and was promoted out of pending. The <b>prlimit</b> command is back 2480(a configuration bug was always disabling it), and <b>ascii</b> now defaults y 2481in defconfig (an oversight).</p> 2482 2483<p><u>New options</u>: 2484The <b>ifconfig</b> command added -S (short view) giving one line of 2485information per interface, in "name ipv4/mask macaddr [ipv6/mask@type...]" 2486format. <b>grep</b> grew binary file detection, with -I and -a options to 2487ignore them or force treating them as ascii (really utf8). 2488Zach Van Rijn asked for <b>find -empty</b>. 2489In the <b>file</b> command, Elliott improved java 2490.class support and added .wav, .bmp, and android .dex. 2491Elliott also added "getconf -a" and a lot of missing getconf symbols the 2492AOSP build uses, including support for pathconf(3) symbols requiring two 2493arguments. This means getconf's arguments work more like ubuntu's 2494now (one or two arguments, not iterating through the supplied list and 2495processing each like last release), and -l grew section identifiers (so 2496the "getconf -l" output is no longer a directly consumable list of symbols 2497it takes). 2498Command line options specifying durations (like top -d or ping -i) can 2499now use fractions and units, like "1.5" and ".1m". See "toybox --help" for 2500details. 2501Toybox now dereferences one layer of symlinks if it doesn't recognize the 2502name it's called under (so if you "ln -s $(which sleep) blah" and then 2503"./blah 30", it should figure out you mean sleep 30).</p> 2504 2505<p><u>Pending</u>: 2506Gavin Howard contributed a large new <b>bc</b> implementation to pending, 2507which covers everything the kernel build needs (and more) but is going to 2508take a while to review. 2509In <b>diff</b> the android guys added --color and timestamps on the 2510+++/--- lines. In <b>mdev</b> Faustas Azuolas Bagdonas added support 2511for mdev.conf's fourth field. Reverend Homer suggested an error message 2512improvement in <b>wget</b>, Kevin Spiteri did a cleanup pass on test.c, 2513and there was a cleanup pass on the passwd command.</p> 2514 2515<p><u>Bugfixes</u>: 2516Elliott responded to a bug in <b>getconf</b>'s sed-based header generation by 2517replacing it with a more conventional array of #defined symbols, with #ifdefs 2518for missing symbols on the three libc implementations (glibc, bionic, musl) 2519we've tested so far. (This requires portability.h work to add new supported 2520build environments, but at least it works on bionic now.) 2521The <b>file</b> command now fails gracefully when reading pathologically 2522broken ELF files (the tests for integer overflow missed one), and 2523<b>date</b> no longer shows a meaningless errno when it can't parse the 2524date format. 2525Several fixes in <b>ping</b> (-w and -W didn't work, -c kept sending packets 2526while waiting for replies, and when no reply packets were received the summary 2527data was printed twice). 2528Elliott fixed <b>wc</b> column widths (traditional reality doesn't match 2529posix), fixed <b>modprobe</b>'s error handling (reported by Wen Xie), and 2530also fixed the units on blockdev --blkraget (reported by Martijn Coenen). 2531Several small fixes to the <b>stat</b> command's output (help text, quoting 2532style, leading zeroes, switch -f from %t to %T), <b>oneit</b> no longer 2533reboots the system when it's not called as PID 1. 2534Last release's commit letting $STRIP be redefined had a typo. 2535Nick Kralevich added an istty() check before calling TIOCGWINSZ (which 2536triggered Android's strict ioctl filtering on non-filesystem fifos). 2537An unstripped build (where the "strip" command fails for a given 2538CROSS_COMPILE) will now overwrite the (read-only) output file instead of 2539failing a rebuild. The dirtree plumbing no longer prints a "No" on front 2540of the filename in error messages (such as "permission denied").</p> 2541 2542<p><u>Docmentation</u> 2543Elliott updated the roadmap with Android AOSP status, towards turning that 2544into a hermetic build (hopefully with an eventual self-hosting option, work 2545is <a href=b33d37d6f735>already underway</a> on the airlock step). 2546Rob added a little more context for LSB and described the move from Aboriginal 2547Linux to mkroot. 2548The <b>ps</b> help text changed: now "ps -o help" shows the fields and the normal 2549--help is much more manageable. Ifconfig's --help text also got tidied up. 2550Our LICENSE file has the SPDX identifier (0BSD) now, and the note about 2551the kconfig/ subdirectory still having (build only, non-shipping) GPL code 2552in it moved into the kconfig subdirectory.</p> 2553 2554<p><u>Coding style change:</u> 2555GLOBALS() now uses the argument letter as the variable 2556name for variables automatically set by command line arguments via lib/args.c, 2557and no longer requires they be one per line. 2558Removed use of the gcc 2559\e extension in string constants, replacing it with \033.</p> 2560 2561<p><u>Library code:</u> 2562lib/interestingtimes.c was renamed to lib/tty.c. 2563The deflate code from toys/pending/compress.c moved to lib/deflate.c, 2564which gunzip.c now uses when it's not configured to use zlib. (This reduces 2565the bootstrap dependencies for the "airlock" step of a hermetic build.) 2566The compression side remains a todo item. 2567Added "%" to lib/args.c reading seconds (including fractions/units) into a 2568long milliseconds argument. Redid xparsetime() to not need floating point, 2569added xparsemillitime() for milliseconds common case, and 2570parsetime() now treats leading garbage as an error instead of returning zero. 2571The bufgetpwuid()/pufgetgrgid() functons can now handle more than 4k of group 2572data. The set_terminal() function can now set serial speed, and microcom.c 2573uses it. Added xsignal_flags(), and more consistently use xsignal() (which wraps 2574sigaction()) instead of signal(). 2575A new xgetrandom() function calls the new getrandom() system call, falling 2576back to /dev/urandom on older kernel/libc (made to actually _work_ by 2577Patrick Oppenlander).</p> 2578 2579<p><u>Build infrastructure:</u> 2580Updated dependency detection so "make distclean defconfig toybox" now works 2581all on the same command line. (This is nice when $CROSS_COMPILE changes). 2582scripts/config2help.c no longer includes parts of lib/ and just copies what 2583it needs into itself, improving build portability. 2584The build now checks the specified cross compiler exists, hopefully providing 2585a more informative error message when it doesn't. Similarly, when .config is 2586missing the error message suggests running make defconfig.</p> 2587 2588<a name="23-06-2018" /><a href="#23-06-2018"><hr><h2><b>June 23, 2018</b></h2></a> 2589<blockquote> 2590<p>There is a theory which states that if ever anyone discovers exactly what 2591the Universe is for and why it is here, it will instantly disappear and be 2592replaced by something even more bizarre and inexplicable. There is another 2593theory which states that this has already happened.</p> 2594<p> - The Hitchhiker's Guide to the Galaxy</p> 2595</blockquote> 2596 2597<p><a href=downloads/toybox-0.7.7.tar.gz>Toybox 0.7.7</a> 2598(<a href=https://github.com/landley/toybox/releases/tag/0.7.7>git commit</a>) 2599is out.</p> 2600 2601<p><u>New commands</u>: Rob Landley added <b>getconf</b>, Elliott Hughes added 2602<b>uuidgen</b>, and <b>ping</b> and <b>fmt</b> were promoted out of pending.</p> 2603 2604<p><u>New options</u>: The <b>cp</b> command added the --parents option, 2605<b>readlink</b> now has -m to show where a missing path would be, 2606<b>netstat</b> got some updates (better nommu support, it can use /etc/services 2607names, more command line options are bounds checked), 2608Ng Zhi An added --getra and --setra (get/set readahead) to <b>blockdev</b>, 2609Elliott Hughes added <b>xxd</b> -o and made <b>top</b> -d understand fractional 2610sections (because the Linux Test Project uses that), and 2611Minghui Liu added <b>watch</b> -b.</p> 2612 2613<p><u>Bugfixes</u>: Several fixes to <b>vmstat</b>: Elliott Hughes made it work on a 261448-core machine 2615and fixed a glitch with -n, Haroon Maqsood pointed out "vmstat 1 1" didn't 2616stop and "vmstat -n" didn't print the first theader line. 2617Minghui Liu fixed a bug where <b>cp</b> -p didn't preserve 2618timestamps, and pointed out the b, c, and d suffixes on numbers weren't 2619working. Ryan Pritchard pointed out that <b>du</b> -d0 should act like du -s. 2620Filip Perich made an RFC2347 OACK compliance fix to <b>tftp</b> in pending. 2621Zach van Rijn fixed a bug in <b>xxd</b> causing incorrect translation of 2622uppercase characters. 2623Elliott fixed several things in <b>top</b> (removed spurious '\r' characters from -b 2624output, removed interactive flicker, made running processes bold), and 2625pushed Rob to make <b>file</b> work better recognizing things on stdin 2626("cat /bin/ls | file -" still won't work but "file - < /bin/ls" should). 2627Rob fixed a bug in <b>netstat</b> on 64 bit big endian systems, 2628and fixed <b>cut</b> -DF 2629(a posix compliance fix broke its ability to act as a decent awk replacment 2630because outputting all delimiterless lines isn't what you want there, -D 2631now disables that too).</p> 2632</u></p> 2633 2634<p><u>Documentation</u>: Rob added a buildroot section to the roadmap with the 2635prequisites that needs to run, and reformatted more of the ps --help text 2636to two columns.</p> 2637 2638<p><u>Library</u>: 2639FLAGS_NODASH is now set in toys.optargs when an optstring starting 2640with & has no dash in its first argument. (This lets "ps -ax" and "ps ax" 2641behave differently.) Factored out xtestfile() into lib/. 2642The comma-separated-list parsing infrastructure moved to lib/commas.c. 2643Added mkpath() for the common case of mkpathat() and #defined MKPATHAT_* 2644constants for the uncommon cases. Elliott added better error checking 2645to xparsetime() and fixed a bug in names_to_pid() (used in <b>pidof</b> 2646and <b>killall</b> and such). 2647Rob inlined the old toys/e2fs.h into pending's mke2fs, 2648and removed a rogue semicolon from the WOULD_EXIT() macro that screwed 2649up if() statement levels. The do_lines() semantics changed adding a 2650callback(0,0) at the end of each file.</p> 2651 2652<p><u>Tests</u>: 2653The test_* commands under toys/example were all renamed demo_* so 2654they don't share a namespace with running a standalone command 2655through the test suite (ala "make test_sed"), then 2656demo_human_readable became demo_number so it can also test atolx() 2657suffixes. The readlink.test was getting confused by a 2658behavior difference between toybox and ubuntu (Ubuntu's ln -sf wouldn't 2659replace a symlink that pointed to itself, toybox's would), now there's 2660an explicit rm so it works on ubuntu too. The test suite plumbing's 2661error message for a test with the wrong number of arguments now 2662correctly identifies which test it's complaining about, and it can now 2663run a test on a command name beginning with "-" (none yet but it 2664comes up in toysh).</p> 2665 2666<p><u>Building</u>: You can now override strip with $STRIP. The set of 2667"make install_airlock" commands got tweaked for the 4.17 kernel (which 2668needs bison and flex because the new kconfig plumbing doesn't have _shipped 2669versions of those generated files like the old kconfig did), and for 2670the new release (remove ping, toybox provides it now). 2671$LDFLAGS now applies to library probing (the android NDK provides 2672dynamic libraries without corresponding --static versions). 2673Elliott took a CFG_TOYBOX_DEBUG check out of an error_exit() to make 2674clang stop complaining about an error that should never happen.</p> 2675 2676<a name="24-02-2018" /><a href="#24-02-2018"><hr><h2><b>February 24, 2018</b></h2></a> 2677<blockquote><p> 2678"Arthur," said Ford.<br /> 2679"Hello? Yes?" said Arthur.<br /> 2680"Just believe everything I tell you, and it will all be very, very simple."<br /> 2681"Ah, well, I'm not sure I believe that."<br /> 2682</p> 2683<p> - The Hitchhiker's Guide to the Galaxy</p> 2684</blockquote> 2685 2686<p><a href=downloads/toybox-0.7.6.tar.gz>Toybox 0.7.6</a> 2687(<a href=https://github.com/landley/toybox/releases/tag/0.7.6>git commit</a>) 2688is out, a month and change late.</p> 2689 2690<p>This release adds a <b>crc32</b> command and 2691cleans up and promotes <b>iconv</b>, <b>logger</b>, and <b>setfattr</b>. 2692Pending added <b>stty</b> and <b>fmt</b>, the other half of <b>ping</b>, 2693and got a lot of cleanup on <b>gzip</b>/<b>gunzip</b>/<b>zcat</b>.</p> 2694 2695<p>The <b>cut</b> command got a complete rewrite adding support for multiple 2696ranges, utf8/unicode characters, and new -CDFO options. Use -C to measure 2697columns rather than characters (for double width chars), and "cut -DF 3-5" 2698selects words (like "awk '{print $3 $4 $5}'").</p> 2699 2700<p>This release adds <b>top -m</b> and <b>iotop -H</b> (and fixed top width truncating for narrow terminal windows), 2701added <b>iconv -c</b>, fixed <b>ps -T 1234</b> to show threads belonging to 2702that PID, improved <b>file</b>'s executable identification (properly recognizing 2703both endiannesses and adding sh/frv fdpic, bpf, and new microblaze 2704identifiers), and <b>blkid</b> now only shows LABEL= when it isn't blank.</p> 2705 2706<p>Running ./configure now does "make defconfig", help_exit() now shows the 2707"See %s --help" message on the same line, and several web pages 2708(about, code, roadmap, cleanup) got updates.</p> 2709 2710<p>Elliott and the Android devs added <b>df -i</b>, <b>uptime -p</b>, 2711<b>time -v</b>, and <b>xxd -i -ri</b>, 2712fixed <b>id -G</b> to only output the group IDs, fixed the <b>xargs -0 -n</b> 2713combination, made <b>find -exec +</b> obey ARG_MAX (just like xargs), made 2714<b>killall</b> kill scripts too, fixed an off by one error in <b>pidof</b> 2715that could cause false positive matches in 15 character names, added 2716gzip/gunzip/zcat tests, and taught <b>file</b> to identify more stuff 2717(ogg, TrueType font, LLVM bitcode, PEM certs, and PE executables), and 2718for some reason needed TOYBOX_VENDOR in the version stuff. 2719Android <b>getprop</b>/<b>setprop</b> now allows ro. properties to have 2720arbitrary lengths, <b>losetup</b> can now find loop devices under /dev/block (where android puts 2721them for historical reasons), and Andreas Gampe pointed out a thinko in 2722<b>top</b> that triggered llvm's address sanitizer checks.</p> 2723 2724<p>Gael PORTAY asked for mdev to work when Linux has the 2725block layer configured out, Patrick Oppenlander pointed out a problem 2726with make bloatcheck's build dependencies, and 2727xuphung on github fixed config2help building on MacOS X.</p> 2728 2729<p>Commands with locale support now setlocale(LC_TYPE, "C.UTF-8") for more 2730targeted locale support (enable character parsing with 2731utf8 support, use "C" semantics otherwise). The toys/example/test_*.c 2732commands got renamed demo_*.c.</p> 2733 2734<p>In lib.c the millitime() function returns current unix time in milliseconds 2735(which as a 64 bit number is good for + or - 292 million years from 2736midnight Jan 1 1970), and the old xgetty() function became tty_fd() returning 2737-1 instead of erroring out.</p> 2738 2739<p>The "make help" text moved from the Makefile to scripts/help.txt, 2740ls.c now lists some of its deviations from posix, and work is underway 2741to make ps.c more intelligible (and figure out how to break it up into 2742multiple files).</p> 2743 2744<a name="12-10-2017" /><a href="#12-10-2017"><hr><h2><b>October 12, 2017</b></h2></a> 2745<blockquote><p>Would it save you a lot of time if I just gave up and went 2746mad now?</p> 2747<p> - The Hitchhiker's Guide to the Galaxy</p> 2748</blockquote> 2749 2750<p><a href=downloads/toybox-0.7.5.tar.gz>Toybox 0.7.5</a> 2751(<a href=https://github.com/landley/toybox/releases/tag/0.7.5>git commit</a>) 2752is out.</p> 2753 2754<p>This is another <b>mostly bugfix release</b>. I delayed it a bit trying to get some 2755of the many half-finished projects (dd, ping, lsof, iconv, cut, sysconf, 2756the cp --parents option...) finished/promoted/tested, but decided adding big 2757things at the end of the dev cycle would be too destabilizing.</p> 2758 2759<p>A couple <b>new features</b> slipped in anyway. Ilya Kuzmich added head -c 2760and strings -t (and corrected strings' output in a couple places). Elliott 2761Hughes taught file to recognize Macintosh (Mach-O) binaries.</p> 2762 2763<p><b>NOTE</b>: the chrt command is broken when built against musl-libc 2764because that project's maintainer decided he didn't like the system calls 2765it depends on, so he 2766<a href=https://git.musl-libc.org/cgit/musl/commit/?id=1e21e78bf7a5>removed 2767them from his libc</a>. The prebuilt binaries are built against musl, 2768so the chrt command there will always say -ENOSYS. (It works fine built against 2769glibc, and presumably bionic or uClibc.)</p> 2770 2771<p><b>Bugfixes</b>: Remounting (mount -o remount,rw) should work properly now, 2772and we fixed another place ps was segfaulting when /proc 2773entries vanished out from under us (a hard to hit race condition 2774mostly noticeable in long-running "top" exiting; error value was treated 2775as a pointer). Elliott Hughes fixed a -Wformat warning in expr.c, 2776and made xargs split the command 2777line at ARG_MAX (which is 1/4 the stack size ulimit, which normal users 2778can arbitrarily increase but execve() will complain if they don't). 2779Josh Gao made netcat -l exit after handling a request, and reported a 2780seq bug (seq "1000000 10000001" output 1e+06, fixed now). Zach Riggle fixed a 2781broken URL in the README. Rob fixed env -0 checking the flag wrong so 2782it didn't work in combination with other flags (typo).</p> 2783 2784<p><b>Plumbing</b>: 2785Replaced mbrtowc() with a new utf8towc() that doesn't have a context 2786struct or care about locale. A bugfix in comma_scan() (wasn't removing an entry at the end 2787of the list) is why remount _sometimes_ worked (depending on argument 2788order), xgetaddrinfo() is now separate from xconnect(), 2789atolx_range() learned the w (word=2) suffix, and b is now (block=512) instead 2790of (byte=1).</p> 2791 2792<p><b>Build</b>: Patrick Oppenlander added a workaround for a bug in config2help.c 2793that resulted in segfaults on newer toolchains. (It was actually a use 2794after free error; the build infrastructure isn't nearly as heavily audited as 2795code that gets installed on the target; oops.) A new GITHASH 2796environment variable can force the build version from 2797the command line, and the build checks for .git in the top directory 2798before asking git what our version is (so it isn't confused by ../../.git 2799in an enclosing directory). Added a build #warning about musl intentionally 2800breaking chrt (as with nommu fork() musl now provides a broken 2801stub function so compile-time probes for its existence think it's there, and 2802you can't run the result to test behavior when cross compiling).</p> 2803 2804<a name="19-06-2017" /><a href="#19-06-2017"><hr><h2><b>June 19, 2017</b></h2></a> 2805<blockquote><p>It is a well-known fact that those people who most want to rule people are, ipso facto, those least suited to do it. 2806To summarize the summary: anyone who is capable of getting themselves made President should on no account be allowed to do the job.</p> 2807<p>- The Hitchhiker's Guide to the Galaxy</p> 2808</blockquote> 2809 2810<p><a href=downloads/toybox-0.7.4.tar.gz>Toybox 0.7.4</a> 2811(<a href=https://github.com/landley/toybox/releases/tag/0.7.4>git commit</a>) 2812is out. No new commands this time, but 2813<b>chrt</b> and <b>dmesg</b> got promoted out of pending.</p> 2814 2815<p><u>New features</u>: 2816Rob rewrote paste, which should work much better now, and added grep 2817-M and -S to match and skip wildcards respectively (useful with -r). 2818Elliott's updated dmesg has -T and --color. The file 2819command can recognize gzip now, uptime grew -s, date grew %N, env knows - as a first argument 2820means -i (posix!) and grew -0, ls defaults to -b 2821instead of -q now when there's a tty, and ls has a new -ll option (with 2822--full-time as a compatibility synonym) showing nanoseconds and (for some 2823reason) timezone. (Why do individual files have timezones?) Elliott added 2824"uudecode -o -" support. Illya Kuzmich taught head -v and -q. The cpio 2825code no longer adds the "TRAILER!!!" entry by default (initramfs extractor 2826doesn't care) without which you can concatenate cpio archives with "cat". 2827(Use the new --trailer option if you want the legacy behavior.) 2828In pending, fdisk compiles now and tar understands bzip2.</p> 2829 2830<p><u>Build</u>: 2831The "make install_airlock" target now symlinks bc from the host because 2832the kernel <a href=https://landley.net/notes-2013.html#28-03-2013>inexplicably</a> 2833needs that to build. This was motivated by 2834<a href=https://github.com/landley/mkroot>mkroot</a>, which builds under 2835a toybox airlock directory.</p> 2836 2837<p>Lots of work on the test suite, mostly from the Android guys who are now 2838running it under Android. This fixed several existing tests that didn't 2839pass, made more tests run on a toybox-only system, and so on. The test suite 2840infrastructure now has a second testing function, 2841"testcmd", which supplies the command name being tested (bypassing 2842shell builtins).</p> 2843 2844<p>Various android build and config fixes, getting closer to being able 2845to let android someday use scripts/make.sh instead of generated/* snapshots. 2846Also more work into building under android's NDK; not quite there yet 2847but much closer. 2848Use nproc in scripts/make.sh detect available processors (so you can control the SMP level with taskset). 2849Removed the old uClibc compatibility glue, it's been 5 years since their 2850<a href=http://lists.busybox.net/pipermail/buildroot/2016-December/180102.html>last release</a>.</p> 2851 2852<p>The new config option TOYBOX_PEDANTIC_ARGS checks arguments when there 2853are no arguments, so things like "uptime" no longer silently ignore arguments 2854you pass but instead refuse to run.</p> 2855 2856<p><u>Docs</u>: 2857The FAQ now has more than one entry. Commands no longer output the full 2858help text for argument errors but instead just say "See %s --help" with the 2859command name (in addition to the actual error message). 2860Elliott did a big period-ectomy on all the --help text, and 2861we cleaned up some tab/space inconsistency. The 2862non-html help -a output now has separators with the command name. 2863The top/iotop and pkill/pgrep help text now describe a lot more of what 2864the commands can do. Twitter's code of conduct page went down so we 2865mirrored the text locally.</p> 2866 2867<p><u>Bugfixes</u>: 2868Fixed a race condition in ps/top where a process that exited right as we 2869read its data returned a different error value than we were expecting (which 2870was causing long-running top instances to occasionally exit), 2871mount now gives an error if it can't autodetect the filesystem 2872type, ps no longer queries the terminal size when output isn't to a tty 2873(so "ps -A | cat" doesn't vary), date's chkmktime() was replaced with 2874simple range checks for fields (to avoid false positives from things like 2875timezones and daylight savings time), removed %s from date's help (we 2876didn't implement it, we have @seconds[.nanoseconds] instead), fixed 2877zcat's buffer flush logic (which was always failing on files larger 2878than 32k), and factor now detects requests for numbers >64 bits and fails 2879loudly instead of producing incorrect answers. 2880Elliott fixed touch -a/-m (they were backwards), and allowed ':' in 2881setprop's property names. Grep now exits with 2 for errors (so -q can 2882distinguish "didn't find" from "didn't work"), doesn't stop on symlinks 2883that point nowhere (there was an error_exit() that should just be a warning), 2884and provides error messages for files we could open but not read.</p> 2885 2886<p><u>Library</u>: 2887New library functions: strend() complements strstart(), minof()/maxof() 2888are min/max macros that evalute arguments once and autodetect type (why 2889isn't this in libc?), xmmap() checks MAP_FAILED (which is not NULL).</p> 2890 2891<a name="21-02-2017" /><a href="#21-02-2017"><hr><h2><b>February 21, 2017</b></h2></a> 2892<blockquote><p>Only six people in the Galaxy knew that the job of the 2893Galactic President was not to wield power but to attract attention 2894away from it. Zaphod Beeblebrox was amazingly good at his job.</p> 2895<p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 2896 2897<p>Despite everything, <a href=downloads/toybox-0.7.3.tar.gz>Toybox 0.7.3</a> 2898(<a href=https://github.com/landley/toybox/releases/tag/0.7.3>git commit</a>) 2899is out. The <u>new commands</u> this time are <b>ftpget</b>, <b>ftpput</b>, <b>microcom</b>, and <b>ascii</b>.<p> 2900 2901<p>We also had two command _demotions_ out of defconfig: 2902<b>hostid</b> got moved to toys/example and 2903switched to "default n" because despite <a href=http://pubs.opengroup.org/onlinepubs/9699919799/functions/gethostid.html>still being in posix</a> 2904the concept of a unique 32 bit number identifying a system is something 2905Linux outgrew about the time Pauline Middelink wrote the first IP 2906Masquerading code. And Elliott did a complete rewrite of <b>dmesg</b> introducing 2907two codepaths that I didn't get a chance to unify and didn't want to 2908hold up the release for, so that's back in pending.</p> 2909 2910<p><u>New features</u>: Rob added units to <b>find</b> -atime and friends 2911(with the legacy -amin alias). Elliott added color and -w to dmesg, fallocate 2912-o, and improved file's ELF parsing. Steve Muckle added -d and finit_module 2913support to modprobe. Rob and Elliott tweaked the 2914ps/top display format a bit more (extending the USER field from 8 to 18 chars 2915and putting + at the end of string fields that got truncated). 2916df -a isn't entirely new, but wasn't documented and needed a bugfix.</p> 2917 2918<p><u>Bugfixes</u>: 2919Last release broke oneit because -c didn't get moved to xopen_stdio() (oops). 2920Rob and Elliott simultaneously spotted ps padding each line to 99999 2921chars when there's no tty (serial console or adb); now it pads to 80 in 2922that case but also switches on -w to avoid field truncation. The "tty" 2923field also sometimes had trailing debris (that's fixed now). And "top" was 2924endlessly redrawing with out tty because receipt of the ANSI size probe 2925results would set SIGWINCH, and handling that sent another ansi probe. (Sigh.) 2926And while we're there, replace "ADDR" with "BIT" in ps -l so there are 2927more than 4 chars left for the "CMD" field on 64 bit systems.</p> 2928 2929<p>Izabera pointed out that split -b and -l can't mix, and suggested seq should 2930multiply to avoid accumulating rounding errors from repeated fractional 2931increments. Wang Xiao Jian fixed a bug in sort -k. 2932Elliott let getprop use the @ character in property names, and 2933Dimitry Ivanov removed the name length limit for system properties. 2934Elliott also improved some error reporting and improved top -H's display 2935of thread names. 2936 2937<p>Josh Gao pointed out that recursive operations on . and .. could be ignored 2938in chmod -R (and the resulting generic fix to dirtree_notdotdot() fixed 2939it in several other places).</p> 2940 2941<p>Justin Cormack caught tar producing a warning to stdout that screwed up 2942"tar c" to stdout. 2943Rob fixed an option parsing bug (where switching off a --longopt in menuconfig 2944confused the parser), and another one where an option excluding itself 2945(ala "abc[-ab][!abc]" with "command -a -b") would segfault.</p> 2946 2947<p>There's some sort of gcc stack over-optimization bug where musl-libc's 2948version of vfork() doesn't get marked with attribute(returns_twice) so 2949stack varabiles in the same function after that get semi-randomly overwritten 2950when the optimizer decides to reclaim the space. So add the attribute 2951to the function the XVFORK() wrapper macro calls. (It's a nommu thing.)</p> 2952 2953<p>Fixed a couple variable size mismatch bugs that were only tested on 64 bit 2954(printf %x 64) or only tested on 32 bit (modprobe), removed some 2955unnecessary casts in stat.</p> 2956 2957<p>Continuing attempts to build under Android NDK brought up that posix 2958defines the global 'stdout' as a macro, which older versions of bionic 2959turned into an array member, but a function was using it as an argument 2960name. (This worked in the AOSP build because it only builds against current 2961bionic, where there's a global 'stdout'.)</p> 2962 2963<p>Several commits argued with clang's warning generation, eventually 2964settling on a variant of __attribute__((__shut_up__)).</p> 2965 2966<p>Android should no longer give spurious error messages 2967when you "ps -A | head" about EPIPE on output. (Older versions of bionic 2968set an error handler on SIGPIPE, but it shouldn't do that now. More recent 2969versions of adb set the SIGPIPE handler to SIGIGN instead of SIGDFL, 2970leading to write returning an error message instead of silently killing 2971the program. So we set it back to the default.)</p> 2972 2973<p><u>Docs</u>: 2974Removed website link to the gmane archive (which didn't survive gmane's 2975change of ownership). The FAQ now answers a _second_ question. (Woo!) 2976Some roadmap updates.</p> 2977 2978<p><u>Build tweaks</u>: 2979Upgraded "make install_airlock" target to only warn about missing 2980commands (unless $PEDANTIC is set) when it sets up the hermetic build 2981path. (The plan is still to implement everything but the toolchain 2982binaries in toybox, but in the meantime we're symlinking other stuff from 2983the $HOST that isn't ready yet. See 2984<a href=https://github.com/landley/mkroot>mkroot</a> for an example using 2985this.)</p> 2986 2987<p>Elliott and Rob continue to poke at building toybox with Android's NDK, 2988but it's a work in progress (<a href=http://lists.landley.net/pipermail/toybox-landley.net/2016-December/008767.html>thread</a>). Various changes 2989removing libcutils dependencies and adding an selinux dependency to getprop 2990are fallout from this.</p> 2991 2992<p>Cross-compiling from Macs needs to use "gsed" instead of apple's 2993version, so teach the build to use that name if it exists in the $PATH. 2994If you try to build without running config first, you should get better 2995error reporting now. Added a workaround for Centos' broken "which" command 2996producing output when it _can't_ find a name in the $PATH.</p> 2997 2998<p><u>Library</u>: 2999The new dirtree flag DIRTREE_PROC skips non-numeric entries so things 3000like ps and top can scan /proc more efficiently.</p> 3001 3002<a name="21-10-2016" /><a href="#21-10-2016"><hr><h2><b>October 21, 2016</b></h2></a> 3003<blockquote><p>Probability factor of one to one. We have normality. I repeat, 3004we have normality. Anything you still can't cope with is therefore your 3005own problem.</p><p> - The Hitchhiker's Guide to the Galaxy</p></blockquote> 3006 3007<p><a href=downloads/toybox-0.7.2.tar.gz>Toybox 0.7.2</a> 3008(<a href=https://github.com/landley/toybox/releases/tag/0.7.2>git commit</a>) 3009is out.</p> 3010 3011<p>During this development cycle, Elliott Hughes <a href=http://androidbackstage.blogspot.com/2016/07/episode-53-adb-on-adb.html>got interviewed 3012on the ADB podcast</a> 3013and Rob Landley <a href=http://linuxluddites.com/shows/episode-88/>got interviewed on Linux Luddites</a> (<a href=http://linuxluddites.com/shows/episode-11/>again</a>). 3014Both talk about toybox and many other things. 3015The web page also grew a new <a href=faq.html>FAQ page</a>, currently with 3016just the one.</p> 3017 3018<p>New comands added to defconfig are <b>tunctl</b>, 3019<b>log</b>, <b>start</b>, <b>stop</b>, and <b>sendevent</b>. 3020The commands <b>file</b> and <b>netstat</b> got promoted out of pending. 3021Pending added <b>chrt</b>, <b>setfattr</b>, and <b>getfattr</b>, and saw 3022a lot of cleanups to diffstat and dd but not enough to promote them to 3023defconfig yet. A new toys/net directory was added, moving ifconfig, netcat, 3024netstat, rfkill, and tunctl there so far.</p> 3025 3026<p><b>Upgrades</b>: All commands now parse --version when they understand 3027--help, but "true" and "false" should now ignore their arguments entirely. 3028We taught stat to handle "%12x" and "%.12x" printf-style escapes, which 3029apparently other versions do. The ifconfig output now shows the interface's 3030device driver. Added patch -d and --dry-run, wc can now do -cm together, 3031find has a NOP -noleaf so scripts that use that don't break, add -c to md5sum 3032and sha1sum. Elliott taught ps to treat extra aguments as additional -p 3033pids, implemented xxd -s, did a number of upgrades to file (added -HL, 3034support for ar files, improved ELF support to report android API level 3035and stripped/not stripped and it no longer prints a guessed build ID type). 3036Elliott also added optional build-time support for using openssl's 3037assembly-optimized md5sum/sha1sum implementations (leading to a new 3038<a href=design.html>design</a> policy on shared libraries).</p> 3039 3040<p><b>Bugfixes</b>: Too many fixes to "ps" and "touch" to list, from both 3041Elliott and Rob. Rob taught sed to handle s/[[:space:]/]// type sequences 3042properly, switched grep to a better 3043workaround for <a href=https://sourceware.org/bugzilla/show_bug.cgi?id=17829>glibc bug 17829</a>, made sed -i preserve ownership when run as root, 3044made du max out at 2 terabytes instead of 2 gigabytes on 32-bit systems 3045(it was always designed to, but was missing a typecast), 3046fixed the option parsing infrastructure (config options that remove command 3047line options got the placeholders wrong), fix to printf for printing 3048octal digits and handling the (posix-mandated) difference between %b and 3049non-%b octal output, reading from "-" no longer closes stdin when done, 3050netcat -L works with nommu (although it may need more portability work), 3051and you can now "make test_scankey" if you want to. Several commands 3052(stat, makedeves, chgrp, cp, find) handled user name lookup failure badly 3053(stat was segfaulting if you interrogated a file belonging to a nonexistent 3054user, "chown 12345 file" errored out if you didn't have that user 3055in /etc/passwd... now they should all print/accept the number when 3056appropriate). "LC_ALL=C ls -Cs --color" produces the same output 3057as other versions (two spaces padding, -k hardwired on).<p> 3058 3059<p>Kyungsik Lee fixed a bug 3060in cp (readlink() doesn't actually null terminate the string it reads in), 3061Elliott Hughes made pgrep/pkill return success/failure, fixed trailing 3062whitespace in netstat, fixed a SMACK symbol conflict due to linux/xattr.h 3063changing, fixed ls -sh, and added a lot of 3064stuff to the <a href=roadmap.html>roadmap page</a>. 3065Izabera pointed out cmp -l and -s can't be selected at the same time, 3066that timeout was never actually checking -v, that ls should default to -q 3067when output is to a tty, and that "file -" would sometimes try to open "-" 3068instead of stdin. 3069Usischev Yury pointed out a use after free error, and that id shouldn't 3070call exit() directly. Matthias Urhahn pointed out that stat(2) returns 3071hardwired 512-byte units, so stat.c was wrong. David Hedges pointed out 3072that route could only handle 10 character interface names when the kernel 3073can do 15 (it's still in pending for a reason, but fixed). Evgenii 3074Stepanov found and helped diagnose one of the more subtle ps bugs fixed 3075this time around.</p> 3076 3077<p>Calling "make test_blah" no longer causes make to error out if the last 3078test fails (and thus returns a nonzero error code). 3079Building single commands and the multiplexer used to require a "make clean" 3080between them (because they had different config files both of which were older 3081than generated/config.h so it didn't get rebuilt; now it just always 3082rebuilds it).</p> 3083 3084<p>The defconfig build is now slightly less broken on older centos versions 3085(although <a href=http://lists.landley.net/pipermail/toybox-landley.net/2016-September/008664.html>the consensus</a> is that Centos is just generally broken).<p> 3086 3087<p>Several commands were over-using xprintf(), which flushes its output 3088to check for error (something you only need to do maybe once per line, 3089and even then maybe only in loops because xexit() flushes and checks 3090ferror() for you and adjusts the exit code if we wrote stuff to stdout 3091that couldn't be printed). Lots of little flushes are inefficient, 3092so most things can use normal printf(). (Retransmission 3093of short writes is presumably libc's problem since it's buffering the 3094output and all.)</p> 3095 3096<p><b>Library:</b> 3097New library functions readlink0() and readlinkat0() which properly null 3098terminates the symlink value (which the stock libc function inexplicably 3099doesn't). 3100The new do_lines() function interates reading lines from a filehandle 3101and calling a function on each line. 3102New function pollinate() factoring out netcat's poll() loop so things 3103like telnet can use it. 3104New functions getusername() and getgroupname() return a 3105char * given a uid/gid (and return a string representation of the number 3106if the lookup fails), and xgetpwnamid/xgetgrnamid were renamed to 3107xgetuid/xgetgid and now return an integer instead of a struct (also helping 3108handle lookup failures, you can still return the uid/get for "12345").</p> 3109 3110<p>Switched atolx() to use long long internally. 3111Renamed xopen() to xopen_stdio() and made a new xopen() that never returns 3112stdin, stdout, or stderr (duping /dev/null into the filehandles as necessary). 3113New function xopenro() opens a file read only with one less argument, and 3114understands that "-" means stdin. New flag WARN_ONLY tells these functions 3115to just print a warning on failure, and return -1 instead of exiting. 3116Misc new functions like openro() which defaults to the WARN_ONLY behavior 3117and notstdio() which dup()s a filehandle up beyond stdin/out/err backfilling 3118with /dev/null as necessary. The WARN_ONLY flag let us remove the failok 3119argument from loopfiles().</p> 3120 3121<p>New TOYFLAG_NOHELP disables --help processing (which "true" and "false" 3122should not do).</b> 3123 3124<p>The test suite now has NOSPACE=1 to ignore whitespace (using diff -b to 3125check results), which helps TEST_HOST pass the same tests as toybox. 3126Fixes to chattr and date tests. It also has a new variable $C with the 3127absolute path to the command being tested (bypassing shell builtins), 3128and a function testcmd() which is just like testing() except it prepends 3129the command name ($C) to the test command line as well as the test 3130description.</p> 3131 3132<a name="02-06-2016" /><a href="#02-06-2016"><hr><h2><b>June 2, 2016</b></h2></a> 3133<blockquote><p>When the 'Drink' button is pressed it makes an instant but 3134highly detailed examination of the subject's taste buds, a spectroscopic 3135analysis of the subject's metabolism, and then sends tiny experimental 3136signals down the neural pathways to the taste centres of the subject's 3137brain to see what is likely to be well received. However, no-one knows 3138quite why it does this because it then invariably delivers a cupful of 3139liquid that is almost, but not quite, entirely unlike tea.</p> 3140<p> - The Hitchhiker's Guide to the Galaxy</p></blockquote> 3141 3142<p><a href=downloads/toybox-0.7.1.tar.gz>Toybox 0.7.1</a> 3143(<a href=https://github.com/landley/toybox/releases/tag/0.7.1>git commit</a>) 3144is out. (Yes, I forgot to update the --version string, but I already 3145uploaded the <a href=downloads/binaries/0.7.1>binaries</a>.)</p> 3146 3147<p>The website has https support now, you can "make cat ps ls" 3148to get standalone commands (and "make list list_pending" to see what's 3149available), and a whole lot of bugfixes and new options to existing 3150commands.</p> 3151 3152<h2>New Commands</h2> 3153<p>Rob implemented <b>ulimit</b>. In pending, Elliott Hughes implemented 3154file. and Lipi Lee implemented a simple wget. (Pending also had minor 3155cleanups to more and lsof, but no promotions this time around.)</p> 3156 3157<h2>New Options</h2> 3158<p>Izabera implemented env -u, suggested adding seq -w, made factor 3159use full unsigned 64 bit math even on 32 bit platforms, pointed out base64 3160-w0 should disable wrapping, and sped up wc -c. 3161Elliott Hughes added mount -o relatime, xxd -p -r, and od -w. 3162Sameer Pradhan (or possibly Bilal Qureshi) suggested adding stat -tL -c %m%t%T. 3163Tom Cherry added getprop -Z. Paul Barker added hostname -b and -F. 3164Rob added ls -b, made ls -q work with utf8, 3165made sed -f - read from stdin, and added top -O (like ps -O).</p> 3166 3167<p>Elliott and Rob <b>added Thread support to ps and top</b>, 3168with -o TID, TNAME. We also added -o PCY (android scheduling policy), 3169-o BIT (process is 32 or 64 bit), and -o TNAME now shows the parent 3170command name for threads.</p> 3171 3172<h2>Documentation</h2> 3173 3174<p>Rob added the sed invocations to convert tabs/spaces and back to 3175design.html. Isaac Dunham updated hexedit's help text. 3176Jakob Flierl pointed out a broken URL in the README.</p> 3177 3178<p>Rob also redid the naming scheme of sed's pattern manipulation code to 3179remove the gratuitous references to Roger Zelazny's "Amber" series, since it 3180was confusing people.</p> 3181 3182<h2>Bugfixes</h2> 3183 3184<p>Grep -H and -n should now work properly with -ABC. Andy Chu pointed out an 3185out of bounds access for zero length lines in rev, fixed a buffer overflow 3186in diff -r, and fixed operator precedence in expr (although Rob is rewriting 3187chunks of expr so toysh can use its plumbing for $(( )) ). 3188Patrick Ohly fixed the too-aggressive suid permission dropping logic. 3189Josh Gao fixed a segfault when find -iname got no argument, and 3190made tail -f work right with just one file argument. 3191Tom Marshall cleaned up tar's long filename support and improved 3192the tar tests, and reported another find bug (with -iname -o -iname not 3193tracking copy lifetimes properly) that got fixed.</p> 3194 3195<p>Elliott Hughes fixed wc -c to not trust zero length files to actually be 3196zero length (/proc does that), fixed "mount -o rw,remount /system" on 3197Android, removed trailing spaces on ps -o cmdline, fixed pkill -9 and 3198the corresponding tests, made "insmod -" work, fixed top -b and tail -NUM, 3199pointed out that ps shouldn't trim numeric fields for display size limits, 3200and added some more 3201explicit "sort" calls to make pipelines so build tempfiles are easier to cache. 3202Rob <a href=https://github.com/landley/toybox/commit/32b3587af261>fixed an insane sed thing</a> the perl 5.22 build was doing. 3203Fixed mount -o to properly pass in leftover string data, and 3204<a href=http://lists.landley.net/pipermail/toybox-landley.net/2016-March/004790.html>documented how to use toybox to mount nfs</a> (warning: kernel patch 3205to fix some bitrot in the kernel NFS driver's string parsing attached 3206to that message).</p> 3207 3208<p>George Burgess IV corrected some variable types in traceroute. 3209Base64 now wraps == properly. Fixed two bzcat segfaults reported by 3210John Regehr. Andy Chu found a segfault in "sed -e 'c\'" with no trailing 3211line, and implemented mv -n and cp -n. The cyanogenmod guys pointed out that 3212cp -a shouldn't complain if a non-root user can't chown, and we added 3213the output path to cp -r error messages while we were there (before was just 3214the filename).</p> 3215 3216<p>Samuel Holland fixed blkid's handling of vfat labels, and 3217fixed a segfault when basename was passed an empty string and an empty 3218suffix. Davis Mosenkovs fixed touch -t seconds parsing. 3219Rob fixed a bunzip bug reported by John Regehr (the bad CRC 3220error message was printing a NUL argument).</p> 3221 3222<p>Not all of the commands build standalone, but more of them do now; 3223scripts/single.sh can now build a "mv" that isn't actually "cp". 3224The dependencies are more granualr, so "make top; make ps" no longer 3225produces a broken ps that ignores -A (because ps.o wasn't getting rebuilt 3226even though top had the FLAG macros for -A zeroed).</p> 3227 3228<h2>Build</h2> 3229<p>Rob added a <b>warning when building commands out of 3230pending</b>. (The pending directory is full of code that hasn't been 3231properly vetted. Use at your own risk.)</p> 3232 3233<p><b>New build targets let you build individual commands by name</b>, ala 3234"make ls cat ps", and you can run the test suite for each standalone 3235command with "make test_ls" and such. 3236"make list" shows all such standalone commands in defconfig, and 3237"make list_pending" shows unfinished commands from toys/pending 3238("make list list_pending" shows both together). "make clean" now deletes 3239these filenames at the top level, and the corresponding unstripped files 3240live in the directory generated/unstripped.</p> 3241 3242<p>Nicholas Boichat suggested switching make.sh to use $! for process 3243enumeration during parallel builds (which is both more efficient and more 3244portable), and suggested shell builtin replacements for wc/awk/sed so the 3245build loop has fewer forks now.</p> 3246 3247<p>Lots of work on the test suite, much of it due to Andy Chu. It now 3248consistently prints the name of the command being tested at the start of each 3249test (and the common infrastructure does that, not each individual test), and 3250"make tests" actually runs all the available tests now. 3251Seperated pgrep and pkill tests, split lsattr/chattr, added fstype and base64 3252tests. The "tests/files" directory now collects files for tests to 3253use, with blkid, bzcat, and utf8 subdirectories: the $FILES variable 3254gives a path to it, so "$FILES/blkid/ext2.bz2" and so on. 3255The testsuite now has test files with 3 different types of "not utf8 output" 3256sequences that require escaping, plus some combining character torture 3257tests, direction reversals, and so on.</p> 3258 3259<p>Added dependencies on TOYBOX_FORK to various pending commands that need 3260nommu conversion (which should fix the allyesconfig build).</p> 3261 3262<p>Static builds with selinux should work again.</p> 3263 3264<h2>Library</h2> 3265 3266<p>New bufgetgrgid() and bufgetpwuid() functions cache previous lookup info 3267rather than repeatedly traversing /etc/passwd and /etc/group (which is slow). 3268Added xpipe() to lib to catch pipe creation failure. 3269The HELP_ macros generated by config2help.c now use a capital prefix 3270to avoid collicing with help_exit() and such.</p> 3271 3272<p>The dirtree infrastructure got a cleanup pass in preparation for adding 3273infinite recursion depth support (needed by rm -r), updated the 3274<a href=code.html#ib_dirtree>documentation</a> to describe the new 3275semantics (removing dirtree_start() and adding dirtree_flagread()). 3276Now dirtree_recurse() takes the new dirfd as an argument.</p> 3277 3278<p>Split out _xexit() from xexit() and let sigatexit() set multiple 3279callbacks.</p> 3280 3281<p>For years the man pages have said to #include <sys/types.h> to get 3282major/minor/makedev but now that glibc 3283<a href=https://sourceware.org/ml/libc-alpha/2015-11/msg00253.html>has vowed 3284to break existing programs</a> and replace it with another nonstandard header 3285not in posix or lsb, we added our own functions to lib/ to do the transform 3286ourselves (based on what the kernel actually expects).</p> 3287 3288<h2>Portability</h2> 3289 3290<p>Debian unstable started needing an extra header #include for some reason, and 3291although printf("%.*s", INT_MAX, s) worked fine on Ubuntu 12.04 it 3292didn't on 14.04, so added a workaround for that. Typecast a printf because 3293wchar_t isn't a rigidly defined size. RLIMIT_RTTIME was 3294added to the kernel in 2008 but you can't expect uClibc to have noticed yet, 3295nor did it #define MS_RELATIME (added in 2006), or prlimit (2010)... 3296(Given the improvements in musl and bionic, uClibc support may be dropped 3297in a future release.) Given that the xattr functions were added during 3298linux 2.5, we can #include its header unconditionally.</p> 3299 3300<p>Renamed basename_r() to something else to avoid conflicting with freebsd's 3301libc, and both scripts/install.h and scripts/config2help.c no longer include 3302toys.h (to make cross-compiling from systems we don't run on easier).</p> 3303 3304<p>Debian bug 635570 did something unspeakably nonportable, depending on 3305"sed -e 'a\'" (with no next line of the pattern, so an unterminated 3306continuation) to add a newline to the last line of input if and only if 3307that last line of the input didn't have a newline, and to take no other 3308action. This is well into "depending on a bug" territory, but we implemented 3309it because otherwise Debian's install broke. (Of course this behavior 3310is undocumented, non-obvious, and doesn't really make logical sense.)</p> 3311 3312<p>CONFIG_TOYBOX_NORECURSE now disables the stack measuring logic (which 3313was giving some "security" code fits). Also we typecast pointers to (long) 3314before comparing them to avoid spurious compiler "optimizations" that 3315break the code.</p> 3316 3317<a name="02-02-2016" /><a href="#02-02-2016"><hr><h2><b>February 2, 2016</b></h2></a> 3318<blockquote><p>"I checked it very thoroughly," said the computer, "and that 3319quite definitely is the answer. I think the problem, to be quite honest with 3320you, is that you've never actually known what the question is." 3321</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 3322 3323<p><a href=downloads/toybox-0.7.0.tar.gz>Toybox 0.7.0</a> 3324(<a href=https://github.com/landley/toybox/releases/tag/0.7.0>git commit</a>) 3325is out.</p> 3326 3327<p>The new commands in defconfig are <b>iotop</b>, <b>top</b>, <b>pgrep</b>, 3328and <b>pkill</b> 3329(most replacing corresponding versions from pending). Added grep -ABC, 3330swapon -d (discard), mkswap -L (label) and UUID support, and find -delete. 3331Izabera added free -h and unshare -f. Josh Gao implemented tail -f. 3332Jose Bollo submitted cp --preserve=context,attr. Kylie McClain added 3333mktemp -u.</p> 3334 3335<p>In pending there's the start of a vi command, and Sameer Pradhan contributed 3336a new dhcp6. This cycle saw several rounds of route cleanup and a little dhcp 3337cleanup, but neither are complete yet. Lipi Lee did some cleanup to netstat.c 3338and Elliott Hughes removed warnings from traceroute.</p> 3339 3340<p>Lots of updates to ps: several new -o options, -k (--sort) -O and -M, 3341improved compatibility with Android's historical behavior, and 3342extensive internal code cleanup (including the removal of all 3343the magic constants).</p> 3344 3345<h3><b>Website</b></h3> 3346 3347<p>Dreamhost restored the <a href="#12-21-2015">missing 11 months</a> 3348to the mailing list archive, in the process deleting the month after 3349that. Now they've asked if I have mbox files archiving the new 3350gap (between December 20, 2015 to January 21, 2016, and presumably they 3351could also fill in the gap from December 14, 2014 to January 3, 2015 that's 3352been there since the last time they did this), 3353but due to some gmail filtering I've 3354<a href=http://landley.net/notes-2012.html#15-10-2012>never 3355been able to disable</a>, my copy of those files is spread among 3 different 3356mbox files I'd have to sort/filter/collate. (It's on the todo list.)</p> 3357 3358<p>Added a code of conduct to the README (we're 3359<a href=https://engineering.twitter.com/opensource/code-of-conduct>borrowing twitter's</a>) because somebody 3360made it necessary.</p> 3361 3362<h3><b>Bugfixes</b></h3> 3363<p>Fixed another sed bug where any ] right after [ was skipped (not just the 3364first one in the range, so [[] didn't terminate). Fixed sort -f and added test cases. 3365Assume 80 columns in "ls -m | cat", ls -L is no longer backwards, 3366and ls of files with no paths no longer uses an uninitialized (zero) dirfd. 3367Several bugfixes 3368to find (Gilad Arnold fixed -perm, Daniel K. Levy fixed "find . -exec echo {}", 3369and while we're there I fixed find --prune, made "find . -execdir 3370echo {} + -execdir ls {} +" work, and ripped out the environment size 3371measuring code that checked for a 128k limit removed back in linux 2.6.22). 3372Elliott Hughes fixed the date command's parsing of 4 digit 3373years and documented the %s escape, fixed hwclock -u, and pointed out 3374that runcon needs to exec to do its job (not recursively call another 3375command_main() in the same process). Tom Marshall reported that blkid was 3376handling ext2 wrong. Mike Moreton corrected cpio extraction's uid and gid 3377values, and added a --no-preserve-owner option. Fixed the SUID permission 3378dropping logic (which was a bit over-zealous, preventing some commands from 3379running at all).</p> 3380 3381<p>I'm told that debian-testing broke its libc so the nsenter build breaks, 3382but my attempts to install the debian-testing network cd image under 3383qemu keep breaking. Maybe someday they'll fix it enough I can actually 3384reproduce the problem. (Debootstrap under unbuntu builds an ubuntu-flavored 3385chroot in which toybox builds fine.)</p> 3386 3387<h3><b>Documentation</b></h3> 3388<p>Rewrite of the about.html page, tweaks to design.html, and a re-triage of 3389sbase in roadmap.html. Update to mkstatus.py to collate multiple span 3390tags with the same id, resulting in a larger status.html page (which 3391was previously ignoring some commands in the roadmap).</p> 3392 3393<p>Expanded the defconfig/allyesconfig/allnoconfighelp text in "make help" 3394to explain what they're for.</p> 3395 3396<h3><b>infrastructure</b></h3> 3397<ul> 3398<li><p>Expanded toys.optargs to 64 bits so a command can have more than 32 options.</p></li> 3399<li><p>Added NOEXIT() wrapper to turn xwrap() functions into warning versions 3400using the existing longjump(toys.rebound) infrastructure.</p></li> 3401<li><p>Renamed dirtree->data to dirfd and stopped storing symlink length 3402into it (this fixed a bug where following symlinks to directories 3403didn't give a valid directory filehandle, noticeable with ls -Z).</p></li> 3404<li><p>New TAGGED_ARRAY() infrastructure generates index and bitmask macros 3405for arrays of structures starting with a name string.</p></li> 3406<li><p>New lib/linestack.c for utf8 fontmetrics (draw_str() and utf8len() 3407and so on), and for tracking multiple lines of text 3408(vi, less, shell history) that need wordwrapping and scrolling up/down.</p></li> 3409<li><p>Upgrades to lib/interestingtimes.c: scan_key() now has a timeout 3410in milliseconds and recognizes more sequences including ANSI 3411window size probes. New utf8 test files in tests/files/utf8 including 3412sequence reversing, stacked combining chars, and all three types of 3413unprintable sequences (low ascii <32 ala ^X, invalid utf8 sequences ala 3414<AB><CD>, and invalid unicode code points ala U+1234).</p></li> 3415<li><p>More comma handling code in lib.c: comma_args()</p></li> 3416<li><p>Added error_msg_raw() to shut up fortify's endless static checking false 3417positives.</p></li> 3418<li><p>readfileat() can now realloc() in a loop to read long files 3419("zcat | insmod" needed it).</p></li> 3420</ul> 3421 3422<h3><b>Roadmap</b></h3> 3423<p>We're getting close to having a self-hosting development environment 3424using toybox for the command line. The remaining busybox commands in 3425<a href=http://landley.net/aboriginal/about.html>Aboriginal Linux</a> are:</p> 3426 3427<blockquote><p><b> 3428awk bunzip2 bzcat bzip2 dd diff expr fdisk ftpd ftpget ftpput gunzip gzip 3429less ping route sh sha512sum tar test tr unxz vi wget xzcat zcat 3430</b></p></blockquote> 3431 3432<p>And the remaining non-busybox commands in Aboriginal Linux's build/host 3433directory (from the distcc, genext2fs, e2fsprogs, zlib, and squashfs packagesi) 3434are:</p> 3435 3436<blockquote><p><b> 3437mke2fs fsck.ext2 resize2fs distcc genext2fs unsquashfs distccd mksquashfs tune2fs 3438</b></p></blockquote> 3439 3440<p>Squashfs and distcc are probably out of scope for toybox, but mke2fs, 3441fsck.ext2, resize2fs, genext2fs, and tune2fs should all be added to the 3442above "busybox" replacement list.</p> 3443 3444<p>Remind me to include this countdown in future releases. Once they've all 3445been replaced, the next goal is <a href=http://landley.net/aboriginal/about.html#selfhost>building AOSP under itself</a>.</p> 3446 3447<p>See the full <a href=roadmap.html>roadmap</a> and <a href=status.html>status</a> 3448pages for more details.</p> 3449 3450<a name="12-21-2015" /><a href="#12-21-2015"><hr><h2><b>December 21, 2015</b></h2></a> 3451 3452<p>Yes, 11 months have gone missing from the mailing list web archive.</p> 3453 3454<p>Yesterday evening Dreamhost's mailman server went down (timing out trying 3455to connect). I poked them about it, they 3456<a href=https://twitter.com/landley/status/678781271670149121>blamed 3457DNS</a>, I explained that the hang was _after_ the DNS lookup and 3458entered the dig info into the trouble ticket showing the IPs the DNS 3459queries were returning, they reinstalled the server at that IP from what I 3460assume was their most recent backup, and that's how 11 months of messages 3461vanished out of the archive.</p> 3462 3463<p>I've <a href=https://twitter.com/landley/status/679114451975467008>continued to poke them about it</a> but I honestly believe that's the best they 3464can do. <a href=http://landley.net/dreamhost.txt>Last time</a> 3465this sort of thing <a href=http://landley.net/dreamhost2.txt>happened</a> 3466we went back and forth for months, so I added a link to a backup web 3467archive (in the nav bar on the left) that isn't controlled by dreamhost, 3468and thus doesn't gratuitously lose data on a regular basis. (I note 3469the earlier hole in Dreamhost's archive was never fixed either. That 3470was data never getting archived, this is a year's worth of data that 3471was in the archive until yesterday vanishing after the fact.)</p> 3472 3473<p>If you're wondering why the <a href=http://lists.landley.net>top level</a> 3474list page has been "temporarily disabled" for multiple years now... you'd have 3475to ask Dreamhost. I know I have. More than a dozen times.</p> 3476 3477<a name="03-11-2015" /><a href="#03-11-2015"><hr><h2><b>November 3, 2015</b></h2></a> 3478<blockquote><p>"Alright," said Ford. "How would you react if I said that I'm 3479not from Guildford after all, but from a small planet somewhere in the vicinity 3480of Betelgeuse?" Arthur shrugged in a so-so sort of way. "I don't know," he 3481said, taking a pull of beer. "Why - do you think it's the sort of thing you're 3482likely to say?"</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 3483 3484<p><a href=downloads/toybox-0.6.1.tar.gz>Toybox 0.6.1</a> 3485(<a href=https://github.com/landley/toybox/releases/tag/0.6.1>git commit</a>) 3486is out.</p> 3487 3488<p>We have a new <b>ps</b> command with all the -o fields posix wants (although 3489it doesn't accept BSD non-dash option syntax yet), and <b>bunzip2</b> (not just 3490bzcat but the proper extract-in-place command). 3491Sameer Pradhan added <b>hostid</b> and <b>fsync</b>. 3492Elliott Hughes added <b>flock</b>. 3493 3494<p>The people waiting for <b>human readable number support</b> (du -hH, ls -h, 3495and so on) can thank Elliott Hughes for implementing it. (Our output doesn't 3496exactly match others' because we our "binary" mode will say 1.0G instead of 34971024M, which is a bug in the other one we didn't emulate.)</p> 3498 3499<p>The other big news is <b>nommu support</b>, tested on the new 3500<a href=http://nommu.org/jcore>jcore</a> processor but presumaby working 3501on any nommu system. A few commands don't support nommu yet, but those 3502are disabled by dependencies on TOYBOX_FORK in menuconfig when building 3503for nommu. The roadmap now has a large section analyzing the uClinux 3504project (note that <a href=http://nommu.org>nommu.org</a> is slowly replacing 3505<a href=http://uclinux.org>uclinux.org</a> as the standard repository of 3506all knowledge and wisdom about nommu. The old site <a href=#12-02-2012>contains 3507much that is apocryphal</a>, or at least wildly inaccurate, and the new one 3508is trying to improve on that).</p> 3509 3510<p>Both "make change" and scripts/single.sh (for building standalone commands 3511without the multiplexer logic) now use the top level .config 3512for toybox global settings such as Linux Security Blanket Module selection, 3513(so make defconfig before change now).</p> 3514 3515<p>Documentation updates to the <a href=code.html>code</a> and 3516<a href=roadmap.html>roadmap</a> pages.</p> 3517 3518<h3>pending</h3> 3519 3520<p>In the pending directory Sameer Pradhan added tftp, 3521and Elliott Hughes sent lsof. Isaac Dunham upgraded mdev, 3522reboot, init, login, and modprobe, and fixed a distro-specific build break in 3523scripts/mkflags.h. Elliott Hughes and Lipi Lee made netstat -p handle 3524command lines longer than 21 characters, and Elliott fixed netstat -e and 3525some build warnings. Yeongdeok Suh fixed a warning in dhcpd. 3526I started cleanup on pgrep/pkill.</p> 3527 3528<h3>Command updates, bugfixes, and infrastructure</h3> 3529 3530<p>The multiplexer's "command not found" error exit is now 127, so now you can't 3531distinguish between a command not being found in the multiplexer and 3532the multiplexer itself not being found by the shell, because people wanted 3533that for some reason.</p> 3534 3535<p>Elliott Hughes made date reject invalid dates rather 3536than set the clock to something weird (setting the clock 100 years into the 3537future makes most Linux desktops surprisingly unhappy, and ntpdate won't fix it 3538either), fixed several ls -l display issues (user/group field ordering, 3539make user/group/lsmcontext left aligned), did the aforementioned 3540extensive work on human readable number output, fixed ionice's default 3541class, fixed a mv overwrite bug, made df's columns auto-size, added 3542--ppid and -Z to ps, and teamed up with Daniel K. Levy to fix 3543a segfault in find's handling of -newer -group or -user.</p> 3544 3545<p>Hyejin Kim added stat -c %T support. Colin Cross worked 3546on vmstat fixing 3547a header printing bug and calculating the bi and bo columns in the right 3548units. Isabella Parakiss reported that sed -e "/x/c\" -e "y" added an extra 3549newline and that grep -w '\(x\)\1' didn't work, both now fixed. 3550Alistair Strachan fixed several problems with switch_root. Kylie McClain 3551pointed out env should be able to clear variables via NAME= syntax. 3552Dima Krasner added support for running blkid without a partition (so it shows 3553all partitions). Hyejin Kim sent in a bunch of static analysis bug reports.</p> 3554 3555<p>Isabella Parakiss reported that sed -e "/x/c\" -e "y" added an extra 3556newline and that grep -w '\(x\)\1' didn't work, both now fixed. 3557Alistair Strachan fixed several problems with switch_root. Kylie McClain 3558pointed out env should be able to clear variables via NAME= syntax. 3559Dima Krasner added support for running blkid without a partition (so it shows 3560all partitions). Hyejin Kim sent in a bunch of static analysis bug reports.</p> 3561 3562<p>Two large thinko fixes in oneit: -3 was always enabled (which would 3563eventually block if the child never read the exiting PID numbers from its file 3564descriptor #3 until the pipe filled up), and the signal handlers weren't 3565set up right (for requesting semi-graceful halt/poweroff/reboot). 3566Calling install without a mode is now 0755, and install -g 0 no longer clashes 3567with cp --preserve. Better error message for ls -r on unreadable 3568directories, and ls -Z now uses O_PATH (with the /proc/self/fd/%d 3569workaround for kernel stupidity as necessary).</p> 3570 3571<p>Date now understands @unixtime[.fraction] and uses -D for 3572the set-side format (matching busybox's extension for this). The seq -f 3573string now checks that it's got exactly one %f escape with the correct 3574attributes (and a whole bunch of test cases for it). Fixed a bug 3575in od that screwed up the position indicator on arm and mips. 3576In stat the d/h units moved from %d %D to the default string. 3577And patch can now correctly apply hunks with trailing context to the start of 3578the file.</p> 3579 3580<p>The prompt argument moved out of yesno() (the caller can print the prompt 3581themselves). Replaced toys.exithelp with help_exit(). Added new 3582XVFORK() macro, and xpopen_both() calls /proc/self/exe when passed 3583a NULL argv (see cpio -p for example usage). Replaced toys.recurse 3584with toys.stacktop so the recurse or re-exec decision is now based 3585on bytes of stack space used. Marked a bunch of command-local functions 3586static.</p> 3587 3588<p>New additions to lib/ include strlower(), xconnect(), and the 3589aforementioned help_exit(). 3590The testsuite now has some infrastructure tests based on "example" 3591commands such as toys/examples/test_human_readable.c. 3592The login command finally got a long-overdue cleanup (it's one of the 3593commands that predate the "pending" directory but were part of the reason 3594for it). Hexedit had an 3595uninitialized variable (of course gcc didn't spot it, it was too busy 3596warning about "may be used uninitialized but never actually is" variables).</p> 3597 3598<p>Tweaked makefile so 3599"make CROSS_COMPILE=prefix-" (as well as "CROSS_COMPILE=prefix- make", 3600which still works). Toybox is now installed chmod -w so broken installers 3601(like the bunzip2 package's) that try to overwrite existing binaries won't 3602knock out the whole of toybox. 3603GCC 5.2.0 stopped being able to compile Linux 2.6.12's kconfig, but 3604we added a workaround. You can now build uptime without utmpx.h. 3605Alejandro Joya pointed out that enabling smack required smack on the host 3606as well as target when cross compiling, which is now fixed.</p> 3607 3608<p>Note: toybox can autodetect nommu support when building with a uClibc 3609toolchain such as <a href=http://landley.net/aboriginal/downloads/binaries/old/1.4.3/cross-compiler-sh2eb.tar.gz>the one from Aboriginal Linux</a>, 3610but <a href=http://github.com/richfelker/musl-cross-make>with musl-libc</a> 3611you'll have to enable CONFIG_TOYBOX_MUSL_NOMMU_IS_BROKEN to work around the 3612fact they provide a non-functional fork() implementation that always returns 3613-ENOSYS, to prevent you from compile-time probing for nommu support when 3614cross-compiling. Unfortunately "preventing you from probing" seems to be 3615an explicit policy with musl, they also don't provide an "#ifdef __MUSL__" 3616because their library is perfect and you're only ever allowed to work around 3617other people's bugs, not theirs. So we have to use menuconfig to manually 3618enable musl-specific bug workarounds.</p> 3619 3620<a name="23-07-2015" /><a href="#23-07-2015"><hr><h2><b>July 23, 2015</b></h2></a> 3621<p>I recreated the <a href=downloads/toybox-0.6.0.tar.gz>0.6.0 source tarball</a> 3622(new sha1sum 08fb1c23f520c25a15f262a8a95ea5b676a98d54) 3623because I forgot to add --prefix to the git archive command when I updated 3624my release script from mercurial, so the files weren't in an enclosing 3625directory. (Ooops.)</p> 3626 3627<a name="19-07-2015" /><a href="#19-07-2015"><hr><h2><b>July 19, 2015</b></h2></a> 3628<blockquote><p> 3629The reason why it was published in the form of a micro sub meson electronic 3630component is that if it were printed in normal book form, an interstellar 3631hitchhiker would require several inconveniently large buildings to carry it 3632around in.</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 3633 3634<p><a href=downloads/toybox-0.6.0.tar.gz>Toybox 0.6.0</a> 3635(<a href=https://github.com/landley/toybox/releases/tag/0.6.0>git commit</a>) 3636is out. (Yes, git. See the <a href=#05-04-2015>previous news entry</a>.)</p> 3637 3638<p>Sorry for the unusually long gap between releases. Since last release Ye 3639Olde Project Maintainer traveled to japan twice and had two more "once 3640a century" floods at home. (Probably a coincidence.) Still catching up.</p> 3641 3642<h3><b>CELF/ELC talk and Wikipedia[citation needed] article</b></h3> 3643 3644<p>I gave another State Of The Toybox talk 3645(<a href=https://www.youtube.com/watch?v=04XwAbtPmAg>video</a> 3646<a href=http://landley.net/talks/celf-2015.txt>outline</a>), in which I 3647repeat my <a href=http://landley.net/notes-2013.html#07-11-2013>perennial</a> 3648<a href=https://twitter.com/landley/status/557309224535851009>complaint</a> 3649that Wikipedia[citation needed] 3650<a href=http://en.wikipedia.org/wiki/Toybox>still</a> 3651<a href=https://en.wikipedia.org/wiki/BusyBox#Controversy_over_Toybox>says</a> 3652toybox was relicensed before its hiatus, when relicensing was why 3653the hiatus ended.</p> 3654 3655<p>Since Wikipedia[citation needed] seems unable to do the 3656<a href=#15-11-2011>most</a> 3657<a href=http://landley.net/hg/toybox/log/tip/LICENSE>basic</a> 3658<a href=http://landley.net/notes-2011.html#13-11-2011>research</a> on 3659this point, and has stuck to an incorrect sequence of events for years, 3660I've been gradually escalating my attempts to correct them. Toybox 3661came out of mothballs in November 2011 <b>because</b> it could be 3662relicensed. That's what opened up a new niche busybox wasn't already 3663filling with a 10 year headstart.</p> 3664 3665<a name="asterisk_back" /> 3666<p>The article has plenty of smaller issues<a href=#asterisk>*</a>, but 3667given that I gave an entire talk at Ohio LinuxFest in 2013 3668(<a href=http://landley.net/talks/ohio-2013.txt>outline</a>, 3669<a href=https://archive.org/download/OhioLinuxfest2013/24-Rob_Landley-The_Rise_and_Fall_of_Copyleft.mp3>audio</a>) on why I switched away from GPL for 3670my projects, that one bugs me.</p> 3671 3672<h3><b>New stuff this release</b></h3> 3673 3674<p>There's a new android menu in menuconfig, and rather a lot of Linux 3675Security Module support (Smack for Tizen from Xavier Roche and José Bollo, 3676and SELinux for Android from Elliott Hughes; see 3677the Security Blanket menu under global settings in menuconfig) has 3678trickled in, although there's still more to come.</p> 3679 3680<p><b>New commands:</b> Added reset, nproc, ionice, and iorenice. 3681Elliott Hughes contributed xxd, runcon, 3682restorecon, load_policy, getenforce, setenforce, getprop, and setprop. 3683Promoted shred, nsenter, and hwclock.</p> 3684 3685<p>You can once again build catv now the flag infrastructure's been updated to 3686let it coexist with cat -v. 3687And on a long plane flight I wrote 3688hexedit, an interactive hex editor that implements the start of 3689cursor control infrastructure (for eventual use by less and vi and shell 3690command history and so on).</p> 3691 3692<p><b>New options:</b> Added sed -E as a BSD-compatible synonym for -r. 3693Upgraded oneit with -r (restart), -3 (send exiting PID values to child), 3694and signal handling. Added -v option to timeout, -m to mknod, -u to shred, 3695-t to dmesg, and -123 to head and tail. Added implicit "." to grep -r without 3696any files to work on. Hyejin Kim requested prefix support for truncate -s. 3697Greg Hackman added -inum to find. 3698Jan Cybulski added the smack side of ls -Z support. Various patches also 3699added -Z to mkdir, mknod, and mkfifo. 3700Basic cp --preserve support went in, but not yet the xattr/LSM parts.</p> 3701 3702<p>The toybox command now has a --version option, 3703which uses "git describe" if available.</p> 3704 3705<p><b>Build infrastructure:</b> 3706The "make change" target now saves the output of each failed standalone 3707command build in a .bad file, and "make defconfig" is quieter now.</p> 3708 3709<p>Paul Barker submitted a large patch changing command install paths so 3710"toybox can be installed alongside busybox without confusing 3711update-alternatives". (There's some argument over 3712what the right paths should be, and I'm waiting for 3713people to tell me what else needs fixing because I have no idea. I've 3714been symlinking /bin to /usr/bin since 2002 3715<a href=http://landley.net/writing/hackermonthly-issue022-pg33.pdf>for 3716historical reasons</a>.)</p> 3717 3718<p><b>Docs:</b> The repository link now goes to github, with another link 3719to the commit rss feed.</p> 3720 3721<p>Elliott Hughes updated the Android section of the roadmap 3722(and he would know). Redid bits of scripts/mkstatus.py to make updating 3723status.html easier, and the README is larger.</p> 3724 3725<p>More description of option parsing in code.html, which now describes the 3726FLAG_x macros, switching flag macro sets with FOR_newcommand, how 3727configuration zeroes flag macros and using FORCE_FLAGS to suppress the 3728zeroing of options shared between commands. Also added description of ";" 3729to make --longopts take an optional =value part, and more about TOYBOX_DEBUG 3730to check NEWTOY() option strings (otherwise a bad option string makes 3731lib/args.c obviously segfault, but doesn't explain why).</p> 3732 3733<p>Added a "Why 0BSD?" section to license.html when submitting zero clause bsd 3734to SPDX (according to the pending license spreadsheet, it's been approved for 3735SPDX 2.2).</p> 3736 3737<p>The old list of commands needing cleanup but not in pending was 3738removed from toys/pending/README and instead the issues were added 3739as TODO comments in the individual commands.</p> 3740 3741<p><b>Bugfixes:</b> 3742Fixed mount -a segfaulting without -O (reported by Janus Troelsen), 3743and made it try a "become rw" ioctl() on the block device before falling 3744back to mounting read only (because Android expects that). 3745Fixed printf -- and printf ---. Lots of tweaks to ls -l spacing with 3746different options. Make touch -d and -t actually set time when you don't 3747specify nanoseconds. 3748Fixed a subtle bug where recursive calls (toybox commands that run other 3749toybox commands) weren't resetting all their state. (This manifested as 3750a "no }" error from "find | xargs sed", but could cause other problems.) 3751And David Halls reported another sed bug trying to compile libiconv (which 3752left extra \ at the start of lines in a generated shell script, breaking 3753the build). Output an error message for "cat /mnt".</p> 3754 3755<p>Kylie McClain reported that mktemp broke when $TMPDIR was set to an empty 3756string (which is not the same as unset), that install/find didn't support 3757numeric uid/gids, and that sort -z affects both input and output. 3758Isabella Parakiss fixed a printf.c bug. 3759David Halls fixed bugs in install -D and find -exec. Samuel Holland 3760fixed unshare -r. Hyejin Kim fixed makedevs with a count of 1, fold -w 3761range checking, an error path in scripts/mkflags.c, added -i to dhcpd, 3762and stopped su from prompting the root user for the new user's password. 3763Jan Cybulski spotted wrong indentation when combining ls -s and -i with -C and 3764-x. José Bollo fixed stat %G. Sameer Pradhan fixed a bug in mkfifo -Z.</p> 3765 3766<p>Elliott Hughes asked for a default SIGPIPE handler to disable 3767the signal handler bionic's dynamic loader installs (yes really). Still not 3768100% sure what the correct behavior is there. (Posix is 3769(<a href=http://permalink.gmane.org/gmane.comp.standards.posix.austin.general/10915>actively unhelpful</a>, but at least they're taking 3770<a href=http://austingroupbugs.net/view.php?id=789#c1976>years to 3771make up their mind</a>. Elliott also sent patches to fix a typo in 3772useradd.test, add missing arguments to error_exit() calls and clean up 3773printf() format strings, fix an off by one error in human_readable(), 3774fix dmesg -c error reporting, fix a segfault in comma_scan where the option 3775was the last item in optlist (triggered by mount -o ro,remount), fix 3776hwclock -w, made ifconfig print lowercase MAC addresses (it was bothering 3777him), and make terminal_size() read the right environment variable 3778(LINES, not ROWS). And he suggested the test suite notice high command exit 3779values (corresponding to segfault or other signals).</p> 3780 3781<p>People are apparently using toys/pending commands, despite the police tape 3782and flashing lights, so added louder warnings to toys/pending/README. 3783Elliott Hughes fixed various problems with tar, dd, more, and top. 3784Hyejin Kim cleaned up syslogd and dumpleases. Isaac Dunham added hotplug 3785support to mdev. Yeongdeok Suh added RFC-3315 ipv6 support to dhcpd.</p> 3786 3787<p>I rewrote ps.c from scratch (in pending), but it's not ready for real use 3788yet.</p> 3789 3790<p><b>Portability:</b> 3791On the portability front Bernhard Rosenkranzer fixed a problem where the 3792menuconfig code wouldn't compile in C99 mode. (This led to me documenting 3793the craptacular nature of kconfig in a README, and the plan to replace it 3794sometime before 1.0.) Some extra flags to shut up overzealous llvm warnings 3795were added (and have to be probed for because gcc complains about 3796arguments it doesn't recognize even when they switch stuff _off_ using 3797a standard syntax). Don't depend on malloc(0) to return non-null in ls. 3798David Halls fixed some mac/ios portability issues, 3799implying somebody's built at least part of toybox on a mac.</p> 3800 3801<p>Added basename_r() to lib/lib.c because the posix semantics for basename() 3802are stupid but what the gnu guys did to it was appalling. 3803Turns out bionic already had a basename_r(), but posix still doesn't. 3804Fixed it up in portability.h, but this 3805could break more stuff in future. (Correct fix is to lobby posix to add it, 3806which would probably take about 15 years...)</p> 3807 3808<p><b>Infrastructure:</b> 3809The build now checks $LDFLAGS for linker-only flags, and allows the strip 3810command to fail (binflt toolchains provide a strip that doesn't work). 3811Since time.c uses floating point, added TOYBOX_FLOAT dependency in config.</p> 3812 3813<p>There's a lib/lsm.h defining varous inline functions for linux 3814security modules stuff, if (lsm_enabled()) should turn into a compile-time 3815constant 0 and let code drop out when TOYBOX_LSM_NONE selected, but 3816testing against CFG_TOYBOX_LSM_NONE or derived symbols is still useful 3817becuase when it _is_ enabled the probe turns into a system call you 3818don't want to repeat too much.</p> 3819 3820<p>Switched a bunch of commands from signal() to xsignal(). Factored out 3821xgetgrnamid() and xgetpwnamid() into xwrap.c. Make time.c depend on 3822TOYBOX_FLOAT (since it always uses float so shouldn't be available on 3823build targets without even software float). Added readfileat() to lib/lib.c.</p> 3824 3825<p>The dirtree infrastructure now passes in full flags for the old symlink 3826field, and the new DIRTREE_SHUTUP flag disables warnings if a file vanishes 3827out from under you during traverse. New dirtree_start() wrapper to 3828create dirtree root with only two arguments.</p> 3829 3830<p>The not-curses infrastructure introduced by hexedit mostly moved to 3831lib/interestingtimes.c.</p> 3832 3833<a name="asterisk" /> 3834<a href="#asterisk_back" />Asterisk:</a> such when 3835Tim contacted me (my blog says a couple days before nov 13, 2011, I.E. 383611/11/11 not some specific day 2 months later) to ask if I wanted to work 3837on a new project he was proposing called 3838<a href=http://www.elinux.org/Busybox_replacement_project>BentoBox</a> 3839(because I used to do busybox, he'd forgotten toybox existed 3840until I brought it up). And don't ask me what "focuses not on compatibility 3841with its GNU counterparts" means when CP_MORE adds 7 non-posix options 3842and toys/other has 84 commands in neither posix nor LSB. I think they're 3843struggling to explain the difference having dismissed "licensing" as being 3844the reason it started up again after a long hiatus? The reason I don't think 3845GNU is special is there are a half-dozen other independent 3846implementations of the same unix command tools out there (AT&T, 3847BSD, Coherent, Minix, plan 9, busybox, toybox, and several more analyzed in 3848the <a href=roadmap.html>roadmap</a>, and that's ignoring the implementations 3849written for DOS or in assembly over the years). But I do care what 3850Linux From Scratch expects, and if it's 3851<a href=http://archive.linuxfromscratch.org/lfs-museum/7.6/LFS-BOOK-7.6-NOCHUNKS.html#ch-tools-gcc-pass1>calling mv -v</a> 3852then I impelement mv -v 3853even if <a href=http://landley.net/toybox/roadmap.html>posix hasn't got 3854it</a>. And I don't know why "gnu counterparts" would describe this when 3855util-linux isn't a gnu package, nor are info-zip, e2fsprogs, kmod, less, 3856procps, shadow, sysklogd, vim, zlib, sudo, dhcpcd...</p> 3857 3858<a name="05-04-2015" /><a href="#05-04-2015"><hr><h2><b>April 5, 2015</b></h2></a> 3859<p>Since <a href=https://android.googlesource.com/platform/external/toybox/>android</a> and 3860<a href=https://git.tizen.org/cgit/platform/upstream/toybox.git>tizen</a> 3861and <a href=https://github.com/kraj/meta-musl/tree/master/recipes-core/toybox>openembedded</a> 3862and <a href=https://packages.gentoo.org/package/sys-apps/toybox>gentoo</a> 3863and so on have all been using Georgi Chorbadzhiyski's git mirror rather 3864than the mercurial repository, I bit the bullet and switched the project's repo 3865<a href=https://github.com/landley/toybox>to git</a>. Georgi's 3866<a href=https://github.com/gfto/toybox>mirror</a> is now pulling from that.</p> 3867 3868<a name="25-02-2015" /><a href="#25-02-2015"><hr><h2><b>February 25, 2015</b></h2></a> 3869<blockquote><p>"A common mistake that people make when trying to design 3870something completely foolproof is to underestimate the ingenuity of 3871complete fools."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 3872 3873<p><a href=downloads/toybox-0.5.2.tar.gz>Toybox 0.5.2</a> 3874(<a href=/hg/toybox/shortlog/1702>commit 1702</a>) is out.</p> 3875 3876<p>New promoted commands: sed (finally fixed enough it builds Linux From 3877Scratch), printf (cleaned up and promoted), shred and 3878base64 (the Tizen guys wanted them), getenforce, setenforce, and chcon (android), 3879mix (promoted with fixes from Isaac Dunham), nsenter (from 3880Andy Lutomirski, merged into unshare).</p> 3881 3882<p>Elliott Hughes submited a bunch of patches to support Android (to 3883both toybox and Bionic libc, which he maintains). On toybox's end this 3884involved a lot of fixups to portability.[ch] and fixes to over a dozen 3885commands, plus several new ones. Other portability fixes included working 3886with buildroot's uclibc fork and building for nommu targets.</p> 3887 3888<p>The new "make change" target builds each toybox command as a standalone 3889binary. Rather a lot of commands that didn't build by themselves (mv depending 3890on cp and so on) were hit with a large rock until they built standalone. 3891This involved rewriting bits of option parsing, more elaborate dependency 3892generation, making each command have its own config 3893symbol and main() function (even when it's just a wrapper calling another 3894command's main()), and so on. Also, some commands can't be built standalone 3895at a conceptual level: "help" describes other enabled commands and "sh" 3896has a number of bulitin commands (cd, exit, set) that require the 3897multiplexer infrastructure, so "make change" filters them out.</p> 3898 3899<p>The mailing list's web archive is still screwed up. Dreamhost has 3900been trying to fix it since approximately September. There are 3901<a href=http://www.mail-archive.com/[email protected]/>two</a> 3902<a href=http://news.gmane.org/gmane.linux.toybox>other</a> less broken 3903archives, but neither has quite the same UI as mailman.</p> 3904 3905<h3>Bugfixes and tweaks</h3> 3906 3907<p>Cynt Rynt sent in tests for ifconfig, 3908Robert Thompson taught factor to accept whitespace separated arguments, 3909Hyejin Kim pointed out that some of mktemp's longopts were attached to 3910the wrong short options, 3911Luis Felipe Strano Moraes fixed a wrong free() call in bootchartd in pending. 3912Patches from Ashwini Sharma to make "df /dev/node" work, prevent du from 3913looping endlessly following symlinks, and to make expr.c 3914(in pending) understand == and regex matches. (Speaking of expr, it gets 3915priority groupings wrong but the bug was actually in the posix spec's 3916HTML conversion. They fixed the posix spec upstream for us. Still need 3917to fix the expr code, but it's in pending for a reason...)</p> 3918 3919<p>Some commands grew new option flags, such as cp --remove-destination 3920and touch -h.</p> 3921 3922<p>The parallel build has better error reporting now. When toybox needs to 3923re-exec itself to regain suid root permissions and hasn't got the suid bit, 3924it now gives the right error message ("not root" instead of "no such command"). 3925 3926<p>Added a test to "mount" to not mount the same device/directory combination 3927over itself (the OS catches this for block devices, but not for tmpfs). 3928Make blkid distinguish ext3 from ext4. Added catv back into cat (because 3929the Android guys wanted it, and they have historical usage on their side, 3930so...). Handle nanoseconds in touch.</p> 3931 3932<p>Fixed a segfault when CP_MORE was disabled (the resulting option flag list 3933no longer defined -d but still had it in option groups at the end). 3934Workaround for glibc redefining dirname() and basename() to random non-posix 3935semantics because gnu. (They could have created dirname_r() but didn't want 3936to.)</p> 3937 3938<p>Fix an ifconfig test that was preventing assigning an ipv4 address to 3939interface aliases. Several cleanup passes on hwclock but not quite 3940promoted out of pending yet.<p> 3941 3942<p>Fixed a wrong error message in rm (if you had a chmod 000 directory and 3943did rm -r on it without -f, after the prompt it would complain it was a 3944directory, which was not the problem).</p> 3945 3946<p>The gzip compression code now does "store only" output to stdout, for 3947what that's worth.</p> 3948 3949<p>Cleanup mountpoint and expand, and remove them from toys/pending/README 3950(a list of commands that predate the toys/pending directory but needed 3951another pass).</p> 3952 3953<h3>Library and infrastructure:</h3> 3954 3955<p>Reworked the option parsing infrastructure so more commands build 3956standalone (via scripts/single.sh or "make change"). The option flag bit 3957values are no longer packed, it leaves spaces where currently disabled 3958flags go, and you can #define FORCE_FLAGS so disabled flags aren't zeroed. 3959This allows multiple commands to more easily share infrastructure, even if 3960your current flag context is for a disabled command (switched off in config), 3961you can force them to stay on and as long as the flags read the same right 3962to left they'll have the same values.</p> 3963 3964<p>We've started removing use of strncpy() because it's a hugely broken 3965standard C function: the length is the maximum length to _append_, not 3966the size of the destination buffer. It memsets the remaining space it didn't 3967copy ala "memset(dest+strlen(dest), 0, len);" so 3968if you think len is the size of dest you're guaranteed to stomp memory off the 3969end). And if it runs out of space it won't null terminate because reasons. 3970(Meanwhile sprintf("%*s", len, str) is counting wide characters in your current 3971locale, so if you set a locale other than "C" it will also go past your 3972allocated buffer size. Whoever is maintining the C library standards is really 3973bad at strings.) 3974Instead we have xstrncat() which will error_exit() if src+dest+1 doesn't 3975fit in the buffer. (Because randomly truncating input data isn't necessarily 3976an improvement.) And there's always xmprintf().</p> 3977 3978<p>Similarly, strtol() doesn't return an error indicator on overflow, 3979you have to clear and then check errno. So new xstrtol() that cares 3980about overflow.</p> 3981 3982<p>The bionic and musl guys agree faccessat(AT_SYMLINK_NOFOLLOW) is not 3983supported, so stop using it.</p> 3984 3985<p>Fixed toy_exec() to detect when argc is in optargs, so we don't 3986need a separate xexec_optargs().</p> 3987 3988<a name="18-02-2015" /><a href="#18-02-2015"><hr><h2><b>February 18, 2015</b></h2></a> 3989<p>Dreamhost continues to be unable to make mailing list archives work, so 3990here's <a href=http://www.mail-archive.com/[email protected]/>another 3991list archive</a> with a less awkward interface than gmane.</p> 3992 3993<p>(Neither gives you the convenient historical monthly views of mailman, 3994but I still have hopes dreamhost will someday figure out what they're doing 3995wrong. They've only been trying since October. Last month they did a 3996<a href=http://www.dreamhoststatus.com/2015/01/14/discussion-list-hardware-maintenance/>hardware upgrade to fix a software problem</a>, and the stale 3997data loads much faster now, so that's something.)</p> 3998 3999<p>Update (Feb 19): the archive started updating again, by discarding 4000all the pending data. So there are now _two_ giant holes in Dreamhost's 4001web archive, from Dec 15-Jan 3, and then another hole from Jan 16-Feb 18. 4002The relevant messages are in both of the other archives. Here's hoping 4003the chronic archive constipation problem won't happen a sixth time.</p> 4004 4005<a name="30-12-2014" /><a href="#30-12-2014"><hr><h2><b>December 30, 2014</b></h2></a> 4006<p>Due to Dreamhost's <a href=http://landley.net/dreamhost.txt>ongoing</a> 4007<a href=http://landley.net/dreamhost2.txt>inability</a> to make mailman 4008work reliably, I've added a link to a backup web archive at 4009<a href=http://news.gmane.org/gmane.linux.toybox>gmane</a> to the nav bar 4010on the left.</p> 4011 4012<p>You still subscribe to the list through 4013<a href=http://lists.landley.net/listinfo.cgi/toybox-landley.net>the first link</a>.</p> 4014 4015<p>Update (January 27, 2015): they're <a href=https://twitter.com/landley/status/558428839462703104>still working on it</a>.</p> 4016 4017<a name="19-11-2014" /><a href="#19-11-2014"><hr><h2><b>November 19, 2014</b></h2></a> 4018 4019<blockquote><p>"This time it was right, it would work, and no one would have to get nailed to anything."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4020 4021<p><a href=downloads/toybox-0.5.1.tar.bz2>Toybox 0.5.1</a> 4022(<a href=/hg/toybox/shortlog/1566>commit 1566</a>) is out.</p> 4023 4024<p>It's an interim release, mostly bugfixes. There are several new commands, 4025but they're all in pending.</p> 4026 4027<h3>Development</h3> 4028 4029<p>Finally implemented sed, which is still in pending because although 4030it's feature complete according to posix, and even passes the parts of 4031Busybox's sed test suite that aren't explicitly testing for gnu bugs we 4032don't want to copy, it's not yet good enough to build Linux From Scratch. 4033(The ./configure stages use very long sed scripts. 20 commits worth of 4034implementation and debugging, just under 1000 lines of code, and there's 4035still more to do. We're definitely up to some of the "fiddly" commands now. 4036Did you know "echo hello | sed p - -" segfaults gnu sed in Ubuntu 12.04? 4037Yeah...)</p> 4038 4039<p>Talked with the Tizen developers to follow up on their desire to 4040make toybox a part of the base Tizen system, and got a list of commands 4041to add to the roadmap. The tizen todo list is:</p> 4042 4043<blockquote><p> 4044wget, sha256*, gzip, gunzip, bunzip2, rsync, zdiff*, 4045less, ar, arch, base64, csplit, dir, fmt, join, 4046nproc, shred, shuf, stdbuf, stty, test, tr, unexpand, 4047users, vdir, diff3, sdiff, dosfsck (fsck.vfat), awk, fdisk 4048</p></blockquote> 4049 4050<p>(Most of which was already on the todo list, but it helps prioritize.)</p> 4051 4052<p>Fixed md5sum and sha1sum on big endian systems (reported by James McMechan). 4053Andy Lutomirski fixed unshare's help text and option parsing, 4054and submitted nsenter (a tool to use setns(2)) to pending. 4055Isaac Dunham implemented acpi -ctV options, and spotted the bug that ls -d 4056was inappropraitely following command line symlinks without -H or -L (it 4057should act like ls -l does), and ls -F handles symlinks wrong too. 4058Lukasz Szpakowski sent in two bugfixes to tail.c. Cynt Rynt spotted an 4059unnecessary assignment in lib/password.c.</p> 4060 4061<p>Ashwini Sharma's team was as busy as usual, submitting tr, crontab, and 4062ipcrm, and hwclock to pending, more features to the pending ip.c, and a 4063pile of bugfixes (to chgrp, killall, ifconfig, insmod, 4064losetup, comm, cp, id, xwrap, netcat, modprobe, nohup...) mostly found by 4065static analysis. (These fixes are mostly to seldom-used codepaths like the 4066TOYBOX_FREE config option, but test coverage is always appreciated.) Ashwini 4067also suggested upgrading ln -f to leave the original target alone if link 4068creation fails, and reported that mv -f and -i weren't implemented (now fixed).</p> 4069 4070<p>New config option: TOYBOX_NORECURSE prevents xexec() from making internal 4071function calls (for nommu systems with a finite stack).</p> 4072 4073<p>The "toybox" multiplexer command no longer adds a trailing space to each 4074line of command names, so things like "./toybox | tr ' \n' '|'" to create 4075a grep pattern snippet are easier to do. (Why you'd want to is your business, 4076but the output is tidier now.)</p> 4077 4078<h3>Infrastructure</h3> 4079 4080<p>Isaac Dunham added Android support to portability.h, including compile 4081probes for functions missing from bionic-libc, and annotated the commands that 4082use those functions. We haven't really tested building against bionic, 4083but in theory it's possible now.</p> 4084 4085<p>Running the test suite now color codes the PASS/SKIP/FAIL notifications 4086if output is to a tty. (And in case you missed it last time, VERBOSE=fail 4087to stop at the first failure is really useful.)</p> 4088 4089<p>In loopfiles_rw() use O_CLOEXEC instead of O_RDONLY to request the loop 4090function close filehandles for us. (Otherwise the callback function must 4091close each supplied filehandle itself.)</p> 4092 4093<p>The printf-style escape parsing ("\n" and friends) got factored out into 4094a new unescape() function.</p> 4095 4096<a name="02-10-2014" /><a href="#02-10-2014"><hr><h2><b>October 2, 2014</b></h2></a> 4097<blockquote><p>"There is an art, it says, or rather, a knack to flying. 4098The knack lies in learning how to throw yourself at the ground and miss... 4099Clearly, it is this second part, the missing, which presents the 4100difficulties."</p><p>- The Hitchhiker's Guide to the Galaxy<p></blockquote> 4101 4102<p><a href=downloads/toybox-0.5.0.tar.bz2>Toybox 0.5.0</a> 4103(<a href=/hg/toybox/shortlog/1512>commit 1512</a>) is out.</p> 4104 4105<h3>New commands</h3> 4106 4107<p>The new commands are find, install, factor, and mount. Promoted commands 4108(cleaned up and moved out of "pending") are lspci, inotifyd, and blockdev.</p> 4109 4110<p>cp now implements -HL and -F to force delete of pending files, cpio now 4111ignores -m and implements -p, ls -C now has utf8 support (using wcwidth 4112instead of strlen), and umount got a number of upgrades involving 4113looking things up in /proc/mounts. Other minor cleanups happend to 4114cut, touch, free, and id.</p> 4115 4116<p>In pending: Bradley Controy submitted mix (adjusts OSS sound volume). Ashwini 4117Sharma submitted diff, userdel, blockdev, ipcs, and crond, upgraded 4118fdisk, fsck, and ftpget, and ran a static analyzer on a lot of other code. 4119Partial cleanup was done to useradd, userdel, groupadd, and groupdel.</p> 4120 4121<h3>Build infrastructure</h3> 4122 4123<p><b>Parallel builds</b></p> 4124 4125<p>The build now takes advantage of SMP, autodetecting the number of 4126processors. (Export the environment variable CPUS to pick a specific number.) 4127Other build changes: split out $LDOPTIMIZE because old compilers complain 4128about linker options passed with -c, and the entire "generated" directory now 4129gets deleted by clean (the README that was in there got merged into code.html).</p> 4130 4131<p><b>Standalone builds</b></p> 4132 4133<p>The standalone build infrastructure (scripts/single.sh) got upgraded to 4134build more commands as standalone executables. In make.sh the source file 4135selection uses a regex to find the source files with the NEWTOY/OLDTOY macro 4136for the command. It enables each command's 4137sub-options (so CP has CP_MORE), enables I18N and FLOAT support to build 4138full-featured commands, and includes --help text (at least when 4139the command doesn't use another command's help). The OLDTOY() macro 4140now produces (redundant) function prototypes so you can build an OLDTOY 4141without the NEWTOY</p> 4142 4143<p>It doesn't quite have complete coverage yet, the defconfig entries that 4144aren't building standalone yet are:</p> 4145 4146<blockquote><p>chown, egrep, fgrep, fstype, halt, mv, nc, poweroff, unix2dos, 4147whoami</p></blockquote> 4148 4149<p>The main reason for standalone build failures is NEWTOY() or OLDTOY() 4150entries that don't have their own config symbol. Another problem is entries 4151that depend on another entry in kconfig, usually because common infrastructure 4152is using one command's flags (which the other commands copy): if that command 4153is disabled, the FLAG macros become 0 so dead code elimination can remove the 4154code. It's <a href=http://landley.net/hg/toybox/rev/1503>possible 4155to untangle</a> this, but a bit awkward. (It boils down to conflicting 4156design goals in the two contexts.)</p> 4157 4158<p>Standalone builds are used by the test suite when testing individual 4159commands. 4160 4161<p><b>Snapshot builds</b></p> 4162 4163<p>A new addition to the "generated" directory is generated/build.sh 4164containing a single compiler command line to build toybox in its current 4165configuration. Combined with the generated/*.{h,sh} files from an 4166exisiting build, this may let you build on a new system that hasn't quite 4167got enough OS bits working to run a full configureand make.</p> 4168 4169<h3>Internals</h3> 4170 4171<p>Library code: xcreate/xopen now O_CLOEXEC by default to avoid leaking 4172filehandles to child processes. DIRTREE_COMEAGAIN's second callback is now 4173done with the directory filehandle still open (new dir->again variable added 4174to distinguish first from second callback, and requesting DIRTREE_RECURSE now 4175requires passing in the specific macro value, not just a true/false). 4176Use daemon() out of libc instead of hand-rolled daemonize() in various 4177pending commands. string_to_mode() now passes through type bits so you can 4178use it to more easily modify a file's existing mode. 4179Split xpopen() into xpopen_both(), xopen(), and xrun() depending on whether 4180we want to redirect both, one, or neither of stdin/stdout.</p> 4181 4182<p>Bugfixes: Better error message when TOYBOX_SUID option can't drop priviliges 4183(which happens when you suid something _other_ than root). 4184The old pending version of nbd_client.c wasn't deleted when the 4185command was promoted (and the build would break if both were enabled), 4186toy_exec() sometimes needs to re-exec from $PATH rather than recurse 4187internally (to gain dropped root permissions or limit stack depth), 4188always call setlocale() when I18N is enabled to switch it back _off_ when 4189we run commands that expect sscanf("%n") to return bytes, 4190dirtree() had a memory leak in an error path, patch.c had some bugs in 4191error paths (didn't report problem clearly). Ashwini Sharma spotted an 4192option parsing bug where [-abc] would forget _all_ command line arguments 4193saved in the GLOBALS() block (not just the ones for options being switched 4194off), plus various minor fixes to nbd_client and cpio. 4195Lukasz Szpakowski fixed rm -f on a broken symlink (failed), and killall 4196with no arguments (segfaulted).</p> 4197 4198<p><b>Portability</b></p> 4199 4200<p>A somewhat fiddly fix to rm -rf (which needs to chmod directories to u+rwx 4201to descend into them) which hit a musl bug in faccessat() which the musl 4202maintainer refuses to fix. (He literally wants the man page changed 4203instead, despite other libcs working.) Added an #ifdef __MUSL__ section 4204to portability.h with a workaround, you may need CFLAGS=-D__MUSL__ in your 4205build if your musl build's features.h doesn't #define that. (I may do 4206a different workaround in future, but sometimes you've just got to make 4207it work so you can ship. Also, toybox grep with multiple patterns 4208requires <a href=http://landley.net/hg/aboriginal/rev/1692>a patch 4209to musl's regex engine</a>, which applies to 1.1.4 but not to the current 4210musl source control.)</p> 4211 4212<p>More portability.h fixes for uClibc too. (I don't expect that to ever have 4213another release, so locally patching around posix-2008 violations is silly).</p> 4214 4215<p><b>Change to username filtering</b></p> 4216 4217<p>Posix recommends the username creation logic filter usernames to a small 4218allowed set of characters (which even Red Hat breaks by explicitly allowing 4219"$" at the end), but this prevents UTF-8 usernames. Posix' stated logic 4220is to allow filesystems to create the user's home directory, but Linux 4221filesystems can accept any character but NUL and "/". The only characters 4222we actually _need_ to filter out are ":" (field separator in passwd), 4223newline (line separator in passwd), and "/" (directory separator in 4224filesystem).</p> 4225 4226<h3>Documentation</h3> 4227 4228<p>Web pages updated: cleanup.html documents more cleanup, code.html 4229documents more code, and about.html now capitalizes "toybox" consistently 4230(it's just a word, capitalize at start of sentence).</p> 4231 4232<p>The pending/README file now lists commands that needed review/cleanup 4233before the pending directory was added.</p> 4234 4235<h3>Test Suite</h3> 4236 4237<p>Moved out of scripts/test into top level "tests" directory, and the 4238testing.sh script is now in scripts rather than mixed into the *.test files.</p> 4239 4240<p>Johan Bergström requested VERBOSE=fail to make tests (telling it to 4241stop at the first failure), and spotted a build bug where using gnu 4242sort on the host broke in non-C locales.</p> 4243 4244<p>Divya Kothari submitted tests for chmod, link, tar, bzcat, xzcat, zcat, 4245and hostname. (And more, but that's all that's merged so far.)</p> 4246 4247<a name="07-07-2014" /><a href="#07-07-2014"><hr><h2><b>July 7, 2014</b></h2></a> 4248<blockquote><p>"This planet has - or rather had - a problem, which was this: 4249most of the people living on it were unhappy for pretty much of the time. Many 4250solutions were suggested for this problem, but most of these were largely 4251concerned with the movement of small green pieces of paper, which was odd 4252because on the whole it wasn't the small green pieces of paper that were 4253unhappy."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4254 4255<p><a href=downloads/toybox-0.4.9.tar.bz2>Toybox 0.4.9</a> (<a href=/hg/toybox/shortlog/1385>commit 1385</a>) is out.</p> 4256 4257<p><b>New commands</b> added to pending include: 4258lsattr, chattr, inotifyd, rfkill, sulogin, strings, makedevs, 4259killall5, and tar from Ashwini Sharma, arp from Kyungwan Han, 4260sysctl by Bilal Qureshi, partprobe from Bertold Van den Bergh, 4261host from Rich felker, and I did nbd-client and the first 2/3 of mount.</p> 4262 4263<p>Finished cleanups (commands promoted out of pending): 4264sysctl, rfkill, strings, mkpasswd, makedevs, partprobe, killall5, 4265fallocate, and nbd-client.</p> 4266 4267<p>(Along the way partial cleanups got made to: last, fold, lspci, ps, 4268bootchartd, init, fsck, telnetd, telnet, vconfig, toysh, iconv, useradd, 4269login, host, openvt, deallocvt, getty, tftpd, and modprobe. But there's 4270still more to do on all of those.)</p> 4271 4272<p>This time around the <a href=bin>static binaries</a> are linked against 4273musl instead of uClibc. (That's why there's no sparc version, musl doesn't 4274support that target yet.)</p> 4275 4276<p><b>Documentation:</b></p> 4277 4278<p>The help text parser expects lower case "usage:" lines with 4279a blank line after them, so go through and regularize those. Expand the 4280"coding style" section in the docs and move it to design.html. (Not a show 4281stopper for incoming 4282contributions, just an explanation of some of the things I'll do to them 4283during cleanup.) The help text for the "toybox" command now includes 4284the shell script snippet to install symlinks to the toybox binary.</p> 4285 4286<p>The <a href=cleanup.html>cleanup page</a> now has descriptions for the 4287full ifconfig cleanup series, among others.</p> 4288 4289<p>The new toys/examples directory contains hello.c and skeleton.c. The first is 4290a simple hello world program in toybox style, the second is a much more 4291elaborate example program using showing how to use the command line option 4292parsing and how to provide multiple commands in the same C file.</p> 4293 4294<p><b>Fixes</b>:</p> 4295 4296<p>Fix od bug reported by Samuel Holland ("od -v -b" was appending the default 4297output type even though an output type was specified). Ashwini Sharma reported 4298bugs where readfile() was incorrectly freeing its buffer, and where toy_init() 4299was zeroing the wrong data because the field it was using to measure (rebound) 4300had moved (when I moved it back I added a comment why the field needs to be 4301there), fixed a segfault in the dhcp client, and made a 0 length read at 4302the start of password entry count as EOF. Make the "we are not root" test 4303in the init code show the help text. Posix implies that fflush() can return 4304success even when the stream's error bit is set, so call both fflush() and 4305ferror() from xprintf().</p> 4306 4307<p>Isaac Dunham pointed out that bloatcheck couldn't deal with diff 4308implementations that only implement "unified diff" format, and that some 4309diff implementations can't handle nonseekable input (I.E. reading from 4310a pipe). Bugfix so "help -a" works again. Option parsing on nohup now stops 4311at first nonoption argument. Fix segfault in "which" if PATH wasn't set, 4312which was actually a bug in lib function find_in_path(). Made rm -rf of 4313chmod 000 directories actually remove them.</p> 4314 4315<p>The build now passes the same $CFLAGS to the library probe as the final 4316build, because arch linux is so broken it provides different sets of 4317libraries for static and dynamic linking.</p> 4318 4319<p>It turns out sprintf("%.123s", str) is counting characters, not bytes, 4320so globally enabling locale support opens stack smashing vulnerabilities. 4321So there's a new TOYFLAGS_LOCALE you set in toyflags when you want the 4322setup code to setlocale().</p> 4323 4324<p><b>Upgrades:</b></p> 4325 4326<p>Isaac Dunham extended cpio to archive unreadable empty files, and I taught it 4327to set uid/gid and timestamp when extracting archives. Isaac also 4328added tests for cpio, link, and du, added lspci -i, made the pci database 4329parsing skip # comment lines, merged logname and whoami into id.</p> 4330 4331<p>Daniel Verkamp sped up md5sum about 30% with some loop unrolling, making 4332it actually smaller in the process. I added -b flags to md5sum and sha1sum 4333for "brief" output that's just the hash with no filename. (I'm aware other 4334implementations use that for MSDOS "binary" mode, and don't care.)</p> 4335 4336<p>When building standalone commands (scripts/singleconfig.sh commandname), 4337the build now switches on all the sub-options of the command so we get 4338a standalone version with all the bells and whistles enabled.</p> 4339 4340<p>Add -ds flags to date and document +FORMAT escapes. Add the shell NOP 4341command ":" as an alias for true (for toysh).</p> 4342 4343<p>Add uClibc probe for iconv() and fallocate. (The fact it didn't always 4344build against uClibc is why fallocate wasn't enabled in defconfig before.)</p> 4345 4346<p>The umount command now does an losetup -d on the device by default, so 4347we don't leak loopback devices. Bugfix to losetup so "losetup /dev/loop0 4348filename" actually works again.</p> 4349 4350<p>Divya Kothari sent in test suite entries for ls, ln, rm, mv, printf, dd, 4351and renice. Then a second round for lsattr/chattr, mount, chmod, pgrep/pkill, 4352groupadd, groupdel, and useradd. Several of these uncovered bugs, still 4353working to fix them.</p> 4354 4355<p>There are now free() functions for the predefined llist types and a 4356dlist_terminate() function to break doubly linked lists. The new 4357generic_signal() handler either sets "toys.signal" or writes a byte 4358to toys.signalfd with the signal number if signalfd isn't -1 (which it's 4359initialized to in toy_init).</p> 4360 4361<p>The option parsing logic can now detect when a double fits in a long and 4362use the more precise type for floating point arguments (the FLOAT macro 4363contains the type used). The human_readable() function now just outputs 4364decimal kilo/mega/gigabytes (so when du -u says 5.0G it means 5.0 billion 4365bytes). The build infrastructure now notices duplicate commands (so if you 4366cp toys/pending/command.c toys/other/command.c and forget to delete the 4367first one, the build break is now more informative).</p> 4368 4369<a name="20-04-2014" /><a href="#20-04-2014"><hr><h2><b>April 20, 2014</b></h2></a> 4370<blockquote><p>And to this end they built themselves a stupendous supercomputer 4371which was so amazingly intelligent that even before the data banks 4372had been connected up it had started from "I think therefore I am" and got as 4373far as the existence of rice pudding and income tax before anyone managed to 4374turn it off.</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4375 4376<p><a href=downloads/toybox-0.4.8.tar.bz2>Toybox 0.4.8</a> is based on 4377<a href=http://landley.net/hg/toybox/shortlog/1262>commit 1262</a>. And 4378about time too.</p> 4379 4380<p>The big news is that the build no longer needs python to generate help.h, 4381that's now done in C. The help text generation is also collating help text 4382from multiple options, merging command line option blocks and usage: lines. 4383There's even a new <a href=help.html>help web page</a>.</p> 4384 4385<p><b>New commands:</b> Ifconfig, cpio, and su were cleaned up the rest of the 4386way and promoted out of pending. That saga is mostly explained on the 4387<a href=cleanup.html>cleanup page</a>. Vivek Bhagat's freeramdisk, 4388Isaac Dunham's fsfreeze, and Felix Janda's iconv are also new.</p> 4389 4390<p><b>In pending:</b> 4391Ashwini Sharma's team submitted tcpsvd, udpsvd, telnet, telnetd, last, more, 4392groupdel/delgroup, arping, brctl, ftpget, ftpput, printf, reset, and added 4393ipv6 support to traceroute. Kyungwan Han's team submitted modprobe and getty. 4394Vivek Bhagat submitted openvt and deacllocvt. Samuel Holland submitted fold. 4395I wrote a new inflate (zip/zlib/gzip decompression) implementation in 4396compress.c, and still need to do a corresponding deflate (compression-side) 4397and plug them into gzip and zip and so on. (Right now it does zcat.)</p> 4398 4399<p>Several commands (vmstat, login, du, vconfig, mountpoint, free, chroot, 4400cut, touch, modinfo, expand) predate the "pending" directory, and are thus 4401in other directories but still need cleanup. Of these, vmstat got some 4402work this time (which would be much easier other vmstat implementations 4403documented what their output actually meant).</p> 4404 4405<p><b>Upgrades:</b> Ifconfig grew /prefix netmask support (ala 1.2.3.4/24). Grep now has -zZ to 4406handle null terminated data, cksum grew -H for hex output. Upgraded od so the 4407fields align better when producing multiple output types. Help has -a and -h 4408options (all commands, html output). 4409Bugfix to blkid building for a 32 bit target. The date command can actually 4410set dates now. The O_NOFOLLOW compile time probe didn't work with cross 4411compiling, so it's back to an #ifdef test in portability.h. Nathan McSween 4412sent in a bugfix to od and a portability fix in the common library code. 4413Ashwini Sharma spotted a bug in pidof -o, and added verbose (-v) options 4414to mkdir and ln, and suggested killall should have an -s option and 4415allow -l to take zero arguments. Ashwini Sharma and Felix Janda upgraded 4416tftpd. Fixed dumpleases still using toynet.h after 4417that was removed. Corrected killall return code and error reporting. 4418Isacc Dunham fixed bugs all over the tree, did cleanup on a bunch of 4419pending commands (getty, ftpget, init, openvt, modprobe...), and clarified 4420find's help text. Tom Sparrow ran three different static analyzers on 4421the code, which resulted in a few cleanups. The peek()/poke() functions 4422now use "volatile" to prevent broken compiler "optimizations" to do with 4423aliasing.</p> 4424 4425<p><b>Build stuff:</b> Each FOR_xxxx macro now has a complementary CLEANUP_xxxx macro, so you 4426can put multiple commands with different command line options in the same 4427.c file, so they can share infrastructure outside of lib. (This let the 4428bunzip logic move out of lib into bzcat.c.) See XXX for example. 4429i 4430<p>The headers #included in toys.h are now grouped by standard, and headers 4431not listed in Posix or LSB were moved to portability.h. The old xregcomp.h 4432was folded into lib.h because it's posix (and supporting oddball uClibc 4433configurations isn't as important as it once was).</p> 4434 4435<p>Regression tested against Ubuntu 8.04 to fix up bit-rot in defconfig 4436build on older systems. (We depend on Posix-2008, but not necessarily 4437the absolute latest build environment.)</p> 4438 4439<p><b>In lib</b>: lib/xwrap.c added xgetpwnam(), xchroot(), and lib/lib.c now has names_to_pid(). 4440xsetuid() was replaced with xsetuser() which takes a struct passwd 4441and sets both gid and uid, mkpathat() got factored out into a library command, 4442get_int_value() became atolx_range(), and 4443xmsprintf() is now just xmprintf(). The bunzip2 logic moved from lib into 4444bzcat.c.</p> 4445 4446<p><b>Documentation</b>: new <a href=help.html>help page</a> with the 4447help text for all the defconfig commands, using the new help -ah output. 4448The <a href=code.html>source code walkthrough</a> now says more about 4449#including header files, and how the generated/* directory works. The 4450<a href=design.html>design page</a> has some new paragraphs about trading 4451of different kinds of simplicity, and why comments aren't a substitute for 4452good code. The README no longer trails off into obvious unfinished confusion 4453at the end. Each page on the website should now have its own title.</p> 4454 4455<a name="18-11-2013" /><a href="#18-11-2013"><hr><h2><b>November 18, 2013</b></h2></a> 4456<blockquote><p>"Space," it says, "is big. Really big. You just won't believe how vastly, hugely, mindbogglingly big it is. I mean, you may think it's a long way down the street to the chemist's, but that's just peanuts to space."</p><p>- 4457The Hitchhiker's Guide to the Galaxy</p></blockquote> 4458 4459<p><a href=downloads/toybox-0.4.7.tar.bz2>Toybox 0.4.7</a> is based on 4460<a href=http://landley.net/hg/toybox/shortlog/1122>commit 1122</a>.</p> 4461 4462<p>New commands: Brad Conroy submitted blkid. Elie De Brauwer submitted 4463reboot, halt, and poweroff. Strake's nl got cleaned up and promoted from 4464pending to posix. In addition, the existing chvt and vconfig got some 4465cleanup.</p> 4466 4467<p>That said, I haven't nearly kept up with the flood of new commands going 4468into pending: Ashwini Sharma's team submitted 4469dd, dumpleases, traceroute, top, useradd, groupadd, mkpasswd, tftpd, and 4470an fsck wrapper (with no filesystem drivers yet). Isaac Dunham sent in cpio.</p> 4471 4472<p>Bugfixes: Jeroen van Rijn added a user count to uptime. Elie De Brauwer 4473added -e to watch, removed a memory leak, and fixed a terminal size problem. 4474William Haddon made xargs call its command line once even with blank input 4475(the standard is vague, but builds expect it), and fixed an off by one bug 4476where grep didn't malloc enough space with -E (leading to a segfault). 4477I fixed a glitch in bunzip2 (same one as went into busybox since they're using 4478the code I wrote), in od to fix -t co, -J, and -c options. Add uname -o as a 4479synonym for -s. Build fix to never use $CC without prefixing it with 4480$CROSS_COMPILE (since $HOSTCC could be different). Anca Emanuel spotted 4481a typo in the web page.</p> 4482 4483<p>The compile-time command line option parsing got rewritten (ported from 4484bash to C), which should speed up builds a bit and allow code controlled by 4485--longopts to drop out properly when disabled in the configuration. Terminal 4486querying got refactored. Patch's -x option is now more informative (a 4487debug thing if you're trying to figure out why a patch didn't apply). 4488The "toynet.h" file got folded into toys.h since musl supports it and 4489micromanging uClibc options isn't very interesting anymore. The test suite 4490now uses scripts/single.sh when testing a single command.</p> 4491 4492<a name="17-09-2013" /><a href="#17-09-2013"><hr><h2><b>September 17, 2013</b></h2></a> 4493<blockquote><p>"Think of a number," said the computer, "any number." 4494Arthur told the computer the telephone number of King's Cross railway 4495station passenger inquiries, on the grounds that it must have some function, 4496and this might turn out to be it.</p><p>- The Hitchhiker's Guide to the Galaxy</p> 4497</blockquote> 4498 4499<p><a href=downloads/toybox-0.4.6.tar.bz2>Toybox 0.4.6</a> is based on 4500<a href=http://landley.net/hg/toybox/shortlog/1068>commit 1068</a>.</p> 4501 4502<p>This release adds 4503several new commands: Felix Janda wrote paste and fallocate, Kyungwan Han 4504submitted eject, Strake contributed grep, Ashwini Sharma added pmap (and 4505a testsuite entry for grep), Lukasz Skalski sent pwdx, Isaac Dunham posted 4506acpi, and I did timeout and umount.</p> 4507 4508<p>The ls command now has a --color=auto option (suggested by Rich Felker). 4509The multiplexer now has a --help option so you can say "./toybox --help blah" 4510instead of using the built-in "help" command. (Which is a shell built-in. 4511Try it on your command line, it's like man for shell builtins. But a certain 4512other project has conditioned people to expect --help, so...) I forget who 4513heehooman at gmail is but they pointed out unshare needed PID and UID 4514namespace support.</p> 4515 4516<h3>Pending</h3> 4517 4518<p>A lot of new commands in toys/pending, to the point the next release should 4519probably just focus on cleanup and review of this backlog. We've got klogd, 4520dhcp, dhcpd, watch, route, and ps from 4521Ashwini Sharma (and an fsck wrapper but no fsck.fstype engines yet), 4522syslogd, pgrep, and pkill from Madhur Verma, netstat by Ranjan Kumar, 4523test by Felix Janda, lspci by Isaac Dunham, nl, su, and renice by strake (I.E. 4524M. Farkas-Dyck), and sysvinit by Kyungwan Han.</p> 4525 4526<p>Some cleanup work on existing pending commands that aren't 4527ready to promote yet: I did a few more rounds on ifconfig 4528and Isaac Dunham's did several cleanups to xzcat, Felix Janda cleaned up 4529logger and syslogd...</p> 4530 4531<p>Also some cleanup work on commands that predate the pending directory, 4532but weren't quite polished when they went in, most prominently du, 4533expand, and touch.</p> 4534 4535<h3>Infrastructure</h3> 4536 4537<p>The new scripts/single.sh builds a standalone command without the 4538multiplexer, although not all commands can be built that way yet (NEWTOY yes, 4539OLDTOY no) and the space savings aren't anything to write home about. (If a 4540command needs the option parsing logic at all, it needs all of it.) If 4541you're curious, you can do:</p> 4542 4543<blockquote><pre> 4544make defconfig 4545make 4546mkdir singles 4547for i in $(./toybox) 4548do 4549 echo $i 4550 PREFIX=singles/ scripts/single.sh $i || break 4551done 4552</pre> 4553<p>(And then wait a long time and watch almost half the builds fail.)</p> 4554</blockquote> 4555 4556<p>There is now libbuf analogous to toybuf, another global 4k buffer this 4557time for use by lib/ code instead of command code.</p> 4558 4559<p>The lib directory got split up a bit, lib/pending.c contains functions 4560not yet used by anything outside of toys/pending/*, and lib/xwrap.c contains 4561functions that wrap other functions and handle failures (via error_exit). 4562This leaves lib/lib.c containing actual new functions.</p> 4563 4564<p>General improvements and bug fixes to argument parsing. The [-abc] exclude 4565logic should now clear arguments slots when disabling options. Bare --longopts 4566should work now and be able to report errors using their name, the new ; 4567option allows optional arguments to longopts only suppliable with = (I.E. 4568--color and --color=auto but not --color auto).</p> 4569 4570<p>I'm gradually weaning the code off of itoa()/utoa() because sprintf 4571does this already. In this case "simple" probably means "let libc do it 4572for us".</p> 4573 4574<p>Rewrote for_each_pid_with_name_in() and renamed it to just names_to_pid(). 4575It shouldn't get confused trying to compare absolute and relative paths quite 4576so much anymore.</p> 4577 4578<p>lib/llist.c grew a new dlist_pop() function for removing a doubly 4579linked list entry while maintaining a circular list; tail and patch are 4580using it now.</p> 4581 4582<p>The musl guys suggested a new optimization flag 4583(-fno-asynchronous-unwind-tables) 4584that shaves about 10% off the binary size by removing a C++ism that crept 4585into gcc's idea of C. While I don't normally try to micromanage the compiler, 4586"-fstop-being-stupid" is a thing you have to hit gcc with from time to time.</p> 4587 4588<h3>Bugfixes</h3> 4589 4590<p>Felix Janda and I did a largeish rewrite of tail to 4591finally make it work right (we think). Still need to implement tail -f 4592someday (the tricky bit is making -f follow multiple files at once). 4593Felix also reported a bug in xpidfile.</p> 4594 4595<p>Juhani Haverinen pointed out that 4596python 3 doesn't work with config2help.py, so the detection logic looks 4597for python2 (until I get around to rewriting that in C). Elie De Brauwer 4598then fixed our first attempt at this, and also fixed uname's help string.</p> 4599 4600<p>Ashwini Sharma 4601pointed out the build was making a FLAG_ macro for " " which broke some 4602configurations. (That's a control character, not a command line option.)</p> 4603 4604<p>Jacek Bukarewicz pointed out a bug in chdir permission handling, and 4605a way to make env segfault. Both should be fixed now.</p> 4606 4607<p>The new function xexec_optargs() 4608replaces calls to xexec(toys.optargs) to avoid freeing and reusing optargs 4609during option parsing screwing stuff up (such as netcat's exec mode).</p> 4610 4611<p>The stat command's %a output was padded with leading zeroes, which 4612didn't match anybody else's behavior and thus made the test suite hiccup 4613between TEST_HOST and testing toybox. (If you go "TEST_HOST=1 scripts/test.sh 4614command" it sanity checks the tests against the host implementation.)</p> 4615 4616<p>Last release, "mkdir sub/sub && chmod 007 sub/sub && rm -rf sub" didn't 4617delete sub and didn't exit with an error either. Neither was correct, rm 4618should now be fixed.</p> 4619 4620<p> 4621<a name="26-07-2013" /><a href="#26-07-2013"><hr><h2><b>July 26, 2013</b></h2></a> 4622<p>Georgi Chorbadzhiyski maintains a <a href=https://github.com/gfto/toybox>git 4623mirror</a> of the repository on github, automatically updated from the 4624mercurial every 6 hours. The mirror is read only, but you can generate patches 4625against it and post them to the list.</p> 4626 4627<a name="02-07-2013" /><a href="#02-07-2013"><hr><h2><b>July 2, 2013</b></h2></a> 4628<blockquote><p>"Time is an illusion. Lunchtime doubly so." "Very deep. You 4629should send that in to the Reader's Digest. They've got a page for people 4630like you."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4631 4632<p><a href=downloads/toybox-0.4.5.tar.bz2>Toybox 0.4.5</a> is based on 4633<a href=http://landley.net/hg/toybox/shortlog/941>commit 941</a>. It adds 4634uuencode and uudecode from Erich Plondke, and enables Luis Morales' "who" by 4635default. Felix Janda and I cleaned up last year's "stat" submission and 4636enabled it. Ivo van Poorten added "groups". 4637Andre Renaud added "lsusb". I implemented "split", "pivot_root", and "mv". 4638</p> 4639 4640<p>The "help" command is implemented differently now (lib/help.c) and 4641each command can now understand --help (including both "toybox --help" 4642and "toybox --help command" in the multiplexer).</p> 4643 4644<p>The "pending" directory has several commands (find, xzcat, nbd-client, 4645logger, expr) which work but are not enabled by default pending further cleanup. 4646Ifconfig is enabled, but still in pending because it's only 2/3 cleaned up. 4647(It's an awkward halfway state but I'm not holding up the release for it.)</p> 4648 4649<p>I'm <a href=cleanup.html>documenting the cleanups</a> to teach 4650more people to do it, but the writeups aren't caught up yet. The 4651<a href=roadmap.html>roadmap</a> also got updated a bit with further analysis 4652of other projects, and the README and about pages got updated.</p> 4653 4654<p>Fixed _another_ "ls -C" segfault when terminal size can't be detected, 4655condensed the ls help text to fit on one page, implented --color, and taught 4656-l to print the major, minor numbers when showing block/char devices. 4657Argument parsing now handles "--" properly (to end option checking), 4658and the infrastructure can now handle bare --longopts that have no 4659corresponding short option (both were implemented before but didn't work). 4660Fixed an old bug in "patch", chmod grew -f, who grew -a. Isaac Dunham 4661fixed "-" vs "_" handling in modinfo, added a "firmware" output 4662field, added -b and -k support, and taught it that the ".ko" extension means 4663to look for the file at the specified path instead of under /lib. Felix Janda 4664moved file permission display code to lib so ls and 4665stat could share it. Ashwini Sharma spotted a bug in xabspath when the 4666last path component exists but we haven't got permissions to open it 4667(ala readlink -f /dev/sda as a normal user). 4668</p> 4669 4670<p>In the build infrastructure, scripts/findglobals.sh finds leaked global 4671variables. (Leaked means they aren't part of the global union: Other than glibc 4672debris, toybox should define "this", "toy_list", "toybuf", and "toys", and 4673that's it; the rest add memory footprint to every command for the benefit of 4674just one command; use GLOBALS() to stick 'em in the union.) Static linking 4675against libraries other than the host's libc now applies to feature probes 4676for unshare and such. Neuter stupid internationalization support that makes 4677various host "sort" commands put things in an order other than alphabetical 4678(breaking the multiplexer's binary search on command names). 4679 4680<p>You should now be able to build from a source control snapshot on a build 4681system that hasn't got python: if you disable CONFIG_TOYBOX_HELP. (The 4682release tarballs ship generated/help.h, but it's not in source control. 4683Eventually I should rewrite that python script in C.)</p> 4684</p> 4685 4686<p><b>LICENSE TWEAK</b>: After <a href=http://lists.landley.net/pipermail/toybox-landley.net/2013-March/000794.html>discussion</a> on the mailing list the "2 clause 4687BSD" <a href=license.html>license</a> got slightly simplified so the first 4688paragraph now says:</p> 4689 4690<blockquote><p>Permission to use, copy, modify, and/or distribute this 4691software for any purpose with or without fee is hereby granted.</p></blockquote> 4692 4693<p>It used to continue "provided that the above copyright notice and this 4694permission notice appear in all copies", but A) what's the point? B) does "all 4695copies" mean binaries, or just source code, or what? C) lots of projects 4696that consider BSD and GPL compatible have <a href=https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/crypto/aes_generic.c>files with 4697both license notices</a> on them (sometimes at <a href=http://git.busybox.net/busybox/tree/shell/ash.c>opposite ends of the file</a> to make the conflict 4698less obvious) because "all copies must include this function" would violate 4699the GPL but "all copies must include this magic text blob" somehow don't?</p> 4700 4701<p>I don't want to have to care about this anymore. The tweaked version is more 4702or less public domain with a liability disclaimer, but we're still calling it 4703BSD (sometimes "0 clause BSD") to avoid explaining.</p> 4704 4705<a name="21-03-2013" /><a href="#21-03-2013"><hr><h2><b>March 21, 2013</b></h2></a> 4706<p>Video of my ELC talk 4707"<a href=http://youtu.be/SGmtP5Lg_t0>Why is Toybox?</a>" 4708is up on youtube. Related materials include the 4709<a href=http://landley.net/talks/celf-2013.txt>talk outline</a> and an 4710<a href=/aboriginal/about.html#selfhost>android self-hosting writeup</a>.</p> 4711 4712<p>[Updated June 4] The following links jump to specific topics in the video. (Sorry about 4713the ads, it's The Linux Foundation.)</p> 4714 4715<ul> 4716<li>0m29s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=0m29s>The smartphone is replacing the PC</a></li> 4717 <ul> 4718 <li>4m22s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=4m22s>Software needed to become self-hosting</a></li> 4719 <li>6m20s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=6m20s>Do we care if android or iphone wins?</a></li> 4720 </ul> 4721<li>9m45s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=9m45s>Android not vanilla: oppose or accept?</a></li> 4722 <ul> 4723 <li>11m30s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=11m30s>Open source can't do User Interfaces</a></li> 4724 </ul> 4725<li>15m09s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=15m09s>Android is not copyleft: oppose or accept?</a></li> 4726<li>18m23s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=18m23s>Security issues</a></li> 4727<li>21m15s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=21m15s>Solutions to the software problems</a></li> 4728 <ul> 4729 <li>22m55s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=22m55s>What toybox needs to be/do</a></li> 4730 <li>28m17s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=28m17s>What is toybox?</a></li> 4731 <ul> 4732 <li>28m58s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=28m58s>Why toybox started...</a></li> 4733 <li>37m50s <a href=http://www.youtube.com/watch?v=SGmtP5Lg_t0#t=37m50s>What does toybox actually implement?</a></li> 4734 </ul> 4735 </ul> 4736</ul> 4737</span> 4738 4739 4740<a name="14-03-2013" /><a href="#14-03-2013"><hr><h2><b>March 14, 2013</b></h2></a> 4741<blockquote><p>"Ford, you're turning into a penguin. Stop it." - 4742</p><p>The Hitchhiker's Guide to the Galaxy</p></blockquote> 4743 4744<p><a href=downloads/toybox-0.4.4.tar.bz2>Toybox 0.4.4</a> is based on 4745<a href=http://landley.net/hg/toybox/shortlog/813>commit 813</a>, adding 4746the "time" and "readahead" commands, plus some bugfixes.</p> 4747 4748<p>The "cp" command now implements the -s symlink option, plus bugfixes 4749getting various corner cases right as used in actual package builds. 4750"id -Gn root" should now print root's groups 4751instead of the current user's. Several build fixes so toybox builds under 4752Ubuntu 8.04 again (which is about as old a build environment as you 4753can expect to find posix-2008 features in).</p> 4754 4755<p>Unfinished commands have generally been moved to "toys/pending". 4756Everything else should "default y" to participate in make defconfig. 4757Several of those pending commands got some basic cleanup so allyesconfig 4758should at least compile (although defconfig is still what's useful).</p> 4759 4760<p>Significant roadmap updates, checking several other multicall binaries 4761(klibc, sash, sbase, s6...) to see what commands they include.</p> 4762 4763<a name="18-01-2013" /><a href="#18-01-2013"><hr><h2><b>January 18, 2013</b></h2></a> 4764<blockquote><p>This must be Thursday. I never could get the hang of Thursdays.</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4765 4766<p><a href=downloads/toybox-0.4.3.tar.bz2>Toybox 0.4.3</a> is based on 4767<a href=http://landley.net/hg/toybox/shortlog/793>commit 793</a>. There 4768are now exactly 100 commands in defconfig (of a little over 220 on the 4769<a href=roadmap.html>todo list</a>).</p> 4770 4771<p>Elie De Brauwer added the rev command, cleaned up tac, implemented the -s 4772and -f flags for seq, added -v and -i to killall (and fixed killall not to 4773kill itself before finishing its pid list), and added to the test suite. 4774Felix Janda added -m to mkdir, pwd -L and -P, and more test suite entries.</p> 4775 4776<p>Rob Landley added the losetup command, and fixed the existing ls, cp, and 4777readlink commands. The segfault in ls 4778happened when it couldn't determine the screen size (last release changed the 4779default to -C and a screen size of 0 made column view unhappy), and cp got an 4780extensive rewrite bringing it up to date with the dirtree changes and fixing 4781a number of things it never did right in the first place. The xabspath() 4782code in the library now handles a symlink after ".." properly (and the test 4783suite checks for it).</p> 4784 4785<p>Infrastructure-wise the code is better about automatically setting the 4786error return code properly. Now error_msg() sets the exit code to 1 if it's 4787still defaulting to 0, and the global exit path does a fflush(NULL) with error 4788bit check rather than trying to be quite so granular about flushing. (That 4789means if we use printf() instead of xprintf() it still exits with the right 4790error code, it just doesn't end the program early on an output error.) 4791Minor bugfix so TOYBOX_DEBUG 4792doesn't always warn about the lack of suid bit when toybox is built with 4793at least one STAYROOT command. Bugfix for the option [grouping] logic 4794(and then further fixes to the error reporting pointed out by Ashwini Sharma). 4795dirtree_handle_callback() now has a prefix like the rest of the dirtree 4796functions. A lot of stuff doing manual path handling was switched to using 4797libc basename() (including, embarassingly, the basename command), which means 4798it now correctly detects "/trailing/slash/" which the previous code didn't.</p> 4799 4800<p>Also, last release included some accidentally checked in debug code that 4801disabled compiler optimization, so the binary size bloated a bit. It's back 4802to -Os by default now.</p> 4803 4804<a name="15-12-2012" /><a href="#15-12-2012"><hr><h2><b>December 15, 2012</b></h2></a> 4805<blockquote><p>"The major difference between a thing that might go wrong and a 4806thing that cannot possibly go wrong is that when a thing that cannot possibly 4807go wrong goes wrong it usually turns out to be impossible to get at or repair." 4808</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4809 4810<p><a href=downloads/toybox-0.4.2.tar.bz2>Toybox 0.4.2</a> is based on 4811<a href=http://landley.net/hg/toybox/shortlog/749>commit 749</a> and is 4812just a resync. Linux 3.7 came out, meaning it's time to do an Aboriginal 4813Linux release, and that should use a stable version of toybox. So here's 4814a new stable version.</p> 4815 4816<p>The new commands are cut (from Jason Kyungwan Han), touch 4817(from Choubey Ji), expand (from Jonathan Clairembault, and he fixed a 4818bug in login), and rm (from Rob Landley). Felix Janda added UTF-8 4819support infrastructure (for non-ascii character sets) with a config option. 4820Elie De Brauwer added tests for cat and sha1sum, and -so options to pidof. 4821The "ls" command defaults to -C (column view) now, and "readlink" now supports 4822-fenq.</p> 4823 4824<p>Portability work: toybox should now build against the musl C library, 4825and against older glibc versions (circa 2008, much before that and kernel 4826features we depend on start to drop out).</p> 4827 4828<p>The whole codebase got reindented from "one tab" to "two spaces" per 4829level. The option parsing logic now understands [groups] of commands (when more 4830than one in a group is selected it can switch the others off, or error out, 4831or other things). The error_exit() infrastructure can now longjmp back to an 4832earlier point instead of exiting. Each toys/* directory now has a README, 4833the first line of which is the fancy name menuconfig uses for the directory 4834(so no more hardwired directory list in scripts/genconfig.sh).</p> 4835 4836<p>Fixed a filehandle leak in getmountlist(). 4837Pass parent pointer to dirtree_add_node() so it can give error messages with 4838full path. The yesno() function now always reads from stdin and writes to 4839stderr (we can retry tty checking complexity once we've got commands needing 4840it).</p> 4841 4842<p>The open group broke their website so the 4843<a href=http://opengroup.org/onlinepubs/9699919799>old links</a> to POSIX 2008 4844now <a href=http://pubs.opengroup.org/onlinepubs/9699919799>need to start with 4845pubs</a>. Some of the links in the tree have been updated, others haven't while 4846I wait to see if their webmaster notices and fixes it.</p> 4847 4848<p>(I note that the current rm implementation is not technically posix compliant 4849because the standard requires infinite recursion depth and the current 4850implementation uses one filehandle per level. I can add a config option 4851to do it Posix's way, which is more brittle and needs extra security checks, 4852but am waiting for somebody to complain first. The default "ulimit -n" is 1024 4853filehandles, so drilling down over 1000 nested subdirectories).</p> 4854 4855<a name="13-11-2012" /><a href="#13-11-2012"><hr><h2><b>November 13, 2012</b></h2></a> 4856<blockquote><p>"Rule Six: The winning team shall be the first team that wins." 4857</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4858 4859<p><a href=downloads/toybox-0.4.1.tar.bz2>Toybox 0.4.1</a> is based on 4860<a href=http://landley.net/hg/toybox/shortlog/691>commit 691</a>.</p> 4861 4862<p>Elie De Brauwer contributed usleep, Ashwini Kumar contributed du, and 4863Kyungwan Han contributed vconfig. Other new commands include switch_root and 4864md5sum, and the remaining shell wrappers are now proper commands (dos2unix, 4865unix2dos).</p> 4866 4867<p>The patch command now supports -l, and gethostname is now enabled by 4868default. The df command follows symlinks to get the actual device name. 4869Felix Janda added -m support to wc (for utf8).</p> 4870 4871<p>On the infrastructure side, the commands have now been grouped into 4872"posix", "lsb", and "other" subdirectories (for things required by Posix-2008, 4873the Linux Standard Base 4.1, and commands in neither). This affects menuconfig 4874and the actual source layout (toys/cp.c is now toys/posix/cp.c, and so on). 4875An android directory is planned (see the updated 4876<a href=roadmap.html#android>android roadmap analysis</a>).</p> 4877 4878<p>The FLAG_ macros for command option parsing and TT alias for the command's 4879global block are now automatically generated, commands should 4880#define FOR_commandname before #including <toys.h> to get the macros for that 4881command.</p> 4882 4883<p>An upgrade to the build infrastructure now allows commands with _ and - 4884in them, such as switch_root.</p> 4885 4886<p>Bugfixes: Avery Pennarun spotted a case where ls showed uid twice instead of 4887uid and gid, and that nice was using the wrong range of numbers. 4888The ls command also recursed inappropriately last time (not quite 4889properly converted for the dirtree changes last release), and now it's 4890fixed. Roy Tam pointed out a glitch in sh, and fixed df's percentage 4891calculation to match the POSIX spec. The kernel build didn't like our mktemp 4892and it does now. The wc command wasn't quite posix compliant (trailing spaces 4893break stuff). The ls command recursed inappropriately last time (not quite 4894properly converted for the dirtree changes last release), and now it's 4895fixed. The catv command wasn't displaying byte 255 correctly. Some lib 4896fixes (thinko in xpidfile). Fixed uname -m when running a 32 bit x86 binary 4897on an x86-64 host (it lies and says the system is i686, i586, or i486 depending 4898on what the toolchain that built the binary supported. This makes builds in 4899a 32 bit chroot on a 64 bit kernel break less.) The df command was checking 4900partitions in the wrong order (displaying undermounts instead of overmounts: 4901this used to work but some library code changed out from under it and it 4902wasn't updated to match until now). Felix Janda filled out the test suite 4903some more. The patch file creation logic got tweaked several times to 4904successfully apply more patches. Support for older (pre 2.10) glibc 4905versions was added to portability.h.</p> 4906 4907<p>Miscelaneous cleanups all around (mknod, sha1sum, logname), including a 4908rewrite of taskset to be less dependent on libc getting the headers right. All 4909the command headers should now point to the current relevant standards 4910document, where applicable.</p> 4911 4912<p>This news page had old news entries from before the relaunch moved into 4913a separate <a href=oldnews.html>oldnews</a> page.</p> 4914 4915<p>I forgot to create <a href=bin>static binaries</a> last time, but they're 4916back now.</p> 4917</span> 4918 4919<a name="23-07-2012" /><a href="#23-07-2012"><hr><h2><b>July 23, 2012</b></h2></a> 4920<blockquote><p>"Ford", Arthur said. "There's an infinite number of monkeys 4921out here who want to talk to us about this script for Hamlet they've worked 4922out."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4923 4924<p><a href=downloads/toybox-0.4.0.tar.bz2>Toybox 0.4.0</a> is based on 4925<a href=http://landley.net/hg/toybox/shortlog/640>commit 640</a>.</p> 4926 4927<p>The new <a href=status.html>status page</a> is calculated from 4928the roadmap info, and should be easier to keep up to date in future.</p> 4929 4930<p>Andre Renaud contributed od and modinfo. Elie De Brauwer contributed 4931taskset, bugfixes to cmp and tail, and tests for sort and tail. Kyungwan Han 4932contributed passwd. Gaurang Shastri contributed w. Ashwini Sharma spotted a 4933case where dirtree was adding extra slashes to a path.</p> 4934 4935<p>I rewrote od, cleaned up comm, documented the 4936<a href=code.html#lib_llist>llist</a> and 4937<a href=code.html#lib_dirtree>dirtree</a> infrastructure, added an -r option 4938to date (and fixed a bug where -u wouldn't override /etc/localtime), 4939fixed bugs in chmod +stw, fixed ls to show suid bits properly when the 4940corresponding executable bit wasn't set, and worked around a longstanding 4941glibc bug where static linking prevents stdout from automatically flushing 4942pending output on exit.</p> 4943 4944<a name="25-06-2012" /><a href="#25-06-2012"><hr><h2><b>June 25, 2012</b></h2></a> 4945<blockquote><p>"For a moment, nothing happened. Then, after a second or so, nothing continued to happen."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4946 4947<p><a href=downloads/toybox-0.3.1.tar.bz2>Toybox 0.3.1</a> is based on commit 4948<a href=http://landley.net/hg/toybox/shortlog/607>commit 607</a>. It's 4949mostly a bugfix release for ls -l (which was unhappy on targets other than 4950x86-64), plus a new "date" from Andre Renaud and rewritten chgrp/chown which 4951now support the full set of posix flags, plus a little work on the test 4952suite and some more header tweaks towards eventual compatability with the 4953musl libc.</p> 4954 4955<p>The todo list runneth over, but "release early, release often", so here 4956it is. The roadmap and documentation are a bit behind, and I've got ~40 4957pending submissions to review. I need to catch up...</p> 4958</span> 4959 4960<a name="12-06-2012" /><a href="#12-06-2012"><hr><h2><b>June 12, 2012</b></h2></a> 4961<blockquote><p>"For instance, on the planet Earth, man had always assumed that 4962he was more intelligent than dolphins because he had achieved so much - the 4963wheel, New York, wars and so on - whilst all the dolphins had ever done was 4964muck about in the water having a good time. But conversely, the dolphins had 4965always believed that they were far more intelligent than man - for precisely 4966the same reasons."</p><p>- The Hitchhiker's Guide to the Galaxy</p></blockquote> 4967 4968<p>It's well past time for <a href=downloads/toybox-0.3.0.tar.bz2>toybox 0.3.0</a>, 4969so here it is, based 4970on <a href=http://landley.net/hg/toybox/shortlog/595>commit 595</a>, and the 4971statically linked <a href=downloads/binaries>prebuilt binaries</a> should 4972actually be statically linked this time (thanks Ashwini Sharma for spotting 4973that).</p> 4974 4975<p>It's hard to figure out where to cut a release, because development 4976doesn't stop. "Long before now" is the obviuos answer, of course. 4977The project's maintainer also moved house during this development cycle, which 4978threw things off for a bit (so many boxes). Releases should hopefully be a bit 4979more frequent from here on.</p> 4980 4981<p>The big things Rob worked on this time were the new dirtree (directory 4982tree traversal) infrastructure, and a complete rewrite of ls using that 4983which should now implement all 26 posix options.</p> 4984 4985<p>Georgi Chorbadzhiyski added printenv, whoami, mkdir, mkfifo, chmod, chown, 4986chgrp, and uniq. He also added fraction and extension support to sleep (so if 4987you need a quarter-second sleep, it can do that now), and fixed a build bug 4988on slackware.</p> 4989 4990<p>Daniel Walter contributed a string to mode_t parser (in use by chmod and 4991mkdir -m). Ilya Kuzmich contributed comm. Elie De Brauwer added mountpoint, 4992vmstat, logname, login, and mktemp. Kevin Chase did some portability cleanups. 4993Pere Orga fixed some documentation.</p> 4994 4995<p>The "tac" and "clear" commands are now normal commands instead of shell 4996wrappers, and the header #includes have been cleaned up a bit to remove 4997deprecated functions and attempt to increase compatability with the bionic and 4998musl C libraries, "tail" should now use lseek() for large files, and "id" got 4999some cleanups and bugfixes.</p> 5000 5001<p>The new TOYBOX_FLOAT configuration option selects whether or not 5002to include floating point support (for embedded targets where that's 5003problematic).</p> 5004 5005<p>Several random bugfixes: unshare() might actually build portably now, 5006yes 'n' | cp -i should no longer bypass stdin and prompt via the tty, the 5007SUID support no longer drops permissions going through the toybox 5008multiplexer command, and a bugfix to xargs -0 means it should no longer 5009segfault. (I have a pending bug report about xargs not doing the full 5010posix whitespace handling that -0 obsoleted, but I'll deal with that next 5011release.)</p> 5012 5013<p>The build infrastructure is now automatically generating FLAG_ macros 5014for the options, but currently with the wrong names. Some more macro glue 5015is necessary, which I haven't quite figured out how to do yet.</p> 5016 5017<p>A defconfig toybox at the start of the $PATH has successfully built 5018Linux From Scratch (in my Aboriginal Linux project). The commands that 5019'default n' in the config are often still broken, cleanup is ongoing. 5020(The new dirtree stuff broke several of them that haven't been converted 5021yet, but if I wait until everything works we won't have a release before 50221.0, so here's a checkpoint.)</p> 5023 5024 5025<a name="03-03-2012" /><a href="#03-03-2012"><hr><h2><b>March 3, 2012</b></h2></a> 5026 5027<blockquote><p>"They went unnoticed at Goonhilly, passed over Cape Canaveral 5028without a blip, and Woomera and Jodrell Bank looked straight through them. 5029Which was a pity, because it was exactly the sort of thing they'd been looking 5030for all these years."</p><p>- The Hitchhiker's Guide to the Galaxy</p> 5031</p></blockquote> 5032 5033<p>Here's <a href=downloads/toybox-0.2.1.tar.bz2>toybox 0.2.1</a> based 5034on <a href=http://landley.net/hg/toybox/shortlog/512>commit 512</a>. This 5035time around, there are statically linked <a href=downloads/binaries>prebuilt 5036binaries</a> for various embedded targets.</p> 5037 5038<p>It's been a busy few weeks, almost entirely due to new contributors. (I 5039have not quite been keeping up.)</p> 5040 5041<p>Elie De Brauwer contributed free, uptime, swapon, swapoff, lsmod, mknod, 5042insmod, rmmod, and fixed a bug in basename. Andre Renaud contributed ls, ln, 5043realpath, and hostname. Andres Heck contributed pidof and killall. Daniel 5044Walter wrote kill and extended id. Timothy Elliott contributed tail and tests 5045for cmp. Frank Bergmann sent a warning fix. Bryce Fricke added -i to cp. 5046Nathan McSween pointed out an optimization. Georgi Chorbadzhiyski fixed 5047cross compiling to work more reliably.</p> 5048 5049<p>(My own contribution this time around was just tightening up other people's 5050code, a build fix to unshare, some random bugfixes, and so on. My only new 5051code this time around was writing a bash replacement for the existing python 5052bloat-o-meter.)</p> 5053 5054<p>Last time (the 0.2.0 release) included the first pass at an id command from 5055Tim Bird, env and basename from Tryn Mirell, cmp and head from Timothy Elliott, 5056more bugfixes from Nathan McSween and Elie De Brauwer, and Luis Felipe Strano 5057Moraes did a first pass at the who command plus other bugfixes and 5058optimizations.</p> 5059 5060<p>(For that release I did xargs, cal, truncate, unlink, nohup, tty, wc, link, 5061dirname, unshare, and various infrastructure tweaks, but it took me 3 months 5062and those guys did their stuff in a week or so.)</p> 5063 5064 5065<a name="12-02-2012" /><a href="#12-02-2012"><hr><h2><b>February 12, 2012</b></h2></a> 5066<blockquote><p> 5067"for though it has many omissions and contains much that is apocryphal, or at 5068least wildly inaccurate, it scores over the older, more pedestrian work in two 5069important respects..."</p> 5070<p> - The Hitchhiker's Guide to the Galaxy</p></blockquote> 5071 5072<p>Here's the first BSD licensed release, 5073<a href=downloads/toybox-0.2.0.tar.bz2>toybox-0.2.0</a>, more a synchronization 5074point than anything particularly useful. 47 commands in a reasonably 5075ready-to-use state (what "make defconfig" builds), another ten or so partially 5076finished stubs ("make allyesconfig"), and several 5077patches pending on the mailing list I need to review and merge.</p> 5078 5079<p>More to come...</p> 5080 5081<hr> 5082<a name="15-11-2011" /><a href="#15-11-2011"><hr><h2><b>November 15, 2011</b></h2></a> 5083- Back from the dead, Toybox is now under a 2 5084clause BSD license, and aiming to become the default command line 5085implementation of Android systems everywhere.</p> 5086 5087<p>More to come...</p> 5088 5089<hr> 5090 5091<p><a href=oldnews.html>Old news</a> from before the relaunch.</p> 5092 5093<!--#include file="footer.html" --> 5094
