From db51d85f50e44693a5381793db363c62c0a73bc7 Mon Sep 17 00:00:00 2001 From: markmental Date: Wed, 4 Nov 2026 16:21:43 -0500 Subject: [PATCH] Fix the build on minimal tcc/BusyBox systems The schema generator escaped quotes through awk's gsub(), but BusyBox awk drops the backslash in \", so it emitted an unescaped double quote inside a string literal and tcc failed on the generated schema_sql.c. The generator is now printf + POSIX sed only, whose escaping is defined and byte-identical between GNU sed and BusyBox sed. Leading "--" comments and blank lines are stripped before escaping, so the embedded DDL has no quotes left to escape at all (323 bytes instead of 785) while schema/used_networks.sql keeps its comments and stays the source of truth the README block matches. Output is staged in a temp file, asserted to contain the CREATE TABLE, and compiled before it is moved into place; the target also depends on the Makefile now, because a changed generator must re-run even when a stale, already-broken schema_sql.c looks up to date -- which is what kept this failure alive after the first fix. The link also asked for libraries a minimal system has no way to satisfy: -ldl, -lpthread and -lm, needed only because the vendored amalgamation was compiled with SQLite's default configuration. glibc 2.34 and later resolved dl and pthread into libc, and Buildroot-style installs ship no dev library for either or for libm, where tcc aborts the whole link on the first unsatisfiable -l flag. The amalgamation is now built SQLITE_THREADSAFE=0 (single-threaded CLI), SQLITE_OMIT_LOAD_EXTENSION (extensions are never loaded) and SQLITE_OMIT_FLOATING_POINT (fabs is SQLite's only libm reference, and the registry stores INTEGER and TEXT, never REAL), which removes every dl, pthread and math symbol: the binary links against libc alone. The defines reach every translation unit via CFLAGS, since sqlite3.h changes its float typedefs under that last define, and `make print-flags` publishes them so the strict-C89 build and the ASan runner cannot drift out of configuration. Verified: CURRENT_TIMESTAMP unchanged and second-accurate despite the float omission, 105/105 tests with and without sanitizers, strict C89 build clean, generator output identical under GNU sed and BusyBox sed. --- .gitignore | 1 + Makefile | 81 ++++++++++++++++++++++++++++++++++++++----------- README.md | 26 ++++++++++++++-- tests/runner.py | 21 ++++++++++--- 4 files changed, 104 insertions(+), 25 deletions(-) diff --git a/.gitignore b/.gitignore index 42d47b8..4d9dc35 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ # generated from schema/used_networks.sql by the Makefile schema_sql.c +schema_sql.c.tmp # build artifacts /ipvarch diff --git a/Makefile b/Makefile index af030cf..fb39bfb 100644 --- a/Makefile +++ b/Makefile @@ -1,51 +1,96 @@ # IPVarch - Internet Protocol Version Architect -# Primary compiler is tcc (Tiny C Compiler); falls back to gcc automatically. +# Primary compiler is tcc (Tiny C Compiler); the commented-out $(shell ...) +# line below restores an automatic tcc-then-gcc fallback if you want it. # The vendored SQLite amalgamation (vendor/sqlite3.[ch], downloaded from # https://www.sqlite.org/2026/sqlite-amalgamation-3540100.zip) is compiled -# separately with gcc - tcc links the resulting object fine. +# separately with $(VCC) (tcc today; gcc works too). 'make check' and the +# sanitizer runner build their own vendor objects, since they need gcc/clang. #CC := $(shell command -v tcc 2>/dev/null || command -v gcc 2>/dev/null || echo cc) CC := tcc #VCC := gcc VCC := tcc -CFLAGS := -Wall + +# Embedder configuration for the vendored amalgamation, chosen so the link +# needs no libraries at all: SQLITE_OMIT_LOAD_EXTENSION drops dlopen/dlclose, +# SQLITE_THREADSAFE=0 drops the pthread mutex layer (the tool is +# single-threaded and never loads extensions) and SQLITE_OMIT_FLOATING_POINT +# drops the one libm reference left, fabs. glibc >= 2.34 resolved dl and +# pthread into libc, and minimal systems (Buildroot and friends) ship no +# libdl.so/libpthread.so/libm.so dev library, while tcc aborts the entire +# link when a -l flag cannot be satisfied. Nothing is lost here: the +# registry stores INTEGER and TEXT only -- no REAL column, no double in our +# sources, and every sqlite3_snprintf format is %s or %d. If a future +# schema needs REAL, drop the last define and link -lm. +# The defines reach every translation unit, ours included: with +# SQLITE_OMIT_FLOATING_POINT sqlite3.h changes its float typedefs, so our +# declarations must match the compiled object. +SQL_DEFS := -DSQLITE_THREADSAFE=0 -DSQLITE_OMIT_LOAD_EXTENSION -DSQLITE_OMIT_FLOATING_POINT +CFLAGS := -Wall $(SQL_DEFS) TARGET := ipvarch -.PHONY: all check test test-sanitize clean +.PHONY: all check test test-sanitize print-flags clean all: $(TARGET) -$(TARGET): ipvarch.c ipvarch_db.c ipvtbl.c ipvarch_db.h ipvtbl.h schema_sql.c vendor/sqlite3.o +$(TARGET): ipvarch.c ipvarch_db.c ipvtbl.c ipvarch_db.h ipvtbl.h schema_sql.c vendor/sqlite3.o Makefile $(CC) $(CFLAGS) -o $@ ipvarch.c ipvarch_db.c ipvtbl.c schema_sql.c \ - vendor/sqlite3.o -lpthread -ldl -lm + vendor/sqlite3.o # Schema DDL compiled into the binary (single source of truth is # schema/used_networks.sql; regenerate after editing it, e.g. with make). -# Emits one C string literal per line into a NULL-terminated array, so -# no single literal approaches ISO C90's 509-byte limit and the output -# is C89-pedantic-clean. -schema_sql.c: schema/used_networks.sql - awk 'BEGIN{print "/* generated from schema/used_networks.sql - do not edit */"; print "const char *ipvarch_schema_lines[] = {"} {gsub(/\\/,"\\\\"); gsub(/"/,"\\\""); printf " \"%s\\n\",\n", $$0} END{print " 0\n};"}' $< > $@ +# Emits one C string literal per line into a NULL-terminated array, so no +# single literal approaches ISO C90's 509-byte limit. Leading "--" +# comments and blank lines stay in the SQL file for readers and are not +# embedded here: the array then holds only live DDL, which carries no +# double quotes to escape. Escaping uses printf + POSIX sed only, since +# awk's gsub() replacement handling is implementation-defined and BusyBox +# awk drops the backslash in \" -- that produced invalid C on a BusyBox +# system. This target also depends on the Makefile: a changed generator +# must re-run even when the .sql timestamp is older than a stale, +# previously broken schema_sql.c. +# The output is staged in a temp file, asserted non-empty, and compiled +# (-x c -c -o /dev/null; tcc has no real -fsyntax-only and would try to +# link) before being moved into place, so a broken generator fails here +# instead of surfacing later as a confusing compile error. +# Note the doubled $$ in the sed script: make reads $/ as an (empty) +# variable reference, which would silently emit an empty array. +schema_sql.c: schema/used_networks.sql Makefile + { printf '/* generated from schema/used_networks.sql - do not edit */\nconst char *ipvarch_schema_lines[] = {\n'; \ + sed -e '/^--/d' -e '/^$$/d' -e 's/\\/\\\\/g' -e 's/"/\\"/g' \ + -e 's/^/ "/' -e 's/$$/\\n",/' $<; \ + printf ' 0\n};\n'; } > $@.tmp && grep -q 'CREATE TABLE' $@.tmp \ + && $(CC) -x c -c $@.tmp -o /dev/null && mv -f $@.tmp $@ -vendor/sqlite3.o: vendor/sqlite3.c vendor/sqlite3.h - $(VCC) -O1 -c vendor/sqlite3.c -o $@ +# Depends on the Makefile as well: a changed SQL_DEFS/$(VCC) must rebuild the +# object, otherwise a stale one compiled with the old flags survives (the same +# trap schema_sql.c documents above). +vendor/sqlite3.o: vendor/sqlite3.c vendor/sqlite3.h Makefile + $(VCC) -O1 $(SQL_DEFS) -c vendor/sqlite3.c -o $@ # Strict C89 conformance build of our sources (needs gcc; tcc has no such mode). # The vendor object is rebuilt with gcc here: the default tcc-built # vendor/sqlite3.o relies on tcc's libtcc1 runtime helpers, which gcc # cannot link, so check compiles its own copy (this is why check always -# rebuilds it and clean removes the cached object). +# rebuilds it and clean removes the cached object). No -l flags: see +# SQL_DEFS above. check: $(TARGET) schema_sql.c @if command -v gcc >/dev/null 2>&1; then \ mkdir -p /tmp/ipvarchtests && \ - gcc -O1 -c vendor/sqlite3.c -o /tmp/ipvarchtests/sqlite3-check.o && \ - gcc -std=c89 -pedantic -Wall -Wextra -Wno-long-long -D_FORTIFY_SOURCE=2 -O2 \ + gcc -O1 $(SQL_DEFS) -c vendor/sqlite3.c -o /tmp/ipvarchtests/sqlite3-check.o && \ + gcc -std=c89 -pedantic -Wall -Wextra -Wno-long-long -D_FORTIFY_SOURCE=2 -O2 $(SQL_DEFS) \ -o /tmp/ipvarchtests/ipvarch-check ipvarch.c ipvarch_db.c ipvtbl.c schema_sql.c \ - /tmp/ipvarchtests/sqlite3-check.o -lpthread -ldl -lm && \ + /tmp/ipvarchtests/sqlite3-check.o && \ echo "check: strict C89 build OK (/tmp/ipvarchtests/ipvarch-check)"; \ else \ echo "check: gcc not available, skipping (tcc has no strict mode)"; fi +# Build-time facts for the sanitizer runner: one source of truth for the +# SQLite embedder defines, so an ASan build can never be configured +# differently from the real one (same principle as the schema generator). +print-flags: + @printf '%s\n' $(SQL_DEFS) + # Validity test suite (python3, stdlib only); each run uses a fresh temp DB. test: $(TARGET) python3 tests/runner.py --bin ./$(TARGET) @@ -56,5 +101,5 @@ test-sanitize: $(TARGET) clean: rm -f $(TARGET) /tmp/ipvarchtests/ipvarch-asan /tmp/ipvarchtests/ipvarch-check - rm -f schema_sql.c /tmp/ipvarchtests/sqlite3-check.o + rm -f schema_sql.c schema_sql.c.tmp /tmp/ipvarchtests/sqlite3-check.o rm -f vendor/sqlite3.o diff --git a/README.md b/README.md index f3d59e1..253b3c7 100644 --- a/README.md +++ b/README.md @@ -115,7 +115,13 @@ The table layout lives in `schema/used_networks.sql`, the single source of truth. The Makefile compiles it into the binary at build time (as the generated `schema_sql.c`) — a purely development convenience: the shipped binary stays self-contained, and changing the schema means -editing the SQL file and rebuilding. It is created with `IF NOT +editing the SQL file and rebuilding. Only the live statements are +embedded: leading `--` comments and blank lines stay in the file for +readers and are dropped from the generated array, which also keeps the +embedded SQL free of double quotes so C escaping can never be the weak +point of a build. The generated file is refreshed whenever the SQL file +*or* the Makefile changes, so an edited generator can't be undone by a +stale artifact. It is created with `IF NOT EXISTS`, so re-opening an initialised database is a no-op: ```sql @@ -193,10 +199,26 @@ Notes: - `ipvarch_db.c/.h` — used-networks SQLite backend. - `vendor/sqlite3.c`, `vendor/sqlite3.h` — vendored SQLite amalgamation 3.54.1 from https://www.sqlite.org/2026/sqlite-amalgamation-3540100.zip - (public domain — free for any use, no attribution required); compiled separately with gcc and linked in, so the + (public domain — free for any use, no attribution required); compiled + separately with `$(VCC)` (tcc today, gcc works too) and linked in, so the only third-party dependency is vendored and there is nothing to install. +Nothing links against `libdl`, `libpthread` or `libm`. The amalgamation is +built with `SQLITE_THREADSAFE=0`, `SQLITE_OMIT_LOAD_EXTENSION` and +`SQLITE_OMIT_FLOATING_POINT` (the `SQL_DEFS` variable in the Makefile), +which removes every reference those libraries would otherwise serve — +`fabs` is the only math symbol SQLite needs at all. So the binary depends +on `libc` alone, which is what lets a tcc-only Buildroot box build it: +glibc 2.34 and later folded `dl` and `pthread` into `libc` anyway, minimal +distributions often ship no dev libraries at all, and tcc aborts the entire +link when a `-l` flag cannot be satisfied. Nothing IPVarch uses is given +up: the registry stores INTEGER and TEXT and never REAL, and the tool is +single-threaded. If a future schema needs `REAL` columns or `%f` in +`sqlite3_mprintf`, drop the last define and add `-lm` back. `make check`, +the schema generator and the sanitizer runner all read the same flags from +the Makefile, so no build path can drift out of configuration. + ### Code style I write IPVarch in what I'll call a Stroustrup hybrid: an opening brace diff --git a/tests/runner.py b/tests/runner.py index 7bf2bf1..27756fc 100644 --- a/tests/runner.py +++ b/tests/runner.py @@ -306,21 +306,32 @@ def build_sanitize(repo): if proc.returncode != 0 or not os.path.exists(schema_gen): return None, "schema generation failed:\n%s" % proc.stderr.strip() + # The Makefile owns the SQLite embedder defines (SQL_DEFS); ask it for + # them instead of duplicating the list, so the sanitized build can never + # end up configured differently from the real one. With those defines + # the amalgamation references no dlopen/pthread/fabs symbol, so no -l + # flags are needed here -- and minimal systems may not even have libdl, + # libpthread or libm available to link. + proc = subprocess.run(["make", "-s", "print-flags"], cwd=repo, + capture_output=True, text=True) + sql_defs = [f for f in proc.stdout.split() if f.startswith("-D")] + if proc.returncode != 0 or not sql_defs: + return None, "make print-flags failed:\n%s" % proc.stderr.strip() + cmd = [cc, "-O1", "-g", "-fsanitize=address,undefined", - "-fno-omit-frame-pointer", "-c", - os.path.join(repo, "vendor", "sqlite3.c"), "-o", vendor_obj] + "-fno-omit-frame-pointer"] + sql_defs + [ + "-c", os.path.join(repo, "vendor", "sqlite3.c"), "-o", vendor_obj] proc = subprocess.run(cmd, capture_output=True, text=True) if proc.returncode != 0: return None, "sanitized sqlite build failed:\n%s" % proc.stderr.strip() cmd = [cc, "-std=c89", "-O1", "-g", - "-fsanitize=address,undefined", "-fno-omit-frame-pointer", + "-fsanitize=address,undefined", "-fno-omit-frame-pointer"] + sql_defs + [ os.path.join(repo, "ipvarch.c"), os.path.join(repo, "ipvarch_db.c"), os.path.join(repo, "ipvtbl.c"), schema_gen, - vendor_obj, "-o", SAN_BIN, - "-lpthread", "-ldl", "-lm"] + vendor_obj, "-o", SAN_BIN] proc = subprocess.run(cmd, capture_output=True, text=True, cwd=repo) if proc.returncode != 0: return None, "sanitized build failed:\n%s" % proc.stderr.strip()