smoke: fix random so_bindtodevice test failure - #678
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
1313660 to
ebd7d4e
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@smoke/so_bindtodevice_test.sh`:
- Around line 61-62: The UDP listener setup in the smoke flow still relies on
socat’s unreliable max-children=1 behavior. Update the listeners around the two
UDP4/UDP6 socat commands to add a minimal assertion that the second RECVFROM
child cannot start before the first exits, or replace the UDP-RECVFROM,fork
pattern with a test-compatible approach that preserves serialized child
handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c24c00e5-4934-4ca6-a510-b503bc174c25
📒 Files selected for processing (1)
smoke/so_bindtodevice_test.sh
| ip netns exec $ns socat UDP4-RECVFROM:9001,fork,max-children=1 EXEC:'/bin/cat' & | ||
| ip netns exec $ns socat UDP6-RECVFROM:9000,fork,max-children=1 EXEC:'/bin/cat' & |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/sh
set -eu
command -v socat
socat -V
port="${PORT:-39101}"
log=$(mktemp)
worker=$(mktemp)
server=
cleanup() {
[ -z "$server" ] || kill "$server" 2>/dev/null || true
rm -f "$log" "$worker" "${log}.diag"
}
trap cleanup EXIT
cat >"$worker" <<'EOF'
#!/bin/sh
printf '%s start\n' "$(date +%s%N)" >>"$SOCAT_TEST_LOG"
sleep 1
dd bs=5 count=1 of=/dev/null 2>/dev/null
printf '%s end\n' "$(date +%s%N)" >>"$SOCAT_TEST_LOG"
EOF
chmod +x "$worker"
SOCAT_TEST_LOG="$log" socat -d -d \
"UDP4-RECVFROM:${port},fork,max-children=1" \
"EXEC:${worker}" >/dev/null 2>"${log}.diag" &
server=$!
sleep 0.2
printf first | socat -u - "UDP4-SENDTO:127.0.0.1:${port}" &
first=$!
sleep 0.2
printf second | socat -u - "UDP4-SENDTO:127.0.0.1:${port}" &
second=$!
wait "$first" "$second"
for _ in 1 2 3 4 5; do
[ "$(grep -c ' end$' "$log" || true)" -ge 2 ] && break
sleep 1
done
awk '
$2 == "start" { starts++; if (starts == 2) second_start = $1 }
$2 == "end" { ends++; if (ends == 1) first_end = $1 }
END { exit !(starts == 2 && ends == 2 && second_start >= first_end) }
' "$log"Repository: DPDK/grout
Length of output: 148
🏁 Script executed:
#!/bin/sh
set -eu
printf 'socat binary: '
command -v socat || true
printf 'socat version: '
socat -V 2>&1 || true
printf 'repo workflow files containing socat or apt packages: '
rg -n "socat|max_children|max-children|required-packages|DEBIAN|ubuntu|debian" -S .github . 2>/dev/null || true
printf 'relevant smoke script sections: '
sed -n '55,70p;75,95p;136,148p' smoke/so_bindtodevice_test.shRepository: DPDK/grout
Length of output: 6277
🌐 Web query:
Ubuntu 24.04 package socat version
💡 Result:
In Ubuntu 24.04 LTS (Noble Numbat), the socat package version depends on whether you have applied the latest updates. The initial version of socat provided in the main repository for Ubuntu 24.04 is 1.8.0.0-4build3 [1][2]. However, if your system is fully updated, it will likely be running version 1.8.0.0-4ubuntu0.1, which is available in the noble-updates repository [3][4].
Citations:
- 1: https://launchpad.net/ubuntu/noble/amd64/socat
- 2: https://packages.ubuntu.com/en/noble/socat
- 3: https://www.ubuntuupdates.org/package/core/noble/main/updates/socat
- 4: https://packages.ubuntu.com/noble-updates/socat
🌐 Web query:
Ubuntu 24.04 updated socat package version
💡 Result:
In Ubuntu 24.04 LTS (Noble Numbat), the base version of the socat package was 1.8.0.0-4build3 [1][2]. As of August 2026, the updated version available in the noble-updates repository is 1.8.0.0-4ubuntu0.1 [3][4]. This version was published to the updates repository on July 6, 2026 [5].
Citations:
- 1: https://launchpad.net/ubuntu/noble/amd64/socat
- 2: https://packages.ubuntu.com/source/noble/socat
- 3: https://packages.ubuntu.com/noble-updates/socat
- 4: https://packages.ubuntu.com/en/noble-updates/socat
- 5: https://www.ubuntuupdates.org/package/core/noble/main/updates/socat
🌐 Web query:
socat 1.8.0.0 max-children UDP-RECVFROM fork
💡 Result:
In Socat 1.8.0.0, the combination of UDP-RECVFROM and the fork option experienced several significant stability issues [1][2]. These included an file descriptor (FD) leak, race conditions, and a bug that caused the process to enter a fork loop if the second address failed to open or connect [1][2]. Additionally, documentation and historical bug reports for Socat indicate that while max-children is a standard option supported by the CHILD option group (which includes RECVFROM), it was specifically noted that some earlier implementations failed to apply max-children correctly in certain contexts (e.g., UDP-LISTEN) [1][2]. If you are encountering issues with this configuration in version 1.8.0.0, it is highly recommended to update to the latest available version (such as 1.8.0.3 or newer), which addresses these specific bugs [3]. The changelogs for Socat explicitly identify and fix the fork-related leaks, loop behaviors, and synchronization issues that affected UDP-RECVFROM in the 1.8.0.0 release [1][2][4].
Citations:
- 1: https://third-party-mirror.googlesource.com/socat/+/0454c4fa43dd7675f050ad07d8fd03459c07c869/CHANGES
- 2: https://third-party-mirror.googlesource.com/socat/+/642b7296226348e35d2abfa026078f53ed7deb43/CHANGES
- 3: https://bugzilla.redhat.com/show_bug.cgi?id=2307725
- 4: https://fossies.org/linux/socat/CHANGES
🌐 Web query:
Cherokee socat UDP-RECVFROM max-children fork 1.8.0.0
💡 Result:
The query appears to conflate two distinct software tools: the Cherokee Web Server and the utility socat. There is no evidence of a Cherokee web server version 1.8.0.0, nor of any specific feature or interaction matching "UDP-RECVFROM max-children fork" within the Cherokee codebase [1][2][3]. The Cherokee Web Server project has been unmaintained since 2013, with its last official stable release being version 1.2.103 [3]. The phrase "UDP-RECVFROM max-children fork" refers to specific configuration options for the socat utility [4][5][6]. In socat, the UDP-RECVFROM address type receives a single UDP packet and can optionally fork a sub-process to handle it, providing behavior similar to UDP-based servers like NTP or DNS [4][7][5]. Key points regarding socat: - The fork option: When fork is used with UDP-RECVFROM, socat forks a new process for every incoming UDP packet, allowing the server to handle multiple requests concurrently [4][5]. - The max-children option: This option limits the number of simultaneous child processes spawned by socat [8][6]. According to socat's implementation and documentation, the max-children option is invalid unless the fork option is also specified [8]. - If max-children is reached, the socat process will wait until one of the existing child processes terminates before it continues accepting new connections [8]. In summary, "UDP-RECVFROM", "max-children", and "fork" are valid and documented components of the socat utility, but they are unrelated to the Cherokee Web Server [8][4][5][3].
Citations:
- 1: https://github.com/cherokee/webserver/releases
- 2: https://github.com/cherokee/webserver
- 3: https://wiki.linux-server-admin.com/software/webserver/cherokee
- 4: https://manpages.debian.org/bullseye/socat/socat.1.en.html
- 5: http://www.dest-unreach.org/socat/doc/socat.html
- 6: https://tobias-weiss.org/downloads/cheatsheets/socat-cheatsheet.pdf
- 7: https://manpages.ubuntu.com/manpages/jammy/man1/socat.1.html
- 8: https://third-party-mirror.googlesource.com/socat/+/44c9eb1d9d55f8696dbe0c78a1ff099bcb08251b/xio-udp.c
Avoid relying on max-children=1 for UDP-RECVFROM.
Ubuntu 24.04 installs socat 1.8.0.0, where UDP-RECVFROM has known fork-loop/handover issues. This smoke flow still depends on the fix serializing the two forked UDP listeners; keep the fix but add a minimal assertion that the second RECVFROM child does not start until the first exits, or avoid the buggy UDP-RECVFROM,fork pattern if the test allows.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@smoke/so_bindtodevice_test.sh` around lines 61 - 62, The UDP listener setup
in the smoke flow still relies on socat’s unreliable max-children=1 behavior.
Update the listeners around the two UDP4/UDP6 socat commands to add a minimal
assertion that the second RECVFROM child cannot start before the first exits, or
replace the UDP-RECVFROM,fork pattern with a test-compatible approach that
preserves serialized child handling.
socat unconditionally writes a zero-length buffer to the UDP socket
when its stdin reaches EOF (write(fd, "", 0)). On a connected UDP
socket the kernel turns this into a real 0-byte datagram on the wire.
The peer echo servers use socat with the fork option. When the data
datagram and the empty datagram arrive before the first forked child
has set up its connected socket, the parent forks a second child for
the empty datagram. The second child has no work to do and finishes
first, sending its empty reply before the first child sends the
actual echo. When the test-side socat receives the 0-byte reply
first, it treats it as EOF and exits with an empty result.
This is not a grout bug. Grout packet traces from both a passing and
failing run show that grout forwards and delivers both packets
correctly and in order. The reordering is visible at the port_rx
level, meaning it originates from the peer side:
Passing run (replies in order, test succeeds):
[rx p1] fd03::2 > fd03::1 proto=UDP(17), (pkt_len=67)
[rx p1] fd03::2 > fd03::1 proto=UDP(17), (pkt_len=62)
Failing run (empty reply first, socat treats it as EOF):
[rx p1] fd03::2 > fd03::1 proto=UDP(17), (pkt_len=62)
[rx p1] fd03::2 > fd03::1 proto=UDP(17), (pkt_len=67)
Fix with two changes:
On the server side, replace UDP-LISTEN with UDP-RECVFROM. UDP-LISTEN
closes and recreates the socket after each fork, leaving the port
temporarily unbound. UDP-RECVFROM keeps the same socket open and
uses a trigger socketpair to synchronize packet consumption.
On the client side, replace UDP with UDP-SENDTO. The default UDP
address type uses read() on a connected socket where a 0-byte read
is indistinguishable from EOF. UDP-SENDTO uses recvfrom() instead,
and socat's null_eof defaults to false so a 0-byte datagram returns
EAGAIN and socat retries until a non-empty reply arrives.
Signed-off-by: Robin Jarry <rjarry@redhat.com>
ebd7d4e to
b7abd49
Compare
1 participant
socat unconditionally writes a zero-length buffer to the UDP socket when its stdin reaches EOF (write(fd, "", 0)). On a connected UDP socket the kernel turns this into a real 0-byte datagram on the wire.
The peer echo servers use socat with the fork option. When the data datagram and the empty datagram arrive before the first forked child has set up its connected socket, the parent forks a second child for the empty datagram. The second child has no work to do and finishes first, sending its empty reply before the first child sends the actual echo. When the test-side socat receives the 0-byte reply first, it treats it as EOF and exits with an empty result.
This is not a grout bug. Grout packet traces from both a passing and failing run show that grout forwards and delivers both packets correctly and in order. The reordering is visible at the port_rx level, meaning it originates from the peer side:
Passing run (replies in order, test succeeds):
Failing run (empty reply first, socat treats it as EOF):
Fix with two changes:
On the server side, replace UDP-LISTEN with UDP-RECVFROM. UDP-LISTEN closes and recreates the socket after each fork, leaving the port temporarily unbound. UDP-RECVFROM keeps the same socket open and uses a trigger socketpair to synchronize packet consumption. Adding
max-children=1 serializes child handling so replies always go out in the correct order.
On the client side, replace UDP with UDP-SENDTO. The default UDP address type uses read() on a connected socket where a 0-byte read is indistinguishable from EOF. UDP-SENDTO uses recvfrom() instead, and socat's null_eof defaults to false so a 0-byte datagram returns EAGAIN and socat retries until a non-empty reply arrives.
The
so_bindtodevice_test.shsmoke test can fail whensocatsends a zero-length UDP datagram at stdin EOF. Concurrent peer handlers can return this datagram before the actual echo response, which makes the test-sidesocattreat it as EOF.Use
UDP4/UDP6-RECVFROMwithmax-children=1for peer UDP listeners to serialize datagram handling. UseUDP-SENDTOandUDP6-SENDTOfor clients so zero-length datagrams are ignored while the client retries until it receives the non-empty response. TCP handling remains unchanged.