Mon 26 Feb 2024

Previous: Mon 19 Feb 2024 Next: Mon 11 Mar 2024

Shark

We’ve got an obuilder runc container using Linux network namespaces with VPNkit forwarding outgoing http requests to a cohttp-proxy-lwt. I’ve created a monorepo assembling obuilder, VPNkit, and cohttp and got them building. Some issues were VPNkit having an outdated version of cohttp and having it’s own dune dns library which collides with the Mirage DNS library.

Next is assembling a binary to do the container, namespace, VPNkit, and http proxy orchestration.

I’m thinking about the UI of what this will look like. Something like

$ shark create <name>
$ shark attach <name>
$ ...

We also need to implement some kind of store to cache and replay http queries.

Some technical nodes on reproducing this prototype separate-binary setup follow:

build obuilder:

$ cd ~/projects/obuilder
$ git clone git@github.com:ocurrent/obuilder.git
$ nix-shell -p sqlite pkg-config
$ opam install . --deps-only
$ dune build main.exe

build vpnkit:

$ cd ~/projects/vpnkit
$ git clone git@github.com:moby/vpnkit.git
$ opam install . --deps-only
$ dune build vpnkit.exe
$ nix-shell -p glibc.static gcc
$ dune build vpnkit.exe

set up the network namespace proxing traffic to TAP device:

$ sudo ip netns add neto
$ sudo ip netns exec neto ip tuntap add tapo mode tap
$ sudo ip netns exec neto ip link set tapo netns neto
$ sudo ip netns exec neto ip addr add 192.168.65.3/24 dev tapo
$ sudo ip netns exec neto ip link set tapo up
$ sudo ip netns exec neto ip route add default via 192.168.65.1
$ sudo ip netns exec neto ~/projects/vpnkit/c/vpnkit-tap-vsockd/sbin/vpnkit-tap-vsockd --tap tapo --path /tmp/vpnkit-ethernet.sock

run a HTTP proxy:

$ cohttp-proxy-lwt -p 3128 -vv

run VPNkit with a HTTP proxy:

$ cat '{"http": "localhost:3128","https": "localhost:3128","exclude": "*.local"}' > host.json
$ ~/projects/vpnkit/_build/default/vpnkit.exe --ethernet /tmp/vpnkit-ethernet.sock --http host.json

try curling in the namespace:

$ sudo ip netns exec neto curl http://freumh.org

create an obuilder container in namespace:

$ nix shell nixpkgs#runc
$ cat "
((from ocaml/opam)
  (run
   (network /var/run/netns/neto)
   (shell "curl http://freumh.org -v")))
" > example.spec
$ sudo ~/projects/obuilder/_build/default/main.exe build -f example.spec . --store=rsync:`pwd`/rsync/ --rsync-mode=copy
main.exe: [INFO] Architectures for multi-arch system: [SCMP_ARCH_X86_64;
                                                       SCMP_ARCH_X86;
                                                       SCMP_ARCH_X32]
(from ocaml/opam)
---> using "ac36be11f82fb13bcf2d2c33422ae3481389700eb141c54e8f01f9c9740faffb" from cache
/: (run (network /var/run/netns/neto)
        (shell "curl http://freumh.org -v"))
main.exe: [INFO] Exec "rsync" "-aHq" "/tmp/rsync/result/ac36be11f82fb13bcf2d2c33422ae3481389700eb141c54e8f01f9c9740faffb/" "/tmp/rsync/result-tmp/3fa186a54e74aad5a53c2c4dcb26ca0b493e0a4032cf0295ba717ea139bca747"
main.exe: [INFO] Exec "runc" "--root" "/tmp/rsync/state/sandbox" "run" "0"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 135.181.100.27:80...
* Connected to freumh.org (135.181.100.27) port 80 (#0)
> GET / HTTP/1.1
> Host: freumh.org
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Sat, 02 Mar 2024 15:34:26 GMT
< Content-Type: text/html
< Location: https://freumh.org/
< Strict-Transport-Security: max-age=31536000
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; base-uri 'self'; frame-src 'self'; frame-ancestors 'self'; form-action 'self';
< Referrer-Policy: same-origin
< transfer-encoding: chunked
<
{ [173 bytes data]
100   162    0   162    0     0   1264      0 --:--:-- --:--:-- --:--:--  1255
* Connection #0 to host freumh.org left intact
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
main.exe: [INFO] Exec "mv" "/tmp/rsync/result-tmp/3fa186a54e74aad5a53c2c4dcb26ca0b493e0a4032cf0295ba717ea139bca747" "/tmp/rsync/result/3fa186a54e74aad5a53c2c4dcb26ca0b493e0a4032cf0295ba717ea139bca747"
---> saved as "3fa186a54e74aad5a53c2c4dcb26ca0b493e0a4032cf0295ba717ea139bca747"
Got: "3fa186a54e74aad5a53c2c4dcb26ca0b493e0a4032cf0295ba717ea139bca747"

in the VPNkit logs:

[2024-03-02T15:34:26.170356988Z][vpnkit.exe][info] ethernet: Connected Ethernet interface f6:16:36:bc:f9:c6
[2024-03-02T15:34:26.170392036Z][vpnkit.exe][info] udp: UDP layer connected on 100.100.100.100
[2024-03-02T15:34:26.170403957Z][vpnkit.exe][info] tcp.pcb: TCP layer connected on 100.100.100.100
[2024-03-02T15:34:26.189383983Z][vpnkit.exe][info] http: HTTP proxy --> 127.0.0.1:3128 Host:freumh.org:80 (Proxy): GET /
[2024-03-02T15:34:26.189481019Z][vpnkit.exe][info] http: HTTP proxy --> 127.0.0.1:3128 Host:freumh.org:80 (Proxy): Successfully connected to 127.0.0.1:3128
[2024-03-02T15:34:26.189506053Z][vpnkit.exe][info] http: Outgoing.Request.write
[2024-03-02T15:34:26.189560890Z][vpnkit.exe][info] http: Outgoing.Response.read
[2024-03-02T15:34:26.297271966Z][vpnkit.exe][info] http: HTTP proxy <-- 127.0.0.1:3128 Host:freumh.org:80 (Proxy): HTTP/1.1 301 Moved Permanently
[2024-03-02T15:34:26.297477960Z][vpnkit.exe][info] http: HTTP proxy <-- 127.0.0.1:3128 Host:freumh.org:80 (Proxy): proxying body

and the cohttp proxy logs:

[DEBUG][cohttp.lwt.io]: <<< GET http://freumh.org:80/ HTTP/1.1
[DEBUG][cohttp.lwt.io]: <<< host: freumh.org
[DEBUG][cohttp.lwt.io]: <<< Accept: */*
[DEBUG][cohttp.lwt.io]: <<<
[DEBUG][cohttp.lwt.server]: Handle request: ((headers ((host freumh.org) (User-Agent curl/7.88.1) (Accept */*)))
 (meth GET) (scheme ()) (resource http://freumh.org:80/) (version HTTP_1_1)
 (encoding Unknown)).
--> GET http://freumh.org:80/ ((headers ((host freumh.org) (User-Agent curl/7.88.1) (Accept */*)))
 (meth GET) (scheme ()) (resource http://freumh.org:80/) (version HTTP_1_1)
 (encoding Unknown))
[DEBUG][cohttp.lwt.io]: <<< User-Agent: curl/7.88.1
[DEBUG][cohttp.lwt.io]: >>> GET / HTTP/1.1
[DEBUG][cohttp.lwt.io]: >>> host: freumh.org
User-Agent: curl/7.88.1
Accept: */*
accept-encoding: identity
[DEBUG][cohttp.lwt.io]: <<< HTTP/1.1 301 Moved Permanently
[DEBUG][cohttp.lwt.io]: <<< Server: nginx
[DEBUG][cohttp.lwt.io]: <<< Date: Sat, 02 Mar 2024 15:34:26 GMT
[DEBUG][cohttp.lwt.io]: <<< Content-Type: text/html
[DEBUG][cohttp.lwt.io]: <<< Connection: keep-alive
[DEBUG][cohttp.lwt.io]: <<< Content-Length: 162
[DEBUG][cohttp.lwt.io]: <<< X-Content-Type-Options: nosniff
[DEBUG][cohttp.lwt.io]: <<< Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; base-uri 'self'; frame-src 'self'; frame-ancestors 'self'; form-action 'self';
[DEBUG][cohttp.lwt.io]: <<< Referrer-Policy: same-origin
[DEBUG][cohttp.lwt.io]: <<<
<-- http://freumh.org:80/ ((encoding (Fixed 162))
 (headers
  ((Server nginx) (Date "Sat, 02 Mar 2024 15:34:26 GMT")
   (Content-Type text/html) (Content-Length 162) (Connection keep-alive)
   (Location https://freumh.org/)
   (Strict-Transport-Security max-age=31536000) (X-Frame-Options SAMEORIGIN)
   (X-Content-Type-Options nosniff)
   (Content-Security-Policy
    "default-src 'self' 'unsafe-inline' 'unsafe-eval'; base-uri 'self'; frame-src 'self'; frame-ancestors 'self'; form-action 'self';")
   (Referrer-Policy same-origin)))
 (version HTTP_1_1) (status Moved_permanently) (flush false))
[DEBUG][cohttp.lwt.io]: >>> HTTP/1.1 301 Moved Permanently
[DEBUG][cohttp.lwt.io]: >>> Server: nginx
Date: Sat, 02 Mar 2024 15:34:26 GMT
Content-Type: text/html
Location: https://freumh.org/
Strict-Transport-Security: max-age=31536000
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'self' 'unsafe-inline' 'unsafe-eval'; base-uri 'self'; frame-src 'self'; frame-ancestors 'self'; form-action 'self';
Referrer-Policy: same-origin
transfer-encoding: chunked
[DEBUG][cohttp.lwt.io]: >>> <html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
[DEBUG][cohttp.lwt.io]: >>>
[DEBUG][cohttp.lwt.io]: >>> 0
[DEBUG][cohttp.lwt.io]: <<< Location: https://freumh.org/
[DEBUG][cohttp.lwt.io]: <<< Strict-Transport-Security: max-age=31536000
[DEBUG][cohttp.lwt.io]: <<< X-Frame-Options: SAMEORIGIN
[DEBUG][cohttp.lwt.io]: >>> a2
[DEBUG][cohttp.lwt.io]: <<<[162] <html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

[DEBUG][cohttp.lwt.io]: <<< EOF
Connection (TCP ((fd <opaque>) (ip 127.0.0.1) (port 36222))) closed

Lenscap

Got a bit distracted with shark this week, but watch this space.