Every Field Was Correct. It Still Wouldn't Connect.

The VPN I share with a few other people ran out of traffic for the month. I had an idle OCI ARM instance sitting there doing nothing, so I decided to stop waiting and build my own node: one VLESS + Reality endpoint, self-hosted, good enough for personal use through Clash Verge. I drove the whole thing with Claude — chatted through the plan, had Claude Code do the Docker deploy, then went back to chat to walk the panel config step by step. The build was an afternoon. The debugging was the rest of the evening, and every minute of it was spent on failures where the configuration was, provably, correct.

That’s the part worth writing down. Not the happy path — the four traps, and especially the last one, where every field matched on every screen and it still refused to connect.

Standing up the panel

The management panel is 3x-ui, run as a container:

docker run -d \
  --restart unless-stopped \
  -v /etc/x-ui:/etc/x-ui \
  -p 443:443 \
  -p 2096:2096 \
  -p 54321:2053 \
  ghcr.io/mhsanaei/3x-ui:latest

Enter fullscreen mode Exit fullscreen mode

Three things here are not optional. Persist /etc/x-ui to the host, or every container recreate wipes your config. Set a restart policy (unless-stopped or always) so the node comes back on its own after a reboot — a proxy you have to SSH in and restart by hand is a proxy you’ll abandon. And map only the ports you actually use with -p: the node (443), the subscription port (2096 here), and the panel. After it’s running, docker ps and confirm every one shows up in the PORTS column.

The moment it’s up, before anything else: log in and change the default credentials. This panel is exposed to the public internet, and default logins are exactly what scanners look for. While you’re in there, move the panel off its default 2053 to a random port — that’s the 54321:2053 mapping above — and consider changing its base access path. None of this is real security on its own; it just takes you out of the easy-target bucket.

Two firewalls, or nothing works

On a normal VPS this is one step. On OCI it’s two, and missing either one gives you the identical symptom — a plain connection timeout — which makes it genuinely hard to tell which layer is at fault.

1. OCI’s own network layer. In the console, add ingress rules to the instance’s Security List (or NSG) for every port you need: TCP, source 0.0.0.0/0.
2. The instance’s host firewall. Ubuntu’s ufw is separate and just as real:

sudo ufw allow 443/tcp     # the node
sudo ufw allow 2096/tcp    # subscription port, etc.
sudo ufw reload

Enter fullscreen mode Exit fullscreen mode

Open one and not the other and you’ll spend twenty minutes curl-ing from the wrong side of the problem. Open both first.

The inbound — and the button that rewrites your config

Create a VLESS inbound on port 443 (blending in with ordinary HTTPS). Most defaults are fine. Two tabs need care.

⚠️ Trap 1 — the Generate button has a side effect. The Protocol tab has Decryption/Encryption fields; for standard VLESS both should read none. But clicking Generate to create the Reality X25519 key pair also silently overwrites those two fields with an experimental post-quantum encryption string — something like mlkem768x25519plus.native.600s.... This is a newer Xray feature, and Mihomo (the core inside Clash Verge) can’t parse it; it errors out with invaild vless encryption value (yes, misspelled that way in Mihomo itself). The fix: after you generate the Reality keys, go back and manually set Decryption and Encryption to none. And don’t click Generate again afterward — it re-overwrites the fields and regenerates your key pair, which just adds a fresh variable.

The decoy site that’s too big

On the Security tab, set the security type to Reality and pick a camouflage target — a real TLS 1.3 site your handshake will impersonate. Set uTLS to chrome; leave short IDs and SpiderX auto-generated.

⚠️ Trap 2 — some targets are literally too large. With www.microsoft.com as the target, every parameter matched and the keys were confirmed identical, but the server log said:

REALITY: processed invalid connection ... handshake did not complete successfully

Enter fullscreen mode Exit fullscreen mode

This isn’t your config. Xray-core reads the target’s TLS Certificate record during the Reality handshake and rejects anything over a hardcoded 8192 bytes. www.microsoft.com returns a record around 8273 bytes — just over the line — so the handshake dies. It’s a known, reproduced upstream limit (Xray-core #6356, #6402, discussion #6387). The fix is to pick a target with a smaller certificate. www.cloudflare.com works, and the error vanished the instant I switched.

The client and the subscription that Clash refuses

On the Add Client page, attach the client to the port-443 inbound you just made, or it’s linked to nothing. Then, on the Credentials tab, change Flow from its default None to xtls-rprx-vision to match the server’s Reality setup — leave it at None and the connection behaves inconsistently or fails outright.

⚠️ Trap 3 — Clash needs YAML, not the default link. 3x-ui’s default subscription link (http://ip:port/sub/<id>) returns the generic Base64 format that clients like V2RayNG expect. Paste that into Clash Verge and you get:

the remote profile data is invalid yaml: invalid type: string, expected a YAML mapping

Enter fullscreen mode Exit fullscreen mode

Clash/Mihomo needs a YAML subscription, which 3x-ui serves from a different path. In panel settings, confirm the Clash/Mihomo subscription is enabled, check its URI path (usually /clash/), and use:

http://server-ip:subscription-port/clash/<subscription-id>

Enter fullscreen mode Exit fullscreen mode

Import that under Profiles → New → Remote. Only this path returns parseable YAML.

The trap that ate the evening

Subscription imported, node listed, everything green. Every dial failed:

[TCP] dial ... error: server-ip:443 connect error: REALITY authentication failed

Enter fullscreen mode Exit fullscreen mode

We worked it in order — I say “we” because Claude was driving the diagnosis the whole way:

• nc -zv server-ip 443 from my laptop — TCP connects. Network’s fine.
• curl from the server to the Reality target — clean 200. Target’s reachable.
• docker exec ... netstat -tlnp | grep 443 — the port is listening inside the container.
• Then the careful part: read the actual running config.json from inside the container and compare Public Key, Short ID, SNI, UUID, and Flow against the panel UI and against the subscription Clash was consuming. Field by field.

Everything matched. An assistant that will happily read every value in every file had just confirmed all of them correct — and the connection still failed. The thing that was wrong wasn’t in any file.

The answer was the version of the running binary. 3x-ui v3.5.0 ships Xray-core v26.7.11, and that release is deliberately incompatible with Mihomo — a documented breaking change (a new default minClientVer), not a local misconfiguration. Multiple people reproduced the exact REALITY authentication failed symptom on v3.5.0, and Mihomo has stated outright it won’t support Xray v26.7.11+ (3x-ui #5957, #5922; Xray-core #6477). The fix is to downgrade the core to v26.6.27. Recreating clients, regenerating keys, rebuilding the inbound — none of it matters, because none of it touches the version.

There was one more layer to it, and it’s the reason the fix looked like it didn’t work. After downgrading, the error was still there. What I hadn’t accounted for: the panel applies changes through Xray’s core API and reports config changes applied through the core API, no restart needed. That hot-reload path does not swap the running Xray process. The old v26.7.11 binary was still serving traffic. A full docker restart 3x-ui loaded the downgraded core, and it connected immediately.

So the two facts sit together. The fix was the downgrade. It only took effect on a real process restart — because the panel’s “no restart needed” is true for config edits and quietly false for the one thing that actually mattered.

What I’d tell myself at the start

When every file says the config is correct and it still fails, stop re-reading the files. The mismatch you can’t see is the process, not the config: what version is actually running, and did it actually reload. The panel telling you “no restart needed” is a claim about config edits, not a guarantee about the process — so after any change that matters, restart the container and check the running core yourself.

A short checklist, in the order that would have saved me the evening:

1. nc -zv server-ip port — reachability from the client.
2. Confirm both the OCI security list and the host ufw allow the port.
3. docker exec ... netstat -tlnp — the port is really listening inside the container.
4. Decryption/Encryption are none, not the mlkem768... post-quantum string.
5. Camouflage target has a small certificate — avoid www.microsoft.com; www.cloudflare.com is safe.
6. Client Flow is xtls-rprx-vision; subscription is the /clash/ YAML path, not /sub/.
7. Check the running Xray-core version. If it’s v26.7.11 and your client is Mihomo/Clash Verge, downgrade to v26.6.27.
8. After any change, docker restart the container — don’t trust “no restart needed” — then retest.

The config was never the hard part. The hard part was trusting, for one evening too long, that a correct file meant a correct process.