feat(security): add a whitelist-only outbound mode that refuses before DNS (#3393)
* feat(security): add a whitelist-only outbound mode that refuses before DNS A penetration test of a private deployment found that a host outside SSRF_WHITELIST still produces a DNS query: every outbound guard resolves the name first and judges the addresses afterwards, which is what makes it immune to DNS rebinding, but it also means an attacker-chosen name reaches the resolver and can carry data out of an otherwise offline network (#3378). SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress policy, and the check is placed wherever a name is still in hand. Where a URL is available, CheckSSRFWhitelistOnly refuses a host the whitelist does not name, IP literals included: URL validation, the sandbox outbound policy, and the web-fetch target resolver. Where only a dial address is available, CheckDialWhitelistOnly judges hostnames and leaves addresses to the checks that already exist, because an IP arriving at a dialer is either a name something else resolved or a literal the URL layer already refused. That covers the shared SSRF-safe dialer, the web-fetch dialer, the Yunzhijia webhook dialer, and every sandbox transport: the HTTP and websocket data planes, the guarded transport and the Docker engine now share one GuardedDialContext instead of each building its own dialer. Where the client resolves its own target, the name is judged before the client exists. gRPC's resolver runs before any dialer it was given, so the env-configured qdrant and milvus clients are built through small constructors that refuse a non-whitelisted host outright; a whitelisted one is resolved by gRPC as before. docreader enforces the same switch in its own validator, and compose passes the variable to both services. Matching is by name only in this mode: the whitelist's CIDR entries still match an IP literal, but a hostname is no longer resolved to see whether its addresses fall inside one, because that lookup is the query the mode exists to prevent. The value is parsed with ParseBool, and a non-empty value that parses as neither enables the mode: silently leaving an egress lockdown off because an operator wrote "yes" is the one failure mode a security control must not have. Still not covered, and documented as such: the Langfuse OTLP exporter builds its own client and never reaches these guards; and with HTTP(S)_PROXY set the proxy host is dialled unguarded when the dial address equals the proxy URL's host exactly, while a mismatch makes the proxy host itself subject to the whitelist. The switch defaults to off, so nothing changes for deployments that do not set it. * docs(security): document SSRF_DNS_WHITELIST_ONLY in website-docs docs/ has been retired in favour of website-docs/ (#3580), so the notes that lived in docs/QA.md move to the configuration table: the new switch, what to whitelist before turning it on, and the three outbound paths it still does not cover. The existing SSRF_WHITELIST row was also stale: compose gives only the app its SSRF_WHITELIST_EXTRA default (which now includes minio), and docreader's is empty, so "shared by app and docreader" held for SSRF_WHITELIST alone.main
parent
2c6ae0c262
commit
171097f06e
|
|
@ -754,6 +754,11 @@ LANGFUSE_HOST=http://langfuse-web:3000
|
|||
# 列入者会在 URL 校验等地方绕过常规 SSRF 规则,生产环境请谨慎配置。
|
||||
# SSRF_WHITELIST=internal.service,*.corp.example,172.16.0.0/12,2001:db8::1,fd00::/8
|
||||
#
|
||||
# 仅允许白名单出站(默认关闭)。设为 true 时白名单就是全部出站策略:不在 SSRF_WHITELIST /
|
||||
# SSRF_WHITELIST_EXTRA 里的主机(含 IP 直连)在发起 DNS 查询之前直接拒绝,域名只按名字匹配、
|
||||
# 不再通过解析结果匹配 CIDR。离线或内网部署建议开启,并把模型、向量库、对象存储等地址全部列入白名单。
|
||||
# SSRF_DNS_WHITELIST_ONLY=false
|
||||
#
|
||||
# ===== 向量库地址的 SSRF 校验 =====
|
||||
# 通过 API 创建/测试向量库连接时,地址会经过 SSRF 校验。docker-compose 内置服务
|
||||
# (qdrant, milvus, weaviate, doris-fe, doris-be, searxng, minio)已通过 SSRF_WHITELIST_EXTRA 默认放行。
|
||||
|
|
|
|||
|
|
@ -255,6 +255,7 @@ services:
|
|||
# docreader 对外 HTTP 抓取的 SSRF 白名单(逗号分隔;与 app 共用同名变量)
|
||||
- SSRF_WHITELIST=${SSRF_WHITELIST:-}
|
||||
- SSRF_WHITELIST_EXTRA=${SSRF_WHITELIST_EXTRA:-}
|
||||
- SSRF_DNS_WHITELIST_ONLY=${SSRF_DNS_WHITELIST_ONLY:-}
|
||||
# ========== gRPC 服务端 TLS / 认证 ==========
|
||||
- GRPC_TLS_ENABLED=${GRPC_TLS_ENABLED:-false}
|
||||
- GRPC_TLS_CERT=${GRPC_TLS_CERT:-}
|
||||
|
|
|
|||
|
|
@ -317,6 +317,9 @@ services:
|
|||
# doris-be: Stream Load follows FE→BE redirects that must receive Basic
|
||||
# credentials on whitelisted BE hosts.
|
||||
- SSRF_WHITELIST_EXTRA=${SSRF_WHITELIST_EXTRA:-searxng,qdrant,milvus,weaviate,doris-fe,doris-be,minio}
|
||||
# 仅允许白名单出站(默认关闭)。开启前请把所有需要访问的地址加入白名单,
|
||||
# 见 docs/QA.md「仅允许白名单出站」。
|
||||
- SSRF_DNS_WHITELIST_ONLY=${SSRF_DNS_WHITELIST_ONLY:-}
|
||||
- CONCURRENCY_POOL_SIZE=${CONCURRENCY_POOL_SIZE:-5}
|
||||
- JWT_SECRET=${JWT_SECRET:-}
|
||||
# File size limit (in MB)
|
||||
|
|
@ -428,6 +431,7 @@ services:
|
|||
# docreader 对外 HTTP 抓取的 SSRF 白名单(逗号分隔;与 app 共用同名变量)
|
||||
- SSRF_WHITELIST=${SSRF_WHITELIST:-}
|
||||
- SSRF_WHITELIST_EXTRA=${SSRF_WHITELIST_EXTRA:-}
|
||||
- SSRF_DNS_WHITELIST_ONLY=${SSRF_DNS_WHITELIST_ONLY:-}
|
||||
# ========== gRPC 服务端 TLS / 认证 ==========
|
||||
- GRPC_TLS_ENABLED=${GRPC_TLS_ENABLED:-false}
|
||||
- GRPC_TLS_CERT=${GRPC_TLS_CERT:-}
|
||||
|
|
|
|||
|
|
@ -20,6 +20,41 @@ class TestSSRFValidation(unittest.TestCase):
|
|||
self._env_patch.stop()
|
||||
reset_ssrf_whitelist_cache_for_test()
|
||||
|
||||
def _whitelist_only(self, whitelist: str, only: str) -> None:
|
||||
patcher = patch.dict(
|
||||
os.environ,
|
||||
{"SSRF_WHITELIST": whitelist, "SSRF_WHITELIST_EXTRA": "", "SSRF_DNS_WHITELIST_ONLY": only},
|
||||
clear=False,
|
||||
)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop)
|
||||
reset_ssrf_whitelist_cache_for_test()
|
||||
self.addCleanup(reset_ssrf_whitelist_cache_for_test)
|
||||
|
||||
def test_whitelist_only_refuses_before_dns(self):
|
||||
self._whitelist_only("docs.internal", "true")
|
||||
safe, reason = is_ssrf_safe_url("https://outside.invalid/page")
|
||||
self.assertFalse(safe)
|
||||
self.assertIn("not in the SSRF whitelist", reason)
|
||||
self.assertNotIn("DNS resolution failed", reason)
|
||||
|
||||
def test_whitelist_only_admits_whitelisted_host(self):
|
||||
self._whitelist_only("docs.internal", "true")
|
||||
safe, reason = is_ssrf_safe_url("https://docs.internal/page")
|
||||
self.assertTrue(safe, reason)
|
||||
|
||||
def test_whitelist_only_off_leaves_validation_alone(self):
|
||||
self._whitelist_only("docs.internal", "false")
|
||||
safe, reason = is_ssrf_safe_url("http://127.0.0.1:8080/page")
|
||||
self.assertFalse(safe)
|
||||
self.assertNotIn("not in the SSRF whitelist", reason)
|
||||
|
||||
def test_whitelist_only_unparsable_value_enables_it(self):
|
||||
self._whitelist_only("docs.internal", "yes")
|
||||
safe, reason = is_ssrf_safe_url("https://outside.invalid/page")
|
||||
self.assertFalse(safe)
|
||||
self.assertIn("not in the SSRF whitelist", reason)
|
||||
|
||||
def test_blocks_loopback_ip(self):
|
||||
safe, reason = is_ssrf_safe_url("http://127.0.0.1:8080/page")
|
||||
self.assertFalse(safe)
|
||||
|
|
|
|||
|
|
@ -126,6 +126,21 @@ def _load_whitelist() -> Tuple[FrozenSet[str], Tuple[str, ...], Tuple[Union[ipad
|
|||
return frozenset(exact_hosts), tuple(suffix_hosts), tuple(cidr_nets)
|
||||
|
||||
|
||||
def _whitelist_only_enabled() -> bool:
|
||||
"""Whether SSRF_DNS_WHITELIST_ONLY makes the whitelist the whole egress
|
||||
policy: a host outside it is refused before any DNS query (#3378).
|
||||
|
||||
Parsed like the Go side: 1/t/true enable it, 0/f/false disable it, and a
|
||||
non-empty value that is neither enables it — silently leaving an egress
|
||||
lockdown off because someone wrote "yes" is the one failure mode this
|
||||
control must not have.
|
||||
"""
|
||||
raw = os.environ.get("SSRF_DNS_WHITELIST_ONLY", "").strip()
|
||||
if not raw:
|
||||
return False
|
||||
return raw.lower() not in {"0", "f", "false"}
|
||||
|
||||
|
||||
def _is_whitelisted(hostname: str) -> bool:
|
||||
lowered = hostname.lower()
|
||||
exact_hosts, suffix_hosts, cidr_nets = _load_whitelist()
|
||||
|
|
@ -246,6 +261,15 @@ def is_ssrf_safe_url(raw_url: str) -> Tuple[bool, str]:
|
|||
if _is_whitelisted(hostname_lower):
|
||||
return True, ""
|
||||
|
||||
# Whitelist-only mode stops here: everything below resolves the name, and
|
||||
# a host outside the whitelist must not reach DNS at all (#3378).
|
||||
if _whitelist_only_enabled():
|
||||
return (
|
||||
False,
|
||||
f"host is not in the SSRF whitelist: {hostname_lower} "
|
||||
"(SSRF_DNS_WHITELIST_ONLY is on; add it to SSRF_WHITELIST to allow it)",
|
||||
)
|
||||
|
||||
if hostname_lower in RESTRICTED_HOSTNAMES:
|
||||
return False, f"hostname {hostname_lower} is restricted"
|
||||
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import (
|
|||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
|
@ -1263,6 +1264,36 @@ func initRawFileService(_ *config.Config) (interfaces.FileService, error) {
|
|||
// Returns:
|
||||
// - Configured retrieval engine registry
|
||||
// - Error if initialization fails
|
||||
//
|
||||
// newEnvQdrantClient builds the env-configured qdrant client, refusing a host
|
||||
// whitelist-only mode does not admit before the client exists.
|
||||
//
|
||||
// These two clients are the one outbound path the dial-time guard cannot
|
||||
// cover: gRPC resolves its own target through its dns resolver before calling
|
||||
// any dialer, so by the time a dialer sees the address the name is gone and
|
||||
// the query has already happened. Judging the configured name here is what
|
||||
// keeps a non-whitelisted vector store host from ever being resolved (#3378).
|
||||
func newEnvQdrantClient(host string, port int, apiKey string, useTLS bool) (*qdrant.Client, error) {
|
||||
if err := secutils.CheckSSRFWhitelistOnly(host); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return qdrant.NewClient(&qdrant.Config{Host: host, Port: port, APIKey: apiKey, UseTLS: useTLS})
|
||||
}
|
||||
|
||||
// newEnvMilvusClient is newEnvQdrantClient's twin for the env-configured
|
||||
// milvus client; its address carries the port, which the whitelist never
|
||||
// matches on.
|
||||
func newEnvMilvusClient(ctx context.Context, cfg *milvusclient.ClientConfig) (*milvusclient.Client, error) {
|
||||
host := cfg.Address
|
||||
if h, _, err := net.SplitHostPort(host); err == nil {
|
||||
host = h
|
||||
}
|
||||
if err := secutils.CheckSSRFWhitelistOnly(host); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return milvusclient.New(ctx, cfg)
|
||||
}
|
||||
|
||||
func initRetrieveEngineRegistry(
|
||||
db *gorm.DB, cfg *config.Config, auditSvc interfaces.AuditLogService,
|
||||
storeRepo interfaces.VectorStoreRepository, engineFactory interfaces.EngineFactory,
|
||||
|
|
@ -1391,12 +1422,7 @@ func initRetrieveEngineRegistry(
|
|||
|
||||
log.Infof("Connecting to Qdrant at %s:%d (TLS: %v)", qdrantHost, qdrantPort, qdrantUseTLS)
|
||||
|
||||
client, err := qdrant.NewClient(&qdrant.Config{
|
||||
Host: qdrantHost,
|
||||
Port: qdrantPort,
|
||||
APIKey: qdrantAPIKey,
|
||||
UseTLS: qdrantUseTLS,
|
||||
})
|
||||
client, err := newEnvQdrantClient(qdrantHost, qdrantPort, qdrantAPIKey, qdrantUseTLS)
|
||||
if err != nil {
|
||||
log.Errorf("Create qdrant client failed: %v", err)
|
||||
} else {
|
||||
|
|
@ -1476,7 +1502,7 @@ func initRetrieveEngineRegistry(
|
|||
if milvusDBName != "" {
|
||||
milvusCfg.DBName = milvusDBName
|
||||
}
|
||||
milvusCli, err := milvusclient.New(context.Background(), &milvusCfg)
|
||||
milvusCli, err := newEnvMilvusClient(context.Background(), &milvusCfg)
|
||||
if err != nil {
|
||||
log.Errorf("Create milvus client failed: %v", err)
|
||||
} else {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,57 @@
|
|||
package container
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
milvusclient "github.com/milvus-io/milvus/client/v2/milvusclient"
|
||||
|
||||
secutils "github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
func whitelistOnly(t *testing.T, whitelist, only string) {
|
||||
t.Helper()
|
||||
t.Setenv("SSRF_WHITELIST", whitelist)
|
||||
t.Setenv("SSRF_WHITELIST_EXTRA", "")
|
||||
t.Setenv("SSRF_DNS_WHITELIST_ONLY", only)
|
||||
secutils.ResetSSRFWhitelistForTest()
|
||||
t.Cleanup(secutils.ResetSSRFWhitelistForTest)
|
||||
}
|
||||
|
||||
// gRPC resolves its own target before any dialer runs, so an env-configured
|
||||
// vector store host can only be judged before the client exists (#3378).
|
||||
// Both helpers must refuse without opening a connection.
|
||||
func TestEnvVectorStoreClientsRefuseNonWhitelistedHost(t *testing.T) {
|
||||
whitelistOnly(t, "qdrant,milvus", "true")
|
||||
|
||||
_, err := newEnvQdrantClient("outside.invalid", 6334, "", false)
|
||||
if !errors.Is(err, secutils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("qdrant: expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
|
||||
cfg := &milvusclient.ClientConfig{Address: "outside.invalid:19530"}
|
||||
if _, err := newEnvMilvusClient(context.Background(), cfg); !errors.Is(err, secutils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("milvus: expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
|
||||
// The address carries a port the whitelist never matches on; the host
|
||||
// inside it is what must be compared. Past the name check the real client
|
||||
// tries to connect, so this one gets a deadline of its own.
|
||||
dialCtx, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer cancel()
|
||||
cfg = &milvusclient.ClientConfig{Address: "milvus:19530"}
|
||||
if _, err := newEnvMilvusClient(dialCtx, cfg); errors.Is(err, secutils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("a whitelisted host must pass the name check, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnvVectorStoreClientsIgnoreTheWhitelistWhenTheModeIsOff(t *testing.T) {
|
||||
whitelistOnly(t, "qdrant", "false")
|
||||
|
||||
_, err := newEnvQdrantClient("outside.invalid", 6334, "", false)
|
||||
if errors.Is(err, secutils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off nothing is refused for being off-whitelist, got %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -8,6 +8,7 @@ import (
|
|||
"strings"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/ipclass"
|
||||
"github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
func deriveWebSocketURL(sendMsgURL, allowedHostSuffix string) (string, error) {
|
||||
|
|
@ -68,6 +69,11 @@ func safeDialContext(ctx context.Context, network, address string) (net.Conn, er
|
|||
if err != nil {
|
||||
return nil, fmt.Errorf("split dial address: %w", err)
|
||||
}
|
||||
// Whitelist-only mode refuses a non-whitelisted name before the lookup
|
||||
// (#3378).
|
||||
if err := utils.CheckDialWhitelistOnly(host); err != nil {
|
||||
return nil, fmt.Errorf("endpoint host %q refused: %w", host, err)
|
||||
}
|
||||
addresses, err := net.DefaultResolver.LookupIPAddr(ctx, host)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve endpoint host %q: %w", host, err)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,41 @@
|
|||
package yunzhijia
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
func whitelistOnly(t *testing.T, whitelist, only string) {
|
||||
t.Helper()
|
||||
t.Setenv("SSRF_WHITELIST", whitelist)
|
||||
t.Setenv("SSRF_WHITELIST_EXTRA", "")
|
||||
t.Setenv("SSRF_DNS_WHITELIST_ONLY", only)
|
||||
utils.ResetSSRFWhitelistForTest()
|
||||
t.Cleanup(utils.ResetSSRFWhitelistForTest)
|
||||
}
|
||||
|
||||
// The webhook endpoint is tenant-supplied, so its dialer resolves whatever it
|
||||
// is given; in whitelist-only mode a host outside the whitelist is refused
|
||||
// before that lookup (#3378).
|
||||
func TestSafeDialContextRefusesNonWhitelistedHostBeforeResolving(t *testing.T) {
|
||||
whitelistOnly(t, "yunzhijia.example", "true")
|
||||
|
||||
_, err := safeDialContext(context.Background(), "tcp", "outside.invalid:443")
|
||||
if !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
// An address reaches the endpoint's own public-IP check instead.
|
||||
_, err = safeDialContext(context.Background(), "tcp", "198.51.100.7:443")
|
||||
if errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("an address must be left to the address checks, got %v", err)
|
||||
}
|
||||
|
||||
whitelistOnly(t, "yunzhijia.example", "false")
|
||||
_, err = safeDialContext(context.Background(), "tcp", "outside.invalid:443")
|
||||
if errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off the host is resolved as before, got %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -265,6 +265,11 @@ func (f *Fetcher) pinnedDialContext() func(context.Context, string, string) (net
|
|||
if utils.IsSystemProxy(address) || utils.IsSSRFWhitelisted(host) {
|
||||
return (&net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}).DialContext(ctx, network, address)
|
||||
}
|
||||
// Whitelist-only mode refuses a non-whitelisted name before the
|
||||
// lookup (#3378).
|
||||
if err := utils.CheckDialWhitelistOnly(host); err != nil {
|
||||
return nil, fmt.Errorf("connection blocked: %w", err)
|
||||
}
|
||||
ips, err := f.resolveIPs(ctx, host)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("DNS resolution failed for %s: %w", host, err)
|
||||
|
|
@ -305,6 +310,11 @@ func (f *Fetcher) resolvePinnedTarget(ctx context.Context, rawURL string) (pinne
|
|||
port = "80"
|
||||
}
|
||||
}
|
||||
// Whitelist-only mode refuses a host outside the whitelist before the
|
||||
// lookup (#3378); a whitelisted one is still resolved to pin the render.
|
||||
if err := utils.CheckSSRFWhitelistOnly(parsedURL.Hostname()); err != nil {
|
||||
return pinnedTarget{}, newFetchError(ErrorSSRFRejected, false, "%w", err)
|
||||
}
|
||||
ips, err := f.resolveIPs(ctx, parsedURL.Hostname())
|
||||
if err != nil {
|
||||
return pinnedTarget{}, newFetchError(ErrorDNS, true, "DNS lookup failed for %s: %v", parsedURL.Hostname(), err)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,92 @@
|
|||
package web_fetch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
func whitelistOnly(t *testing.T, whitelist, only string) {
|
||||
t.Helper()
|
||||
t.Setenv("SSRF_WHITELIST", whitelist)
|
||||
t.Setenv("SSRF_WHITELIST_EXTRA", "")
|
||||
t.Setenv("SSRF_DNS_WHITELIST_ONLY", only)
|
||||
utils.ResetSSRFWhitelistForTest()
|
||||
t.Cleanup(utils.ResetSSRFWhitelistForTest)
|
||||
}
|
||||
|
||||
// countingResolver stands in for DNS so a test can prove the lookup never ran.
|
||||
func countingResolver(ips ...string) (func(context.Context, string) ([]net.IP, error), *int) {
|
||||
calls := 0
|
||||
return func(context.Context, string) ([]net.IP, error) {
|
||||
calls++
|
||||
parsed := make([]net.IP, 0, len(ips))
|
||||
for _, ip := range ips {
|
||||
parsed = append(parsed, net.ParseIP(ip))
|
||||
}
|
||||
return parsed, nil
|
||||
}, &calls
|
||||
}
|
||||
|
||||
func TestResolvePinnedTargetRefusesNonWhitelistedHostWithoutDNS(t *testing.T) {
|
||||
whitelistOnly(t, "docs.internal", "true")
|
||||
resolve, calls := countingResolver("93.184.216.34")
|
||||
f := &Fetcher{resolveIPs: resolve}
|
||||
|
||||
_, err := f.resolvePinnedTarget(context.Background(), "https://outside.invalid/page")
|
||||
|
||||
if !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
var fetchErr *FetchError
|
||||
if !errors.As(err, &fetchErr) || fetchErr.Code != ErrorSSRFRejected {
|
||||
t.Fatalf("the refusal must be classified as an SSRF rejection, got %v", err)
|
||||
}
|
||||
if fetchErr.Retryable {
|
||||
t.Fatal("a whitelist refusal is permanent, not retryable")
|
||||
}
|
||||
if *calls != 0 {
|
||||
t.Fatalf("the host must not be resolved at all, got %d lookups", *calls)
|
||||
}
|
||||
}
|
||||
|
||||
// The dialer behind the fetch client is a second entry point into DNS.
|
||||
func TestPinnedDialContextRefusesNonWhitelistedHostWithoutDNS(t *testing.T) {
|
||||
whitelistOnly(t, "docs.internal", "true")
|
||||
resolve, calls := countingResolver("93.184.216.34")
|
||||
f := &Fetcher{resolveIPs: resolve}
|
||||
|
||||
_, err := f.pinnedDialContext()(context.Background(), "tcp", "outside.invalid:443")
|
||||
|
||||
if !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
if *calls != 0 {
|
||||
t.Fatalf("the host must not be resolved at all, got %d lookups", *calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolvePinnedTargetStillResolvesWhitelistedAndOrdinaryHosts(t *testing.T) {
|
||||
whitelistOnly(t, "docs.internal", "true")
|
||||
resolve, calls := countingResolver("93.184.216.34")
|
||||
f := &Fetcher{resolveIPs: resolve}
|
||||
|
||||
// A whitelisted host is still resolved: the render pins one address.
|
||||
if _, err := f.resolvePinnedTarget(context.Background(), "https://docs.internal/page"); err != nil {
|
||||
t.Fatalf("a whitelisted host must be fetchable: %v", err)
|
||||
}
|
||||
if *calls != 1 {
|
||||
t.Fatalf("expected one lookup for the whitelisted host, got %d", *calls)
|
||||
}
|
||||
|
||||
whitelistOnly(t, "docs.internal", "false")
|
||||
if _, err := f.resolvePinnedTarget(context.Background(), "https://outside.invalid/page"); err != nil {
|
||||
t.Fatalf("with the mode off an ordinary public host must still fetch: %v", err)
|
||||
}
|
||||
if *calls != 2 {
|
||||
t.Fatalf("expected the ordinary host to be resolved, got %d lookups", *calls)
|
||||
}
|
||||
}
|
||||
|
|
@ -16,7 +16,6 @@ import (
|
|||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
|
|
@ -220,11 +219,9 @@ func newDockerEngineClient(endpoint dockerEndpoint) (*client.Client, error) {
|
|||
// "allow private endpoints" switch. Must come after WithHost, which
|
||||
// installs its own dialer.
|
||||
if dockerHostNeedsDialGuard(host) {
|
||||
opts = append(opts, client.WithDialContext((&net.Dialer{
|
||||
Timeout: 10 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
Control: SafeDialControlForPolicy(OutboundURLPolicy{AllowPrivate: endpoint.AllowPrivate}),
|
||||
}).DialContext))
|
||||
opts = append(opts, client.WithDialContext(
|
||||
GuardedDialContext(OutboundURLPolicy{AllowPrivate: endpoint.AllowPrivate}),
|
||||
))
|
||||
}
|
||||
|
||||
built, err := client.New(opts...)
|
||||
|
|
|
|||
|
|
@ -157,17 +157,17 @@ func (p *SandboxGatewayTransportPool) dataTransport(target string) http.RoundTri
|
|||
// newGatewayDataTransportWithPolicy dials target regardless of the request's
|
||||
// authority, preserving the gateway rewrite and the configured outbound guard.
|
||||
func newGatewayDataTransportWithPolicy(target string, policy OutboundURLPolicy) *http.Transport {
|
||||
dialer := &net.Dialer{
|
||||
Timeout: 10 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
Control: SafeDialControlForPolicy(policy),
|
||||
}
|
||||
// The same guarded dial as the websocket data plane: in whitelist-only
|
||||
// mode a non-whitelisted gateway hostname is refused before this resolves
|
||||
// it, and the policy's Control hook then judges the address (#3378). The
|
||||
// SDK's authority is ignored — the gateway is addressed by `target`.
|
||||
guardedDial := GuardedDialContext(policy)
|
||||
return &http.Transport{
|
||||
// The gateway is addressed directly; an ambient HTTP proxy would
|
||||
// defeat the rewrite.
|
||||
Proxy: nil,
|
||||
DialContext: func(ctx context.Context, network, _ string) (net.Conn, error) {
|
||||
return dialer.DialContext(ctx, network, target)
|
||||
return guardedDial(ctx, network, target)
|
||||
},
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 4,
|
||||
|
|
|
|||
|
|
@ -27,7 +27,6 @@ import (
|
|||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
|
@ -141,11 +140,7 @@ func NewGuardedTransport() *http.Transport {
|
|||
|
||||
func NewGuardedTransportWithPolicy(policy OutboundURLPolicy) *http.Transport {
|
||||
return &http.Transport{
|
||||
DialContext: (&net.Dialer{
|
||||
Timeout: 10 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
Control: SafeDialControlForPolicy(policy),
|
||||
}).DialContext,
|
||||
DialContext: GuardedDialContext(policy),
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 4,
|
||||
IdleConnTimeout: 90 * time.Second,
|
||||
|
|
|
|||
|
|
@ -21,14 +21,17 @@
|
|||
package sandbox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/ipclass"
|
||||
"github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
// ErrUnsafeOutboundURL is returned for any endpoint that uses an unsupported
|
||||
|
|
@ -93,6 +96,11 @@ func (p OutboundURLPolicy) Validate(raw string) error {
|
|||
if host == "" {
|
||||
return fmt.Errorf("%w: missing host", ErrUnsafeOutboundURL)
|
||||
}
|
||||
// Whitelist-only mode refuses a host outside the whitelist before any
|
||||
// lookup, IP literals and the loopback opt-in included (#3378).
|
||||
if err := utils.CheckSSRFWhitelistOnly(host); err != nil {
|
||||
return fmt.Errorf("%w: %w", ErrUnsafeOutboundURL, err)
|
||||
}
|
||||
// ".local" is mDNS; "localhost" is only acceptable under the opt-in.
|
||||
lower := strings.ToLower(host)
|
||||
if strings.HasSuffix(lower, ".local") {
|
||||
|
|
@ -125,6 +133,28 @@ func (p OutboundURLPolicy) Validate(raw string) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// GuardedDialContext is the DialContext every guarded transport uses: in
|
||||
// whitelist-only mode a non-whitelisted hostname is refused before the dialer
|
||||
// resolves it (#3378), and the policy's Control hook then judges the address
|
||||
// the kernel is about to connect to.
|
||||
func GuardedDialContext(policy OutboundURLPolicy) func(context.Context, string, string) (net.Conn, error) {
|
||||
dialer := &net.Dialer{
|
||||
Timeout: 10 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
Control: SafeDialControlForPolicy(policy),
|
||||
}
|
||||
return func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
host := addr
|
||||
if h, _, err := net.SplitHostPort(addr); err == nil {
|
||||
host = h
|
||||
}
|
||||
if err := utils.CheckDialWhitelistOnly(host); err != nil {
|
||||
return nil, fmt.Errorf("%w: %w", ErrUnsafeOutboundURL, err)
|
||||
}
|
||||
return dialer.DialContext(ctx, network, addr)
|
||||
}
|
||||
}
|
||||
|
||||
// DialControl refuses connections to forbidden addresses. It inspects the
|
||||
// concrete address the kernel is about to connect to, which is what makes it
|
||||
// immune to DNS rebinding.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,106 @@
|
|||
package sandbox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Tencent/WeKnora/internal/utils"
|
||||
)
|
||||
|
||||
func whitelistOnly(t *testing.T, whitelist, only string) {
|
||||
t.Helper()
|
||||
t.Setenv("SSRF_WHITELIST", whitelist)
|
||||
t.Setenv("SSRF_WHITELIST_EXTRA", "")
|
||||
t.Setenv("SSRF_DNS_WHITELIST_ONLY", only)
|
||||
utils.ResetSSRFWhitelistForTest()
|
||||
t.Cleanup(utils.ResetSSRFWhitelistForTest)
|
||||
}
|
||||
|
||||
// In whitelist-only mode a tenant-supplied endpoint outside the whitelist is
|
||||
// refused by name, before Validate resolves it (#3378). The refusal keeps the
|
||||
// package's own error so callers classify it as they always did.
|
||||
func TestPolicyRefusesNonWhitelistedHostBeforeResolving(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal,10.0.0.0/8", "true")
|
||||
|
||||
for _, raw := range []string{
|
||||
"https://outside.invalid/hook",
|
||||
"https://198.51.100.7/hook",
|
||||
"http://localhost:8080/hook",
|
||||
} {
|
||||
err := denyPrivate.Validate(raw)
|
||||
if !errors.Is(err, ErrUnsafeOutboundURL) || !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("%s: expected an unsafe-URL whitelist refusal, got %v", raw, err)
|
||||
}
|
||||
}
|
||||
|
||||
// The loopback opt-in does not reopen what the whitelist closed.
|
||||
if err := allowPrivate.Validate("http://localhost:8080/hook"); !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("AllowPrivate must not bypass the whitelist, got %v", err)
|
||||
}
|
||||
|
||||
// A whitelisted IP literal still goes through the address checks.
|
||||
if err := denyPrivate.Validate("http://10.1.2.3:8080/hook"); errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("a whitelisted host must reach the usual checks, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPolicyLeavesNonWhitelistOnlyDeploymentsAlone(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal", "false")
|
||||
|
||||
if err := denyPrivate.Validate("https://outside.invalid/hook"); errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off nothing is refused for being off-whitelist, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Every guarded transport dials through GuardedDialContext, so a name outside
|
||||
// the whitelist is refused before the dialer resolves it; an address is left
|
||||
// to the policy's own address checks.
|
||||
func TestGuardedDialContextRefusesNonWhitelistedName(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal", "true")
|
||||
dial := GuardedDialContext(denyPrivate)
|
||||
|
||||
_, err := dial(context.Background(), "tcp", "outside.invalid:443")
|
||||
if !errors.Is(err, ErrUnsafeOutboundURL) || !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected an unsafe-URL whitelist refusal, got %v", err)
|
||||
}
|
||||
|
||||
_, err = dial(context.Background(), "tcp", "198.51.100.7:443")
|
||||
if errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("an address must be left to the address checks, got %v", err)
|
||||
}
|
||||
|
||||
whitelistOnly(t, "api.internal", "false")
|
||||
_, err = GuardedDialContext(denyPrivate)(context.Background(), "tcp", "outside.invalid:443")
|
||||
if errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off nothing is refused for being off-whitelist, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// The HTTP data plane dials the gateway the same way the websocket one does,
|
||||
// so whitelist-only mode has to reach it too (#3378).
|
||||
func TestGatewayDataTransportRefusesNonWhitelistedGateway(t *testing.T) {
|
||||
whitelistOnly(t, "gateway.internal", "true")
|
||||
|
||||
transport := newGatewayDataTransportWithPolicy("outside.invalid:443", denyPrivate)
|
||||
_, err := transport.DialContext(context.Background(), "tcp", "ignored.example:443")
|
||||
|
||||
if !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
|
||||
// The SDK's authority is ignored, so a whitelisted authority cannot be
|
||||
// used to smuggle a non-whitelisted gateway past the check.
|
||||
_, err = newGatewayDataTransportWithPolicy("outside.invalid:443", denyPrivate).
|
||||
DialContext(context.Background(), "tcp", "gateway.internal:443")
|
||||
if !errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("the target, not the authority, must be judged; got %v", err)
|
||||
}
|
||||
|
||||
whitelistOnly(t, "gateway.internal", "false")
|
||||
_, err = newGatewayDataTransportWithPolicy("outside.invalid:443", denyPrivate).
|
||||
DialContext(context.Background(), "tcp", "ignored.example:443")
|
||||
if errors.Is(err, utils.ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off nothing is refused for being off-whitelist, got %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -144,22 +144,20 @@ func (d *WebsocketDialer) Dial(
|
|||
}
|
||||
}
|
||||
|
||||
netDialer := &net.Dialer{
|
||||
Timeout: 10 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
Control: SafeDialControlForPolicy(d.policy),
|
||||
}
|
||||
guardedDial := GuardedDialContext(d.policy)
|
||||
dialer := &websocket.Dialer{
|
||||
HandshakeTimeout: 15 * time.Second,
|
||||
// Only "binary" — see the doc comment.
|
||||
Subprotocols: []string{"binary"},
|
||||
NetDialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
|
||||
// Ignore addr and dial the gateway, exactly as
|
||||
// newGatewayDataTransportWithPolicy does for HTTP.
|
||||
// newGatewayDataTransportWithPolicy does for HTTP — both go
|
||||
// through GuardedDialContext, so the two data planes judge the
|
||||
// same name at the same point.
|
||||
if d.target != "" {
|
||||
addr = d.target
|
||||
}
|
||||
return netDialer.DialContext(ctx, network, addr)
|
||||
return guardedDial(ctx, network, addr)
|
||||
},
|
||||
}
|
||||
dialer.NetDialTLSContext = nil // let the TLS handshake use NetDialContext
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package utils
|
|||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"html"
|
||||
"log"
|
||||
|
|
@ -12,6 +13,7 @@ import (
|
|||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
|
|
@ -846,6 +848,11 @@ func SSRFSafeDialContext(ctx context.Context, network, addr string) (net.Conn, e
|
|||
}
|
||||
return dialer.DialContext(ctx, network, addr)
|
||||
}
|
||||
// Whitelist-only mode refuses a non-whitelisted NAME before the lookup
|
||||
// below; an address is left to the checks that follow (#3378).
|
||||
if err := CheckDialWhitelistOnly(host); err != nil {
|
||||
return nil, fmt.Errorf("connection blocked: %w", err)
|
||||
}
|
||||
if restrictedPorts[port] {
|
||||
return nil, fmt.Errorf("connection blocked: port %s is restricted", port)
|
||||
}
|
||||
|
|
@ -1105,7 +1112,39 @@ func mergeSSRFWhitelistRaws(primary, extra string) string {
|
|||
// IsSSRFWhitelisted checks whether the given hostname (or IP string) is
|
||||
// covered by the SSRF_WHITELIST environment variable.
|
||||
func IsSSRFWhitelisted(hostname string) bool {
|
||||
// One read of the whitelist for the whole decision: SetSSRFWhitelistFromRaw
|
||||
// can swap it between two loads, which would mix generations.
|
||||
wl := loadSSRFWhitelist()
|
||||
if whitelistedByName(wl, hostname) {
|
||||
return true
|
||||
}
|
||||
// Also resolve and check resolved IPs against CIDR whitelist — never in
|
||||
// whitelist-only mode, where a hostname outside the whitelist must not
|
||||
// reach DNS at all.
|
||||
if wl == nil || len(wl.cidrNets) == 0 || net.ParseIP(hostname) != nil || SSRFWhitelistOnlyEnabled() {
|
||||
return false
|
||||
}
|
||||
if ips, err := net.LookupIP(hostname); err == nil {
|
||||
for _, ip := range ips {
|
||||
for _, cidr := range wl.cidrNets {
|
||||
if cidr.Contains(ip) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// isSSRFWhitelistedByName matches a host against the whitelist without any
|
||||
// DNS query: exact and wildcard names, and CIDR entries for an IP literal.
|
||||
func isSSRFWhitelistedByName(hostname string) bool {
|
||||
return whitelistedByName(loadSSRFWhitelist(), hostname)
|
||||
}
|
||||
|
||||
// whitelistedByName is isSSRFWhitelistedByName against an already-loaded
|
||||
// whitelist.
|
||||
func whitelistedByName(wl *ssrfWhitelistConfig, hostname string) bool {
|
||||
if wl == nil {
|
||||
return false
|
||||
}
|
||||
|
|
@ -1131,23 +1170,58 @@ func IsSSRFWhitelisted(hostname string) bool {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Also resolve and check resolved IPs against CIDR whitelist
|
||||
if net.ParseIP(hostname) == nil && len(wl.cidrNets) > 0 {
|
||||
if ips, err := net.LookupIP(hostname); err == nil {
|
||||
for _, ip := range ips {
|
||||
for _, cidr := range wl.cidrNets {
|
||||
if cidr.Contains(ip) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// SSRFWhitelistOnlyEnabled reports whether SSRF_DNS_WHITELIST_ONLY turns the
|
||||
// whitelist into the whole egress policy: a host outside it is refused before
|
||||
// any DNS query, and a hostname counts as whitelisted by name only, never by
|
||||
// what it resolves to (#3378).
|
||||
//
|
||||
// The value is parsed with strconv.ParseBool, so 1/t/T/TRUE/true/True enable
|
||||
// it and 0/f/F/FALSE/false/False disable it. A non-empty value that parses as
|
||||
// neither enables it: silently leaving an egress lockdown off because an
|
||||
// operator wrote "yes" is the one failure mode this control must not have.
|
||||
func SSRFWhitelistOnlyEnabled() bool {
|
||||
raw := strings.TrimSpace(os.Getenv("SSRF_DNS_WHITELIST_ONLY"))
|
||||
if raw == "" {
|
||||
return false
|
||||
}
|
||||
parsed, err := strconv.ParseBool(raw)
|
||||
return err != nil || parsed
|
||||
}
|
||||
|
||||
// ErrSSRFHostNotWhitelisted is what whitelist-only mode returns for a host
|
||||
// outside SSRF_WHITELIST. It is raised before the name is resolved.
|
||||
var ErrSSRFHostNotWhitelisted = errors.New("host is not in the SSRF whitelist")
|
||||
|
||||
// CheckSSRFWhitelistOnly refuses a host that whitelist-only mode does not
|
||||
// admit. Paths that still hold the URL run it before resolving a name, so a
|
||||
// host outside the whitelist never produces a DNS query; with the mode off it
|
||||
// admits everything and the usual checks apply. An IP literal is judged here
|
||||
// too: the URL layer is where "no direct IP" belongs.
|
||||
func CheckSSRFWhitelistOnly(host string) error {
|
||||
if !SSRFWhitelistOnlyEnabled() || isSSRFWhitelistedByName(host) {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("%w: %s (SSRF_DNS_WHITELIST_ONLY is on; add it to SSRF_WHITELIST to allow it)",
|
||||
ErrSSRFHostNotWhitelisted, host)
|
||||
}
|
||||
|
||||
// CheckDialWhitelistOnly is CheckSSRFWhitelistOnly for a dial address, and it
|
||||
// judges hostnames only. An IP literal arriving at a dialer is either a name
|
||||
// something else already resolved — gRPC's own resolver runs before the
|
||||
// dialer it was given, so a whitelisted service name reaches this point as an
|
||||
// address — or a literal the URL layer already refused. Refusing it again
|
||||
// here would break clients whose whitelisted host was resolved for them,
|
||||
// while adding nothing the URL layer does not already do.
|
||||
func CheckDialWhitelistOnly(host string) error {
|
||||
if net.ParseIP(host) != nil {
|
||||
return nil
|
||||
}
|
||||
return CheckSSRFWhitelistOnly(host)
|
||||
}
|
||||
|
||||
// ResetSSRFWhitelistForTest resets the whitelist singleton so tests in any
|
||||
// package can re-read the SSRF_WHITELIST environment variable after changing
|
||||
// it. Exported (rather than unexported) because callers exist outside
|
||||
|
|
@ -1182,6 +1256,15 @@ func FormatSSRFError(label, rawURL string, err error) string {
|
|||
if parsed, perr := parseHostForHint(rawURL); perr == nil && parsed != "" {
|
||||
host = parsed
|
||||
}
|
||||
if errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
return fmt.Sprintf(
|
||||
"%s 未通过安全校验:服务端开启了「仅允许白名单出站」(SSRF_DNS_WHITELIST_ONLY),"+
|
||||
"%s 不在白名单内,因此在解析域名之前就被拒绝。如该地址确实可信,请联系运维把它加入 "+
|
||||
"SSRF_WHITELIST_EXTRA(支持精确域名 / *.example.com 通配 / IP / CIDR),"+
|
||||
"示例:SSRF_WHITELIST_EXTRA=%s,*.example.com,10.0.0.0/8",
|
||||
label, host, host,
|
||||
)
|
||||
}
|
||||
return fmt.Sprintf(
|
||||
"%s 未通过安全校验:%v。如该地址确实可信,请联系运维在服务端环境变量 "+
|
||||
"SSRF_WHITELIST_EXTRA 中加入该主机(支持精确域名 / *.example.com 通配 / IP / CIDR),"+
|
||||
|
|
@ -1250,6 +1333,12 @@ func ValidateURLForSSRF(rawURL string) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// Whitelist-only mode stops here: the validation below resolves the name,
|
||||
// and a host outside the whitelist must not reach DNS (#3378).
|
||||
if err := CheckSSRFWhitelistOnly(hostname); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Delegate to the full SSRF validation (uses the normalised URL).
|
||||
if safe, reason := isSSRFSafeURL(normalized); !safe {
|
||||
return fmt.Errorf("SSRF validation failed: %s", reason)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,179 @@
|
|||
package utils
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// whitelistOnly configures the whitelist and the whitelist-only switch for
|
||||
// one test and restores the cached whitelist afterwards.
|
||||
func whitelistOnly(t *testing.T, whitelist, only string) {
|
||||
t.Helper()
|
||||
t.Setenv("SSRF_WHITELIST", whitelist)
|
||||
t.Setenv("SSRF_WHITELIST_EXTRA", "")
|
||||
t.Setenv("SSRF_DNS_WHITELIST_ONLY", only)
|
||||
ResetSSRFWhitelistForTest()
|
||||
t.Cleanup(ResetSSRFWhitelistForTest)
|
||||
}
|
||||
|
||||
func TestCheckSSRFWhitelistOnly(t *testing.T) {
|
||||
const whitelist = "api.internal,*.corp.example,10.0.0.0/8,203.0.113.5"
|
||||
cases := []struct {
|
||||
name string
|
||||
only string
|
||||
host string
|
||||
refused bool
|
||||
}{
|
||||
{"off: any domain passes", "false", "outside.invalid", false},
|
||||
{"off: any IP passes", "false", "198.51.100.7", false},
|
||||
{"unset counts as off", "", "outside.invalid", false},
|
||||
{"on: exact name", "true", "api.internal", false},
|
||||
{"on: exact name, any case", "true", "API.Internal", false},
|
||||
{"on: wildcard suffix", "true", "svc.corp.example", false},
|
||||
{"on: wildcard root", "true", "corp.example", false},
|
||||
{"on: IP in CIDR", "true", "10.20.30.40", false},
|
||||
{"on: exact IP", "true", "203.0.113.5", false},
|
||||
{"on: other domain refused", "true", "outside.invalid", true},
|
||||
{"on: other IP refused", "true", "198.51.100.7", true},
|
||||
{"on: loopback refused unless listed", "true", "127.0.0.1", true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
whitelistOnly(t, whitelist, tc.only)
|
||||
err := CheckSSRFWhitelistOnly(tc.host)
|
||||
if tc.refused {
|
||||
if !errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected ErrSSRFHostNotWhitelisted, got %v", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), tc.host) {
|
||||
t.Fatalf("error should name the host: %v", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The switch is a security control, so a value that is neither true nor false
|
||||
// must not silently leave it off.
|
||||
func TestSSRFWhitelistOnlyEnabledParsesLikeTheRestOfTheRepo(t *testing.T) {
|
||||
cases := map[string]bool{
|
||||
"": false,
|
||||
"false": false,
|
||||
"FALSE": false,
|
||||
"0": false,
|
||||
"f": false,
|
||||
"true": true,
|
||||
"TRUE": true,
|
||||
"True": true,
|
||||
"1": true,
|
||||
"t": true,
|
||||
" true ": true,
|
||||
"yes": true, // unparsable: fail closed rather than silently off
|
||||
"on": true,
|
||||
"enabled": true,
|
||||
}
|
||||
for raw, want := range cases {
|
||||
t.Run("value="+raw, func(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal", raw)
|
||||
if got := SSRFWhitelistOnlyEnabled(); got != want {
|
||||
t.Fatalf("SSRF_DNS_WHITELIST_ONLY=%q: enabled=%v, want %v", raw, got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A dialer sees an address, not always a name: gRPC resolves its own target
|
||||
// before calling the dialer it was given, so a whitelisted service name
|
||||
// arrives as an IP. Judging names is the dialer's job; judging IP literals is
|
||||
// the URL layer's.
|
||||
func TestCheckDialWhitelistOnlyJudgesNamesOnly(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal", "true")
|
||||
|
||||
if err := CheckDialWhitelistOnly("outside.invalid"); !errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("a non-whitelisted name must be refused, got %v", err)
|
||||
}
|
||||
if err := CheckDialWhitelistOnly("api.internal"); err != nil {
|
||||
t.Fatalf("a whitelisted name must pass, got %v", err)
|
||||
}
|
||||
for _, ip := range []string{"198.51.100.7", "10.20.30.40", "2001:db8::1"} {
|
||||
if err := CheckDialWhitelistOnly(ip); err != nil {
|
||||
t.Fatalf("%s: an address must be left to the address checks, got %v", ip, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// With the mode on the refusal comes from the whitelist, before the name is
|
||||
// ever resolved; with it off the host takes the ordinary path.
|
||||
func TestValidateURLForSSRF_WhitelistOnlyRefusesBeforeDNS(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal", "true")
|
||||
err := ValidateURLForSSRF("https://outside.invalid/v1")
|
||||
if !errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "DNS resolution failed") {
|
||||
t.Fatalf("the refusal must come before DNS: %v", err)
|
||||
}
|
||||
if err := ValidateURLForSSRF("https://api.internal/v1"); err != nil {
|
||||
t.Fatalf("a whitelisted host must pass: %v", err)
|
||||
}
|
||||
|
||||
whitelistOnly(t, "api.internal", "false")
|
||||
if err := ValidateURLForSSRF("https://outside.invalid/v1"); errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off nothing is refused for being off-whitelist, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSSRFSafeDialContext_WhitelistOnlyRefusesNamesBeforeDNS(t *testing.T) {
|
||||
whitelistOnly(t, "api.internal", "true")
|
||||
_, err := SSRFSafeDialContext(context.Background(), "tcp", "outside.invalid:443")
|
||||
if !errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("expected the whitelist refusal, got %v", err)
|
||||
}
|
||||
// An address reaches the restricted-IP checks instead of this one.
|
||||
_, err = SSRFSafeDialContext(context.Background(), "tcp", "198.51.100.7:443")
|
||||
if errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("an address must be left to the address checks, got %v", err)
|
||||
}
|
||||
|
||||
whitelistOnly(t, "api.internal", "false")
|
||||
_, err = SSRFSafeDialContext(context.Background(), "tcp", "outside.invalid:443")
|
||||
if errors.Is(err, ErrSSRFHostNotWhitelisted) {
|
||||
t.Fatalf("with the mode off the name is resolved as before, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// With CIDR entries the whitelist resolves a hostname to see whether its
|
||||
// addresses fall inside; whitelist-only mode matches by name only, because
|
||||
// that lookup is the query the mode exists to prevent.
|
||||
func TestIsSSRFWhitelisted_WhitelistOnlyMatchesByNameOnly(t *testing.T) {
|
||||
ips, err := net.LookupIP("localhost")
|
||||
loopback := false
|
||||
for _, ip := range ips {
|
||||
if ip.IsLoopback() {
|
||||
loopback = true
|
||||
}
|
||||
}
|
||||
if err != nil || !loopback {
|
||||
t.Skip("localhost does not resolve to a loopback address here; the CIDR fallback cannot be exercised")
|
||||
}
|
||||
|
||||
whitelistOnly(t, "127.0.0.0/8,::1/128,api.internal", "false")
|
||||
if !IsSSRFWhitelisted("localhost") {
|
||||
t.Fatal("with the mode off a hostname is matched by what it resolves to")
|
||||
}
|
||||
|
||||
whitelistOnly(t, "127.0.0.0/8,::1/128,api.internal", "true")
|
||||
if IsSSRFWhitelisted("localhost") {
|
||||
t.Fatal("whitelist-only mode must match by name, never by resolved address")
|
||||
}
|
||||
if !IsSSRFWhitelisted("api.internal") || !IsSSRFWhitelisted("127.0.0.1") {
|
||||
t.Fatal("name and IP-literal CIDR entries must still match")
|
||||
}
|
||||
}
|
||||
|
|
@ -235,9 +235,25 @@ AWS S3 的 `S3_ACCESS_KEY` / `S3_SECRET_KEY` 可以**同时留空**,此时走
|
|||
| `WEKNORA_AUDIT_RETENTION_DAYS` | 90 | 审计日志保留天数 |
|
||||
| `WEKNORA_BOOTSTRAP_SYSTEM_ADMIN_EMAIL` | 空 | 引导第一个系统管理员。**不会创建用户**:该邮箱需先自行注册,下次启动时若部署内还没有任何系统管理员,才把它提升;已有管理员后本变量不再生效。详见[租户、用户与认证授权](../03-features/01-tenant-auth.md) |
|
||||
| `OIDC_AUTH_ENABLE` 及 `OIDC_AUTH_*` / `OIDC_USER_INFO_MAPPING_*` | false / 空 | OIDC 单点登录全套配置 |
|
||||
| `SSRF_WHITELIST` / `SSRF_WHITELIST_EXTRA` | 空 / `searxng,qdrant,milvus,weaviate,doris-fe,doris-be` | 出站请求 SSRF 白名单(app 与 docreader 共用) |
|
||||
| `SSRF_WHITELIST` / `SSRF_WHITELIST_EXTRA` | 空 / `searxng,qdrant,milvus,weaviate,doris-fe,doris-be,minio`(仅 app) | 出站请求 SSRF 白名单。`SSRF_WHITELIST` 为 app 与 docreader 共用;compose 只给 app 的 `SSRF_WHITELIST_EXTRA` 设了默认值,docreader 的同名变量默认为空 |
|
||||
| `SSRF_DNS_WHITELIST_ONLY` | false | 仅允许白名单出站。开启后,不在白名单的主机在 **DNS 查询前**即被拒绝,URL 校验处连 IP 直连也一并拒绝;域名只按名字匹配,写在白名单里的 CIDR 不再对域名生效。取值按布尔解析(`1/t/true` 开、`0/f/false` 关),**非空且无法解析的取值按「开」处理**。开启前的准备见下文 |
|
||||
| `IMAGE_HOST_KEEP_URL` | 空 | 保留原始 URL 的图片域名白名单 |
|
||||
|
||||
#### 开启 `SSRF_DNS_WHITELIST_ONLY` 之前
|
||||
|
||||
开启后白名单就是全部出站策略,需要先把所有出站地址写进 `SSRF_WHITELIST` 或 `SSRF_WHITELIST_EXTRA`。docreader 也要访问 compose 内的主机时,请写进两个服务共用的 `SSRF_WHITELIST`(它的 `SSRF_WHITELIST_EXTRA` 默认为空)。通常还要补上:
|
||||
|
||||
- 模型服务地址(chat / embedding / rerank / VLM / ASR,含本机 Ollama 的 `localhost`)
|
||||
- OIDC 登录的 `dex`(或你的 IdP 域名)、MCP 服务地址、`docreader`
|
||||
- 对象存储(外部 S3/COS/OSS 等)、外部向量库、Langfuse 地址
|
||||
- 沙箱控制面地址:开启后「允许私网端点」不再能绕过白名单
|
||||
|
||||
仍未覆盖的出站路径,按影响排序:
|
||||
|
||||
1. **gRPC 向量库的运行时解析**:qdrant / milvus 客户端由 gRPC 自己的 resolver 解析 target,拨号器拿到的已是地址,因此这类主机是在**建客户端之前按名字**判断的(环境变量配置在启动时判断,控制台保存的配置走 URL 校验),而不是每次连接前。
|
||||
2. **Langfuse 的 OTLP 导出器**自带 HTTP 客户端,完全不走本机制。`LANGFUSE_HOST` 默认是 SaaS 地址,离线部署请关闭追踪或改成内网地址。
|
||||
3. **`HTTP(S)_PROXY`**:拨号器对代理主机的放行条件是「拨号地址与代理 URL 的 host 完全相等」。相等时代理主机即使不在白名单也会被解析和连接;不相等时(例如代理 URL 没写端口)会被当成非白名单直接拒绝。离线部署请 unset 代理,或把代理主机一并写进白名单。
|
||||
|
||||
### Docreader 解析(docreader 容器)
|
||||
|
||||
| 名称 | 默认值 | 说明 |
|
||||
|
|
|
|||
Loading…
Reference in New Issue