Networking Layer

Understanding AvalancheGo's P2P networking, peer management, and message protocols.

AvalancheGo uses a custom peer-to-peer (P2P) networking layer designed for high-throughput consensus messaging. This page covers the network architecture, peer management, and message protocols.

Network Interface

The core network interface (network/network.go) handles all P2P operations:

network/network.go

type Network interface {
    // Message sending (from consensus)
    sender.ExternalSender

    // Health monitoring
    health.Checker

    // Peer management
    peer.Network

    // Lifecycle
    StartClose()
    Dispatch() error

    // Manual peer tracking
    ManuallyTrack(nodeID ids.NodeID, ip netip.AddrPort)

    // Peer information
    PeerInfo(nodeIDs []ids.NodeID) []peer.Info

    // Uptime tracking
    NodeUptime() (UptimeResult, error)
}

Peer Discovery

AvalancheGo discovers peers through multiple mechanisms:

Initial connections to known bootstrap nodes are configured per-network (sampled from genesis) and can be overridden with --bootstrap-ips/--bootstrap-ids. See config/config.go:getBootstrapConfig.

Peer Exchange

Nodes share known peers with each other:

IP Tracking

The network maintains IP information for reconnection:

network/ip_tracker.go

type ipTracker struct {
    // Tracked nodes and their most recent signed IPs
    tracked map[ids.NodeID]*trackedNode

    // Bloom filter of the most recent tracked IPs, for efficient gossip
    bloom *bloom.Filter
}

Connection Lifecycle

Establishing Connections

TLS Authentication

All connections use mutual TLS with staking certificates:

// Each node has a staking keypair
type Node struct {
    StakingTLSSigner crypto.Signer
    StakingTLSCert   *staking.Certificate
    ID               ids.NodeID  // Derived from certificate
}

Node ID Derivation:

// NodeID is derived from the staking certificate
nodeID := ids.NodeIDFromCert(stakingCert)

Peer States

Message Protocol

Message Types

Messages are defined using Protocol Buffers (proto/p2p/p2p.proto):

proto/p2p/p2p.proto

message Message {
  reserved 1;
  oneof message {
    // Optional compression for supported message types
    bytes compressed_zstd = 2;

    // Handshake & peering
    Ping ping = 11;
    Pong pong = 12;
    Handshake handshake = 13;
    GetPeerList get_peer_list = 35;
    PeerList peer_list = 14;

    // State sync
    GetStateSummaryFrontier get_state_summary_frontier = 15;
    StateSummaryFrontier state_summary_frontier = 16;
    GetAcceptedStateSummary get_accepted_state_summary = 17;
    AcceptedStateSummary accepted_state_summary = 18;

    // Bootstrapping
    GetAcceptedFrontier get_accepted_frontier = 19;
    AcceptedFrontier accepted_frontier = 20;
    GetAccepted get_accepted = 21;
    Accepted accepted = 22;
    GetAncestors get_ancestors = 23;
    Ancestors ancestors = 24;

    // Consensus
    Get get = 25;
    Put put = 26;
    PushQuery push_query = 27;
    PullQuery pull_query = 28;
    Chits chits = 29;

    // Application-level
    AppRequest app_request = 30;
    AppResponse app_response = 31;
    AppGossip app_gossip = 32;
    AppError app_error = 34;

    // Simplex consensus (experimental)
    Simplex simplex = 36;
  }
}

Consensus Messages

Message	Purpose
`PushQuery`	Send block and request vote
`PullQuery`	Request vote without sending block
`Chits`	Vote response with preferences
`Get`	Request a specific block
`Put`	Send a requested block
`GetAcceptedFrontier` / `AcceptedFrontier`	Bootstrap frontier exchange
`GetAccepted` / `Accepted`	Request/return accepted containers for heights
`GetAncestors` / `Ancestors`	Fetch a container and its ancestors
`GetStateSummaryFrontier` / `StateSummaryFrontier`	State sync frontier
`GetAcceptedStateSummary` / `AcceptedStateSummary`	State summaries at heights

Application Messages

VMs send custom messages through the common.AppSender provided at initialization:

type AppSender interface {
    SendAppRequest(ctx context.Context, nodeIDs set.Set[ids.NodeID],
        requestID uint32, appRequestBytes []byte) error
    SendAppResponse(ctx context.Context, nodeID ids.NodeID,
        requestID uint32, appResponseBytes []byte) error
    SendAppError(ctx context.Context, nodeID ids.NodeID,
        requestID uint32, errorCode int32, errorMessage string) error
    SendAppGossip(ctx context.Context, config common.SendConfig,
        appGossipBytes []byte) error
}

Inbound app traffic is delivered to the VM's AppRequest/AppResponse/AppGossip handlers via common.AppHandler.

Message Routing

The router (snow/networking/router) directs messages to appropriate handlers:

snow/networking/router/router.go

type Router interface {
    // Route messages to chains
    HandleInbound(ctx context.Context, msg message.InboundMessage)

    // Register chain handlers
    AddChain(ctx context.Context, chain handler.Handler)

    // Health checking
    health.Checker
}

Chain Message Handler

snow/networking/handler/handler.go

type Handler interface {
    // Queue a message for the chain's consensus engine
    Push(ctx context.Context, msg Message)

    // Lifecycle
    Start(ctx context.Context, recoverPanic bool)
    Stop(ctx context.Context)
}

Throttling & Rate Limiting

Inbound Throttling

Protects nodes from message floods (network/throttling):

network/throttling/inbound_msg_throttler.go

type InboundMsgThrottler interface {
    // Block until the node can read a message of msgSize bytes from nodeID
    Acquire(ctx context.Context, msgSize uint64, nodeID ids.NodeID) ReleaseFunc
}

Throttling Dimensions:

Bytes: Bytes of messages from a peer that are being processed
Message count: Messages from a peer that are being processed
Bandwidth: Bytes per second per peer (token bucket)
CPU time: CPU usage caused by a peer
Disk: Disk usage caused by a peer

Outbound Throttling

Prevents overwhelming peers:

network/throttling/outbound_msg_throttler.go

type OutboundMsgThrottler interface {
    // Returns false if the message must not be sent
    Acquire(msg *message.OutboundMessage, nodeID ids.NodeID) bool

    // Release the bytes that Acquire reserved
    Release(msg *message.OutboundMessage, nodeID ids.NodeID)
}

Connection Throttling

Limits connection attempts:

type InboundConnUpgradeThrottler interface {
    // Check if connection upgrade should proceed
    ShouldUpgrade(ip netip.AddrPort) bool
}

Peer Scoring

Nodes track peer behavior for connection prioritization:

type PeerInfo struct {
    IP             netip.AddrPort
    PublicIP       netip.AddrPort
    ID             ids.NodeID
    Version        string
    LastSent       time.Time
    LastReceived   time.Time
    ObservedUptime uint32
    TrackedSubnets []ids.ID
}

Benchlisting

Misbehaving peers are temporarily blacklisted (snow/networking/benchlist):

snow/networking/benchlist/benchlist.go

type Benchlist interface {
    // Add peer to benchlist
    RegisterFailure(validatorID ids.NodeID)

    // Check if peer is benched
    IsBenched(validatorID ids.NodeID) bool
}

Benchlist Triggers:

Request timeouts (AppRequest timeouts do not count). The node keeps a decaying average of request failures for each validator (--benchlist-halflife, default 1m).
The node benches a validator when the average is above --benchlist-bench-probability (default 0.5).
It unbenches the validator when the average is below --benchlist-unbench-probability (default 0.2), or after --benchlist-duration (default 5m).

Health Monitoring

Network health is exposed via the health API:

type UptimeResult struct {
    // Percent of stake that sees us as meeting uptime requirement
    RewardingStakePercentage float64

    // Weighted average of observed uptimes
    WeightedAveragePercentage float64
}

Health Metrics

const (
    ConnectedPeersKey          = "connectedPeers"
    TimeSinceLastMsgReceivedKey = "timeSinceLastMsgReceived"
    TimeSinceLastMsgSentKey     = "timeSinceLastMsgSent"
    SendFailRateKey            = "sendFailRate"
)

Example Health Response:

{
  "healthy": true,
  "checks": {
    "network": {
      "message": {
        "connectedPeers": 45,
        "timeSinceLastMsgReceived": "1.2s",
        "timeSinceLastMsgSent": "0.8s",
        "sendFailRate": 0.001
      }
    }
  }
}

Network Configuration

Key configuration options:

network/config.go

type Config struct {
    HealthConfig         // send fail rate, peer and message timing checks
    PeerListGossipConfig // PeerListNumValidatorIPs, PeerListPullGossipFreq, PeerListBloomResetFreq
    TimeoutConfig        // PingPongTimeout, ReadHandshakeTimeout
    DelayConfig          // InitialReconnectDelay, MaxReconnectDelay
    ThrottlerConfig      ThrottlerConfig // inbound/outbound message and connection throttlers

    PingFrequency   time.Duration
    AllowPrivateIPs bool
    TrackedSubnets  set.Set[ids.ID]
    // ...
}

Command Line Flags

# Connection settings
--network-max-reconnect-delay=1m
--network-initial-reconnect-delay=1s
--network-peer-list-pull-gossip-frequency=2s

# Throttling
--throttler-inbound-at-large-alloc-size=6291456
--throttler-outbound-at-large-alloc-size=33554432

Subnet Networking

Validators can participate in multiple subnets:

// network.Config (network/config.go)
// TrackedSubnets of the node, from --track-subnets.
// It must not include the Primary Network ID.
TrackedSubnets set.Set[ids.ID]

Subnet-Specific Peers:

Handshake carries tracked subnet IDs (and all_subnets flag)
Nodes connect preferentially to same-subnet validators
Gossip is scoped to relevant subnets
Cross-subnet communication uses explicit routing

Debugging Network Issues

Common Issues

Symptom	Possible Cause	Solution
No peers	Firewall blocking 9651	Open port 9651
High latency	Geographic distance	Add regional bootstrappers
Disconnections	Rate limiting	Increase throttle limits
Benchmark failures	Peer misbehavior	Check peer logs

Useful Endpoints

# Get connected peers
curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"info.peers"
}' -H 'content-type:application/json;' localhost:9650/ext/info

# Get network health
curl localhost:9650/ext/health

Network Overview