The operational paradigm of enterprise network monitoring has undergone a radical transformation. For more than three decades, network health was assessed through periodic polling using the Simple Network Management Protocol (SNMP). While SNMPv3 introduced User-Based Security (USM) and View-Based Access Control (VACM) to encrypt credentials and MIB payloads, pull-based 5-minute polling cycles are fundamentally incapable of capturing sub-second microbursts, transient interface buffer exhaustion, and ephemeral BGP route dampening. Modern hyperscale architectures demand true Network Observability: moving from reactive polling to push-based gRPC/gNMI streaming telemetry, wire-rate flow export via NetFlow v9 and IPFIX (RFC 7011), deep in-kernel tracing with eBPF, and lossless hardware TAP aggregation. In this architectural monograph, we analyze the telemetry pipeline required to achieve full-stack packet-to-application visibility.
Enterprise Network Telemetry Baseline
Engineered by the Enterprise Network & Cloud Infrastructure Editorial Board. Architecture baselines: RFC 7011 (IPFIX Information Model), RFC 3414 (SNMPv3 USM), gRPC Network Management Interface (gNMI OpenConfig v0.8), and Linux eBPF/BCC Observability Hooks. Target: Sub-second anomaly detection with line-rate flow sampling across 100G/400G spine-leaf fabrics.
1. The Limitations of Legacy SNMP: Pull-Based Polling vs. Push Telemetry
SNMP operates on an imperative pull model where a centralized Network Management Station (NMS) periodically transmits UDP requests to query specific Management Information Base (MIB) Object Identifiers (OIDs) on target switches. This architecture suffers from severe systemic limitations:
- Temporal Blind Spots: Polling intervals are typically configured for 1 to 5 minutes to prevent overloading the control plane CPU of network switches. Any microburst lasting 50 milliseconds that exhausts switch buffer memory and drops critical VoIP or database packets is invisible to SNMP averages.
- Control Plane CPU Saturation: When an NMS walks hundreds of interface MIB tables on a 128-port spine chassis, the switch CPU must process SNMP encoding and serialization, competing directly with BGP and OSPF keepalive processes.
- Streaming Telemetry (gNMI / OpenConfig): Modern switches utilize gRPC Network Management Interface (gNMI) to stream telemetry continuously. The switch ASIC directly streams hardware counter updates (e.g., interface byte counts, dropped packet queues, optical laser temperatures) over an HTTP/2 gRPC session using Protocol Buffers (protobuf) at cadence intervals as fast as 100 milliseconds.
2. Flow Telemetry: NetFlow v9, IPFIX (RFC 7011) & sFlow Mechanics
While streaming telemetry reports interface throughput and packet queue drops, it provides zero insight into who is talking to whom. Flow telemetry decodes traffic composition without capturing full packet payloads:
- NetFlow v9 & IPFIX (RFC 7011): Flow-based state tracking. The switch ASIC monitors incoming packets and identifies unique communication flows based on a 7-tuple key (Source IP, Destination IP, Source Port, Destination Port, Protocol, Ingress Interface, IP ToS/DSCP). When a flow expires (inactivity timeout or active 60-second timer), an IPFIX template-based export packet is sent to the telemetry collector. IPFIX extends NetFlow with custom variable fields (e.g., BGP Next-Hop, AS-Path, VXLAN VNI, TCP flags).
- sFlow (RFC 3176): Packet sampling technology implemented purely in switch silicon. The ASIC copies 1 out of every $N$ packets (e.g., 1:1000 or 1:4096 sampling rate) along with the first 128 bytes of the packet header and emits it immediately via UDP without maintaining state tables in memory, ensuring zero impact on switch CPU even during heavy DDoS saturation.
3. Kernel-Level Visibility: eBPF for Network & Host Observability
In containerized cloud-native environments (Kubernetes, AWS EKS), traffic between microservices on the same virtual host never traverses a physical top-of-rack switch, blinding traditional NetFlow probes. Extended Berkeley Packet Filter (eBPF) provides in-kernel network visibility:
eBPF In-Kernel Network Telemetry Hooks
By attaching lightweight sandboxed C programs to kernel tracepoints (such as sock:sock_ops, kprobe:tcp_v4_connect, and tc traffic control filters), eBPF instruments every TCP socket state transition at the kernel layer. It measures microsecond TCP round-trip times (RTT), detects TCP window stalls, records connection retransmissions, and maps Layer 7 HTTP/gRPC request-response latencies directly to Kubernetes pod namespaces without injecting sidecar proxies.
4. Lossless Packet Analysis: TAP Aggregation & NPBs
When investigating forensic cybersecurity breaches, complex application performance anomalies, or financial trading errors, flow summaries are insufficient; engineers require raw, uncompressed PCAP packet traces. Traditional Switch Port Analyzer (SPAN / Port Mirroring) drops mirrored packets when the mirror destination port becomes congested, corrupting packet capture timing.
Modern enterprise data centers deploy dedicated Network Packet Brokers (NPBs) and TAP Aggregation fabrics (using Arista DANZ or Gigamon). Passive optical TAPs split physical fiber signals at the physical layer (e.g., 70/30 optical splitters), routing a permanent copy of photons directly into an isolated TAP aggregator switch. The NPB hardware performs hardware-accelerated packet de-duplication, payload slicing (stripping payload bytes for GDPR/PCI compliance while preserving IP headers), and timestamping at nanosecond precision before forwarding to intrusion detection systems and packet capture buffers.
5. Telemetry Architecture Pipeline: From Edge to OpenTelemetry
| Telemetry Tier | Protocol / Standard | Data Frequency | CPU / Performance Impact | Primary Operational Insight |
|---|---|---|---|---|
| SNMPv3 (USM) | Pull UDP/161 (MIB polling) | 1 to 5 minutes | Medium (Control plane polling overhead) | Legacy device health, interface admin status, power supplies. |
| Streaming Telemetry | Push gRPC/gNMI (Protobuf) | 100ms to 10 seconds | Negligible (Hardware ASIC streamed) | Microburst detection, queue buffer depths, optical RX/TX levels. |
| IPFIX / NetFlow v9 | Stateful Push UDP/2055 | 30 to 60 second flushes | Low (ASIC flow table tracking) | Bandwidth attribution, top talkers, WAN utilization, billing. |
| sFlow (RFC 3176) | Stateless Packet Sampling | Real-time (1:1000 sample) | Zero (Stateless ASIC copy) | DDoS volumetric detection, transit AS traffic matrix profiling. |
| eBPF Host Tracing | Kernel Ring Buffer hooks | Event-driven (per-socket) | Sub-1% host CPU overhead | Microservice east-west latency, TCP socket stalls, pod-to-pod metrics. |
| TAP Aggregation | Optical Splitting / NPB | Continuous 100% Wire-Rate | Zero (Physical passive splitting) | Forensic packet capture, TLS handshake timing, IDS inspection. |
6. Production Switch Configuration: Arista EOS gNMI & IPFIX Export
Below is a production configuration enabling gNMI streaming telemetry and hardware IPFIX flow export on an Arista EOS enterprise leaf switch:
! ==============================================================================
! Arista EOS Streaming Telemetry & IPFIX Flow Tracking Configuration
! ==============================================================================
management api gnmi
transport grpc default
port 6030
ssl profile SSL_TELEMETRY
!
provider eos-native
!
flow tracking hardware
tracker TRACKER_IPFIX_WAN
record export on inactive timeout 15
record export on interval 60
default exporters
exporter COLLECTOR_LOKI
collector 10.250.0.50 port 2055
local interface Loopback0
template interval 300
!
!
interface Ethernet1/1
description UPLINK_TO_SPINE_1
flow tracking hardware input TRACKER_IPFIX_WAN
!Frequently Asked Questions (FAQ)
Why does SNMP polling fail to detect microburst congestion?
Microbursts are rapid surges in traffic lasting between 10 microseconds and 100 milliseconds that completely fill switch port packet buffer memory, causing packet drops for latency-sensitive applications. Because SNMP polls averages over 60 to 300-second windows, the mathematical average remains low (e.g., 20% link utilization), masking the fact that the port dropped thousands of packets during transient sub-second spikes.
What is the difference between NetFlow and sFlow?
NetFlow/IPFIX is stateful flow tracking: the switch ASIC maintains a cache of active sessions in memory, aggregates packet counts and byte sizes, and exports summarized records when flows terminate. sFlow is stateless packet sampling: the switch ASIC samples 1 out of every N packets, clips the initial header, and transmits it immediately to a collector without maintaining any connection state in switch memory.
How does eBPF capture network metrics without slowing down production applications?
eBPF programs run inside the Linux kernel utilizing Just-In-Time (JIT) compilation directly into native machine instructions. They access socket buffers and tracepoints directly in kernel space and write metrics to shared memory ring buffers. Because no context switches between kernel space and user space occur during packet processing, eBPF introduces less than 1% CPU overhead.
What is an optical TAP and why is it preferred over a SPAN port?
An optical TAP (Test Access Point) is a passive hardware device spliced into a fiber optic cable that splits a percentage of the laser light (e.g., 70% continues to destination, 30% to monitoring) with zero latency and zero electronics. Unlike switch SPAN ports—which drop mirrored packets during high switch CPU load and alter packet timing—an optical TAP delivers a 100% physically accurate replica of all wire-rate traffic.