Go's net/http server employs a goroutine-per-connection model combined with the runtime's M:N scheduling strategy. When the server accepts a TCP connection, it immediately spawns a lightweight goroutine to handle that connection's entire lifecycle, allowing the main accept loop to return and receive the next connection immediately. These goroutines are multiplexed onto a limited pool of OS threads by the Go scheduler, which parks goroutines performing blocking I/O and reschedules runnable ones onto available threads. This architecture allows the server to maintain hundreds of thousands of concurrent connections using only a handful of kernel threads, avoiding the memory overhead of traditional thread-per-connection servers.
We needed to build a real-time telemetry gateway capable of ingesting data from 50,000 IoT devices simultaneously over persistent HTTP/1.1 connections.
Problem description: Our initial prototype using Python with Twisted provided the necessary concurrency but quickly became unmaintainable due to complex callback chains and deeply nested error handling. When we attempted a Java thread-per-connection approach to simplify the code, we encountered the operating system's thread limit at approximately 32,000 connections, causing the JVM to crash with OutOfMemoryError: unable to create new native thread because each thread consumed over 1MB of virtual memory.
Different solutions considered:
Asyncio with explicit state machines: We evaluated migrating to Python's asyncio to use a single event loop with coroutines. This would significantly reduce memory footprint compared to threads, but it would require rewriting all our protocol parsing logic into async/await syntax and introduced the risk of accidentally blocking the event loop with CPU-intensive operations. Debugging stack traces across asynchronous boundaries also proved notoriously difficult for our development team.
Horizontal sharding of JVM instances: We considered running ten smaller Java instances behind a load balancer, with each instance handling 5,000 threads. This approach solved the per-process thread limit but introduced substantial operational complexity, required additional hardware resources, and complicated the management of shared state and connection stickiness across the cluster. The operational overhead of maintaining this micro-cluster outweighed the benefits of staying with Java.
Go's goroutine-per-connection model: We chose to reimplement the gateway in Go, leveraging the standard library's net/http and net packages. The server's Serve method automatically spawns a lightweight goroutine for each accepted TCP connection, and the Go runtime's scheduler transparently multiplexes these onto a limited pool of OS threads. This allowed us to write straightforward, synchronous-looking I/O code that would scale to hundreds of thousands of connections without manual state machine management.
Chosen solution and why: We selected the Go implementation because it offered the scalability of event-driven systems combined with the simplicity of threaded programming. The runtime handles the complexity of scheduling and non-blocking I/O automatically, allowing our developers to focus on business logic rather than concurrency primitives. Additionally, the goroutine's 2KB initial stack size meant we could theoretically handle millions of connections within our memory budget.
Result: The production system successfully managed 75,000 concurrent persistent connections on a single 8-core server, consuming less than 4GB of RAM. CPU utilization remained stable at 35-40% because the scheduler efficiently hid I/O latency, and we eliminated the operational burden of managing a cluster of sharded Java instances.
How does the Go scheduler prevent a thundering herd problem when thousands of goroutines block on the same channel receive?
The Go scheduler uses a first-in-first-out (FIFO) waiting queue for channels, not a semaphore-style wake-all. When a sender writes to a channel, the scheduler wakes exactly one waiting goroutine from the receive queue (the one that has been waiting longest). This ensures that only one goroutine consumes the value, preventing the thundering herd where multiple goroutines wake, compete for the lock, and all but one go back to sleep. Candidates often incorrectly assume that channel operations broadcast to all waiters like condition variables.
Why might increasing GOMAXPROCS beyond the number of physical CPU cores degrade the performance of an I/O-bound Go HTTP server?
While Go's scheduler is preemptive since version 1.14, having more OS threads (M) than cores increases kernel-level context switch overhead. For I/O-bound servers, excessive threads can lead to the scheduler spending more time managing runqueues and thread handoffs than executing user code. Additionally, each OS thread consumes kernel resources (memory for thread-local storage and kernel stacks), which can pressure the operating system when scaled excessively beyond necessary parallelism.
How does Go's net/http server handle the TCP SO_BACKLOG queue when the goroutine acceptance rate temporarily lags behind the connection arrival rate?
The server relies on the kernel's listen backlog queue (controlled by net.ListenConfig's Backlog or system defaults). If goroutines are slow to spawn or handlers are slow to accept connections from the listener, the kernel queues incoming SYNs in the backlog. Once the backlog fills, the kernel rejects new connections via TCP RST. Go's Accept() loop runs in its own goroutine and should ideally spawn handler goroutines quickly. However, if the handler spawning is delayed (e.g., due to GC pauses or mutex contention in middleware), connections drop. Candidates often miss that Go doesn't implement user-space connection queuing; it depends entirely on the kernel backlog, and tuning SOMAXCONN or ListenConfig.Backlog is crucial for burst absorption.