Golang Client Timeout exceeded while awaiting headers

Recently, I have an issue in Golang:

This happens when I tried to fetch data from Coingecko to have the token market price.

The thing is, Coingecko only allows 10-50 requests/minute for Free account, so we have to use a Proxy to rotate the IP (so 20 IPs mean that we have 200-1000 requests/minute under Free account).

This setup works for months without any issue … until now.

But in order to understand the root cause, let’s talk about what is timeout?

There are 2 kinds of timeout: server timeouts and client timeouts:

Server Timeouts

The “So you want to expose Go on the Internet” post has more information on server timeouts, in particular about HTTP/2 and Go 1.7 bugs.

It’s critical for an HTTP server exposed to the Internet to enforce timeouts on client connections. Otherwise very slow or disappearing clients might leak file descriptors and eventually result in something along the lines of:

http: Accept error: accept tcp [::]:80: accept4: too many open files; retrying in 5ms

There are two timeouts exposed in http.Server: ReadTimeout and WriteTimeout. You set them by explicitly using a Server:

srv := &http.Server{
    ReadTimeout: 5 * time.Second,
    WriteTimeout: 10 * time.Second,
}
log.Println(srv.ListenAndServe())

ReadTimeout covers the time from when the connection is accepted to when the request body is fully read (if you do read the body, otherwise to the end of the headers). It’s implemented in net/http by calling SetReadDeadline immediately after Accept.

WriteTimeout normally covers the time from the end of the request header read to the end of the response write (a.k.a. the lifetime of the ServeHTTP), by calling SetWriteDeadline at the end of readRequest.

However, when the connection is HTTPS, SetWriteDeadline is called immediately after Accept so that it also covers the packets written as part of the TLS handshake. Annoyingly, this means that (in that case only) WriteTimeout ends up including the header read and the first byte wait.

You should set both timeouts when you deal with untrusted clients and/or networks, so that a client can’t hold up a connection by being slow to write or read.

Finally, there’s [http.TimeoutHandler](https://golang.org/pkg/net/http/#TimeoutHandler). It’s not a Server parameter, but a Handler wrapper that limits the maximum duration of ServeHTTP calls. It works by buffering the response, and sending a 504 Gateway Timeout instead if the deadline is exceeded. Note that it is broken in 1.6 and fixed in 1.6.2.

Client Timeouts

Client-side timeouts can be simpler or much more complex, depending which ones you use, but are just as important to prevent leaking resources or getting stuck.

The easiest to use is the Timeout field of [http.Client](https://golang.org/pkg/net/http/#Client). It covers the entire exchange, from Dial (if a connection is not reused) to reading the body.

c := &http.Client{
    Timeout: 15 * time.Second,
}
resp, err := c.Get("https://hoangtrinhj.com/")

Like the server-side case above, the package level functions such as http.Get use a Client without timeouts, so are dangerous to use on the open Internet.

For more granular control, there are a number of other more specific timeouts you can set:

• net.Dialer.Timeout limits the time spent establishing a TCP connection (if a new one is needed).
• http.Transport.TLSHandshakeTimeout limits the time spent performing the TLS handshake.
• http.Transport.ResponseHeaderTimeout limits the time spent reading the headers of the response.
• http.Transport.ExpectContinueTimeout limits the time the client will wait between sending the request headers when including an Expect: 100-continue and receiving the go-ahead to send the body. Note that setting this in 1.6 will disable HTTP/2 (DefaultTransport is special-cased from 1.6.2).

c := &http.Client{
    Transport: &http.Transport{
        Dial: (&net.Dialer{
                Timeout:   30 * time.Second,
                KeepAlive: 30 * time.Second,
        }).Dial,
        TLSHandshakeTimeout:   10 * time.Second,
        ResponseHeaderTimeout: 10 * time.Second,
        ExpectContinueTimeout: 1 * time.Second,
    }
}

As far as I can tell, there’s no way to limit the time spent sending the request specifically. The time spent reading the request body can be controlled manually with a time.Timer since it happens after the Client method returns (see below for how to cancel a request).

Finally, new in 1.7, there’s http.Transport.IdleConnTimeout. It does not control a blocking phase of a client request, but how long an idle connection is kept in the connection pool.

Note that a Client will follow redirects by default. http.Client.Timeout includes all time spent following redirects, while the granular timeouts are specific for each request, since http.Transport is a lower level system that has no concept of redirects.

In our use case, we need to care about the Client Timeouts (Coingecko is the server, and we are a client that fetches data from the Coingecko’s server)

I tried to increase the Client.Timeout and also the TLSHandshakeTimeout but the issue is not resolved. So the reason must be somewhere else

Reference

https://blog.cloudflare.com/the-complete-guide-to-golang-net-http-timeouts/

https://joshishantanu.com/posts/timeout/