Unintended Side Effects of Using http.DefaultClient in Go

The Internet is plenty of articles that telling why you should not be using http.DefaultClient in Golang (one, two) but they refer to Timeout and MaxIdleConns settings.

Today I want to share with you another reason why you should avoid using http.DefaultClient in your code.

The Story

As an SRE at Criteo, I both read and write code. Last week, I worked on patching Updatecli — an upgrade automation tool written in Go.

The patch itself was just ~15 lines of code. But then I spent three days debugging a strange authorization bug in an unrelated part of the code.

It happened because of code like this:

client := http.DefaultClient
client.Transport = &transport.PrivateToken{
    Token: s.Token,
    Base:  client.Client.Transport,
}

Since http.DefaultClient is a reference, not a value:

var DefaultClient = &Client{}

The code above is effectively the same as:

http.DefaultClient.Transport = &transport.PrivateToken{
    Token: s.Token,
    Base:  http.DefaultClient.Transport,
}

Later, in a third-party library, I found this:

if opts.Client == nil {
    opts.Client = http.DefaultClient
}

The Fix

To prevent this, I had to change the code to:

client := &http.Client{}
client.Transport = &transport.PrivateToken{
    Token: s.Token,
    Base:  client.Transport,
}

As a result, the patched client with the authorization transport got injected into the third-party library, causing unexpected failures.

Bugs like this are hard to catch just by reading the code, since they involve global state mutation. But could they be detected by linters?

What do you think? How do you find or prevent such issues in your projects?