Fixing toomanyrequests error when using Docker with Travis-CI

Students of our course Build a Backend REST API with Python & Django – Advanced have recently been getting the following issue when pulling Docker images in Travis-CI jobs:

ERROR: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
The command "docker-compose run --rm app sh -c "python manage.py wait_for_db && python manage.py test && flake8"" exited with 1.

The problem occurs because Docker is now rate limiting image pulls.

There is a limit of 100 pulls for anonymous users, which are tracked via their IP address. Since services like Travis-CI may serve thousands of customers from the same IP, the free allocation is quickly consumed.

How to fix it?

Fortunately, Docker provide a generous 200 pulls every 6 hours to those with a registered account, which should be more than enough for your average project.

However, in order to use Docker on a shared CI/CD service (like Travis-CI), you need to make a few changes to authenticate.

In this guide, I’ll show you how to do that!

Create an account on Docker Hub

First, create a new free account at hub.docker.com.

Once logged in, click on your username (top right) and Account Settings:

From the Account Settings page, choose Security and then click New Access Token.

On the New Access Token screen, enter a name for your token (I suggest using the name of your GitHub project), and click Create.

On the Copy Access Token screen, take note of the access token and store it somewhere securely (we’ll need this further down), and click Copy and Close.

Note: There is no way to retrieve the token once this screen is closed. If you lose it, simply delete the token and create a new one.

Remember your username and token, because you’re going to need that next!

Add credentials to Travis-CI

Login to travis-ci.com.

Then select your project, and choose More options > Settings.

Locate the Environment Variables section, then add two new variables:

• DOCKERHUB_USER – The username for your Docker Hub account.
• DOCKERHUB_TOKEN – The token for your Docker Hub account.

Ensure you leave DISPLAY VALUE IN BUILD LOG unchecked for both values.

When done, it should look something like this:

Modify Travis-CI Config

In your project, open the .travis-ci.yml file, and add the following block:

before_install:
  - echo $DOCKERHUB_TOKEN | docker login --username $DOCKERHUB_USER --password-stdin

This will do the following:

• echo $DOCKER_PASSWORD prints the password to the screen, and the | (pipe) will send that output to the proceeding command.
• docker login --username $DOCKER_USERNAME will call the docker login command with the username we set in the environment variables.
• --password-stdin is used to accept the password in a way that prevents it being printed to the screen (it’s required with the | syntax).

If you’re following our course, the full .travis-ci.yml file would look something like this:

language: python
python:
  - "3.6"

services:
  - docker

before_install:
  - echo $DOCKERHUB_TOKEN | docker login --username $DOCKERHUB_USER --password-stdin

before_script: pip install docker-compose

script:
  - docker-compose run --rm app sh -c "python manage.py wait_for_db && python manage.py test && flake8"

Now your Docker build should work as expected.

Other Notes

Some people (like myself) may be uncomfortable with putting their Docker Hub credentials directly inside a project configuration, as this is not best practice from a security perspective, especially if you’re working on a shared project.

My current solution is to have a specific Docker Hub account that is only used for CI purposes.

If you need to push images, you can give this CI account permissions to push to the registry on your main account.

If anyone has any better approaches, I would love to hear about them in the comments or on Reddit!