Tutorial

How To Work with Docker Data Volumes on Ubuntu 14.04

Updated on November 17, 2016
How To Work with Docker Data Volumes on Ubuntu 14.04

Introduction

In this article we’re going to run through the concept of Docker data volumes: what they are, why they’re useful, the different types of volumes, how to use them, and when to use each one. We’ll also go through some examples of how to use Docker volumes via the docker command line tool.

By the time we reach the end of the article, you should be comfortable creating and using any kind of Docker data volume.

Prerequisites

To follow this tutorial, you will need the following:

Note: Even though the Prerequisites give instructions for installing Docker on Ubuntu 14.04, the docker commands for Docker data volumes in this article should work on other operating system as long as Docker is install.

Explaining Docker Containers

Working with Docker requires understanding quite a few Docker-specific concepts, and most of the documentation focuses on explaining how to use Docker’s toolset without much explanation of why you’d want to use any of those tools. This can be confusing if you’re new to Docker, so we’ll start by going through some basics and then jump into working with Docker containers. Feel free to skip ahead to the next section if you’ve worked with Docker before and just want to know how to get started with data volumes.

A Docker container is similar to a virtual machine. It basically allows you to run a pre-packaged “Linux box” inside a container. The main difference between a Docker container and a typical virtual machine is that Docker is not quite as isolated from the surrounding environment as a normal virtual machine would be. A Docker container shares the Linux kernel with the host operating system, which means it doesn’t need to “boot” the way a virtual machine would.

Since so much is shared, firing up a Docker container is a quick and cheap operation — in most cases you can bring up a full Docker container (the equivalent of a normal virtual machine) in the same time as it would take to run a normal command line program. This is great because it makes deploying complex systems a much easier and more modular process, but it’s a different paradigm from the usual virtual machine approach and has some unexpected side effects for people coming from the virtualization world.

Learning the Types of Docker Data Volumes

There are three main use cases for Docker data volumes:

  1. To keep data around when a container is removed
  2. To share data between the host filesystem and the Docker container
  3. To share data with other Docker containers

The third case is a little more advanced, so we won’t go into it in this tutorial, but the first two are quite common.

In the first (and simplest) case you just want the data to hang around even if you remove the container, so it’s often easiest to let Docker manage where the data gets stored.

Keeping Data Persistent

There’s no way to directly create a “data volume” in Docker, so instead we create a data volume container with a volume attached to it. For any other containers that you then want to connect to this data volume container, use the Docker’s --volumes-from option to grab the volume from this container and apply them to the current container. This is a bit unusual at first glance, so let’s run through a quick example of how we could use this approach to make our byebye file stick around even if the container is removed.

First, create a new data volume container to store our volume:

docker create -v /tmp --name datacontainer ubuntu

This created a container named datacontainer based off of the ubuntu image and in the directory /tmp.

Now, if we run a new Ubuntu container with the --volumes-from flag and run bash again as we did earlier, anything we write to the /tmp directory will get saved to the /tmp volume of our datacontainer container.

First, start the ubuntu image:

docker run -t -i --volumes-from datacontainer ubuntu /bin/bash

The -t command line options calls a terminal from inside the container. The -i flag makes the connection interactive.

At the bash prompt for the ubuntu container, create a file in /tmp:

  1. echo "I'm not going anywhere" > /tmp/hi

Go ahead and type exit to return to your host machine’s shell. Now, run the same command again:

docker run -t -i --volumes-from datacontainer ubuntu /bin/bash

This time the hi file is already there:

  1. cat /tmp/hi

You should see:

Output of cat /tmp/hi
I'm not going anywhere

You can add as many --volumes-from flags as you’d like (for example, if you wanted to assemble a container that uses data from multiple data containers). You can also create as many data volume containers as you’d like.

The only caveat to this approach is that you can only choose the mount path inside the container (/tmp in our example) when you create the data volume container.

Sharing Data Between the Host and the Docker Container

The other common use for Docker containers is as a means of sharing files between the host machine and the Docker container. This works differently from the last example. There’s no need to create a “data-only” container first. You can simply run a container of any Docker image and override one of its directories with the contents of a directory on the host system.

As a quick real-world example, let’s say you wanted to use the official Docker Nginx image but you wanted to keep a permanent copy of the Nginx’s log files to analyze later. By default the nginx Docker image logs to the /var/log/nginx directory, but this is /var/log/nginx inside the Docker Nginx container. Normally it’s not reachable from the host filesystem.

Let’s create a folder to store our logs and then run a copy of the Nginx image with a shared volume so that Nginx writes its logs to our host’s filesystem instead of to the /var/log/nginx inside the container:

  1. mkdir ~/nginxlogs

Then start the container:

  1. docker run -d -v ~/nginxlogs:/var/log/nginx -p 5000:80 -i nginx

This run command is a little different from the ones we’ve used so far, so let’s break it down piece by piece:

  • -v ~/nginxlogs:/var/log/nginx — We set up a volume that links the /var/log/nginx directory from inside the Nginx container to the ~/nginxlogs directory on the host machine. Docker uses a : to split the host’s path from the container path, and the host path always comes first.

  • -d — Detach the process and run in the background. Otherwise, we would just be watching an empty Nginx prompt and wouldn’t be able to use this terminal until we killed Nginx.

  • -p 5000:80 — Setup a port forward. The Nginx container is listening on port 80 by default, and this maps the Nginx container’s port 80 to port 5000 on the host system.

If you were paying close attention, you may have also noticed one other difference from the previous run commands. Up until now we’ve been specifying a command at the end of all our run statements (usually /bin/bash) to tell Docker what command to run inside the container. Because the Nginx image is an official Docker image, it follows Docker best practices, and the creator of the image set the image to run the command to start Nginx automagically. We can just drop the usual /bin/bash here and let the creators of the image choose what command to run in the container for us.

So, we now have a copy of Nginx running inside a Docker container on our machine, and our host machine’s port 5000 maps directly to that copy of Nginx’s port 80. Let’s use curl to do a quick test request:

curl localhost:5000

You’ll get a screenful of HTML back from Nginx showing that Nginx is up and running. But more interestingly, if you look in the ~/nginxlogs folder on the host machine and take a look at the access.log file you’ll see a log message from Nginx showing our request:

cat ~/nginxlogs/access.log

You will see something similar to:

Output of `cat ~/nginxlogs/access.log`
172.17.42.1 - - [23/Oct/2015:05:22:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.35.0" "-"

If you make any changes to the ~/nginxlogs folder, you’ll be able to see them from inside the Docker container in real-time as well.

Conclusion

That about sums it up! We’ve now covered how to create data volume containers whose volumes we can use as a way to persist data in other containers as well as how to share folders between the host filesystem and a Docker container. This covers all but the most advanced use cases when it comes to Docker data volumes.

If you are using Docker Compose, Docker data volumes can be configured in your docker-compose.yml file. Check out How To Install and Use Docker Compose on Ubuntu 14.04 for details.

Good luck and happy Dockering!

Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.

Learn more about us


About the authors

Default avatar
Tammy Fox

editor


Still looking for an answer?

Ask a questionSearch for more help

Was this helpful?
 
10 Comments


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

I am not native english speaker. But i had understood every single row of your excellent article. You have a talent of explanation abstract thing in simple way, thanks a lot Nik!

Hi~ Thank you for your sharing! I’m new to Docker, but I think that there is something different from what I know about how the modifications to the container will be dealt with. I noticed that you use “docker run” during the whole process with out an argument “–name <name>”, thus, Docker will create a new container every time, whose name is a random one generated by the docker itself, eg: if you run

docker run -d busybox && docker ps -a

there comes a container whose name is “sleepy_liskov” on my machine, you run it again, another container called “fervent_lalande” is created, they exit at once cause the busybox dose nothing. So, my point is, in your scenario, when you created the file “byebye” in a container without a name given by you, you stop it, and you “docker run” again, the new container is a new one which is definitely not the one with the “byebye” file. But if you use

docker ps -a

you’ll find the container(let’s call it foo_bar) with the “byebye” file, it’s not gone, just stopped. you can use

docker start foo_bar

to restart it. So, when you say stop the container and start it up again, I would think that you mean "docker start " and “docker stop” not “docker run”, the container will be always there with any of your modification until you remove them with

docker rm <name>

So, I guess maybe you try to find the "byebye " file in the wrong place, and it may be a good practice to use the “–name” argument or use the “–rm”. Again, I’m new to Docker, I’ve been stuck for days about how to use volume properly on OS X, 'cause you know, there is another VirtualBox layer between your host and the container, the volume mounted to the container with “-v” has some permission problem, which drives me crazy. Do you have some clue about it~

Is this still up to date? I think the changes are now persisted in the top RW layer, even if no data volume is mounted. Right?

I tried creating a file an restarting the container. The file was still there…

Quite possibly the clearest and most comprehensive overview of containers I have ever read. Thanks for making the comparison to virtual machines - as a long time VMware guy (since 2002!), I have struggled to wrap my head around the concept and learn where the similarities and differences were. Now, if you could explain how containers compare to, say, a VMware-based virtual application, you’d really win me over :-). BTW…HUGE Digital Ocean fan - I love your site!

Something definitely has changed regarding volumes. If i don’t specify a custom host path to the volume, i have no issues.

But when i want to run a container with a volume at a custom location on my host, i am not able to use that volume. This happens to me with al images i have tested it with:

[mru@dockster ~]$ docker run -d -v ~/nginxlogs:/var/log/nginx -p 5000:80 -i nginx [mru@dockster ~]$ ls nginxlogs/ [mru@dockster ~]$ docker logs brave_gates nginx: [alert] could not open error log file: open() “/var/log/nginx/error.log” failed (13: Permission denied) 2018/10/27 17:18:14 [emerg] 1#1: open() “/var/log/nginx/error.log” failed (13: Permission denied)

So, the directory gets created, but i am not able to write to it for whatever reason. This happens also when i start docker as root, and also when i specify a directory which is chmodded 777. Has anyone an idea what goes wrong here?

Hi, you wrote: “There’s no way to directly create a “data volume” in Docker, so instead we create a data volume container with a volume attached to it” Actually, from docker 1.9.0 you can directly create volumes without the need of creating data volume containers. See: https://docs.docker.com/engine/reference/commandline/volume_create/

Hi, thank you for contributing such a clear and comprehensive tutorial for how to set up docker data volumes for what I presume are the most common use cases. Would you be able to give some pointers for how to address the third, more advanced, scenario i.e. sharing the contents of a persistent data volume between multiple containers? I’ve found a few resources about GlusferFS and NFS, but so far nothing as clearly outlined as your article. Cheers

Clearer than this explanation you won’t find thanks Nik this is awesome :)

I do not really get what does the following mean. 3. To share data with other Docker containers

I want to know more about it. Any more info or ref link?

Hi,

First of all thanks for the great article. I am particularly interested in sharing data between host and container. I am trying to create a volume that will store screenshots for failed capybara tests but I am having some issues.

My docker file has the following line:

VOLUME ./tmp/screenshot:/tmp/screenshot

I can see the screenshot being saved in the container:

Entonces veo un mensaje de confirmación         # features/step_definitions/information_load_steps.rb:24
    HTML screenshot: /tmp/screenshot/screenshot_2016-05-24-14-35-14.938.html
    Image screenshot: /tmp/screenshot/screenshot_2016-05-24-14-35-14.938.png

But I can’t see anything being created in the host.

Could you please help me figure out what is going on here?

Thanks

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up

Join the Tech Talk
Success! Thank you! Please check your email for further details.

Please complete your information!

Get our biweekly newsletter

Sign up for Infrastructure as a Newsletter.

Hollie's Hub for Good

Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.

Become a contributor

Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.

Welcome to the developer cloud

DigitalOcean makes it simple to launch in the cloud and scale up as you grow — whether you're running one virtual machine or ten thousand.

Learn more
DigitalOcean Cloud Control Panel