Skip to content

Developing With Docker (Day 2), caching debs using squid-deb-proxy

Today’s step is going to take a bit of a detour. One of the things that kills you when rebuilding a box is that you have to download dependencies every time from the internet. With big files this can be quite slow, and can keep you from working at all if you are offline.

So I’m going try and cache as many of these files on my host box as possible so this isn’t an issue. Of course if I add new dependencies and I’m offline, it won’t work unless I’ve somehow cached them before. But I should be able to rebuild the machine.

So if you pull down the repo, there will be a tag for Day 2.

Diff of the changes from day1 to day2 (git diff day1 day 2)

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..909475b
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+30proxy
diff --git a/Dockerfile b/Dockerfile
index d643899..641429e 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,5 +1,9 @@
 from ubuntu:12.10
 
+# set your host up as the apt cache for speed
+
+add 30proxy /etc/apt/apt.conf.d/30proxy
+
 # update system and install dependencies
 
 run apt-get update
diff --git a/hostconfig.sh b/hostconfig.sh
new file mode 100755
index 0000000..45754e8
--- /dev/null
+++ b/hostconfig.sh
@@ -0,0 +1,3 @@
+sudo apt-get install squid-deb-proxy
+HOST_IP=`ifconfig docker0 |grep inet|head -1|sed 's/\:/ /'|awk '{print $3}'`
+echo "Acquire::http::Proxy \"http://$HOST_IP:8000\";" > 30proxy
\ No newline at end of file
diff --git a/shell.sh b/shell.sh
new file mode 100755
index 0000000..50a0f87
--- /dev/null
+++ b/shell.sh
@@ -0,0 +1,2 @@
+#! /bin/bash
+docker run -i -t u1210_nodebase /bin/bash
\ No newline at end of file

* Ignore 30proxy which is a generated file so we don’t check it in accidentally
* In the Dockerfile, tell docker to copy the 30proxy file over to /etc/apt/apt.conf.d/30proxy, so that it’ll configure apt.
* Create a hostconfig.sh script which you can (and should) run to install squid-deb-proxy to your host and correctly configure the container to point at the host for it’s apt cache.
* Add a shell.sh script which makes it easier to shell into this instance.

Testing

So we can see this is working by opening a second terminal window and doing a “tail -f /var/log/squid-deb-proxy/access.log”.

And then running build.sh in the original terminal. You should see lines like:
[code]
1375291602.249 2235 172.16.42.1 TCP_MISS/200 861368 GET http://archive.ubuntu.com/ubuntu/pool/universe/o/openssl098/libssl0.9.8_0.9.8o-7ubuntu3.1_amd64.deb - DIRECT/91.189.92.176 application/x-debian-package
[/code]

Which means a cache miss, and a fetch from the real source (TCP_MISS). Or lines like this:
[code]
1375295650.284 4 172.16.42.137 TCP_MEM_HIT/200 202735 GET http://archive.ubuntu.com/ubuntu/pool/main/m/mpfr4/libmpfr4_3.1.0-3ubuntu3_amd64.deb - NONE/- application/x-debian-package
[/code]

Which means squid served the file from memory (no external network access).

That’s about it. Tomorrow we’ll get back into

One Comment

  1. Interesting…The only problem I see with this however is that it breaks down when the container runs on a different machine.

    Could there be a way to have a more transparant way of caching packages?

    Friday, October 11, 2013 at 2:46 pm | Permalink

Post a Comment

Your email is never published nor shared.