{"id":2468,"date":"2019-09-22T22:23:00","date_gmt":"2019-09-22T22:23:00","guid":{"rendered":"http:\/\/wiki.thomasandsofia.com\/?p=2468"},"modified":"2020-04-22T01:25:47","modified_gmt":"2020-04-22T01:25:47","slug":"docker-engine","status":"publish","type":"post","link":"https:\/\/wiki.thomasandsofia.com\/?p=2468","title":{"rendered":"7: Docker Engine, Storage and Networking"},"content":{"rendered":"<p><a href=\"\/docker-registry\/\">&lt; 6 Docker Registry<\/a> | <a href=\"\/8-docker-on-mac-and-windows\/\">8 Docker on Mac and Windows &gt;<\/a><\/p>\n<p>59% Complete<\/p>\n<h1>33: Docker Engine<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829078#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829078#content<\/a><\/p>\n<h2>Architecture<\/h2>\n<ul>\n<li>The &#8220;Engine&#8221; is the host with Docker installed on it.\n<ul>\n<li>Docker Daemon\n<ul>\n<li>Backgroud process that manages the\n<ul>\n<li>Images<\/li>\n<li>Containers<\/li>\n<li>Volumes<\/li>\n<li>Networks<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>REST API server\n<ul>\n<li>Provides the interface to talk to the daemons<\/li>\n<\/ul>\n<\/li>\n<li>Docker CLI\n<ul>\n<li>Command line interface<\/li>\n<li>Uses the REST API to talk to the daemon<\/li>\n<li>Does not need to be installed on the same Host!<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre>docker -h=&lt;remote-docker-engine&gt;:2375 run &lt;image&gt;<\/pre>\n<h2>Containerization<\/h2>\n<ul>\n<li>Docker uses namespaces to isolate\n<ul>\n<li>workspace<\/li>\n<li>process ids<\/li>\n<li>networks<\/li>\n<li>interprocess communications<\/li>\n<li>mounts<\/li>\n<li>Unix timesharing systems<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Process ID namespaces PID<\/h2>\n<ul>\n<li>When a Linux system boots up, it starts with a single process with an ID of 1.<\/li>\n<li>This is the &#8216;root&#8217; process.<\/li>\n<li>The root process starts up all the other processes.\n<ul>\n<li>All process IDs are unique. No two processes can share an ID<\/li>\n<li>To list all the process, use the <code>ps<\/code> command.<\/li>\n<\/ul>\n<\/li>\n<li>A container is like a child system in the current system\n<ul>\n<li>Needs to think it is an independent system with its own root process with a process ID of 1.<\/li>\n<li>However, processes running inside the container are actually processes running inside the host.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h3>Process ID Namespaces<\/h3>\n<p><a href=\"http:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2471\" src=\"http:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1.png\" alt=\"\" width=\"531\" height=\"367\" srcset=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1.png 531w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-300x207.png 300w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-150x104.png 150w\" sizes=\"auto, (max-width: 531px) 100vw, 531px\" \/><\/a><\/p>\n<ul>\n<li>Each process id namespace has it&#8217;s own set of processes and each can have it&#8217;s own root process with a process ID of 1.<\/li>\n<li>A process started in a contain is just another process on the host machine and will get the next available process ID.<\/li>\n<li>It will also get the next available process id from the container.\u00a0 This is how the container can have it&#8217;s own root process and process list.<\/li>\n<li>To verify this\n<ul>\n<li>Check the last PID on the host.<\/li>\n<li>Now run a docker container and then run PS to see the name of the service with PID 1.<\/li>\n<li>Run PS again on the host. You&#8217;ll see the same process name, but with a different ID<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>cgroups &#8211; Control Groups<\/h2>\n<ul>\n<li>By default, there are no restrictions regarding the amount of resources a container may use.<\/li>\n<li>Control Groups are used to restrict the amount of hardware resources allocated to each container.<\/li>\n<li>In the example below, we&#8217;ll restrict the CPU to 50% and the RAM to 100MB.<\/li>\n<\/ul>\n<pre>docker run --cpus=.5 --memory=100m ubuntu<\/pre>\n<h1>Namespace Demo<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829688#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829688#content<\/a><\/p>\n<p>Prove Processes are on both the host and container and they have different PIDs.<\/p>\n<h1>Docker Storage<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829082#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829082#content<\/a><\/p>\n<h2>Docker File Systems<\/h2>\n<ul>\n<li>\/var\/lib\/docker\n<ul>\n<li>aufs<\/li>\n<li>containers<\/li>\n<li>image<\/li>\n<li>volumes<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Layered Architecture<\/h2>\n<p><a href=\"http:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-1.png\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-full wp-image-2476\" src=\"http:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-1.png\" alt=\"\" width=\"642\" height=\"411\" srcset=\"https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-1.png 642w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-1-300x192.png 300w, https:\/\/wiki.thomasandsofia.com\/wp-content\/uploads\/2019\/09\/a1-1-150x96.png 150w\" sizes=\"auto, (max-width: 642px) 100vw, 642px\" \/><\/a><\/p>\n<ul>\n<li>Docker Images are created in a layered architecture with only the changes to the layer above it applied,\n<ul>\n<li>FROM ubuntu<\/li>\n<li>RUN apt-get update -y<\/li>\n<li>&#8230;<\/li>\n<li>This method cuts disk space and speeds creation time.<\/li>\n<li>These layers are read only<\/li>\n<\/ul>\n<\/li>\n<li>When a container is created, it uses the Image layers, then creates a new &#8216;container&#8217; layer on top of them.\n<ul>\n<li>The image layers are then copied to the Container layer.<\/li>\n<li>This layer is lost when the container is exited.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Volumes<\/h2>\n<h3>Create a persistent volume<\/h3>\n<pre>docker volume create my_volume<\/pre>\n<ul>\n<li>This will create a volume at \/var\/lib\/docker\/volumes\/my_volume<\/li>\n<\/ul>\n<p>To use your volume,<\/p>\n<pre>docker run -v my_volume:\/docker\/container\/path<\/pre>\n<ul>\n<li>If you do not create the volume first, it will create it for you.<\/li>\n<\/ul>\n<h3>Using bind points<\/h3>\n<ul>\n<li>Use this system to mount specific drive locations, such as an external SSD<\/li>\n<li>Simply identify the full path to the location.<\/li>\n<\/ul>\n<h4>Example:<\/h4>\n<ul>\n<li>The default location for these volumes is \/var\/lib\/docker\/volumes<\/li>\n<li>If you do not create the folder prior to running the command, it will create it for you.<\/li>\n<\/ul>\n<pre>docker run -v \/persistent\/folder:\/docker\/container\/folder &lt;image&gt;<\/pre>\n<pre>docker run --mount type=bind,source=\/persistent\/folder,target=\/docker\/container\/folder &lt;image&gt;<\/pre>\n<ul>\n<li>&#8211;mount is preferred method over -v\n<ul>\n<li>It is more verbose and more inline with JSON key=value pairs.<\/li>\n<li>See below for volume vs bind types.<\/li>\n<\/ul>\n<\/li>\n<li>Using this system will mount the \/persistent\/volume\/ folder to the docker container at the location specified.<\/li>\n<li>When the container is destroyed, the volume will remain and any data written to it by the container will not be lost.<\/li>\n<\/ul>\n<h3>Volume Mounting vs. Bind Mounting<\/h3>\n<p><strong>Volume mounting:<\/strong> Not specifying a direct path such that docker uses the default \/var\/lib\/docker\/volumes folder.<\/p>\n<p><strong>Bind Mounting:<\/strong> Specifying an existing path from the root folder.<\/p>\n<h2>Storage Drivers<\/h2>\n<p>Supported drivers:<\/p>\n<ul>\n<li>Storage drivers are responsible for creating the folders.<\/li>\n<li>Docker will chose the best storage drive depending on the underlying operating system.<\/li>\n<\/ul>\n<ul>\n<li>AUFS<\/li>\n<li>ZFS<\/li>\n<li>BTRFS<\/li>\n<li>Device Mapper<\/li>\n<li>Overlay<\/li>\n<li>Overlay2<\/li>\n<\/ul>\n<h1>36: Docker Storage Demo<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829694#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829694#content<\/a><\/p>\n<p>Get Docker Info<\/p>\n<pre>docker info\r\nClient:\r\n Debug Mode: false\r\n\r\nServer:\r\n Containers: 1\r\n  Running: 1\r\n  Paused: 0\r\n  Stopped: 0\r\n Images: 48\r\n Server Version: 19.03.8\r\n Storage Driver: overlay2\r\n  Backing Filesystem: \r\n  Supports d_type: true\r\n  Native Overlay Diff: true\r\n Logging Driver: json-file\r\n Cgroup Driver: cgroupfs\r\n Plugins:\r\n  Volume: local\r\n  Network: bridge host ipvlan macvlan null overlay\r\n Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog\r\n Swarm: inactive\r\n Runtimes: runc\r\n Default Runtime: runc\r\n Init Binary: docker-init\r\n containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429\r\n runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd\r\n init version: fec3683\r\n Security Options:\r\n  apparmor\r\n  seccomp\r\n   Profile: default\r\n Kernel Version: 5.3.0-46-generic\r\n Operating System: Ubuntu 18.04.4 LTS\r\n OSType: linux\r\n Architecture: x86_64\r\n CPUs: 2\r\n Total Memory: 1.941GiB\r\n Name: docker01\r\n ID: TLSU:IN33:XSIE:3LS5:VIV7:IDY2:DW6M:VKTL:27JV:PODP:6PYQ:4TDT\r\n Docker Root Dir: \/var\/lib\/docker\r\n Debug Mode: false\r\n Registry: https:\/\/index.docker.io\/v1\/\r\n Labels:\r\n Experimental: false\r\n Insecure Registries:\r\n  127.0.0.0\/8\r\n Live Restore Enabled: false\r\n\r\nWARNING: No swap limit support\r\n<\/pre>\n<h4>To see how an image is built<\/h4>\n<pre>docker history ImageId<\/pre>\n<p>Points to ponder<\/p>\n<ul>\n<li>When making changes to an image build:\n<ul>\n<li>Steps that have not changed, up to the change, are used from cache<\/li>\n<li>Once a step changes, that step, along ALL following steps are recreated.\n<ul>\n<li>This includes following steps that may or may not have changed.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<li>If only a minor change is made, the original and new image will display approximately the same size.\n<ul>\n<li>This does not mean the total disk space consumed is the total of the two.<\/li>\n<li>Since many of the layers may be shared, those layers are NOT replicated to create the new image.<\/li>\n<li>The image size represents the amount of space required if you were to pull it to another system.<\/li>\n<\/ul>\n<\/li>\n<li>To see total disk space used, run the docker system df command.<\/li>\n<\/ul>\n<pre>docker system df &lt;-v&gt;<\/pre>\n<p>Example:<\/p>\n<pre>root@docker01:\/var\/lib\/docker\/overlay2# docker images\r\nREPOSITORY                  TAG                 IMAGE ID            CREATED             SIZE\r\nexample-voting-app_worker   latest              b22add4acbb9        21 hours ago        1.72GB\r\nexample-voting-app_result   latest              4fb7cc75482d        21 hours ago        146MB\r\nexample-voting-app_vote     latest              a027647f0ba4        21 hours ago        84.2MB\r\nresult-app                  latest              86b97d2a7378        22 hours ago        146MB\r\nvoting-app                  latest              b39ff7f319fa        22 hours ago        84.2MB\r\npython                      2.7-alpine          8579e446340f        23 hours ago        71.1MB\r\nredis                       alpine              f727a777bc34        25 hours ago        29.8MB\r\nworker-app                  latest              4af0bf7ffcd1        31 hours ago        1.72GB\r\n                                    8689b56902f4        33 hours ago        84.1MB\r\nwordpress                   latest              895c3d7568db        4 days ago          540MB\r\npostgres                    latest              4b59e7d1ba1c        4 days ago          314MB\r\nredis                       latest              975fe4b9f798        4 days ago          98.3MB\r\nnode                        10-slim             af55a61c87a1        5 days ago          135MB\r\npostgres                    9.4                 ed5a45034282        2 months ago        251MB\r\nregistry                    2                   708bc6af7e5e        2 months ago        25.8MB\r\nmicrosoft\/dotnet            2.0.0-sdk           fde8197d13f4        2 years ago         1.64GB\r\n# ~4.0GB\r\nroot@docker01:\/var\/lib\/docker\/overlay2# docker system df\r\nTYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE\r\nImages              16                  1                   3.246GB             3.22GB (99%)\r\nContainers          1                   0                   0B                  0B\r\nLocal Volumes       19                  0                   979.2MB             979.2MB (100%)\r\nBuild Cache         0                   0                   0B                  0B\r\n<\/pre>\n<p>&nbsp;<\/p>\n<h1>37: Lab<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/course\/learn-docker\/learn\/lecture\/15829086#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/course\/learn-docker\/learn\/lecture\/15829086#content<\/a><\/p>\n<p>&nbsp;<\/p>\n<h1>38: Docker Networking<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/course\/learn-docker\/learn\/lecture\/7894034#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/course\/learn-docker\/learn\/lecture\/7894034#content<\/a><\/p>\n<h2>Default Networks<\/h2>\n<ul>\n<li>Bridge\n<ul>\n<li>docker run IMAGE<\/li>\n<li>By default, all docker images get attached to this network\n<ul>\n<li>Usually 172.17.x.x<\/li>\n<\/ul>\n<\/li>\n<li>Containers can access each other via this network.<\/li>\n<li>To access these containers from the outside world, map the ports to the ports on the host.<\/li>\n<\/ul>\n<\/li>\n<li>None\n<ul>\n<li>docker run &#8211;network=none IMAGE<\/li>\n<li>No access to the host network, nor to any other container.<\/li>\n<\/ul>\n<\/li>\n<li>Host\n<ul>\n<li>Docker run &#8211;network=host IMAGE<\/li>\n<li>This removes distinction between the host and the container.<\/li>\n<li>Removes ability to run multiple containers on the same port.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<h2>Create custom networks<\/h2>\n<pre>docker network create --driver bridge --subnet 182.18.0.0\/16 custome-isolated-network<\/pre>\n<h3>To view all networks:<\/h3>\n<pre>docker network ls<\/pre>\n<h3>To find additional information about a network<\/h3>\n<pre>docker network inspect NETWORK<\/pre>\n<h3>To view a container&#8217;s network information<\/h3>\n<p>(you should already know this one)<\/p>\n<pre>docker inspect CONTAINER\r\n...\r\n            \"Gateway\": \"172.17.0.1\",\r\n            \"GlobalIPv6Address\": \"\",\r\n            \"GlobalIPv6PrefixLen\": 0,\r\n            \"IPAddress\": \"172.17.0.2\",\r\n            \"IPPrefixLen\": 16,\r\n            \"IPv6Gateway\": \"\",\r\n            \"MacAddress\": \"02:42:ac:11:00:02\",\r\n            \"Networks\": {\r\n                \"bridge\": {\r\n                    \"IPAMConfig\": null,\r\n                    \"Links\": null,\r\n                    \"Aliases\": null,\r\n                    \"NetworkID\": \"a09b6de4c1e5a5cd4eb5e4537d80c603c9ea4491c2f466deba1291d3d52aecf8\",\r\n                    \"EndpointID\": \"1528588ed06a50c1055bb8407609f7f6e48fee5e1363283744898b67e5f66699\",\r\n                    \"Gateway\": \"172.17.0.1\",\r\n                    \"IPAddress\": \"172.17.0.2\",\r\n                    \"IPPrefixLen\": 16,\r\n                    \"IPv6Gateway\": \"\",\r\n                    \"GlobalIPv6Address\": \"\",\r\n                    \"GlobalIPv6PrefixLen\": 0,\r\n                    \"MacAddress\": \"02:42:ac:11:00:02\",\r\n                    \"DriverOpts\": null<\/pre>\n<h2>Embedded DNS<\/h2>\n<p><span style=\"color: #ff0000;\"><strong>Always use the container names to connect to one another!<\/strong><\/span><\/p>\n<ul>\n<li>All containers have built in DNS that allows them to connect to each other via their container names.<\/li>\n<li>This allows them to re-connect in case of a reboot\/shutdown\/etc and the containers are not assigned the same IP address they had initially.<\/li>\n<li>The DNS server is located at 127.0.0.11<\/li>\n<\/ul>\n<h1>39: Labs<\/h1>\n<p><a href=\"https:\/\/www.udemy.com\/course\/learn-docker\/learn\/lecture\/15828672#content\" target=\"_blank\" rel=\"noopener\">https:\/\/www.udemy.com\/course\/learn-docker\/learn\/lecture\/15828672#content<\/a><\/p>\n<p>&nbsp;<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>&lt; 6 Docker Registry | 8 Docker on Mac and Windows &gt; 59% Complete 33: Docker Engine https:\/\/www.udemy.com\/learn-docker\/learn\/lecture\/15829078#content Architecture The &#8220;Engine&#8221; is the host with Docker installed on it. Docker Daemon Backgroud process that manages the Images Containers Volumes Networks REST API server Provides the interface to talk to the daemons Docker CLI Command line ..<\/p>\n<div class=\"clear-fix\"><\/div>\n<p><a href=\"https:\/\/wiki.thomasandsofia.com\/?p=2468\" title=\"read more...\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[52],"tags":[],"class_list":["post-2468","post","type-post","status-publish","format-standard","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/2468","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2468"}],"version-history":[{"count":15,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/2468\/revisions"}],"predecessor-version":[{"id":2697,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=\/wp\/v2\/posts\/2468\/revisions\/2697"}],"wp:attachment":[{"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2468"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2468"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wiki.thomasandsofia.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2468"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}