Docker Architecture

Introduction

Docker architecture is based on a Client–Server model that enables developers and system administrators to create, build, ship, and run applications inside containers. The main goal of Docker architecture is to provide a consistent environment where applications can run reliably across:

• Developer machines
• Testing servers
• Data centers
• Cloud platforms

Docker architecture consists of three major components: Docker Client, Docker Host (Docker Engine), and Docker Registry.

Docker Architecture Overview

    Docker Client
         |
         | REST API
         |
         ↓
    Docker Daemon (Docker Engine)
         |
    --------------------
    |                  |
 Containers          Images
    |                  |
  Volumes           Networks
         |
         ↓
    Docker Registry (Docker Hub)
    

Docker Client

The Docker Client is the command-line interface (CLI) that users interact with to communicate with Docker. Users execute Docker commands through the client. Examples: docker run nginx, docker build -t myapp ., docker ps. The Docker Client sends requests to the Docker Daemon through the Docker REST API.

Responsibilities: Sends commands to Docker Engine, Manages containers, Builds images, Pulls images, Starts/Stops containers.

Docker Host

The Docker Host is the machine where Docker Engine runs (Physical Server, Virtual Machine, or Cloud Instance like AWS EC2, Azure VM, Linux Server). It contains: Docker Daemon, Containers, Images, Volumes, and Networks.

Docker Engine (Docker Daemon)

The Docker Engine is the core component responsible for managing Docker objects. The Docker Daemon runs as a background service (process: dockerd). It listens for API requests from the Docker Client.

• Container Management: Creates, starts, stops, and removes containers.
• Image Management: Downloads and manages images.
• Network Management: Creates and manages container networks.
• Storage Management: Manages Docker volumes.

Docker Objects

1. Docker Images

An image is a read-only template used to create containers. Contains: Application code, Libraries, Dependencies, Runtime environment.

2. Docker Containers

A container is a running instance of a Docker image.

3. Docker Volumes

Volumes provide persistent storage for containers.

4. Docker Networks

Networks allow containers to communicate with each other.

Docker Registry

A Docker Registry stores and distributes Docker images (e.g., Docker Hub). Users can upload (push) and download (pull) images.

Docker Image Workflow

Developer -> Dockerfile -> Docker Build -> Docker Image -> Docker Registry -> Docker Pull -> Docker Container.

Docker REST API

Docker Client communicates with Docker Daemon using REST API via UNIX Socket, TCP Socket, or HTTP API.

Docker Container Runtime

Docker uses containerd (lifecycle management, image transfer, storage) and runc (creating/running container processes) to execute containers.

Docker Networking Architecture

Supports Bridge Network (default), Host Network, and Overlay Network (for Swarm/Kubernetes).

Docker Storage Architecture

Uses storage drivers like Overlay2, AUFS, or Device Mapper to stack the writable container layer on top of image layers.

Docker Architecture in DevOps

Pipeline: Developer -> Git -> Jenkins CI/CD -> Docker Build -> Image -> Registry -> Kubernetes Cluster -> Production.

Docker Architecture Advantages

• Lightweight application deployment
• Faster startup time
• Application isolation
• Easy scaling
• Portable environments
• Cloud-ready architecture
• Supports Microservices
• Enables CI/CD automation

Real-World Example

A typical production application: Users -> Load Balancer -> Nginx Container -> (Backend Container + Database Container) -> Docker Host (AWS EC2).

Summary

Docker Architecture consists of Docker Client, Docker Engine, Docker Host, Images, Containers, Volumes, Networks, and Registry. These components work together to build, distribute, and run containerized applications efficiently.

Docker Installation Guide

Introduction

Docker installation is the process of setting up the Docker Engine on an OS (Linux, Windows, macOS, or Cloud servers). For DevOps, Linux-based installation (Ubuntu Server) is the industry standard.

Docker Components Installed

• Docker Engine: Core service for running containers.
• Docker CLI: Command-line tool for management.
• Container Runtime: Includes containerd and runc.
• Docker Compose: Tool for managing multi-container applications.

System Requirements (Linux)

Requirement	Specification
CPU	64-bit Processor
RAM	Minimum 2 GB
OS	Ubuntu 20.04/22.04/24.04
Storage	Minimum 20 GB

Installing Docker on Ubuntu

Follow these essential steps:

1. Update: sudo apt update && sudo apt upgrade -y
2. Remove old versions: sudo apt remove docker docker-engine docker.io containerd runc
3. Install Dependencies: sudo apt install ca-certificates curl gnupg lsb-release -y
4. Add GPG Key & Repo: Configure Docker's official repository.
5. Install Engine: sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

Run Docker Test Container

Verify installation by running: sudo docker run hello-world

Configure Docker Without sudo

To avoid using sudo, add your user to the docker group:

    sudo usermod -aG docker $USER
    newgrp docker
    

Install on CentOS / RHEL

1. Install utils: sudo yum install -y yum-utils
2. Add Repo: sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
3. Install: sudo yum install docker-ce docker-ce-cli containerd.io -y

Docker Installation Verification

Use these commands to confirm health:

• docker version
• docker info
• docker ps (List running containers)

Docker Directory Structure

Docker stores data in /var/lib/docker/.

• containers/: Stores container data.
• images/: Stores downloaded images.
• volumes/: Persistent storage.
• overlay2/: Container filesystem layers.

Docker Service Management

Use systemctl commands:

• Start: systemctl start docker
• Stop: systemctl stop docker
• Status: systemctl status docker

Common Docker Installation Errors

• Permission Denied: Solve by adding user to docker group.
• Service Not Running: Run systemctl start docker.

Docker Installation in DevOps

A typical DevOps server setup runs the Docker Engine on Ubuntu to host Nginx, MySQL, and application containers.

Summary

Docker installation is the foundational step for building modern DevOps, CI/CD, and Microservices-based environments.

Docker Images & Containers

Introduction

In Docker, Images and Containers are the two most important concepts.

• A Docker Image is a blueprint/template used to create containers.
• A Docker Container is a running instance of a Docker image.

Simple analogy: Docker Image = Application Template, Docker Container = Running Application.

What is a Docker Image?

A Docker Image is a read-only package that contains everything required to run an application, including application code, OS files, runtime environment, libraries, dependencies, and configuration files.

Characteristics of Docker Images

Read-Only: Images cannot be modified after creation; changes are stored in new layers.

Portable: The same image can run on developer laptops, testing servers, cloud servers, and production.

Layer-Based: Built using multiple layers, which allows for faster builds, image reuse, and less storage usage.

Docker Image Lifecycle

Dockerfile → docker build → Docker Image → docker run → Docker Container.

Docker Image Commands

List Images: docker images

Download from Hub: docker pull nginx

Search: docker search nginx

Remove: docker rmi nginx

Inspect: docker inspect nginx

History: docker history nginx

Creating Docker Image Using Dockerfile

A Dockerfile defines instructions to build an image.

    FROM ubuntu:22.04
    RUN apt update
    RUN apt install nginx -y
    EXPOSE 80
    CMD ["nginx","-g","daemon off;"]
    

Build image: docker build -t my-nginx .

Docker Image Tags

Tags identify different versions (e.g., nginx:latest, nginx:1.25). Command: docker tag nginx nginx:v1

Docker Registry

Stores and distributes images (e.g., Docker Hub). Commands: docker pull mysql (download), docker push username/image-name (upload).

What is a Docker Container?

A running environment created from a Docker image. It contains application process, file system, network interface, environment variables, and runtime configuration.

Characteristics of Containers

• Lightweight: Share host OS kernel, consuming fewer resources than VMs.
• Isolated: Each container runs independently.
• Fast Startup: Start within seconds.
• Portable: Run consistently across environments.

Container Lifecycle

Create → Start → Running → Stop → Remove.

Container Commands

Create: docker create nginx

Run: docker run nginx (use -d for detached mode).

List Running: docker ps

List All: docker ps -a

Stop/Start/Restart: docker stop/start/restart container_id

Remove: docker rm container_id

Logs: docker logs container_id

Access Shell: docker exec -it container_id bash

Difference Between Docker Image and Container

Feature	Docker Image	Docker Container
Definition	Blueprint	Running instance
Nature	Static	Dynamic
Persistence	Read-only	Writable layer

Image and Container Relationship

One image can create multiple containers (e.g., one Ubuntu image creating Web, DB, and API containers).

Container Writable Layer

Containers have a writable layer on top of image layers. Data in the writable layer is lost when the container is deleted. Use Volumes or Bind Mounts for permanent storage.

Docker Image vs VM

Feature	Docker Container	Virtual Machine
Size	MB	GB
Startup	Seconds	Minutes
OS	Shares Host Kernel	Full OS

Real-World DevOps Example

Pipeline: Git → Dockerfile → Image → Registry → Container → Production Server.

CI/CD Integration

CI/CD Pipeline: Push code to GitHub → Jenkins builds image → Pushes image → Deploys container.

Best Practices

Images: Use official images, keep small, use version tags, scan for vulnerabilities.

Containers: Use env variables, store data in volumes, monitor logs, avoid running as root.

Summary

Docker Image is the blueprint. Docker Container is the running instance. Understanding these is essential for all DevOps and Cloud Engineers.

Dockerfile Creation

Introduction to Dockerfile

A Dockerfile is a text-based configuration file that contains a set of instructions used to automatically build a Docker image. It defines the base OS, application installation steps, dependencies, environment variables, and startup commands. It acts as a blueprint for creating Docker images.

Dockerfile Naming

The default filename is simply Dockerfile. A typical project structure looks like this:

    my-web-app/
    ├── Dockerfile
    ├── index.html
    ├── app.py
    ├── requirements.txt
    └── config/
    

Basic Syntax

The general format is INSTRUCTION arguments. Example:

    FROM ubuntu:22.04
    RUN apt update
    COPY app /app
    CMD ["./app"]
    

Important Dockerfile Instructions

• FROM: Defines the base image (e.g., python:3.12).
• LABEL: Adds metadata (maintainer, version).
• RUN: Executes commands during build (installing/updating software).
• COPY: Copies files from the local host to the image.
• ADD: Like COPY, but supports URLs and extracting compressed files.
• WORKDIR: Sets the working directory inside the container.
• EXPOSE: Documents the port the application listens on.
• CMD: Defines the default startup command.
• ENTRYPOINT: Defines the main executable of the container.
• ENV: Sets environment variables.
• USER: Sets the user context to avoid running as root.
• VOLUME: Creates a mount point for persistent storage.
• ARG: Defines variables used only during the build process.

Creating a Simple Dockerfile

Example for an Nginx Web Server:

    FROM nginx:latest
    COPY index.html /usr/share/nginx/html/
    EXPOSE 80
    CMD ["nginx","-g","daemon off;"]
    

Build and Run

Build: docker build -t my-nginx .

Run: docker run -d -p 8080:80 my-nginx

Application Examples

Python Application: Uses FROM python:3.12, WORKDIR /app, pip install, and CMD ["python", "app.py"].

Node.js Application: Uses FROM node:20, npm install, and CMD ["npm", "start"].

Multi-Stage Builds

Used to reduce image size by separating the build environment from the production runtime.

Dockerfile Build Process

Developer Code → Dockerfile → Build → Image Layers → Docker Image → Container.

Best Practices

• Use small base images like alpine.
• Combine RUN commands to reduce layers.
• Always use a .dockerignore file.
• Avoid secrets in the Dockerfile; use Environment variables or Docker Secrets.
• Use specific image versions instead of latest.

Dockerfile Debugging

Use docker history image_name to view layers, docker inspect to check details, and docker exec -it container_id bash to enter a running container.

Dockerfile in DevOps Pipeline

GitHub → Jenkins Pipeline → Build Image → Push to Docker Hub → Deploy Container.

Summary

A Dockerfile automates the installation and packaging of applications. Mastering it is essential for DevOps Engineers, Cloud Engineers, and CI/CD professionals.

Docker Volumes

Introduction to Docker Volumes

Docker Volumes are a storage mechanism used to persist data generated and used by Docker containers. By default, data inside a container exists only in the writable layer and is lost when the container is removed. Docker Volumes solve this by storing data outside the container lifecycle.

Why Do We Need Docker Volumes?

Containers are temporary. Without a volume, database data or application logs are deleted when a container is removed. Volumes are essential for database storage, application data, logs, configuration files, and backups.

Docker Storage Architecture

Docker uses a layered architecture where the container's writable layer sits on top of read-only image layers, while volume mounts provide a bypass for persistent data storage.

Types of Docker Storage

• Docker Volumes: Managed by Docker (Recommended).
• Bind Mounts: Maps a host directory directly into a container.
• tmpfs Mounts: Stores data in memory (for temporary or sensitive data).

Docker Volume Architecture

Creating Docker Volumes

Create: docker volume create database-data

List: docker volume ls

Inspect: docker volume inspect database-data

Remove: docker volume rm database-data or docker volume prune for unused.

Using Volumes with Containers

Use the -v flag to attach a volume:

    docker run -d --name mysql-db \
    -v mysql-data:/var/lib/mysql \
    mysql
    

Bind Mounts

Connects a host directory to a container directory:

    docker run -v /home/user/project:/app nginx
    

Docker Volume vs Bind Mount

Feature	Docker Volume	Bind Mount
Managed by Docker	Yes	No
Location	Docker directory	User-defined
Production Use	Recommended	Dev use

Volumes in Docker Compose

    services:
      database:
        image: mysql
        volumes:
          - mysql-data:/var/lib/mysql
    volumes:
      mysql-data:
    

Sharing Volumes Between Containers

Multiple containers can mount the same volume, allowing shared access to data files or logs.

Backup & Restore

Back up data by mounting the volume to a temporary Ubuntu container and running tar commands to archive the data to the host.

Security Best Practices

• Use read-only volumes (:ro) when possible.
• Monitor disk usage with docker system df.
• Never store sensitive credentials/keys directly in volumes; use Docker Secrets.

Docker Volumes in Cloud Environments

In production (AWS, Azure, etc.), volumes often map to cloud-native storage like AWS EBS, ensuring data persists even if the EC2 instance is rebooted or replaced.

Docker Volume Interview Questions

Common questions: What is a volume? Where are they stored? How are they different from bind mounts? Which one is recommended for production?

Summary

Docker Volumes decouple data from the container lifecycle, ensuring application data remains safe even if containers are stopped or deleted. They are a fundamental building block for production-grade DevOps environments.

Docker Networking

Definition & Importance

Docker Networking enables communication between containers, the host, and external networks. It provides isolation, security, and connectivity.

Why is it required? It allows container-to-container communication within an application stack, supports distributed applications, and secures environments by isolating services.

Network Architecture

Docker uses a virtual networking layer called CNM (Container Network Model) to manage how containers connect to each other and the outside world.

Network Components

• Network Sandbox: Provides isolation for each container (interfaces, IP, routing table, DNS).
• Endpoint: Connects a container to a Docker network.
• Network: A virtual construct that allows containers to communicate.

Docker Network Drivers

• Bridge Network: The default driver; used when containers run on a single host.
• Host Network: Removes network isolation; containers share the host's IP/network stack directly.
• None Network: No network access; provides maximum security and isolation.
• Overlay Network: Connects containers across multiple Docker hosts (ideal for Swarm/Kubernetes).
• Macvlan Network: Assigns a MAC address to containers, making them appear as physical devices on your network.

Port Mapping

To access an isolated container from the outside world, you use port mapping: docker run -p [HostPort]:[ContainerPort] nginx.

Docker DNS Service Discovery

Docker provides automatic DNS resolution. Containers can communicate using their container names instead of IP addresses within a custom bridge network (e.g., mysql:3306).

Docker Network Commands

• List: docker network ls
• Create: docker network create [name]
• Inspect: docker network inspect [name]
• Connect/Disconnect: docker network connect/disconnect [net] [container]

Docker Compose Networking

Docker Compose automatically creates a network for your services, allowing them to communicate via service names defined in the docker-compose.yml file.

Best Practices

• Use custom bridge networks, not the default bridge.
• Only expose ports that are absolutely necessary.
• Use DNS names (service/container names) instead of hardcoding IPs.
• Separate frontend and backend networks for better security.

Real-World DevOps Example

In production, traffic flows through a Load Balancer to an Nginx container (Frontend Network), which talks to an Application container (Backend Network), which finally accesses a Database container (Database Network).

Summary Table

Network Type	Purpose
Bridge	Default, single-host communication
Host	High-performance, no isolation
None	Complete network isolation
Overlay	Multi-host container communication
Macvlan	Physical network integration

Docker Compose

Definition & Purpose

Docker Compose is a tool used to define, configure, and manage multi-container applications using a YAML configuration file (compose.yaml or docker-compose.yml). It replaces manual docker run commands with a single automated configuration.

Benefits: Manages multi-container stacks, automates deployments, simplifies networking/volumes, and creates reproducible development environments.

Docker Compose Architecture

Compose takes your configuration file and manages the entire lifecycle of services, networks, and volumes as a single unit.

File Structure

Commonly named docker-compose.yml. A simple example:

    version: "3.8"
    services:
      web:
        image: nginx
        ports:
          - "8080:80"
      database:
        image: mysql
        environment:
          MYSQL_ROOT_PASSWORD: password
    

Main Components

• Services: The containers that make up your app (e.g., web, api, db).
• Images: The source templates (e.g., node:20).
• Build: Path to a Dockerfile for custom images.
• Ports: Maps host ports to container ports.
• Volumes: Ensures persistent storage for databases/files.
• Environment Variables: Configuration settings like passwords or API keys.

Useful Commands

Start: docker compose up -d (Run in background)

Stop: docker compose down

Logs: docker compose logs -f

Status: docker compose ps

Real-World Example (WordPress)

A WordPress stack links a wordpress container to a mysql container using internal networking, with persistent data managed via Volumes.

Networking & Scaling

Compose creates a default private network for all services, allowing them to communicate via service names (e.g., mysql:3306). You can also scale services easily: docker compose up --scale web=3.

.env Configuration

Keep sensitive data out of the YAML file by using a .env file. Access variables in the compose file using ${VARIABLE_NAME}.

Docker Run vs Compose

Feature	Docker Run	Docker Compose
Scale	Single container	Multiple containers
Config	Command-based	YAML-based
Management	Manual	Automated

Docker Compose in DevOps

Docker Compose is widely used in development environments and CI/CD pipelines to spin up entire application stacks for integration testing before deploying to production (Kubernetes/Swarm).

Best Practices

• Use version-controlled compose files.
• Store secrets securely (don't hardcode passwords).
• Always use named volumes for database persistence.
• Use Healthchecks to ensure containers are ready before depending on them.
• Pin image versions (don't use latest).

Summary

Docker Compose is essential for managing multi-container microservices. It transforms complex manual deployments into a single, declarative YAML configuration, making it a cornerstone skill for DevOps engineers.

Docker Multi-Stage Builds

Definition

Docker Multi-Stage Builds is a technique that allows you to use multiple FROM statements in a single Dockerfile. You create different stages for building and copying only the required final output into a clean runtime image.

Why Multi-Stage Builds Are Needed?

Traditional builds include compilers, source code, and build tools, leading to large images and security risks. Multi-stage builds separate the Build Environment from the Production Environment, ensuring only necessary files exist in the final image.

Multi-Stage Build Architecture

The build stage compiles the code, and the final production stage takes only the binary or static files, leaving behind the heavy tools.

Basic Syntax

    FROM image AS stage-name
    # Build instructions
    FROM another-image
    COPY --from=stage-name files destination
    

Node.js Application Example

Instead of keeping the entire Node source and build tools, you build the app and copy only the final dist folder into a lightweight Nginx image.

Java Application Example

Use a maven image to compile the source into a .jar file, then copy that .jar into a lightweight openjdk runtime image.

Go Application Example

Go apps are perfect for this: use a golang image to compile the binary, then copy it into an alpine image, resulting in a tiny, secure executable.

Naming Build Stages

You can name stages (e.g., FROM ubuntu AS development) to make the Dockerfile more readable and to target specific stages during the build process using --target.

Multi-Stage Build with Testing

You can add a test stage in the Dockerfile. If the tests fail, the image build fails, ensuring only tested code reaches the production stage.

Advantages

• Smaller Images: Reduces size from GBs to MBs.
• Improved Security: Removes source code and sensitive build tools from production.
• Faster Deployment: Quicker downloads and startup times.
• CI/CD Friendly: Standardized flow for Jenkins, GitHub Actions, and GitLab CI.

Best Practices

• Use small base images like Alpine.
• Separate build and runtime stages clearly.
• Copy only necessary artifacts.
• Use .dockerignore to exclude unnecessary files like node_modules or .git.

Multi-Stage Build in DevOps

The workflow moves from Git Repository → Docker Build → Testing Stage → Production Image → Registry → Kubernetes Deployment.

Multi-Stage Build vs Single-Stage

Feature	Single Stage	Multi Stage
Image Size	Large	Small
Security	Lower	Higher
Production Ready	Less optimized	Highly optimized

Summary

Multi-stage builds are essential for creating optimized, secure, and production-ready images. They are a mandatory skill for modern DevOps engineering.