Container Forensics

Last major content update: 13 April, 2021
 

Introduction

The target audience for this article is seasoned forensicators that have a beginner/intermediate knowledge of containers. If you need to brush up on the basics, I recommend the following:


On to the forensicating part…

Due to their self-healing nature and the fact that containers are usually running something business-critical, be prepared for the very high likelihood that containers will be terminated before you get to the forensic-part. Containers are a newer IT paradigm, in which diagnosing a problem takes longer than just blowing it all out and starting all new containers from the gold images. Consequently, IT admins often do not bother with diagnosis. Moreover, many of the health/status tools automatically identify and kill malfunctioning containers. All of that sucks for the forensic examiner, but is unquestionably valuable to the business. Even worse, containers are ephemeral and do not have inherently persistent storage. So unless the logs are writing to an external location you will not be able to quickly diagnose a root cause once that container has been terminated.

With those points in mind, let us address the obvious question, “If containers heal themselves, and the IT folks remediate by cleanly refreshing them, why bother learning container forensics?” Because there are use cases that require investigation, even if you might have to do it with one forensic arm tied behind your back. Some examples are a rogue IT team member running a coin mining pod, ransomware that has broken out of container isolation, or a new zero-day vulnerability of which we are woefully unprepared (that one doesn’t happen, right?) In summary, and as mentioned in previous articles, knowledge and preparation are key, and the middle of a time-sensitive incident response is not the time to be learning how containers function.

A couple disclaimers:

  • I am not an expert in container architecture so if you think I am wrong about anything, please comment below and I will correct it.
  • Almost none of this information represents original research on my part, but rather consolidating others’ excellent research into a guide of sorts. If you see anything that I failed to cite please let me know at dfwforensics@gmail.com and I will fix it immediately.
  • Although I am adding content for AWS and Azure, I have not had much experience in GCP so I apologize, both to the reader and to Google….who actually created Kubernetes. The good part is a lot of this will apply to GCP as well.
Terms

DOCKER

  • Docker – a popular container format
  • Image – the container’s gold copy
  • Manifest – the container’s configuration information (how many layers, size, OS, etc.) This is the description of the docker image. Below is an example of a Docker Manifest taken from here.
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 7023,
        "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
    },
    "layers": [
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 32654,
            "digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 16724,
            "digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
        },
        {
            "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
            "size": 73109,
            "digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
        }
    ]
}

KUBERNETES (K8S)

  • Kubernetes – a popular orchestration framework that automates container operations
  • Pod – a Kubernetes term for one or more containers that share the same configuration, networking, etc. K8S assigns a single DNS to a Pod. It helps if you notice Docker’s emblem is a whale and a group of whales is a pod.
  • Deployment / Replica Set – The Deployment object is your desired pod state. It manages the number of pod replicas. In most cases, you are only going to see containers/pods get created through a “kind: Deployment” object, and not directly through a “kind: Pod“.
    To ease the confusion, the hierarchy is….
    Deployment –> Replica Sets –> Pods –> Container
  • The Service – Although pods on the same host can see each other, this is the REST object that provides pod/pod and pods/non-pod interaction. For example, a front-end pod to a backend pod, or a pod to the load balancer, persistent storage, database, etc. Like most things in K8S, the Service is a REST object and will have its own IP/DNS. Moreover, since the point of all this is autonomous orchestration of groups of containers, K8S will build/destroy the containers as needed; that means the IP’s and container states are not persistent. That is what the Service does; it creates a stable IP for networking purposes.
Artifacts

This is not a complete list and will be supplemented over time. Please consider providing your comments via Twitter or LinkedIn at the links below. For such a new technology, any input is most welcome!

Yaml files

YAML is the chosen format for Kubernetes configuration files. As far as I know, there is no fixed path. They can even be piped objects. However, these are the files you should request from the IT team as they contain the number of pods, image types, ports, etc.

Jumping over to Docker for a minute, it is worthwhile to note you will also see YAML files used in the implementation for “Docker compose”. While “Docker build” may be the method to create a docker image from some baseline (as well as add whatever the container needs for the specific use case) “Docker compose” is the current best practice, as it can read a Dockerbuild file, as well as the build files of other relevant containers (db backend, service objects, etc), and spin up the whole stack.

Back to YAML now. An in-depth description of YAML is outside the scope of this article. However, it is worth understanding two points. First, YAML is a superset of JSON, so you may see the system administrators using JSON files. Second, YAML has only two structures, Lists and Maps, so if it does not look like a list it is a map. The third equally important point of our two points is that YAML is nest-able.

The most important name-value pair in these YAML files is the “kind” object. For example, the one below is labeled “kind: Deployment.” That will create a deployment object, which deploys and manages the pods. If that field says “kind: pod“, that would instead create a pod.

NOTE: You may want to look at any command line executions that are baked directly into the creation. See “Post Manifest (Example with Command Execution)” below.

Command: 
kubectl apply -f

Pod Manifest (Web App Example)

A pod manifest is a YAML document that contains the startup parameters and application defaults for when a web application is launched.

Here’s an example manifest for a web server pod named nginx-demo.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-demo
  labels:
    role: myrole
spec:
  containers:
    - name: nginx 
      image:nginx:1.14.2
      ports:
        - name: web
          containerPort: 80
          protocol: TCP

From:https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/

Pod Manifest (Example with Command Execution)

This is a pod manifest that has command line executions baked in.

apiVersion: v1
kind: Pod
metadata:
  name: command-demo
  labels:
    purpose: demonstrate-command
spec:
  containers:
  - name: command-demo-container
    image: debian
    command: ["printenv"]
    args: ["HOSTNAME", "KUBERNETES_PORT"]
  restartPolicy: OnFailure

From: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/

Deployment Manifest (Web App Example)

Here is a manifest that deploys a web app replica set.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mywebapp-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

From: https://k8s.io/examples/controllers/deploy.yaml

Command line history

Commands sent to the orchestrator (Kubernetes) and containers (Docker) will come from a source workstation. So the originating source of the system administrator’s command line history (e.g. bash history, PowerShell Get-History, etc) should be in-scope.

You should also obtain copies of the system administrator’s locally-stored copies of the YAML files that were initially used to set up the containers and orchestration.

Also, if you believe the maliciousness does not originate from the host straight to its child containers, you may want to assign someone to the network forensics portion. See Incident Response in the Cloud – Part I –> Detection –> Network Forensics

The Service Definition

As stated previously, the Service is the piece that allows the specific interaction between pods and between pod/non-pod resources, like an external load balancer, persistent storage, database, etc. The Service is a REST object and will have its own IP/DNS.

This is an example of the creation (via “POST”) of a Kubernetes Service. “mywebapp”, which will get an assigned IP and will map incoming traffic from TCP 80 to the pod’s specified listening port, TCP 9376.

apiVersion: v1
kind: Service
metadata:
  name: myservice
spec:
  selector:
    app: myservice
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376

From: https://kubernetes.io/docs/concepts/services-networking/service/

Cloud Orchestration logs

These are some of the more useful commands that cloud providers have created in order to interact with the containers.

AWS

TODO

Azure

Application logs

Container application logs (Az container logs) can be viewed in the command line

az container logs --resource-group myResourceGroup --name mycontainer
Traceback (most recent call last):
File "wordcount.py", line 11, in
urllib.request.urlretrieve (sys.argv[1], "foo.txt")
File "/usr/local/lib/python3.6/urllib/request.py", line 248, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(args) File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(args)
File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

From: MicrosoftAzurecontainerlogsreference

Container startup logs

Explained further in the forensic workflow section. Essentially, this command can direct the container stdout and stderror to you.

az container attach --resource-group myResourceGroup --name mycontainer
Docker Desktop (WSL 2)

The data sources below capture interactions between:

1) the Windows host –> Docker Desktop (a MobyLinuxVM using WSL 2.)
2) The Docker Desktop VM –> Docker containers
3) NOT the Docker container commands (AFAIK), which are likely the most relevant. For that data source, please refer to the subsequent sections of this article.

Be advised, those paths are reconfigurable by the system administrator.

> The Docker Desktop host VM (.vhdx)
     C:\Users\<username>\AppData\Local\Docker\wsl\data

> Logs, Cache, BlobStorage, Leveldb, Cookies, etc 
     C:\Users\<username>\AppData\Roaming\Docker Desktop
     C:\Users\<username>\AppData\Roaming\Docker
     C:\Users\<username>\AppData\Local\Docker

> POSTs to WSL2 VM (MobyLinuxVM running Docker Desktop)
     C:\ProgramData\DockerDesktop
Data Volumes

As mentioned previously, containers do not have inherently persistent storage. It is necessary to either ask the system administrator for the mount points or look for the mount section in “Docker inspect <container>“.

Mounts come in three flavors

  • Volumes
    • For Linux, they should live in /var/lib/docker/volumes
    • For Windows, (i.e. Docker Desktop) please refer to the specific Docker Desktop (WSL 2) section. But essentially, the volumes should live within its “host” ext4 file system on the vhdx volume.
  • Bind mounts
    • These can exist anywhere. Look for them with the Docker inspect command above
  • tmpfs mounts
    • These are in the host RAM, so acquire it. Especially considering these mounts can be through named pipes as well.

Full PCAP


Given the reliance on the REST API it may be worthwhile to perform a network capture on the host VM, if there is a suspicion of malicious traffic. I have not tried this myself, but I believe you should be able to do this through the use of a “sidecar.”
Another possibility is ksniff, a pretty inventive tool from Eldad Rudich, that uses kubectl to upload tcpdump to your container, then pushes the output to your Wireshark instance.

DFIR Workflow

Please be advised, this will be supplemented/edited whenever I have time. Hopefully, the quick workflow below is, at least, a small amount of help during your cloud response.

Urgent Containment Needed?

Quarantine the container if possible.

Docker containers can be paused/suspended with “docker pause“. On Linux, this leverages the freezer cgroup and the processes will not be aware or able to control it. On Windows, the container can be paused if it is using Hyper-V.

Jonathan Seawright astutely pointed out the container provider may also be able to quarantine a container with a firewall isolation policy. I would add the host EDR tools may have some capability to quarantine the host itself.

It is important to note, these are capabilities you should understand in your environment before employing, as any quarantine attempts, whether at the container or host level, have the potential for instability. Additionally, Docker creates its own iptables and config entries so it is exceedingly difficult to quarantine a container with just a simple command line.

If you cannot pause the container, I recommend stopping it, which will likely kill it. Reference SIGTERM / SIGKILL in the next section’s command list

Not an Urgent Containment

Analyze for Container Escape

StackRox did a short, but outstanding, blog post on Container Escape and I recommend reading it in-full here.

They recommend you review the container’s deployment.yaml to see if the container was launched with any of the flags below. If these flags are present in the launch, then you may want to image the host. A good next step would be to follow the instructions provided by Jonathan Greig in his excellent article on post-host-imaged container analysis.

If the flags below are not present, then you can limit your analysis to the container processes.

  • --cap-add *
  • --device *
  • --device-cgroup-rule
  • --ipc system
  • --mount /
  • --pid system
  • --privileged
  • --security-opt
  • --volume /
  • --volumes-from

No Container Escape
  • Acquire relevant data sources referenced in the “Artifacts” section above and/or in the Azure/AWS sections that follow
  • suspend/pause using
    • docker container pause <container name>.
    • This will keep the memory/differencing intact.
  • Commit the container’s changes to a new instance for spinning up in your forensic environment.
    • docker commit $CONTAINER_ID <source image>
  • [OPTIONAL 1] Remove the misbehaving container from the orchestrator. Kubectl has a couple of ways to do this. Recommend deferring to the business unit’s system administrator or referring to the K8S documents.
  • Launch the new container in an isolated forensic environment for analysis (e.g. your AWS forensic VPC, your Azure forensic Resource Group, or your on-prem forensic lab environment.)
  • [OPTIONAL 2] Consider going old school on the containers and comparing a gold image hash set to the container you are about to acquire.
    • Get a known hash set
      • Spin up a clean container or run the hash command (below) against known clean containers. Make sure to use the original deployment config files. I recommend getting the exact steps from the system administrator.
    • Get the hashes from the suspicious container
    • Compare the hashes
      • Run the below command on the gold containers and the infected containers, then remove all the common files. Or load the known hash set into your forensic tool of choice and decimate the data set. (note the parenthesis)
  • [OPTIONAL 3] As mentioned in the Container Escape section above, if you want to pursue an alternate workflow and you have the time to get an image of the full host, you can follow the instructions provided by Jonathan Greig in his excellent article on host-based container analysis. I have also included some useful commands in the sections that follow.
HASH ALL THE THINGS...

docker exec -it <containerID> sh -c \
  "find . -name '*' -type f -print0 | \
  xargs -0 md5sum" > \ 
  md5sum_<KnownOrSuspiciousContainer>.txt
AWS

TODO

Azure


Container Host Forensics

Microsoft did a great job at explaining Azure container commands here. The forensically-useful points are below.

Container Startup Diagnostics

This points stdout (console) and stderr (error) to the container’s output so you can read the ongoing logs.

az container attach --resource-group myResourceGroup --name <mycontainer>
Example Output

Container 'mycontainer' is in state 'Unknown'…
Container 'mycontainer' is in state 'Waiting'…
Container 'mycontainer' is in state 'Running'…
(count: 1) (last timestamp: 2019-03-21 19:42:39+00:00) pulling image "<imagename>:latest"
Container 'mycontainer' is in state 'Running'…
(count: 1) (last timestamp: 2019-03-21 19:42:39+00:00) pulling image "mcr.microsoft.com/azuredocs/<imagename>:latest"
(count: 1) (last timestamp: 2019-03-21 19:42:52+00:00) Successfully pulled image "mcr.microsoft.com/azuredocs/<imagename>:latest"
(count: 1) (last timestamp: 2019-03-21 19:42:55+00:00) Created container
(count: 1) (last timestamp: 2019-03-21 19:42:55+00:00) Started container
Container Startup Events
az container show --resource-group myResourceGroup --name <mycontainer>

Example output:

{
  "containers": [
    {
      "command": null,
      "environmentVariables": [],
      "image": "mcr.microsoft.com/azuredocs/<aci-myimage>",
      ...
        "events": [
          {
            "count": 1,
            "firstTimestamp": "2019-03-21T19:46:22+00:00",
            "lastTimestamp": "2019-03-21T19:46:22+00:00",
            "message": "pulling image \"mcr.microsoft.com/azuredocs/<aci-myimage>\"",
            "name": "Pulling",
            "type": "Normal"
          },
          {
            "count": 1,
            "firstTimestamp": "2019-03-21T19:46:28+00:00",
            "lastTimestamp": "2019-03-21T19:46:28+00:00",
            "message": "Successfully pulled image \"mcr.microsoft.com/azuredocs/<aci-myimage>\"",
            "name": "Pulled",
            "type": "Normal"
          },
          {
            "count": 1,
            "firstTimestamp": "2019-03-21T19:46:31+00:00",
            "lastTimestamp": "2019-03-21T19:46:31+00:00",
            "message": "Created container",
            "name": "Created",
            "type": "Normal"
          },
          {
            "count": 1,
            "firstTimestamp": "2019-03-21T19:46:31+00:00",
            "lastTimestamp": "2019-03-21T19:46:31+00:00",
            "message": "Started container",
            "name": "Started",
            "type": "Normal"
          }
        ],
        "previousState": null,
        "restartCount": 0
      },
      "name": "<mycontainer>",
      "ports": [
        {
          "port": 80,
          "protocol": null
        }
      ],
      ...
    }
  ],
  ...
}
Container Interactive Session

If you want to query the container in a non-forensically-sound manner, you can launch an Interactive session. Consider then acquiring the data points enumerated by Sandfly Security

az container exec \
--resource-group <my-resourcegroup> \
--name mycontainer \
--exec-command /bin/sh (to launch a shell) 

(or you can just execute a command from right here similar to the hash command in the previous section)
Coinminer?

Check the container metrics using Azure Monitor.

Here is is a list of the currently available container metrics. Be advised, AZ Monitor’s default retention is 93 days

AZ Monitor is set to provide the averages over all the containers in a resource group but you can get the individual container metrics by using an Azure “dimension.”

You can also use the Azure CLI to get the instance-specific container metrics. Microsoft provides the following instructions to do so.

Below is an example of container metrics by resource group (not by individual container). To get the individual container metrics, just add –dimension <containerName>.

CPU metrics

CONTAINER_ID=$(az container show \
--resource-group <my azure resource group> \
--name <mycontainer> \
--query id \
--output tsv

az monitor metrics list \ 
--resource $CONTAINER_ID \ 
--metric CPUUsage \ 
--output table
Example Output:

Timestamp Name Average
2018-08-20 21:39:00 CPU Usage
2018-08-20 21:40:00 CPU Usage
2018-08-20 21:41:00 CPU Usage
2018-08-20 21:42:00 CPU Usage
2018-08-20 21:43:00 CPU Usage 0.375
2018-08-20 21:44:00 CPU Usage 0.875
2018-08-20 21:45:00 CPU Usage 1
2018-08-20 21:46:00 CPU Usage 3.625
2018-08-20 21:47:00 CPU Usage 1.5
2018-08-20 21:48:00 CPU Usage 2.75
2018-08-20 21:49:00 CPU Usage 1.625
2018-08-20 21:50:00 CPU Usage 0.625
2018-08-20 21:51:00 CPU Usage 0.5
2018-08-20 21:52:00 CPU Usage 0.5
2018-08-20 21:53:00 CPU Usage 0.5


Memory metrics

az monitor metrics list \
--resource $CONTAINER_ID \
--metric MemoryUsage \
--output table
Example Output:

Timestamp Name Average
2018-08-20 21:43:00 Memory Usage
2018-08-20 21:44:00 Memory Usage 0.0
2018-08-20 21:45:00 Memory Usage 15917056.0
2018-08-20 21:46:00 Memory Usage 16744448.0
2018-08-20 21:47:00 Memory Usage 16842752.0
2018-08-20 21:48:00 Memory Usage 17190912.0
2018-08-20 21:49:00 Memory Usage 17506304.0
2018-08-20 21:50:00 Memory Usage 17702912.0
2018-08-20 21:51:00 Memory Usage 17965056.0
2018-08-20 21:52:00 Memory Usage 18509824.0
2018-08-20 21:53:00 Memory Usage 18649088.0
2018-08-20 21:54:00 Memory Usage 18845696.0
2018-08-20 21:55:00 Memory Usage 19181568.0



More Useful DFIR Commands

Kubernetes & Docker (also AWS & Azure for that matter) use a base command with options. They can be called with –help for additional instructions.

Kubernetes

Get commands with basic output

# List all services in the namespace
kubectl get services

# List all pods in all namespaces
kubectl get pods --all-namespaces

# List all pods in the current namespace, with more details 
kubectl get pods -o wide

# List a particular deployment
kubectl get deployment <my-deployment>

# List all pods in the namespace
kubectl get pods

# Get a pod's YAML
kubectl get pod my-pod -o yaml

# Describe commands with verbose output
kubectl describe nodes <my-node>
kubectl describe pods <my-pod>

# List Services Sorted by Name
kubectl get services --sort-by=.metadata.name

# List pods Sorted by Restart Count
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'

# List PersistentVolumes sorted by capacity
kubectl get pv --sort-by=.spec.capacity.storage

# Get the version label of all pods with label app=cassandra
kubectl get pods --selector=app=cassandra -o \
jsonpath='{.items[*].metadata.labels.version}'

# Retrieve the value of a key with dots, e.g. 'ca.crt'
kubectl get configmap <myconfig> \
-o jsonpath='{.data.ca\.crt}'

# Get all worker nodes (use a selector to exclude results that have a label
# named 'node-role.kubernetes.io/master')
kubectl get node --selector='!node-role.kubernetes.io/master'

# Get all running pods in the namespace
kubectl get pods --field-selector=status.phase=Running

# Get ExternalIPs of all nodes
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'

# List Names of Pods that belong to Particular RC
# "jq" command useful for transformations that are too complex for jsonpath, it can be found at https://stedolan.github.io/jq/
sel=${$(kubectl get rc my-rc --output=json | jq -j '.spec.selector | to_entries | .[] | "\(.key)=\(.value),"')%?}
echo $(kubectl get pods --selector=$sel --output=jsonpath={.items..metadata.name})

# Show labels for all pods (or any other Kubernetes object that supports labelling)
kubectl get pods --show-labels

# Check which nodes are ready
JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}' \
&& kubectl get nodes -o jsonpath="$JSONPATH" | grep "Ready=True"

# Output decoded secrets without external tools
kubectl get secret my-secret -o go-template='{{range $k,$v := .data}}{{"### "}}{{$k}}{{"\n"}}{{$v|base64decode}}{{"\n\n"}}{{end}}'

# List all Secrets currently in use by a pod
kubectl get pods -o json | jq '.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name' | grep -v null | sort | uniq

# List all containerIDs of initContainer of all pods
# Helpful when cleaning up stopped containers, while avoiding removal of initContainers.
kubectl get pods --all-namespaces -o jsonpath='{range .items[*].status.initContainerStatuses[*]}{.containerID}{"\n"}{end}' | cut -d/ -f3

# List Events sorted by timestamp
kubectl get events --sort-by=.metadata.creationTimestamp

# Compares the current state of the cluster against the state that the cluster would be in if the manifest was applied.
kubectl diff -f ./<my-manifest>.yaml

# Produce a period-delimited tree of all keys returned for nodes. Helpful when locating a key within a complex nested JSON structure
kubectl get nodes -o json | jq -c 'path(..)|[.[]|tostring]|join(".")'

# Produce a period-delimited tree of all keys returned for pods, etc
kubectl get pods -o json | jq -c 'path(..)|[.[]|tostring]|join(".")'

Copied verbatim from: Kubernetes.io’s Kubernete Cheat Sheet

Docker
DFIR-specific

For this section, I thought it would me more useful if I pulled down a Docker image and ran some commands. Hopefully, that will help you understand the commands and outputs later. I have commented the commands as well.

The specific steps below cover:

  • reviewing the locally-stored images on the host
  • checking if any containers are running
  • starting a web app container and binding it to TCP 80
  • noting the SHA256 of the container
  • reading the access logs of the newly-running container
  • executing “ps -a” from the host into the container
  • open a shell in the container and running commands from inside the container

>> docker images
# the images that are available to use
# recommend noting any container that is suspiciously different
    
REPOSITORY                       TAG       IMAGE ID       CREATED       SIZE
dfwforensics/docker101tutorial   latest    a7dc23f996e8   3 hours ago   27.9MB
docker101tutorial                latest    a7dc23f996e8   3 hours ago   27.9MB
alpine/git                       latest    a939554ad0d0   4 weeks ago   25.1MB
---------------------------------------


>> docker ps
# check for running containers. None are running on this system yet


CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

---------------------------------------


>> docker run -dp 80:80 docker101tutorial
#this runs the image above, then binds the container's listening port to TCP 80.
#if the image does not exist locally, docker will download it.
#output is the sha256 of the image    

cdb9128363a9e96c2f954eccbc36f88dbe95c1a675931e3bf8985a4b862af1b1 
---------------------------------------


>> docker ps
# after the container is running you can query its summary information

CONTAINER ID   IMAGE               COMMAND                  CREATED 
cdb9128363a9   docker101tutorial   "/docker-entrypoint.…"   17 minutes ago
   
STATUS          PORTS                NAMES
Up 17 minutes   0.0.0.0:80->80/tcp   <containername>
---------------------------------------

>> docker inspect <mycontainer>
# use "inspect" to read a running container's metadata.
# or use "inspect" on an image with "--verbose" on a suspicious image 


[
    {
        "Id": "sha256:a7dc23f996e88f5787e8dc2112348fb022bd0a25f91644c92bcd50816dd24b6a",
        "RepoTags": [
            "dfwforensics/docker101tutorial:latest",
            "docker101tutorial:latest"
        ],
        "RepoDigests": [
            "dfwforensics/docker101tutorial@sha256:3800b76243ae13fdd3337f92084774457482e6ae33af6b5a51b7ca5d287026e8"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2021-03-25T17:15:37.0510086Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.19.8",
                "NJS_VERSION=0.5.2",
                "PKG_RELEASE=1"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/docker-entrypoint.sh"
            ],
            "Labels": {
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },
            "StopSignal": "SIGQUIT"
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 27941754,
        "VirtualSize": 27941754,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/u95ewzojeausn0e21kdz4dtu9/diff:/var/lib/docker/overlay2/221ca0b88a17dfdfc3f71c8e552a1aefd9213399d38284dde5e2f38b91465f5b/diff:/var/lib/docker/overlay2/d32278f833945f7e0e0a3cf752f631f1594515bde5b4cd426a5e39b7cd1d477a/diff:/var/lib/docker/overlay2/d0c54430881927715c04507b8da79dff77bedec352418b02c30f4839833f7868/diff:/var/lib/docker/overlay2/6c2981beeab6ea01fb679afece6b904fdac5bb6e2e270044521bf9bab3dc3d1e/diff:/var/lib/docker/overlay2/2bc0eee3896be31ea7c4ddc755f8f3fe5a077fec1db928d77899f19ecabc7d09/diff:/var/lib/docker/overlay2/0a09fdf0301e002ee7849200c716ab3cd19ad379041cb6c7d39892aff643cf8e/diff",
                "MergedDir": "/var/lib/docker/overlay2/8j883dxj8a4hvd9lqnmj7ixg0/merged",
                "UpperDir": "/var/lib/docker/overlay2/8j883dxj8a4hvd9lqnmj7ixg0/diff",
                "WorkDir": "/var/lib/docker/overlay2/8j883dxj8a4hvd9lqnmj7ixg0/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:cb381a32b2296e4eb5af3f84092a2e6685e88adbc54ee0768a1a1010ce6376c7",
                "sha256:369b3f326f894cccb059e895c7c3e08d640b51944f67ff87283884930e67b03e",
                "sha256:e7be5ec0da705e9db18584b3ea97fab2187badd90a2e8d130c586a0e85700cb3",
                "sha256:9f5787713b402dfff2f9788943b2067ec2bbad58569710e54fe9da6192770413",
                "sha256:9565c0a5609213d1852bfd029427a462f036b5d22ccb81f451a1c4d9a0753f3e",
                "sha256:b16c410e3fcaba32ea9ec1dcdb5056c7b1c343f676c4de2175f71b2bcea5fccd",
                "sha256:0407828aad85b225cba9d774ce3fb8fb212e1e6679037067d0b0390af676d322",
                "sha256:a32f9c0f1e2f8209fab8e8da5d1fac480af58bfdd89b493e2fa715d33bbdd021"
            ]
        },
        "Metadata": {
            "LastTagTime": "2021-03-25T17:29:34.297585Z"
        }
    }
]
---------------------------------------


>> docker logs cdb9128363a9 #note the container ID
# pulling this (web app) container's logs after web browsing in it a bit

172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET / HTTP/1.1" 200 8713 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/stylesheets/application.adb8469c.css HTTP/1.1" 200 76332 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/stylesheets/application-palette.a8b3c06d.css HTTP/1.1" 200 38773 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/fonts/material-icons.css HTTP/1.1" 200 873 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/javascripts/modernizr.86422ebf.js HTTP/1.1" 200 7296 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
...
...
172.17.0.1 - - [25/Mar/2021:20:21:04 +0000] "GET /tutorial/persisting-our-data/items-added.png HTTP/1.1" 200 63754 "http://localhost/tutorial/persisting-our-data/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:21:04 +0000] "GET /tutorial/persisting-our-data/dashboard-open-cli-ubuntu.png HTTP/1.1" 200 170038 "http://localhost/tutorial/persisting-our-data/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:21:05 +0000] "GET /tutorial/using-bind-mounts/ HTTP/1.1" 200 18697 "http://localhost/tutorial/persisting-our-data/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:21:05 +0000] "GET /tutorial/using-bind-mounts/updated-add-button.png HTTP/1.1" 200 21838 "http://localhost/tutorial/using-bind-mounts/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
---------------------------------------


 >> docker exec cdb9128363a9 ps -a
# running a command in the container

PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process nginx -g daemon off;
   33 nginx     0:00 nginx: worker process
   34 nginx     0:00 nginx: worker process
   35 nginx     0:00 nginx: worker process
   ...
   ...
   47 nginx     0:00 nginx: worker process
   48 nginx     0:00 nginx: worker process
   55 root      0:00 ps -a
---------------------------------------


>> docker ps
# starting a remote shell of the container - first, get the container info

CONTAINER ID   IMAGE               COMMAND                  CREATED          STATUS          PORTS                NAMES
cdb9128363a9   docker101tutorial   "/docker-entrypoint.…"   30 minutes ago   Up 30 minutes   0.0.0.0:80->80/tcp   <containername>
---------------------------------------


>> docker exec -it <containername> /bin/sh
# starting a remote shell of the container - second, start the shell

/ #

---------------------------------------


/ # ps -a
# You are now in the container's shell


PID   USER     TIME  COMMAND
    1 root      0:00 nginx: master process nginx -g daemon off;
   33 nginx     0:00 nginx: worker process
   34 nginx     0:00 nginx: worker process
   35 nginx     0:00 nginx: worker process
...
...
   47 nginx     0:00 nginx: worker process
   48 nginx     0:00 nginx: worker process
   61 root      0:00 /bin/sh
   68 root      0:00 /bin/sh
   75 root      0:00 ps -a
---------------------------------------

/ # pwd


/
---------------------------------------


/ # whoami


root
---------------------------------------


/home # more /etc/passwd

root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
...
...
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
nginx:x:101:101:nginx:/var/cache/nginx:/sbin/nologin
---------------------------------------


/home # more /etc/shadow

root:!::0:::::
bin:!::0:::::
daemon:!::0:::::
...
...
smmsp:!::0:::::
guest:!::0:::::
nobody:!::0:::::
nginx:!:18697:0:99999:7:::
---------------------------------------
Docker Command Help Menu

The full command list

>> docker

Usage:  docker [OPTIONS] COMMAND

A self-sufficient runtime for containers

Options:
      --config string      Location of client config files (default
                           "C:\\Users\\miche\\.docker")
  -c, --context string     Name of the context to use to connect to the
                           daemon (overrides DOCKER_HOST env var and
                           default context set with "docker context use")
  -D, --debug              Enable debug mode
  -H, --host list          Daemon socket(s) to connect to
  -l, --log-level string   Set the logging level
                           ("debug"|"info"|"warn"|"error"|"fatal")
                           (default "info")
      --tls                Use TLS; implied by --tlsverify
      --tlscacert string   Trust certs signed only by this CA (default
                           "C:\\Users\\miche\\.docker\\ca.pem")
      --tlscert string     Path to TLS certificate file (default
                           "C:\\Users\\miche\\.docker\\cert.pem")
      --tlskey string      Path to TLS key file (default
                           "C:\\Users\\miche\\.docker\\key.pem")
      --tlsverify          Use TLS and verify the remote
  -v, --version            Print version information and quit

Management Commands:
  app*        Docker App (Docker Inc., v0.9.1-beta3)
  builder     Manage builds
  buildx*     Build with BuildKit (Docker Inc., v0.5.1-docker)
  config      Manage Docker configs
  container   Manage containers
  context     Manage contexts
  image       Manage images
  manifest    Manage Docker image manifests and manifest lists
  network     Manage networks
  node        Manage Swarm nodes
  plugin      Manage plugins
  scan*       Docker Scan (Docker Inc., v0.5.0)
  secret      Manage Docker secrets
  service     Manage services
  stack       Manage Docker stacks
  swarm       Manage Swarm
  system      Manage Docker
  trust       Manage trust on Docker images
  volume      Manage volumes

Commands:
  attach      Attach local standard input, output, and error streams to a running container
  build       Build an image from a Dockerfile
  commit      Create a new image from a container's changes
  cp          Copy files/folders between a container and the local filesystem
  create      Create a new container
  diff        Inspect changes to files or directories on a container's filesystem
  events      Get real time events from the server
  exec        Run a command in a running container
  export      Export a container's filesystem as a tar archive
  history     Show the history of an image
  images      List images
  import      Import the contents from a tarball to create a filesystem image
  info        Display system-wide information
  inspect     Return low-level information on Docker objects
  kill        Kill one or more running containers
  load        Load an image from a tar archive or STDIN
  login       Log in to a Docker registry
  logout      Log out from a Docker registry
  logs        Fetch the logs of a container
  pause       Pause all processes within one or more containers
  port        List port mappings or a specific mapping for the container
  ps          List containers
  pull        Pull an image or a repository from a registry
  push        Push an image or a repository to a registry
  rename      Rename a container
  restart     Restart one or more containers
  rm          Remove one or more containers
  rmi         Remove one or more images
  run         Run a command in a new container
  save        Save one or more images to a tar archive (streamed to STDOUT by default)
  search      Search the Docker Hub for images
  start       Start one or more stopped containers
  stats       Display a live stream of container(s) resource usage statistics
  stop        Stop one or more running containers
  tag         Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
  top         Display the running processes of a container
  unpause     Unpause all processes within one or more containers
  update      Update configuration of one or more containers
  version     Show the Docker version information
  wait        Block until one or more containers stop, then print their exit codes

Run 'docker COMMAND --help' for more information on a command.

AWS / Azure / GCP

I recommend either looking at the examples in previous sections or simply performing a web search for “<cloudProviderName” AND <commandYouWant>. The online documentation is very good for all of these cloud providers. Additionally, once you understand the AWS/AZ/GCP command syntax, the commands are closely aligned with the standard Docker/K8S syntax.

SIFT Docker Image

update 1 Apr 21: I will leave the previously posted information below, but as expected, Digital Sleuth made a better docker image with greatly expanded capabilities compared to mine. I highly recommend you use that one. Simply clone that GitHub repository, navigate to your local folder that contains them, and launch “docker compose up -d“. That will give you a fully SIFT’d docker container, along with a SIFT subnet, gui, ssh, and mounting capabilities. So…..pretty much exactly like my docker image but cleaner. And smaller. And….better.
If you do it right it, the in-progress install will look like this:

>> cd .\sift-docker\
>> ls


    Directory: ..\sift-docker


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----          4/1/2021   9:31 AM                .git
-a----          4/1/2021   9:31 AM            485 docker-compose.yaml
-a----          4/1/2021   9:31 AM           2335 Dockerfile.bionic
-a----          4/1/2021   9:31 AM           2334 Dockerfile.focal
-a----          4/1/2021   9:31 AM          35149 LICENSE
-a----          4/1/2021   9:31 AM            945 README.md


>> docker compose up -d
[+] Building 628.0s (3/4)
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 73B                                                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => [internal] load metadata for docker.io/digitalsleuth/sift-docker:focal                                                                                                 3.5s
 => [1/1] FROM docker.io/digitalsleuth/sift-docker:focal@sha256:6c57dfb413adca4e10aa47ee56248e7c131905511d1f46275b3b13656ad829cf                                         624.5s
 => => resolve docker.io/digitalsleuth/sift-docker:focal@sha256:6c57dfb413adca4e10aa47ee56248e7c131905511d1f46275b3b13656ad829cf                                           0.0s
 => => sha256:b2055b333b029de2d76a27fbf5ee74e5ce5479467c4b7790b2d1d832a9c9515c 583.01MB / 1.50GB                                                                         624.5s
 => => sha256:728042ecb84323afa057d4d19a3c2ab6d9015829a15ecf49d458c4dbd542ffa4 111B / 111B                                                                                 0.4s
 => => sha256:6c57dfb413adca4e10aa47ee56248e7c131905511d1f46275b3b13656ad829cf 1.36kB / 1.36kB                                                                             0.0s
 => => sha256:e1b967990adc6e002937671ad01ffba85e51c71bcac3dd9fa3aae0e181790658 7.55kB / 7.55kB                                                                             0.0s

——————————————————
My SIFT docker image

Below is a link to a SIFT Docker image that you can download and play with. Please feel free to try it out. Just load Docker on your Linux workstation with a simple apt install, or if you have WSL 2 enabled, install Docker for Windows. Be advised, this SIFT image is not production-tested or SANS-authorized. Also, DigitalSleuth is working on a SIFT image and it will almost certainly be better than mine, so if that is available, use that one.

To start, just Google “Docker pull” and follow the quick instructions. Then, “Docker run <the other stuff you Googled>” and you will be up and running. At that point “Docker exec -it <container> /bin/bash” will give you an interactive shell in the container and you can work with your data just like you normally would in SIFT. Keep in mind, the value here is scalability, small size, compartmentation, and a clean instance for every case. The downside is docker does not have persistent storage so you should map a storage location if that is something you need. Like every other DFIR workflow task, it is only hard the first time you do it.

Once you have the container running, just copy some files to it and you are good to go. Copying files is easy and there are five different Google-able ways to do that. If you are using Docker for Windows, the easiest way is here. You can also access the container directly on your host. If you are running docker under WSL, just look for a hidden share with Windows Explorer “\\wsl$”

—————————————
THE UNAUTHORIZED AND PROBABLY TERRIBLE SIFT DOCKER IMAGE
—————————————

https://hub.docker.com/repository/docker/dfwforensics/sift2021_4_0ubuntu_20_04



Citations

Riadi, Imam & Umar, Rusydi & Sugandi, Andi. (2020). Web Forensic on Kubernetes Cluster Services Using Grr Rapid Response Framework. International Journal of Scientific & Technology Research. 9. 3484-3488.

Grieg, Jonathan (2021). Container Forensics with Docker Explorer. OSDFIR. https://osdfir.blogspot.com/2021/01/container-forensics-with-docker-explorer.html

StackRox. (2017). Forensics in the age of containers. Accessed from: https://www.stackrox.com/post/2017/08/csi-container-edition-forensics-in-the-age-of-containers

Ashnik. Container Forensics. Accessed from: https://www.ashnik.com/sysdig-resources/container-forensics/

Microsoft. (2019). Retrieve container logs and events in Azure Container Instances. Accessed from: https://docs.microsoft.com/en-us/azure/container-instances/container-instances-get-logs

unk author. (2020). Kubernetes Documentation. Accessed from: https://kubernetes.io/docs/home/


Burns, Brenden. (2015). A Technical Overview of Kubernetes – CoreOS Fest 2015. Accessed from: https://www.youtube.com/watch?v=WwBdNXt6wO4&ab_channel=CoreOS