Last major content update: 13 April, 2021
Introduction
The target audience for this article is seasoned forensicators that have a beginner/intermediate knowledge of containers. If you need to brush up on the basics, I recommend the following:
- Docker Overview
- Docker Tutorial
- Kubernetes Overview
- Kubernetes Tutorial
- Container Forensics Jonathan Greig’s excellent post-host-acquisition article
On to the forensicating part…
Due to their self-healing nature and the fact that containers are usually running something business-critical, be prepared for the very high likelihood that containers will be terminated before you get to the forensic-part. Containers are a newer IT paradigm, in which diagnosing a problem takes longer than just blowing it all out and starting all new containers from the gold images. Consequently, IT admins often do not bother with diagnosis. Moreover, many of the health/status tools automatically identify and kill malfunctioning containers. All of that sucks for the forensic examiner, but is unquestionably valuable to the business. Even worse, containers are ephemeral and do not have inherently persistent storage. So unless the logs are writing to an external location you will not be able to quickly diagnose a root cause once that container has been terminated.
With those points in mind, let us address the obvious question, “If containers heal themselves, and the IT folks remediate by cleanly refreshing them, why bother learning container forensics?” Because there are use cases that require investigation, even if you might have to do it with one forensic arm tied behind your back. Some examples are a rogue IT team member running a coin mining pod, ransomware that has broken out of container isolation, or a new zero-day vulnerability of which we are woefully unprepared (that one doesn’t happen, right?) In summary, and as mentioned in previous articles, knowledge and preparation are key, and the middle of a time-sensitive incident response is not the time to be learning how containers function.
A couple disclaimers:
- I am not an expert in container architecture so if you think I am wrong about anything, please comment below and I will correct it.
- Almost none of this information represents original research on my part, but rather consolidating others’ excellent research into a guide of sorts. If you see anything that I failed to cite please let me know at [email protected] and I will fix it immediately.
- Although I am adding content for AWS and Azure, I have not had much experience in GCP so I apologize, both to the reader and to Google….who actually created Kubernetes. The good part is a lot of this will apply to GCP as well.
Terms
DOCKER
- Docker – a popular container format
- Image – the container’s gold copy
- Manifest – the container’s configuration information (how many layers, size, OS, etc.) This is the description of the docker image. Below is an example of a Docker Manifest taken from here.
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 7023,
"digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b86451d9bce1f432a537bc7"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 32654,
"digest": "sha256:e692418e4cbaf90ca69d05a66403747baa33ee08806650b51fab815ad7fc331f"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 16724,
"digest": "sha256:3c3a4604a545cdc127456d94e421cd355bca5b528f4a9c1905b15da2eb4a4c6b"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 73109,
"digest": "sha256:ec4b8955958665577945c89419d1af06b5f7636b4ac3da7f12184802ad867736"
}
]
}KUBERNETES (K8S)
- Kubernetes – a popular orchestration framework that automates container operations
- Pod – a Kubernetes term for one or more containers that share the same configuration, networking, etc. K8S assigns a single DNS to a Pod. It helps if you notice Docker’s emblem is a whale and a group of whales is a pod.
- Deployment / Replica Set – The Deployment object is your desired pod state. It manages the number of pod replicas. In most cases, you are only going to see containers/pods get created through a “kind: Deployment” object, and not directly through a “kind: Pod“.
To ease the confusion, the hierarchy is….
Deployment –> Replica Sets –> Pods –> Container - The Service – Although pods on the same host can see each other, this is the REST object that provides pod/pod and pods/non-pod interaction. For example, a front-end pod to a backend pod, or a pod to the load balancer, persistent storage, database, etc. Like most things in K8S, the Service is a REST object and will have its own IP/DNS. Moreover, since the point of all this is autonomous orchestration of groups of containers, K8S will build/destroy the containers as needed; that means the IP’s and container states are not persistent. That is what the Service does; it creates a stable IP for networking purposes.
Artifacts
This is not a complete list and will be supplemented over time. Please consider providing your comments via Twitter or LinkedIn at the links below. For such a new technology, any input is most welcome!
Yaml files
YAML is the chosen format for Kubernetes configuration files. As far as I know, there is no fixed path. They can even be piped objects. However, these are the files you should request from the IT team as they contain the number of pods, image types, ports, etc.
Jumping over to Docker for a minute, it is worthwhile to note you will also see YAML files used in the implementation for “Docker compose”. While “Docker build” may be the method to create a docker image from some baseline (as well as add whatever the container needs for the specific use case) “Docker compose” is the current best practice, as it can read a Dockerbuild file, as well as the build files of other relevant containers (db backend, service objects, etc), and spin up the whole stack.
Back to YAML now. An in-depth description of YAML is outside the scope of this article. However, it is worth understanding two points. First, YAML is a superset of JSON, so you may see the system administrators using JSON files. Second, YAML has only two structures, Lists and Maps, so if it does not look like a list it is a map. The third equally important point of our two points is that YAML is nest-able.
The most important name-value pair in these YAML files is the “kind” object. For example, the one below is labeled “kind: Deployment.” That will create a deployment object, which deploys and manages the pods. If that field says “kind: pod“, that would instead create a pod.
NOTE: You may want to look at any command line executions that are baked directly into the creation. See “Post Manifest (Example with Command Execution)” below.
Command:
kubectl apply -fPod Manifest (Web App Example)
A pod manifest is a YAML document that contains the startup parameters and application defaults for when a web application is launched.
Here’s an example manifest for a web server pod named nginx-demo.
apiVersion: v1
kind: Pod
metadata:
name: nginx-demo
labels:
role: myrole
spec:
containers:
- name: nginx
image:nginx:1.14.2
ports:
- name: web
containerPort: 80
protocol: TCPFrom:https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/
Pod Manifest (Example with Command Execution)
This is a pod manifest that has command line executions baked in.
apiVersion: v1
kind: Pod
metadata:
name: command-demo
labels:
purpose: demonstrate-command
spec:
containers:
- name: command-demo-container
image: debian
command: ["printenv"]
args: ["HOSTNAME", "KUBERNETES_PORT"]
restartPolicy: OnFailure
From: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/
Deployment Manifest (Web App Example)
Here is a manifest that deploys a web app replica set.
apiVersion: apps/v1
kind: Deployment
metadata:
name: mywebapp-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80From: https://k8s.io/examples/controllers/deploy.yaml
Command line history
Commands sent to the orchestrator (Kubernetes) and containers (Docker) will come from a source workstation. So the originating source of the system administrator’s command line history (e.g. bash history, PowerShell Get-History, etc) should be in-scope.
You should also obtain copies of the system administrator’s locally-stored copies of the YAML files that were initially used to set up the containers and orchestration.
Also, if you believe the maliciousness does not originate from the host straight to its child containers, you may want to assign someone to the network forensics portion. See Incident Response in the Cloud – Part I –> Detection –> Network Forensics
The Service Definition
As stated previously, the Service is the piece that allows the specific interaction between pods and between pod/non-pod resources, like an external load balancer, persistent storage, database, etc. The Service is a REST object and will have its own IP/DNS.
This is an example of the creation (via “POST”) of a Kubernetes Service. “mywebapp”, which will get an assigned IP and will map incoming traffic from TCP 80 to the pod’s specified listening port, TCP 9376.
apiVersion: v1
kind: Service
metadata:
name: myservice
spec:
selector:
app: myservice
ports:
- protocol: TCP
port: 80
targetPort: 9376From: https://kubernetes.io/docs/concepts/services-networking/service/
Cloud Orchestration logs
These are some of the more useful commands that cloud providers have created in order to interact with the containers.
AWS
TODO
Azure
Application logs
Container application logs (Az container logs) can be viewed in the command line
az container logs --resource-group myResourceGroup --name mycontainerTraceback (most recent call last):
File "wordcount.py", line 11, in
urllib.request.urlretrieve (sys.argv[1], "foo.txt")
File "/usr/local/lib/python3.6/urllib/request.py", line 248, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.6/urllib/request.py", line 532, in open
response = meth(req, response)
File "/usr/local/lib/python3.6/urllib/request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.6/urllib/request.py", line 570, in error
return self._call_chain(args) File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain result = func(args)
File "/usr/local/lib/python3.6/urllib/request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
Container startup logs
Explained further in the forensic workflow section. Essentially, this command can direct the container stdout and stderror to you.
az container attach --resource-group myResourceGroup --name mycontainerDocker Desktop (WSL 2)
The data sources below capture interactions between:
1) the Windows host –> Docker Desktop (a MobyLinuxVM using WSL 2.)
2) The Docker Desktop VM –> Docker containers
3) NOT the Docker container commands (AFAIK), which are likely the most relevant. For that data source, please refer to the subsequent sections of this article.
Be advised, those paths are reconfigurable by the system administrator.
> The Docker Desktop host VM (.vhdx)
C:\Users\<username>\AppData\Local\Docker\wsl\data
> Logs, Cache, BlobStorage, Leveldb, Cookies, etc
C:\Users\<username>\AppData\Roaming\Docker Desktop
C:\Users\<username>\AppData\Roaming\Docker
C:\Users\<username>\AppData\Local\Docker
> POSTs to WSL2 VM (MobyLinuxVM running Docker Desktop)
C:\ProgramData\DockerDesktopData Volumes
As mentioned previously, containers do not have inherently persistent storage. It is necessary to either ask the system administrator for the mount points or look for the mount section in “Docker inspect <container>“.
Mounts come in three flavors
- Volumes
- For Linux, they should live in /var/lib/docker/volumes
- For Windows, (i.e. Docker Desktop) please refer to the specific Docker Desktop (WSL 2) section. But essentially, the volumes should live within its “host” ext4 file system on the vhdx volume.
- Bind mounts
- These can exist anywhere. Look for them with the Docker inspect command above
- tmpfs mounts
- These are in the host RAM, so acquire it. Especially considering these mounts can be through named pipes as well.
Full PCAP
Given the reliance on the REST API it may be worthwhile to perform a network capture on the host VM, if there is a suspicion of malicious traffic. I have not tried this myself, but I believe you should be able to do this through the use of a “sidecar.”
Another possibility is ksniff, a pretty inventive tool from Eldad Rudich, that uses kubectl to upload tcpdump to your container, then pushes the output to your Wireshark instance.
DFIR Workflow
Please be advised, this will be supplemented/edited whenever I have time. Hopefully, the quick workflow below is, at least, a small amount of help during your cloud response.
Urgent Containment Needed?
Quarantine the container if possible.
Docker containers can be paused/suspended with “docker pause“. On Linux, this leverages the freezer cgroup and the processes will not be aware or able to control it. On Windows, the container can be paused if it is using Hyper-V.
Jonathan Seawright astutely pointed out the container provider may also be able to quarantine a container with a firewall isolation policy. I would add the host EDR tools may have some capability to quarantine the host itself.
It is important to note, these are capabilities you should understand in your environment before employing, as any quarantine attempts, whether at the container or host level, have the potential for instability. Additionally, Docker creates its own iptables and config entries so it is exceedingly difficult to quarantine a container with just a simple command line.
If you cannot pause the container, I recommend stopping it, which will likely kill it. Reference SIGTERM / SIGKILL in the next section’s command list
Not an Urgent Containment
Analyze for Container Escape
StackRox did a short, but outstanding, blog post on Container Escape and I recommend reading it in-full here.
They recommend you review the container’s deployment.yaml to see if the container was launched with any of the flags below. If these flags are present in the launch, then you may want to image the host. A good next step would be to follow the instructions provided by Jonathan Greig in his excellent article on post-host-imaged container analysis.
If the flags below are not present, then you can limit your analysis to the container processes.
--cap-add *--device *--device-cgroup-rule--ipc system--mount /--pid system--privileged--security-opt--volume /--volumes-from
No Container Escape
- Acquire relevant data sources referenced in the “Artifacts” section above and/or in the Azure/AWS sections that follow
- suspend/pause using
- docker container pause <container name>.
- This will keep the memory/differencing intact.
- Commit the container’s changes to a new instance for spinning up in your forensic environment.
- docker commit $CONTAINER_ID <source image>
- docker commit $CONTAINER_ID <source image>
- [OPTIONAL 1] Remove the misbehaving container from the orchestrator. Kubectl has a couple of ways to do this. Recommend deferring to the business unit’s system administrator or referring to the K8S documents.
- Launch the new container in an isolated forensic environment for analysis (e.g. your AWS forensic VPC, your Azure forensic Resource Group, or your on-prem forensic lab environment.)
- [OPTIONAL 2] Consider going old school on the containers and comparing a gold image hash set to the container you are about to acquire.
- Get a known hash set
- Spin up a clean container or run the hash command (below) against known clean containers. Make sure to use the original deployment config files. I recommend getting the exact steps from the system administrator.
- Get the hashes from the suspicious container
- Compare the hashes
- Run the below command on the gold containers and the infected containers, then remove all the common files. Or load the known hash set into your forensic tool of choice and decimate the data set. (note the parenthesis)
- Run the below command on the gold containers and the infected containers, then remove all the common files. Or load the known hash set into your forensic tool of choice and decimate the data set. (note the parenthesis)
- Get a known hash set
- [OPTIONAL 3] As mentioned in the Container Escape section above, if you want to pursue an alternate workflow and you have the time to get an image of the full host, you can follow the instructions provided by Jonathan Greig in his excellent article on host-based container analysis. I have also included some useful commands in the sections that follow.
HASH ALL THE THINGS...
docker exec -it <containerID> sh -c \
"find . -name '*' -type f -print0 | \
xargs -0 md5sum" > \
md5sum_<KnownOrSuspiciousContainer>.txtAWS
TODO
Azure
Container Host Forensics
Microsoft did a great job at explaining Azure container commands here. The forensically-useful points are below.
Container Startup Diagnostics
This points stdout (console) and stderr (error) to the container’s output so you can read the ongoing logs.
az container attach --resource-group myResourceGroup --name <mycontainer>Example Output
Container 'mycontainer' is in state 'Unknown'…
Container 'mycontainer' is in state 'Waiting'…
Container 'mycontainer' is in state 'Running'…
(count: 1) (last timestamp: 2019-03-21 19:42:39+00:00) pulling image "<imagename>:latest"
Container 'mycontainer' is in state 'Running'…
(count: 1) (last timestamp: 2019-03-21 19:42:39+00:00) pulling image "mcr.microsoft.com/azuredocs/<imagename>:latest"
(count: 1) (last timestamp: 2019-03-21 19:42:52+00:00) Successfully pulled image "mcr.microsoft.com/azuredocs/<imagename>:latest"
(count: 1) (last timestamp: 2019-03-21 19:42:55+00:00) Created container
(count: 1) (last timestamp: 2019-03-21 19:42:55+00:00) Started containerContainer Startup Events
az container show --resource-group myResourceGroup --name <mycontainer>Example output:
{
"containers": [
{
"command": null,
"environmentVariables": [],
"image": "mcr.microsoft.com/azuredocs/<aci-myimage>",
...
"events": [
{
"count": 1,
"firstTimestamp": "2019-03-21T19:46:22+00:00",
"lastTimestamp": "2019-03-21T19:46:22+00:00",
"message": "pulling image \"mcr.microsoft.com/azuredocs/<aci-myimage>\"",
"name": "Pulling",
"type": "Normal"
},
{
"count": 1,
"firstTimestamp": "2019-03-21T19:46:28+00:00",
"lastTimestamp": "2019-03-21T19:46:28+00:00",
"message": "Successfully pulled image \"mcr.microsoft.com/azuredocs/<aci-myimage>\"",
"name": "Pulled",
"type": "Normal"
},
{
"count": 1,
"firstTimestamp": "2019-03-21T19:46:31+00:00",
"lastTimestamp": "2019-03-21T19:46:31+00:00",
"message": "Created container",
"name": "Created",
"type": "Normal"
},
{
"count": 1,
"firstTimestamp": "2019-03-21T19:46:31+00:00",
"lastTimestamp": "2019-03-21T19:46:31+00:00",
"message": "Started container",
"name": "Started",
"type": "Normal"
}
],
"previousState": null,
"restartCount": 0
},
"name": "<mycontainer>",
"ports": [
{
"port": 80,
"protocol": null
}
],
...
}
],
...
}Container Interactive Session
If you want to query the container in a non-forensically-sound manner, you can launch an Interactive session. Consider then acquiring the data points enumerated by Sandfly Security
az container exec \
--resource-group <my-resourcegroup> \
--name mycontainer \
--exec-command /bin/sh (to launch a shell)
(or you can just execute a command from right here similar to the hash command in the previous section)Coinminer?
Check the container metrics using Azure Monitor.
Here is is a list of the currently available container metrics. Be advised, AZ Monitor’s default retention is 93 days
AZ Monitor is set to provide the averages over all the containers in a resource group but you can get the individual container metrics by using an Azure “dimension.”
You can also use the Azure CLI to get the instance-specific container metrics. Microsoft provides the following instructions to do so.
Below is an example of container metrics by resource group (not by individual container). To get the individual container metrics, just add –dimension <containerName>.
CPU metrics
CONTAINER_ID=$(az container show \
--resource-group <my azure resource group> \
--name <mycontainer> \
--query id \
--output tsv
az monitor metrics list \
--resource $CONTAINER_ID \
--metric CPUUsage \
--output tableExample Output:
Timestamp Name Average
2018-08-20 21:39:00 CPU Usage
2018-08-20 21:40:00 CPU Usage
2018-08-20 21:41:00 CPU Usage
2018-08-20 21:42:00 CPU Usage
2018-08-20 21:43:00 CPU Usage 0.375
2018-08-20 21:44:00 CPU Usage 0.875
2018-08-20 21:45:00 CPU Usage 1
2018-08-20 21:46:00 CPU Usage 3.625
2018-08-20 21:47:00 CPU Usage 1.5
2018-08-20 21:48:00 CPU Usage 2.75
2018-08-20 21:49:00 CPU Usage 1.625
2018-08-20 21:50:00 CPU Usage 0.625
2018-08-20 21:51:00 CPU Usage 0.5
2018-08-20 21:52:00 CPU Usage 0.5
2018-08-20 21:53:00 CPU Usage 0.5
Memory metrics
az monitor metrics list \
--resource $CONTAINER_ID \
--metric MemoryUsage \
--output tableExample Output:
Timestamp Name Average
2018-08-20 21:43:00 Memory Usage
2018-08-20 21:44:00 Memory Usage 0.0
2018-08-20 21:45:00 Memory Usage 15917056.0
2018-08-20 21:46:00 Memory Usage 16744448.0
2018-08-20 21:47:00 Memory Usage 16842752.0
2018-08-20 21:48:00 Memory Usage 17190912.0
2018-08-20 21:49:00 Memory Usage 17506304.0
2018-08-20 21:50:00 Memory Usage 17702912.0
2018-08-20 21:51:00 Memory Usage 17965056.0
2018-08-20 21:52:00 Memory Usage 18509824.0
2018-08-20 21:53:00 Memory Usage 18649088.0
2018-08-20 21:54:00 Memory Usage 18845696.0
2018-08-20 21:55:00 Memory Usage 19181568.0More Useful DFIR Commands
Kubernetes & Docker (also AWS & Azure for that matter) use a base command with options. They can be called with –help for additional instructions.
Kubernetes
Get commands with basic output
# List all services in the namespace
kubectl get services
# List all pods in all namespaces
kubectl get pods --all-namespaces
# List all pods in the current namespace, with more details
kubectl get pods -o wide
# List a particular deployment
kubectl get deployment <my-deployment>
# List all pods in the namespace
kubectl get pods
# Get a pod's YAML
kubectl get pod my-pod -o yaml
# Describe commands with verbose output
kubectl describe nodes <my-node>
kubectl describe pods <my-pod>
# List Services Sorted by Name
kubectl get services --sort-by=.metadata.name
# List pods Sorted by Restart Count
kubectl get pods --sort-by='.status.containerStatuses[0].restartCount'
# List PersistentVolumes sorted by capacity
kubectl get pv --sort-by=.spec.capacity.storage
# Get the version label of all pods with label app=cassandra
kubectl get pods --selector=app=cassandra -o \
jsonpath='{.items[*].metadata.labels.version}'
# Retrieve the value of a key with dots, e.g. 'ca.crt'
kubectl get configmap <myconfig> \
-o jsonpath='{.data.ca\.crt}'
# Get all worker nodes (use a selector to exclude results that have a label
# named 'node-role.kubernetes.io/master')
kubectl get node --selector='!node-role.kubernetes.io/master'
# Get all running pods in the namespace
kubectl get pods --field-selector=status.phase=Running
# Get ExternalIPs of all nodes
kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'
# List Names of Pods that belong to Particular RC
# "jq" command useful for transformations that are too complex for jsonpath, it can be found at https://stedolan.github.io/jq/
sel=${$(kubectl get rc my-rc --output=json | jq -j '.spec.selector | to_entries | .[] | "\(.key)=\(.value),"')%?}
echo $(kubectl get pods --selector=$sel --output=jsonpath={.items..metadata.name})
# Show labels for all pods (or any other Kubernetes object that supports labelling)
kubectl get pods --show-labels
# Check which nodes are ready
JSONPATH='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}' \
&& kubectl get nodes -o jsonpath="$JSONPATH" | grep "Ready=True"
# Output decoded secrets without external tools
kubectl get secret my-secret -o go-template='{{range $k,$v := .data}}{{"### "}}{{$k}}{{"\n"}}{{$v|base64decode}}{{"\n\n"}}{{end}}'
# List all Secrets currently in use by a pod
kubectl get pods -o json | jq '.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name' | grep -v null | sort | uniq
# List all containerIDs of initContainer of all pods
# Helpful when cleaning up stopped containers, while avoiding removal of initContainers.
kubectl get pods --all-namespaces -o jsonpath='{range .items[*].status.initContainerStatuses[*]}{.containerID}{"\n"}{end}' | cut -d/ -f3
# List Events sorted by timestamp
kubectl get events --sort-by=.metadata.creationTimestamp
# Compares the current state of the cluster against the state that the cluster would be in if the manifest was applied.
kubectl diff -f ./<my-manifest>.yaml
# Produce a period-delimited tree of all keys returned for nodes. Helpful when locating a key within a complex nested JSON structure
kubectl get nodes -o json | jq -c 'path(..)|[.[]|tostring]|join(".")'
# Produce a period-delimited tree of all keys returned for pods, etc
kubectl get pods -o json | jq -c 'path(..)|[.[]|tostring]|join(".")'Copied verbatim from: Kubernetes.io’s Kubernete Cheat Sheet
Docker
DFIR-specific
For this section, I thought it would me more useful if I pulled down a Docker image and ran some commands. Hopefully, that will help you understand the commands and outputs later. I have commented the commands as well.
The specific steps below cover:
- reviewing the locally-stored images on the host
- checking if any containers are running
- starting a web app container and binding it to TCP 80
- noting the SHA256 of the container
- reading the access logs of the newly-running container
- executing “ps -a” from the host into the container
- open a shell in the container and running commands from inside the container
>> docker images
# the images that are available to use
# recommend noting any container that is suspiciously different
REPOSITORY TAG IMAGE ID CREATED SIZE
dfwforensics/docker101tutorial latest a7dc23f996e8 3 hours ago 27.9MB
docker101tutorial latest a7dc23f996e8 3 hours ago 27.9MB
alpine/git latest a939554ad0d0 4 weeks ago 25.1MB
---------------------------------------
>> docker ps
# check for running containers. None are running on this system yet
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
---------------------------------------
>> docker run -dp 80:80 docker101tutorial
#this runs the image above, then binds the container's listening port to TCP 80.
#if the image does not exist locally, docker will download it.
#output is the sha256 of the image
cdb9128363a9e96c2f954eccbc36f88dbe95c1a675931e3bf8985a4b862af1b1
---------------------------------------
>> docker ps
# after the container is running you can query its summary information
CONTAINER ID IMAGE COMMAND CREATED
cdb9128363a9 docker101tutorial "/docker-entrypoint.…" 17 minutes ago
STATUS PORTS NAMES
Up 17 minutes 0.0.0.0:80->80/tcp <containername>
---------------------------------------
>> docker inspect <mycontainer>
# use "inspect" to read a running container's metadata.
# or use "inspect" on an image with "--verbose" on a suspicious image
[
{
"Id": "sha256:a7dc23f996e88f5787e8dc2112348fb022bd0a25f91644c92bcd50816dd24b6a",
"RepoTags": [
"dfwforensics/docker101tutorial:latest",
"docker101tutorial:latest"
],
"RepoDigests": [
"dfwforensics/[email protected]:3800b76243ae13fdd3337f92084774457482e6ae33af6b5a51b7ca5d287026e8"
],
"Parent": "",
"Comment": "buildkit.dockerfile.v0",
"Created": "2021-03-25T17:15:37.0510086Z",
"Container": "",
"ContainerConfig": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": null,
"Cmd": null,
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": null,
"OnBuild": null,
"Labels": null
},
"DockerVersion": "",
"Author": "",
"Config": {
"Hostname": "",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"80/tcp": {}
},
"Tty": false,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"NGINX_VERSION=1.19.8",
"NJS_VERSION=0.5.2",
"PKG_RELEASE=1"
],
"Cmd": [
"nginx",
"-g",
"daemon off;"
],
"Image": "",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/docker-entrypoint.sh"
],
"Labels": {
"maintainer": "NGINX Docker Maintainers <[email protected]>"
},
"StopSignal": "SIGQUIT"
},
"Architecture": "amd64",
"Os": "linux",
"Size": 27941754,
"VirtualSize": 27941754,
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/u95ewzojeausn0e21kdz4dtu9/diff:/var/lib/docker/overlay2/221ca0b88a17dfdfc3f71c8e552a1aefd9213399d38284dde5e2f38b91465f5b/diff:/var/lib/docker/overlay2/d32278f833945f7e0e0a3cf752f631f1594515bde5b4cd426a5e39b7cd1d477a/diff:/var/lib/docker/overlay2/d0c54430881927715c04507b8da79dff77bedec352418b02c30f4839833f7868/diff:/var/lib/docker/overlay2/6c2981beeab6ea01fb679afece6b904fdac5bb6e2e270044521bf9bab3dc3d1e/diff:/var/lib/docker/overlay2/2bc0eee3896be31ea7c4ddc755f8f3fe5a077fec1db928d77899f19ecabc7d09/diff:/var/lib/docker/overlay2/0a09fdf0301e002ee7849200c716ab3cd19ad379041cb6c7d39892aff643cf8e/diff",
"MergedDir": "/var/lib/docker/overlay2/8j883dxj8a4hvd9lqnmj7ixg0/merged",
"UpperDir": "/var/lib/docker/overlay2/8j883dxj8a4hvd9lqnmj7ixg0/diff",
"WorkDir": "/var/lib/docker/overlay2/8j883dxj8a4hvd9lqnmj7ixg0/work"
},
"Name": "overlay2"
},
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:cb381a32b2296e4eb5af3f84092a2e6685e88adbc54ee0768a1a1010ce6376c7",
"sha256:369b3f326f894cccb059e895c7c3e08d640b51944f67ff87283884930e67b03e",
"sha256:e7be5ec0da705e9db18584b3ea97fab2187badd90a2e8d130c586a0e85700cb3",
"sha256:9f5787713b402dfff2f9788943b2067ec2bbad58569710e54fe9da6192770413",
"sha256:9565c0a5609213d1852bfd029427a462f036b5d22ccb81f451a1c4d9a0753f3e",
"sha256:b16c410e3fcaba32ea9ec1dcdb5056c7b1c343f676c4de2175f71b2bcea5fccd",
"sha256:0407828aad85b225cba9d774ce3fb8fb212e1e6679037067d0b0390af676d322",
"sha256:a32f9c0f1e2f8209fab8e8da5d1fac480af58bfdd89b493e2fa715d33bbdd021"
]
},
"Metadata": {
"LastTagTime": "2021-03-25T17:29:34.297585Z"
}
}
]
---------------------------------------
>> docker logs cdb9128363a9 #note the container ID
# pulling this (web app) container's logs after web browsing in it a bit
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET / HTTP/1.1" 200 8713 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/stylesheets/application.adb8469c.css HTTP/1.1" 200 76332 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/stylesheets/application-palette.a8b3c06d.css HTTP/1.1" 200 38773 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/fonts/material-icons.css HTTP/1.1" 200 873 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:20:59 +0000] "GET /assets/javascripts/modernizr.86422ebf.js HTTP/1.1" 200 7296 "http://localhost/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
...
...
172.17.0.1 - - [25/Mar/2021:20:21:04 +0000] "GET /tutorial/persisting-our-data/items-added.png HTTP/1.1" 200 63754 "http://localhost/tutorial/persisting-our-data/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:21:04 +0000] "GET /tutorial/persisting-our-data/dashboard-open-cli-ubuntu.png HTTP/1.1" 200 170038 "http://localhost/tutorial/persisting-our-data/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:21:05 +0000] "GET /tutorial/using-bind-mounts/ HTTP/1.1" 200 18697 "http://localhost/tutorial/persisting-our-data/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
172.17.0.1 - - [25/Mar/2021:20:21:05 +0000] "GET /tutorial/using-bind-mounts/updated-add-button.png HTTP/1.1" 200 21838 "http://localhost/tutorial/using-bind-mounts/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "-"
---------------------------------------
>> docker exec cdb9128363a9 ps -a
# running a command in the container
PID USER TIME COMMAND
1 root 0:00 nginx: master process nginx -g daemon off;
33 nginx 0:00 nginx: worker process
34 nginx 0:00 nginx: worker process
35 nginx 0:00 nginx: worker process
...
...
47 nginx 0:00 nginx: worker process
48 nginx 0:00 nginx: worker process
55 root 0:00 ps -a
---------------------------------------
>> docker ps
# starting a remote shell of the container - first, get the container info
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cdb9128363a9 docker101tutorial "/docker-entrypoint.…" 30 minutes ago Up 30 minutes 0.0.0.0:80->80/tcp <containername>
---------------------------------------
>> docker exec -it <containername> /bin/sh
# starting a remote shell of the container - second, start the shell
/ #
---------------------------------------
/ # ps -a
# You are now in the container's shell
PID USER TIME COMMAND
1 root 0:00 nginx: master process nginx -g daemon off;
33 nginx 0:00 nginx: worker process
34 nginx 0:00 nginx: worker process
35 nginx 0:00 nginx: worker process
...
...
47 nginx 0:00 nginx: worker process
48 nginx 0:00 nginx: worker process
61 root 0:00 /bin/sh
68 root 0:00 /bin/sh
75 root 0:00 ps -a
---------------------------------------
/ # pwd
/
---------------------------------------
/ # whoami
root
---------------------------------------
/home # more /etc/passwd
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
...
...
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
nginx:x:101:101:nginx:/var/cache/nginx:/sbin/nologin
---------------------------------------
/home # more /etc/shadow
root:!::0:::::
bin:!::0:::::
daemon:!::0:::::
...
...
smmsp:!::0:::::
guest:!::0:::::
nobody:!::0:::::
nginx:!:18697:0:99999:7:::
---------------------------------------
Docker Command Help Menu
The full command list
>> docker
Usage: docker [OPTIONS] COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default
"C:\\Users\\miche\\.docker")
-c, --context string Name of the context to use to connect to the
daemon (overrides DOCKER_HOST env var and
default context set with "docker context use")
-D, --debug Enable debug mode
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level
("debug"|"info"|"warn"|"error"|"fatal")
(default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default
"C:\\Users\\miche\\.docker\\ca.pem")
--tlscert string Path to TLS certificate file (default
"C:\\Users\\miche\\.docker\\cert.pem")
--tlskey string Path to TLS key file (default
"C:\\Users\\miche\\.docker\\key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit
Management Commands:
app* Docker App (Docker Inc., v0.9.1-beta3)
builder Manage builds
buildx* Build with BuildKit (Docker Inc., v0.5.1-docker)
config Manage Docker configs
container Manage containers
context Manage contexts
image Manage images
manifest Manage Docker image manifests and manifest lists
network Manage networks
node Manage Swarm nodes
plugin Manage plugins
scan* Docker Scan (Docker Inc., v0.5.0)
secret Manage Docker secrets
service Manage services
stack Manage Docker stacks
swarm Manage Swarm
system Manage Docker
trust Manage trust on Docker images
volume Manage volumes
Commands:
attach Attach local standard input, output, and error streams to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes to files or directories on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on Docker objects
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart one or more containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive (streamed to STDOUT by default)
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update configuration of one or more containers
version Show the Docker version information
wait Block until one or more containers stop, then print their exit codes
Run 'docker COMMAND --help' for more information on a command.AWS / Azure / GCP
I recommend either looking at the examples in previous sections or simply performing a web search for “<cloudProviderName” AND <commandYouWant>. The online documentation is very good for all of these cloud providers. Additionally, once you understand the AWS/AZ/GCP command syntax, the commands are closely aligned with the standard Docker/K8S syntax.
SIFT Docker Image
update 1 Apr 21: I will leave the previously posted information below, but as expected, Digital Sleuth made a better docker image with greatly expanded capabilities compared to mine. I highly recommend you use that one. Simply clone that GitHub repository, navigate to your local folder that contains them, and launch “docker compose up -d“. That will give you a fully SIFT’d docker container, along with a SIFT subnet, gui, ssh, and mounting capabilities. So…..pretty much exactly like my docker image but cleaner. And smaller. And….better.
If you do it right it, the in-progress install will look like this:
>> cd .\sift-docker\
>> ls
Directory: ..\sift-docker
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 4/1/2021 9:31 AM .git
-a---- 4/1/2021 9:31 AM 485 docker-compose.yaml
-a---- 4/1/2021 9:31 AM 2335 Dockerfile.bionic
-a---- 4/1/2021 9:31 AM 2334 Dockerfile.focal
-a---- 4/1/2021 9:31 AM 35149 LICENSE
-a---- 4/1/2021 9:31 AM 945 README.md
>> docker compose up -d
[+] Building 628.0s (3/4)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 73B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/digitalsleuth/sift-docker:focal 3.5s
=> [1/1] FROM docker.io/digitalsleuth/sift-docker:[email protected]:6c57dfb413adca4e10aa47ee56248e7c131905511d1f46275b3b13656ad829cf 624.5s
=> => resolve docker.io/digitalsleuth/sift-docker:[email protected]:6c57dfb413adca4e10aa47ee56248e7c131905511d1f46275b3b13656ad829cf 0.0s
=> => sha256:b2055b333b029de2d76a27fbf5ee74e5ce5479467c4b7790b2d1d832a9c9515c 583.01MB / 1.50GB 624.5s
=> => sha256:728042ecb84323afa057d4d19a3c2ab6d9015829a15ecf49d458c4dbd542ffa4 111B / 111B 0.4s
=> => sha256:6c57dfb413adca4e10aa47ee56248e7c131905511d1f46275b3b13656ad829cf 1.36kB / 1.36kB 0.0s
=> => sha256:e1b967990adc6e002937671ad01ffba85e51c71bcac3dd9fa3aae0e181790658 7.55kB / 7.55kB 0.0s——————————————————
My SIFT docker image
Below is a link to a SIFT Docker image that you can download and play with. Please feel free to try it out. Just load Docker on your Linux workstation with a simple apt install, or if you have WSL 2 enabled, install Docker for Windows. Be advised, this SIFT image is not production-tested or SANS-authorized. Also, DigitalSleuth is working on a SIFT image and it will almost certainly be better than mine, so if that is available, use that one.
To start, just Google “Docker pull” and follow the quick instructions. Then, “Docker run <the other stuff you Googled>” and you will be up and running. At that point “Docker exec -it <container> /bin/bash” will give you an interactive shell in the container and you can work with your data just like you normally would in SIFT. Keep in mind, the value here is scalability, small size, compartmentation, and a clean instance for every case. The downside is docker does not have persistent storage so you should map a storage location if that is something you need. Like every other DFIR workflow task, it is only hard the first time you do it.
Once you have the container running, just copy some files to it and you are good to go. Copying files is easy and there are five different Google-able ways to do that. If you are using Docker for Windows, the easiest way is here. You can also access the container directly on your host. If you are running docker under WSL, just look for a hidden share with Windows Explorer “\\wsl$”
—————————————
THE UNAUTHORIZED AND PROBABLY TERRIBLE SIFT DOCKER IMAGE
—————————————
Citations
Riadi, Imam & Umar, Rusydi & Sugandi, Andi. (2020). Web Forensic on Kubernetes Cluster Services Using Grr Rapid Response Framework. International Journal of Scientific & Technology Research. 9. 3484-3488.
Grieg, Jonathan (2021). Container Forensics with Docker Explorer. OSDFIR. https://osdfir.blogspot.com/2021/01/container-forensics-with-docker-explorer.html
StackRox. (2017). Forensics in the age of containers. Accessed from: https://www.stackrox.com/post/2017/08/csi-container-edition-forensics-in-the-age-of-containers
Ashnik. Container Forensics. Accessed from: https://www.ashnik.com/sysdig-resources/container-forensics/
Microsoft. (2019). Retrieve container logs and events in Azure Container Instances. Accessed from: https://docs.microsoft.com/en-us/azure/container-instances/container-instances-get-logs
unk author. (2020). Kubernetes Documentation. Accessed from: https://kubernetes.io/docs/home/
Burns, Brenden. (2015). A Technical Overview of Kubernetes – CoreOS Fest 2015. Accessed from: https://www.youtube.com/watch?v=WwBdNXt6wO4&ab_channel=CoreOS
