Nikita’s HPCC Internship Journey 👩🏽‍💻

17 min readAug 1, 2021

Day 1: Setting up HPCC Systems on my Local Machine, Adding Code, Checking in Code

Fork the HPCC-Platform on git into my personal repository (click the Fork button to essentially create a copy of the repository for just you so you can make changes without affecting the main repository. You can sync changes you make with the main repository using something called Pull Requests)
Open Terminal
Mkdir HPCC-Platform (create a separate folder for your code in HPCC)
cd HPCC-Platform
git clone REPOSITORY_URL (https://github.com/nikita-jha/HPCC-Platform.git)
cd NAME_OF_REPO (cd HPCC-Platform)
git remote -v (to see all of the existing remotes set up)
git remote add upstream https://github.com/hpcc-systems/HPCC-Platform.git
git fetch — all
git checkout -b HPCC-24870 upstream/master
cd dockerfiles (to see list of all dockerfiles, type ls)
vim incr.sh
vim buildall-common.sh
Make the code change
:q! — vim command sequence (to exit the code sequence)
git reset — hard upstream/master

Day 2: Building and Running HPCC Image Locally

Local Deployment of HPCC systems

https://cdn.hpccsystems.com/releases/CE-Candidate-8.0.10/docs/EN_US/Containerized_HPCC_Systems_Platform_EN_US-8.0.10-1.pdf

Refer to the notes above to understand what I did in order to locally deploy the HPCC system platform and ECL Watch on my local machine

Day 3: Building and Running HPCC Image with Incr.sh

Open dockerfiles, which should be in the HPCC-Platform in your root/home directory and make sure you are on the home branch (you can check with git branch and switch with git checkout <BRANCHNAME>
brew install md5sha1sum
Install md5sum for mac
Make 2 new repositories in Docker Hub (one called platform-build and the other platform-core)
docker login
./incr.sh -d nikitajha -p
docker images (to see all the images you currently have)
cd ..
vim CMakeLists.text
./incr.sh -d nikitajha -p

Document for ALL Jira Ticket Research: https://docs.google.com/document/d/14sAD1FyitudfCL5MvSujqXh_BsiQKJoD2bIfmyLgu60/edit?usp=sharing

Day 4: JIRA subtask #1 (Investigate image caching in the docker builds)

Final product for this JIRA ticket: a pull request with all code changes
Researched about “Latest Tag” and its problems pertaining to caching
Determined what are the possible solutions and weighed advantages and disadvantages of each (all of this is included in the document linked above
Researched about Trivvy to scan files for potential threats

Day 5: JIRA subtask #4 (Investigate any uses of root access in our image builds that elevate privilege’s unnecessarily)

Investigate root access in all the docker files

Learned about Root access, when to use it, what the dangers/threats are associated with it, and what privileges and permissions it has
Group add, user add, passwd platform-core (is this the right way to do it)
Ensure we are switching to user HPCC before we are doing any of the final run commands
Write up a report about this (If you find any good sites, put in links)

Day 6: Adding ‘no-cache’ option to buildall common script and then re-build image

**Buildall.sh** — Added option to input whether or not the user wants to disable cache (1 if they want to disable and 0 if they do not want to disable)

BUILD_ML=    # all or ml,gnn,gnn-gpu
DISABLE_CACHE= #empty[[ -n ${INPUT_BUILD_ML} ]] && BUILD_ML=${INPUT_BUILD_ML}[[ "${INPUT_DISABLE_CACHE}" == "1" ]] && DISABLE_CACHE="--no-cache"echo "DISABLE_CACHE: ${DISABLE_CACHE}"exit

buildall-common.sh — Added DISABLE_CACHE variable here as well

${DISABLE_CACHE} ${rest} ${name}/

action.yml — added in the description and requirement of disable_cache variable

disable_cache:
	description: 'This will disable the Docker build cache when building an image'
	required: false

build-and-publish.yml — set the default value of disable_cache to 0 since it runs with cache by default

disable_cache: 0 #this should only be set to '1' if you want to disable the docker image cache

To test the edits and re-build the buildall.sh file, I went to terminal and first changed the value of the disable_cache variable to be 1 (signaling no cache) by typing

**export INPUT_DISABLE_CACHE=1**

Then I tried building the buildall.sh script file by saying ./buildall.sh but I got the following error:

fatal: no tag exactly matches 'b75aad0ca715741a86aea51339013e5ddede9765'
Current tag could not be located
Perhaps you meant to run incr.sh ?

buildall-common.sh — The error means we were not putting in the tag exactly as it was supposed to be so I went in the buildall-common.sh file and removed the need for the tag to match exactly to run (line in the comment is what it was before while line above is what I changed it to)

BUILD_TAG=$(git describe --tags || true)  # The git tag for the images we are building
# BUILD_TAG=$(git describe --exact-match --tags || true)  # The git tag for the images we are building

I saved all changes and rebuilt the buildall.sh file after changing the value of disable_cache to be 1 and got the following response

Conclusion:

This edit allows developers to specify whether they want a build to run with or without cache
The default is no cache
If they want cache, they have to type “export INPUT_DISABLE_CACHE=1” in terminal before building the file
To specify with cache again, they have to type “export INPUT_DISABLE_CACHE=0” in terminal before building the file

Day 7: Researching on Azure environment

Researched into Azure environment shell and deploying HPCC on kubernetes

Day 8: Looked into Trivy Scanning

Read about Trivy scanning and how it works (theory)
Trivy: A simple and comprehensive vulnerability scanner for containers, images, and other artifacts
Using this tool to run a scan on the HPCC docker image will help me understand exactly what I should be trying to fix
A software vulnerability is a glitch, flaw, or weakness present in the software or in an Operating System. Trivy detects vulnerabilities of OS packages (Alpine, RHEL, CentOS, etc.) and application dependencies (Bundler, Composer, npm, yarn, etc.).

Day 9: Setup Entire LexisNexis Environment

Called Help Desk and setup my LN email and Workplace environment
I encountered a GRA configuration problem but was ultimately resolved
Got the security journey on my personal computer (in bookmarks)

Day 10: CyberSecurity Training, More About Trivy, Azure platform login and account setup

Started Cybersecurity training module
Setup Trivy on my local machine (https://github.com/aquasecurity/trivy)
https://aquasecurity.github.io/trivy/v0.18.3/
Installed on mac (brew install aquasecurity/trivy/trivy)
trivy image hpccsystems/platform-core:8.0.14–1
Logged into Azure platform using LexisNexis account; Requested an Azure subscription

Day 11: Trivy Analysis and Fixing ‘Cache’ and ‘No-Cache’ Directives in incremental build and buildall.sh

We always want no cache turned on for buildall, but we want an option (cache or no cache) for incremental build

Trivy Security Analysis

trivy image hpccsystems/DOCKER_IMAGE:TAG_NAME

First Medium Level Security Threat

Take Aways:

Medium Security Level Threats

AppArmor Mount rules are automatically widened when the program is compiled (http://manpages.ubuntu.com/manpages/xenial/man5/apparmor.d.5.html)
Kernet: arm straight-line speculation vulnerability
Local DoS by event-busy-loop from writing long lines
lz4: memory corruption due to an integer overflow bug caused by memmove
Improper input validation of octal strings

Day 12: Docker Best Practices — Images

Docker is a complex product, with many configuration options. Security best practices identify how to limit attack surfaces and lock Docker down

Choose minimal base images

Keep smallest possible attack surface — the less number of things you have in the image, the less chances you have of something going wrong
Very small and popular linux distribution hardened for security and includes almost nothing

Don’t build dependencies from source

Container image vulnerability scanners like Trivy that are limited in that they can detect vulnerability by looking at installed packages and their versions but there is a problem with this way of checking for vulnerabilities.
If you build something from source, your chances of a vulnerable piece of source code being build into a dependency for the application not getting picked up by one of these tools is much higher
By not building dependencies from source, you can prevent these image scanning softwares from missing certain vulnerabilities
Install dependencies using existing packages and package management systems that come with base image

Adopt fixed tags for immutability

Docker allows you to reference a container image that you want to base your image off of by using the name and then by labels or you can do it with a sha256 tag

When you know you are going to rebuild the same thing many times, fixing your base image using that tag ensures that any future changes won’t affect it
You have to be careful if the base image is updated or a new version is released that you change the tag naming correspondingly

Employ COPY instead of ADD

Utilize a least privilege user

If you are not using Docker Namespace making, which is disabled by default and usually remains this way because it is difficult to work with, when you run your processes in a container, they are going to run by default as the root user, which is the same root user as on the host
This makes it much more likely for something to go wrong where if there may be some vulnerabilities or opportunities that can be used inside the container to break out of it
It is best to run container processes as an unprivileged user so that if something goes wrong inside the container (maybe software gets exploited by some adversaries), the opportunities to get out of that container are that much more limited

One standard way to do this is to create an unprivileged system user as part of your image build process and then make that the default user to run when the container is executed

Deploy read-only filesystems and volumes

Protect sensitive information

Label and lint

Day 13: Full PR Run-Through

Rebuilt build file (./buildall.sh)
Tried checking history for that platform-build image but it was too large to run on trivy (9 GB)
Rebuilt image
Checked history again and reported results

Day 14: Vulnerability Scanning

Continued looking through Trivy scan and made a brief report of everything I understood from the results
Vulnerability Scanning Link: https://docs.docker.com/engine/scan/
Found another scanner similar to Trivy so you can compare the outputs from both of them (Trivy and the new scanner) — Docker documentation vulnerability scanner
https://geekflare.com/container-security-scanners/

Security Scanners:

Use hpccsystems/platform-core:TAG for the scanning since this is the final HPCC image that will be deployed, etc

Trivy: explanation, installation, and implementation in day 11 notes

Anchore: an open-source project that allows for the deep analysis of docker images and determines whether they are secure or not as well as any security vulnerabilities

Install:
brew tap anchore/grype
brew install grype
Run:
grype <image> -o <format> #I use table as the format

Docker Scan (Provided by Docker Desktop):

Install:
Pre-installed with Docker Desktop
Run:
Docker scan IMAGE_NAME

Day 15: Kubernetes Security Journey Section

Kubernetes Security Journey
Wrote down any questions I had while I watched the Kubernetes security journey
Researched into Kubernetes more

Day 16: HPCC Documentation Run Through and Azure Subscription

Deployed HPCC on local
Went through docker documentation and wrote down anything that is confusing from the documentation
Troubleshooted why I did not have the Azure subscription and asked Lorraine about the issue as well

Day 17: Security Vulnerability Scanning for the platform-core(main image)

Rescanned all the docker images except this time on platform-core instead of platform-build since platform-core is the final HPCC docker image
Trivy:

Anchore Grype:

Docker Scan:

Day 18: Understanding Kubernetes from beginner level

https://docs.google.com/document/d/1wDSTfdKnjNlmZ8-xHDF9TKsl94zwQSexQUVMmwrq1rQ/edit?usp=sharing
Refer to my notes above to get an understanding of what I researched

Day 19: Setting up Github Actions Workflow to build image from there instead of local machine (Second Blog Post)

Since building from local machine was taking too long, we had to setup a Github Actions and run the workflow from there to build the image

cd out of Dockerfiles
cd .github/workflows/
code . (configure VS code to open from terminal)
edit build-and-publish.yml in VS code
Change to this:

6. Make sure your Dockerhub has an exact replica of the workflow — has platform-core, platform-build, platform-build-base

Setup Secrets in github

Go to personal forked repo of HPCC platform
Go to settings
Go to Secrets
Add the following 4 secrets: DOCKER_USERNAME (dockerhub username), DOCKER_PASSWORD (dockerhub password), BUILD_USER (github username), DOCKER_REPO (dockerhub username)
Open buildall-common.sh in VS code
Comment out ‘exit 2’ in line 45
git status
Add all the files that are not tracked by doing git add FILE_NAME (You have to do this individually for each file since git add . does not work for some reason)
git commit -m “Workflow setup”
git push origin master
Go to actions in forked repo and click Build and Publish
Click Run Workflow in master branch

Day 20: Kubernetes Research

List all software (shared objects and executables), compare all 3 scans and combine into 1, coalesce them into one report, write report
This is why we should use minimal distribution — why it will be good or not based on attack surface as compared by the three report scans before and after

Day 21: Building Image with Github Actions instead of from local machine for the cache, no cache JIRA ticket (#25906)

Make sure master branch has the most recent changes from main HPCC branch (git fetch — all)
Make a new branch for the JIRA ticket (git checkout -b HPCC-25906 upstream/master)
Pick the right commit which has the changes made for cache no cache (exact tag number for commit can be found in github commit history) (git cherry-pick 2cee6446de725d68d69280e8ccebd79cd7867de8)
git push origin HPCC-25906
git checkout master
Changed the default cache to 1 in buildall-common.sh
git add build-and-publish.yml
git commit -m “Changing default cache to disabled”
git push origin master

Move the build-and-publish.yml to the branch for this ticket/error

cp build-and-publish.yml ~/build-and-publish.yml
git checkout HPCC-25906
cp ~/build-and-publish.yml build-and-publish.yml
git commit -a -m “Changing build and publish”
git push origin HPCC-25906

Rerun image in Workflow

Go back to the actions in my forked repository
Click build-and-publish and run the workflow in HPCC-25906

Day 22: Coalescing All Initial Security Vulnerability Scanning Results into One Report (Without Ubuntu Minimal)

Report: https://docs.google.com/document/d/1CkqLLt99NriZrrLVOnX_6lqde-StFfneoO8zCbc6pkc/edit?usp=sharing

List of all Vulnerable Software (Shared Objects and Executables) Across All 3 Scanners

Finished first JIRA ticket (List and verify all software that is included in container : #25916)

Day 23: Cache and No Cache Image Building

Checked normal way to compare to no cache — got a clean branch that uses master, created a branch that says master test, set it to mimic master, pushed it, did the build, ensured that is actually visible on Dockerhub
Finished mid-way internship checkpoint

Day 24: Table-Formatted Security Vulnerability Report

Report:https://docs.google.com/document/d/1CkqLLt99NriZrrLVOnX_6lqde-StFfneoO8zCbc6pkc/edit?usp=sharing

(Friday) Day 25: Made another report for ‘No-Cache’ Image Building to compare with the ‘cache’ builds

Report:https://docs.google.com/document/d/1fSJSZYUNUNXMaa_HSvVkVdznwWTX_K0vp9JYepyGhCM/edit

Summary:

We should stick with enabling image caching since there are two new ‘medium’ level security threats that get added when the image is built with disabled cache

libhogweed5
libnettle7

(Monday) Day 26: Ubuntu Minimal Analysis (Deciding whether we need to switch or not for security reasons)

Tried researching into how to determine whether we need ubuntu minimal without actually downloading it and implementing it on the HPCC image
Ask Michael: Could not find any specific information relating to certain smaller packages in ubuntu minimal that I could cross reference with our existing files in HPCC so how should I proceed? Is there another image that I should scan that has ubuntu minimal part of it?

(Tuesday) Day 27: Resolving Ubuntu JIRA ticket and Kubernetes JIRA Ticket

Summary: The latest version that we currently have (20.04) is perfectly fine for our needs. It is as stripped down as it needs to be in terms of attack surface. Here are references that explain how it has a minimal attack surface. According to an article (https://cloudacademy.com/blog/docker-image-security-get-it-in-your-sights/), in Ubuntu Minimal, when there is no USER specified in the dockerfile, the default mechanism is to execute the container using the root user — this increases the exposed attack surface and is bad. To avoid this undesirable scenario, we should create both a dedicated user and a dedicated group in the Docker image. Next, use the USER directive in the Dockerfile to make sure that the container runs the application with the least privileged access possible, as is best practice. Since HPCC systems does this, it complies with this specific part of ubuntu minimal. In addition, according to the article (https://snyk.io/blog/10-docker-image-security-best-practices/), using vulnerability scanners like Synk and Trivy is especially important for open source software to make sure there are no ‘High Risk’ or ‘Danger’ security threats that are coming up. After performing these scans on the current HPCC image, none of these high level threats were coming up which shows that it is as stripped down as it needs to be at this time. Continuous scans should still be run as more software is added to make sure the threats do not appear, but as for now, there is no urgent reason to transfer to Ubuntu. we should do the scans once every point release for the ubuntu distribution that we are using as our base. That is when we are changing any packages. Minimal since the current HPCC systems image seems to be secure enough.

(Wednesday) Day 28: Writing out Kubernetes Components to Secure

Wrote out Kubernetes components to secure and put this as a comment in JIRA to resolve ticket 25919
https://docs.google.com/document/d/1uVHetE1RjkgI4FdgI33A-6HK23BKugVTi8eQbGHnMXw/edit?usp=sharing
Start working on blog

(Thursday) Day 29: Networking Across Namespaces

Networking across namespaces, look up namespaces networking, write up something short for this

Notes: https://docs.google.com/document/d/1uVHetE1RjkgI4FdgI33A-6HK23BKugVTi8eQbGHnMXw/edit?usp=sharing
See starting page 7 for my notes on this(for blog)

(Friday) Day 30: Setting up a default HPCC Systems cluster on Microsoft Azure Cloud Using HPCC Systems and Kubernetes

https://hpccsystems.com/blog/default-azure-setup
Had a meeting with Stew, Michael, and Ming about everything I have done so far and what is to come
Went through tutorial above written by Jake Smith (made sure to delete resource group afterwards)
Wrote my first official blog post (Caching and No-Caching Option)

(Monday) Day 31: Started making Kubernetes list of components to secure (listed in table)

This table can be found in the ‘Kubernetes Notes’ document

(Tuesday) Day 32: Finished table from yesterday

Determined which are highest priority to work on first

(Wednesday) Day 33: Began research for Pod Security Policies and Network Policies

Ran a Pod security policy on my machine and determined what the process was to create my own policy
Took notes on what pod security policies are as well as best practices for these policies

(Thursday) Day 34: Made My Own Pod Security Policy

Created a pod security policy for denying certain privileged requests called “psp-deny-privileged.yaml”
YAML file in my notes

(Friday) Day 35: Researched into Network Policies and took Notes and SharePoint Read-Through

Notes can be found in Notion doc
Look through all the documents in Azure and Kubernetes sharepoint and make sure to find anything relevant to project
Find more information on Encryption at Rest (sharepoint documents that include any regulations or guidelines or suggestions for Best Practice)

(Tuesday) Day 36: HPCC YAML Files Analysis

Look through the HPCC Yaml files (ming set instructions in gitter) for reference and see if you can implement any of the security best practices there

Desired Audience: Everyone on open-source community (generalized audience)

(Wednesday) Day 37: HPCC Pod Security and Encryption (At Rest and in Transit) Best Practices Implementation and Encryption at Rest Read-Through

Read through SharePoint files on Encryption at Rest Best Guidelines
Establish HPCC framework and best practices on Encryption at Rest and Encryption in Transit protocols
Looked through HPCC yaml files and other source code to make sure they were following pod security best practices

High Level Overview of Kubernetes Security Concepts (For Presentation):

Pod Security Policies (Include what they are, what the best practices are, How you check to make sure HPCC was following these best practices and made suggestions for revisions if they did not)
Encryption at Rest and Encryption in Transit (Include what it is, what the best practices are, Make sure HPCC is following these best practices and make suggestions for revisions if they are not)
Include documentation for

(Thursday) Day 38: Hashicorp Vault Installation and TLS Check

Go through the github (https://github.com/hpcc-systems/HPCC-Platform/tree/master/helm/examples/secrets) for installing Hashicorp vault and after installation, go through and suggest changes based on security best practices for encryption (at rest and in transit) directly in the source code
Error when trying to setup environmental variable: after setting up k8s cluster, make sure the ports match (the port you assigned your kube cluster to should be the same as the one your assigning in the environmental variable)
Values.yaml file on TLS (https://github.com/hpcc-systems/HPCC-Platform/blob/community_8.2.0-2/helm/hpcc/values.yaml) ⇒ Is it working or does documentation need more clarity. Resolved (another intern)

(Friday) Day 39: Hashicorp Vault Installation and TLS Check

Testing the vault instruction code below:

Nikita’s HPCC Internship Journey 👩🏽‍💻

xwang2713/HPCC-Platform

See also https://learn.hashicorp.com/tutorials/vault/kubernetes-cert-manager for a more Vault centric tutorial on…

Written by Nikita Jha