Skip to content

Commit

Permalink
Merge branch 'thinx-staging'
Browse files Browse the repository at this point in the history
* thinx-staging: (181 commits)
  log leak fix
  log cleanup, redeploy after changing expired Rollbar Project Access Tokens
  submodule sync
  version bump, log cleanup and base image update after fixing GitHub OAuth
  dependency updates, fixing github login (has code but different object structure)
  test passes, but parsing fails
  spec fix for staging
  fails in tests, because code is B
  removed json
  fixes
  gpg
  debugging broken github-oauth login
  task renamed
  recent test passed, build stable, adding debug logging only for next refactoring steps
  fix for potent. unlinked github login addRoutes
  recent test passed, build stable, adding debug logging only for next refactoring steps
  github fix
  ssrf fix
  fix axios get error
  moved from got to axios
  ...
  • Loading branch information
suculent committed Nov 8, 2023
2 parents 0a5a2c3 + 78eeed6 commit 9e41ced
Show file tree
Hide file tree
Showing 251 changed files with 30,636 additions and 10,451 deletions.
41 changes: 21 additions & 20 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
executor: docker/docker
steps:
- setup_remote_docker:
version: 20.10.12
version: 20.10.18
- checkout
- docker/check
- run:
Expand All @@ -33,7 +33,7 @@ jobs:
executor: docker/docker
steps:
- setup_remote_docker:
version: 20.10.12
version: 20.10.18
- checkout
- run:
name: Fetch all dependencies (esp. console)
Expand Down Expand Up @@ -75,7 +75,7 @@ jobs:
executor: docker/docker
steps:
- setup_remote_docker:
version: 20.10.12
version: 20.10.18
- checkout
- run:
name: Fetch all dependencies (esp. console)
Expand Down Expand Up @@ -111,7 +111,7 @@ jobs:
executor: docker/docker
steps:
- setup_remote_docker:
version: 20.10.12
version: 20.10.18
- checkout
- run:
name: Clean package-lock
Expand Down Expand Up @@ -175,7 +175,7 @@ jobs:
docker compose version
- setup_remote_docker:
version: 20.10.12
version: 20.10.18

- checkout

Expand Down Expand Up @@ -212,7 +212,7 @@ jobs:
command: |
mkdir -p /mnt/data/
mkdir -p /mnt/data && cp -R ./conf /mnt/data
mkdir -p /mnt/data/ssh_keys
mkdir -p /mnt/data/ssh_keys && cp -R ./spec/mnt/data/ssh_keys /mnt/data
mkdir -p /mnt/data/mosquitto/auth
mkdir -p /mnt/data/mosquitto/config
mkdir -p /mnt/data/mosquitto/data
Expand Down Expand Up @@ -316,16 +316,17 @@ jobs:
docker compose logs couchdb
- run:
name: Starting API Test
name: Running Unit and Integration Tests
command: |
export ENVIRONMENT=test
pwd
docker compose up --build api | tee -ia ./test.log
if [ ! $(cat ./test.log | grep "specs, 0 failures") ]; then
echo "Docker test complete."
if [[ ! $(grep "specs, 0 failures" ./test.log) ]]; then
echo "» TEST failed. Should not deploy this commit."
cat ./test.log | grep "Failures:"
grep "Failures:" ./test.log
exit 1
fi
date
- run:
name: Allow inspecting MQTT logs
Expand Down Expand Up @@ -372,7 +373,7 @@ jobs:
executor: docker/docker
steps:
- setup_remote_docker:
version: 20.10.12
version: 20.10.18
- checkout
- docker/check
- run:
Expand Down Expand Up @@ -426,15 +427,15 @@ workflows:
- main
- master

# - build-console-classic:
# context:
# - thinx-docker-repo
# - console
# filters:
# branches:
# only:
# - thinx-staging
# - master
- build-console-classic:
context:
- thinx-docker-repo
- console
filters:
branches:
only:
- thinx-staging
- master

# - build-console-cloud:
# context:
Expand Down
1 change: 0 additions & 1 deletion .coveralls.yml
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
repo_token: nJpg2RHfxQRyMMmHAYmGTUyWa3B7L76ty

15 changes: 4 additions & 11 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,13 +1,6 @@
clair*
.git
node_modules/
**/node_modules/
package-lock.json
tools/arduino-docker-build
tools/platformio-docker-build
tools/micropython-docker-build
tools/mongoose-docker-build
tools/nodemcu-docker-build
tools/nodemcu-firmware
tools/lua-inspect

conf/
tools/
conf/
clair*
3 changes: 3 additions & 0 deletions .env.dist
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,9 @@ WORKER_SECRET=twilight_zone

# Slack Bot Notifications
# SLACK_BOT_TOKEN=
# SLACK_CLIENT_ID=
# SLACK_CLIENT_SECRET=
# SLACK_WEBHOOK=

# Mailgun API Key
# MAILGUN_API_KEY=
34 changes: 34 additions & 0 deletions .github/workflows/checkmarx.ym_
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: "Checkmarx"

on:
push:
branches: [master, thinx-staging, main]
pull_request:
# The branches below must be a subset of the branches above
branches: [master, thinx-staging, main]
schedule:
- cron: '0 18 * * 5'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest

strategy:
fail-fast: false
matrix:
# Override automatic language detection by changing the below list
# Supported options are ['csharp', 'cpp', 'go', 'java', 'javascript', 'python']
language: ['javascript']
# Learn more...
# https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#overriding-automatic-language-detection

steps:
- name: Checkmarx AST Github Action
uses: Checkmarx/[email protected]

with:
base_uri: https://ast.checkmarx.net/
cx_tenant: nfr_nfr_ast_corpus
cx_client_id: ${{ secrets.CX_CLIENT_ID }}
cx_client_secret: ${{ secrets.CX_CLIENT_SECRET }}
4 changes: 0 additions & 4 deletions .gitmodules
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,6 @@
path = builders/nodemcu-docker-build
url = https://github.com/suculent/nodemcu-docker-build.git

[submodule "builders/nodemcu-firmware"]
path = builders/nodemcu-firmware
url = https://github.com/suculent/nodemcu-firmware.git

[submodule "builders/platformio-docker-build"]
path = builders/platformio-docker-build
url = https://github.com/suculent/platformio-docker-build.git
Expand Down
14 changes: 11 additions & 3 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
FROM thinxcloud/base:alpine

LABEL maintainer="Matej Sychra <[email protected]>"
LABEL name="THiNX API" version="1.8.2247"
LABEL name="THiNX API" version="1.9.2451"

ARG DEBIAN_FRONTEND=noninteractive

Expand Down Expand Up @@ -47,9 +47,17 @@ ARG GITHUB_CLIENT_ID
ENV GITHUB_CLIENT_ID=${GITHUB_CLIENT_ID}
ARG GITHUB_CLIENT_SECRET
ENV GITHUB_CLIENT_SECRET=${GITHUB_CLIENT_SECRET}
ARG GITHUB_ACCESS_TOKEN
ENV GITHUB_ACCESS_TOKEN={GITHUB_ACCESS_TOKEN}

ARG SLACK_BOT_TOKEN
ENV SLACK_BOT_TOKEN=${SLACK_BOT_TOKEN}
ARG SLACK_CLIENT_ID
ENV SLACK_CLIENT_ID=${SLACK_CLIENT_ID}
ARG SLACK_CLIENT_SECRET
ENV SLACK_CLIENT_SECRET=${SLACK_CLIENT_SECRET}
ARG SLACK_WEBHOOK
ENV SLACK_WEBHOOK=${SLACK_WEBHOOK}

ARG ENTERPRISE
ENV ENTERPRISE=${ENTERPRISE}
Expand All @@ -66,8 +74,8 @@ WORKDIR /opt/thinx/thinx-device-api
# Install app dependencies
COPY package.json ./

RUN npm install -g npm@8.6.0 \
&& npm install --unsafe-perm --only-prod .
RUN npm install -g npm@10.2.3 \
&& npm install --only-prod .

# THiNX Web & Device API (HTTP)
EXPOSE 7442
Expand Down
22 changes: 13 additions & 9 deletions Dockerfile.test
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
FROM thinxcloud/base:alpine

LABEL maintainer="Matej Sychra <[email protected]>"
LABEL name="THiNX API" version="1.8.2247"
LABEL name="THiNX API" version="1.9.2451"

ARG DEBIAN_FRONTEND=noninteractive

# For test-env node-18
ENV NODE_TLS_REJECT_UNAUTHORIZED=0
ENV NODE_EXTRA_CA_CERTS=/mnt/data/ssl/testRoot.crt

ARG THINX_HOSTNAME
ENV THINX_HOSTNAME=${THINX_HOSTNAME}

Expand All @@ -33,9 +29,6 @@ ENV AQUA_SEC_TOKEN=${AQUA_SEC_TOKEN}
ARG SNYK_TOKEN
ENV SNYK_TOKEN=${SNYK_TOKEN}

ARG GITHUB_ACCESS_TOKEN
ENV GITHUB_ACCESS_TOKEN={GITHUB_ACCESS_TOKEN}

ARG ENVIRONMENT
ENV ENVIRONMENT=${ENVIRONMENT}

Expand All @@ -54,9 +47,17 @@ ARG GITHUB_CLIENT_ID
ENV GITHUB_CLIENT_ID=${GITHUB_CLIENT_ID}
ARG GITHUB_CLIENT_SECRET
ENV GITHUB_CLIENT_SECRET=${GITHUB_CLIENT_SECRET}
ARG GITHUB_ACCESS_TOKEN
ENV GITHUB_ACCESS_TOKEN={GITHUB_ACCESS_TOKEN}

ARG SLACK_BOT_TOKEN
ENV SLACK_BOT_TOKEN=${SLACK_BOT_TOKEN}
ARG SLACK_CLIENT_ID
ENV SLACK_CLIENT_ID=${SLACK_CLIENT_ID}
ARG SLACK_CLIENT_SECRET
ENV SLACK_CLIENT_SECRET=${SLACK_CLIENT_SECRET}
ARG SLACK_WEBHOOK
ENV SLACK_WEBHOOK=${SLACK_WEBHOOK}

ARG GITHUB_SECRET
ENV GITHUB_SECRET=${GITHUB_SECRET}
Expand All @@ -73,6 +74,9 @@ ENV CIRCLE_NODE_TOTAL=${CIRCLE_NODE_TOTAL}
ARG CIRCLE_NODE_INDEX
ENV CIRCLE_NODE_INDEX=${CIRCLE_NODE_INDEX}

ARG NODE_COVERALLS_DEBUG
ENV NODE_COVERALLS_DEBUG=0

# Create app directory
WORKDIR /opt/thinx/thinx-device-api

Expand All @@ -82,7 +86,7 @@ RUN apk add openjdk8-jre p7zip
# Install app dependencies
COPY package.json ./

RUN npm install -g npm@8.6.0 \
RUN npm install -g npm@10.2.3 \
&& npm install .

VOLUME /var/lib/docker
Expand Down
10 changes: 10 additions & 0 deletions HISTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## HISTORY

6/11/2023 1.9.2451

» Updating vulnerable components
» Improving security configuration
» Cleaning vulnerable/unnecessary parts of code

28/1/2023 1.8.2343

» Refactoring fixes, optimizations, dependency updates and cleanup, production fix

16/1/2023 1.8.2247

» Security fixes due to Circle CI leak (moved Mailgun API key to env var)
Expand Down
2 changes: 0 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,7 @@ IoT Device Management Server running on node.js.

[![CodeFactor](https://www.codefactor.io/repository/github/suculent/thinx-device-api/badge)](https://www.codefactor.io/repository/github/suculent/thinx-device-api)
[![codebeat badge](https://codebeat.co/badges/a3b416b1-b53b-4bc5-ae6e-8a2b9ca31880)](https://codebeat.co/projects/github-com-suculent-thinx-device-api-master)
[![Language grade: JavaScript](https://img.shields.io/lgtm/grade/javascript/g/suculent/thinx-device-api.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/suculent/thinx-device-api/context:javascript)
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/9a7d084ad97e430ba12333f384b44255)](https://www.codacy.com/app/suculent/thinx-device-api?utm_source=github.com&utm_medium=referral&utm_content=suculent/thinx-device-api&utm_campaign=badger)
[![Total alerts](https://img.shields.io/lgtm/alerts/g/suculent/thinx-device-api.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/suculent/thinx-device-api/alerts/)
<a href="https://scan.coverity.com/projects/suculent-thinx-device-api">
<img alt="Coverity Scan Build Status"
src="https://scan.coverity.com/projects/18787/badge.svg"/>
Expand Down
28 changes: 28 additions & 0 deletions VeracodeIgnored.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
[
{
"CWEId": "259",
"FlawMatch": {
"ProcedureHash": "1057981634",
"PrototypeHash": "2924686005",
"FlawHash": "2666345062",
"FlawHashCount": "1",
"FlawHashOrdinal": "1",
"CauseHash": "3488685266",
"CauseHashCount": "1",
"CauseHashOrdinal": "1",
"CauseHash2": "0",
"CauseHash2Ordinal": "0"
},
"Files": {
"SourceFile": {
"File": "auth.js",
"Line": "23",
"FunctionName": "add_mqtt_credentials",
"QualifiedFunctionName": "Auth.add_mqtt_credentials",
"FunctionPrototype": "add_mqtt_credentials(: ::Auth, : any, : any, : any, ...) : any",
"Scope": "^::Auth",
"AbsoluteFilePath": "/Users/sychram/Repositories/thinx-device-api/lib/thinx/auth.js"
}
}
}
]
2 changes: 1 addition & 1 deletion base
Submodule base updated 5 files
+1 −0 .cxast
+3 −4 Dockerfile
+3 −0 README.md
+2,162 −2,451 package-lock.json
+19 −19 package.json
2 changes: 1 addition & 1 deletion builders/arduino-docker-build
Loading

0 comments on commit 9e41ced

Please sign in to comment.