Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dependency caching to reduce build time #1973

Draft
wants to merge 3 commits into
base: reduce-docker-buildtime
Choose a base branch
from

Conversation

kcirtapfromspace
Copy link
Contributor

@kcirtapfromspace kcirtapfromspace commented Feb 7, 2024

Description of change

Should something like turbo build be used to cache packages for dev & ci/cd systems?
Has CircleCI integration 👍

The major benefits to this approach would be there would be a shared remote build cache of packages developers and CI can leverage cutting a lengthy node build install process down to a second or two.

(Caching Tasks)[https://turbo.build/repo/docs/core-concepts/caching] 👍
Every JavaScript or TypeScript codebase will need to run package.json scripts, like build, test and lint. Using Turborepo, we call these tasks. Turborepo can cache the results and logs of your tasks - leading to enormous speedups for slow tasks.

Since Turborepo only caches to the local filesystem by default, the same task (turbo run build) must be re-executed on each machine (by you, by your teammates, by your CI, by your PaaS, etc.) even when all of the task inputs are identical — which wastes time and resources. By working with providers like Vercel, Turborepo can securely communicate with a remote cache - a cloud server that stores the results of your tasks.

This can save enormous amounts of time by preventing duplicated work across the organization.

How to test

Issue(s)

Checklists

Every PR

  • Meets issue criteria
  • JIRA ticket status updated
  • Code is meaningfully tested
  • Meets accessibility standards (WCAG 2.1 Levels A, AA)
  • API Documentation updated
  • Boundary diagram updated
  • Logical Data Model updated
  • Architectural Decision Records written for major infrastructure decisions
  • UI review complete

Before merge to main

  • OHS demo complete
  • Ready to create production PR

Production Deploy

  • Staging smoke test completed

After merge/deploy

  • Update JIRA ticket status

@kcirtapfromspace
Copy link
Contributor Author

I was able to get the frontend to cache & build.

The backend is a whole other thing.

@repo/backend:build: ../node_modules/@types/react/ts5.0/index.d.ts:4131:19 - error TS2320: Interface 'IntrinsicAttributes' cannot simultaneously extend types 'Attributes' and 'Attributes'.
@repo/backend:build:   Named property 'key' of types 'Attributes' and 'Attributes' are not identical.
@repo/backend:build: 
@repo/backend:build: 4131         interface IntrinsicAttributes extends React.Attributes {}
@repo/backend:build:                        ~~~~~~~~~~~~~~~~~~~
@repo/backend:build: 
@repo/backend:build: ../node_modules/@types/react/ts5.0/index.d.ts:4132:19 - error TS2320: Interface 'IntrinsicClassAttributes<T>' cannot simultaneously extend types 'ClassAttributes<T>' and 'ClassAttributes<T>'.
@repo/backend:build:   Named property 'key' of types 'ClassAttributes<T>' and 'ClassAttributes<T>' are not identical.
@repo/backend:build: 
@repo/backend:build: 4132         interface IntrinsicClassAttributes<T> extends React.ClassAttributes<T> {}
@repo/backend:build:                        ~~~~~~~~~~~~~~~~~~~~~~~~
@repo/backend:build: 
@repo/backend:build: ../node_modules/umzug/lib/umzug.d.ts:9:45 - error TS2315: Type 'ErrorWithCause' is not generic.
@repo/backend:build: 
@repo/backend:build: 9 export declare class MigrationError extends errorCause.ErrorWithCause<unknown> {
@repo/backend:build:                                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@repo/backend:build: 
@repo/backend:build: 
@repo/backend:build: Found 189 errors in 6 files.
@repo/backend:build: 
@repo/backend:build: Errors  Files
@repo/backend:build:      2  src/lib/cache.ts:40
@repo/backend:build:      2  src/routes/admin/redis.ts:36
@repo/backend:build:      2  tests/utils/dbUtils.ts:60
@repo/backend:build:    176  ../node_modules/@types/react-dom/node_modules/@types/react/index.d.ts:3139
@repo/backend:build:      6  ../node_modules/@types/react/ts5.0/index.d.ts:4112
@repo/backend:build:      1  ../node_modules/umzug/lib/umzug.d.ts:9
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@repo/backend:build: ERROR: command finished with error: command (/Users/patrickdeutsch/repos/Head-Start-TTADP/backend) /opt/homebrew/bin/yarn run build exited (2)
@repo/backend#build: command (/Users/patrickdeutsch/repos/Head-Start-TTADP/backend) /opt/homebrew/bin/yarn run build exited (2)

 Tasks:    1 successful, 2 total
Cached:    1 cached, 2 total
  Time:    18.231s 
Failed:    @repo/backend#build

@kcirtapfromspace kcirtapfromspace mentioned this pull request Feb 7, 2024
13 tasks
@kcirtapfromspace kcirtapfromspace changed the title Got turbo to build the frontend Add dependency caching to reduce build time Feb 7, 2024
Copy link
Contributor Author

@kcirtapfromspace kcirtapfromspace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

=> => # warning Workspaces can only be enabled in private projects.
LOLz. Yarn says its broken and this warning will pop up but to ignore it
yarnpkg/yarn#8580 (comment)

@kcirtapfromspace
Copy link
Contributor Author

Got the cashing to hit with vercel
image

@kcirtapfromspace
Copy link
Contributor Author

kcirtapfromspace commented Feb 9, 2024

Struggling with dependency resolution with a monorepo, turbo & the backend. Straight docker caching can build & serve no issues.

But there is a significant mismatch in installed dependencies 🤷 Which is what I suspect causes the issues for trying to do the backend caching.

https://github.com/GoogleContainerTools/container-diff

Overlap found in the following directories:
/app/node_modules/@rushstack: Added 112 files, Deleted 10 files
/app/node_modules/eslint-plugin-react: Added 17 files, Deleted 4 files
/app/node_modules/@newrelic: Added 12 files, Deleted 125 files
/app/node_modules/jsdom: Added 29 files, Deleted 1 files
/usr/share/doc: Added 513 files, Deleted 36 files
/app/node_modules/function-bind: Added 5 files, Deleted 4 files
/app/node_modules/sane: Added 14 files, Deleted 10 files
/app/node_modules/eslint-plugin-jest: Added 25 files, Deleted 8 files
/app/node_modules/cron-parser: Added 51 files, Deleted 24 files
/app/node_modules/parseley: Added 1 files, Deleted 6 files
/app/node_modules/htmlparser2: Added 22 files, Deleted 69 files
/app/node_modules/tr46: Added 2 files, Deleted 4 files
/app/node_modules/ioredis: Added 45 files, Deleted 14 files
/app/node_modules/pg: Added 6 files, Deleted 1 files
/app/node_modules/axios: Added 54 files, Deleted 10 files
/app/node_modules/browserslist: Added 1 files, Deleted 1 files
/app/node_modules/handlebars: Added 1 files, Deleted 1 files
/app/node_modules/playwright-core: Added 67 files, Deleted 52 files
/app/node_modules/preview-email: Added 83 files, Deleted 8 files
/app/node_modules/depcheck: Added 61 files, Deleted 14 files
/app/node_modules/tsx: Added 29 files, Deleted 12 files
/app/node_modules/nodemailer: Added 3 files, Deleted 2 files
/app/node_modules/enquirer: Added 1 files, Deleted 1 files
/app/node_modules/bull: Added 3 files, Deleted 3 files
/app/node_modules/editorconfig: Added 71 files, Deleted 23 files
/app/node_modules/npm: Added 3 files, Deleted 21 files
/app/node_modules/umzug: Added 43 files, Deleted 3 files
/app/node_modules/node-libs-browser: Added 41 files, Deleted 11 files
/app/node_modules/eslint-plugin-jsx-a11y: Added 40 files, Deleted 21 files
/app/node_modules/undici: Added 16 files, Deleted 2 files
/app/node_modules/js-beautify: Added 136 files, Deleted 14 files
/app/node_modules/events: Added 2 files, Deleted 11 files
/app/node_modules/mailparser: Added 22 files, Deleted 150 files
/app/node_modules/hoek: Added 1 files, Deleted 1 files
/app/node_modules/core-js: Added 347 files, Deleted 8 files
/app/node_modules/web-resource-inliner: Added 22 files, Deleted 6 files
/app/node_modules/eslint-plugin-import: Added 14 files, Deleted 16 files
/app/node_modules/libqp: Added 4 files, Deleted 1 files
/app/node_modules/iconv-lite: Added 12 files, Deleted 1 files
/app/node_modules/import-in-the-middle: Added 41 files, Deleted 1 files
/app/node_modules/istanbul-lib-instrument: Added 8 files, Deleted 545 files
/app/node_modules/triple-beam: Added 5 files, Deleted 4 files
/app/node_modules/@axe-core: Added 4 files, Deleted 30 files
/app/node_modules/entities: Added 56 files, Deleted 5 files
/app/node_modules/@grpc: Added 4 files, Deleted 93 files
/app/node_modules/side-channel: Added 1 files, Deleted 1 files
/app/node_modules/playwright: Added 102 files, Deleted 2 files
/app/node_modules/define-property: Added 9 files, Deleted 27 files
/app/node_modules/@messageformat: Added 2 files, Deleted 1 files
/app/node_modules/@babel: Added 453 files, Deleted 5081 files
/app/node_modules/@jest: Added 65 files, Deleted 232 files
/app/node_modules/rxjs: Added 9 files, Deleted 23 files
/app/node_modules/whatwg-url: Added 13 files, Deleted 15 files
/app/node_modules/.bin: Added 7 files, Deleted 2 files
/usr/share/man: Added 307 files, Deleted 4 files
/app/node_modules/nise: Added 7 files, Deleted 131 files
/app/node_modules/node-expat: Added 2 files, Deleted 51 files
/app/node_modules/just-extend: Added 4 files, Deleted 1 files
/app/node_modules/babel-plugin-istanbul: Added 8 files, Deleted 232 files
/app/node_modules/magic-string: Added 4 files, Deleted 3 files
/app/node_modules/sinon: Added 2 files, Deleted 61 files
/app/node_modules/@types: Added 35 files, Deleted 663 files
/app/node_modules/responselike: Added 4 files, Deleted 4 files
/app/node_modules/registry-auth-token: Added 2 files, Deleted 4 files
/app/node_modules/shell-quote: Added 6 files, Deleted 3 files
/app/node_modules/available-typed-arrays: Added 2 files, Deleted 1 files
/app/node_modules/linkify-it: Added 4 files, Deleted 2 files
/app/node_modules/polished: Added 1 files, Deleted 1 files
/app/node_modules/uc.micro: Added 8 files, Deleted 7 files
/app/node_modules/mobx-react-lite: Added 12 files, Deleted 30 files
/var/lib/dpkg: Added 513 files, Deleted 20 files
/app/node_modules/newrelic: Added 13 files, Deleted 61 files
/app/node_modules/winston-transport: Added 4 files, Deleted 2 files
/app/node_modules/run-applescript: Added 8 files, Deleted 10 files
/app/node_modules/cacheable-request: Added 23 files, Deleted 3 files
/app/node_modules/@elastic: Added 28 files, Deleted 27 files
/app/node_modules/styled-components: Added 2 files, Deleted 214 files
/app/node_modules/@sinonjs: Added 5 files, Deleted 4 files
/app/node_modules/@redocly: Added 78 files, Deleted 106 files
/app/node_modules/@typescript-eslint: Added 129 files, Deleted 973 files
/app/node_modules/http2-wrapper: Added 15 files, Deleted 1 files
/app/node_modules/assert: Added 3 files, Deleted 14 files
/app/node_modules/camelize: Added 5 files, Deleted 2 files
/app/node_modules/write-file-atomic: Added 37 files, Deleted 3 files
/app/node_modules/got: Added 22 files, Deleted 19 files
/app/node_modules/call-me-maybe: Added 3 files, Deleted 6 files
/app/node_modules/aws-sdk: Added 319 files, Deleted 37 files

@kcirtapfromspace
Copy link
Contributor Author

Basically stuck with this error 🤷
Found commentary on deleting your /.yarn to adding permissions to /etc/passwd.

2024-02-09 14:51:16 $ tsx watch --trace-events-enabled --trace-warnings --trace-uncaught ./src/index.ts
2024-02-09 14:51:17 9:51:17 PM [tsx] rerunning
2024-02-09 14:51:16 warning Skipping preferred cache folder "/.cache/yarn" because it is not writable.
2024-02-09 14:51:16 warning Selected the next writable cache folder in the list, will be "/tmp/.yarn-cache-501".
2024-02-09 14:51:16 warning Cannot find a suitable global folder. Tried these: "/usr/local, /.yarn"
2024-02-09 14:51:17 node:os:365
2024-02-09 14:51:17     throw new ERR_SYSTEM_ERROR(ctx);
2024-02-09 14:51:17     ^
2024-02-09 14:51:17 
2024-02-09 14:51:17 SystemError [ERR_SYSTEM_ERROR]: A system error occurred: uv_os_get_passwd returned ENOENT (no such file or directory)
2024-02-09 14:51:17     at new SystemError (node:internal/errors:256:5)
2024-02-09 14:51:17     at new NodeError (node:internal/errors:367:7)
2024-02-09 14:51:17     at Object.userInfo (node:os:365:11)
2024-02-09 14:51:17     at <instance_members_initializer> (/app/node_modules/tsx/dist/index-5f60304b.cjs:21:492)
2024-02-09 14:51:17     at new pA (/app/node_modules/tsx/dist/index-5f60304b.cjs:21:578)
2024-02-09 14:51:17     at Object.<anonymous> (/app/node_modules/tsx/dist/index-5f60304b.cjs:21:1813)
2024-02-09 14:51:17     at Module._compile (node:internal/modules/cjs/loader:1256:14)
2024-02-09 14:51:17     at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
2024-02-09 14:51:17     at Module.load (node:internal/modules/cjs/loader:1119:32)
2024-02-09 14:51:17     at Module._load (node:internal/modules/cjs/loader:960:12) {
2024-02-09 14:51:17   code: 'ERR_SYSTEM_ERROR',
2024-02-09 14:51:17   info: {
2024-02-09 14:51:17     errno: -2,
2024-02-09 14:51:17     code: 'ENOENT',
2024-02-09 14:51:17     message: 'no such file or directory',
2024-02-09 14:51:17     syscall: 'uv_os_get_passwd'
2024-02-09 14:51:17   },
2024-02-09 14:51:17   errno: [Getter/Setter],
2024-02-09 14:51:17   syscall: [Getter/Setter]
2024-02-09 14:51:17 }
2024-02-09 14:51:17 Thrown at:
2024-02-09 14:51:17     at __node_internal_captureLargerStackTrace (node:internal/errors:496:5)
2024-02-09 14:51:17     at SystemError (node:internal/errors:256:5)
2024-02-09 14:51:17     at NodeError (node:internal/errors:367:7)
2024-02-09 14:51:17     at userInfo (node:os:365:11)
2024-02-09 14:51:17     at <instance_members_initializer> (/app/node_modules/tsx/dist/index-5f60304b.cjs:21:492)
2024-02-09 14:51:17     at pA (/app/node_modules/tsx/dist/index-5f60304b.cjs:21:578)
2024-02-09 14:51:17     at /app/node_modules/tsx/dist/index-5f60304b.cjs:21:1813
2024-02-09 14:51:17     at Module._compile (node:internal/modules/cjs/loader:1256:14)
2024-02-09 14:51:17     at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
2024-02-09 14:51:17     at Module.load (node:internal/modules/cjs/loader:1119:32)
2024-02-09 14:51:17 
2024-02-09 14:51:17 
2024-02-09 14:51:17 Node.js v18.18.2
2024-02-09 14:51:17 Could not open trace file node_trace.1.log: permission denied

@thewatermethod
Copy link
Collaborator

First thought is that this appears to me like the kind of error you get when the bindings for your npm packages install doesn't match the bindings for your system...

Could be wrong, but I don't think the warning Cannot find a suitable global folder. Tried these: "/usr/local, /.yarn" errors at the top are related to the fatal one. Looks like you get that when "os" can't find a home directory. Could the fix be as simple as chmod'ing something?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants