Active / Candidate test, missing offchain execution

[0xAustinWang]

CCIP-3372

This PR builds off of the work we've done in https://smartcontract-it.atlassian.net/browse/CCIP-3755, it'll need to be merged first

[github-actions]

AER Report: CI Core

aer_workflow , commit , Detect Changes , Clean Go Tidy & Generate , lint , Core Tests (go_core_tests) , Core Tests (go_core_ccip_deployment_tests) , Core Tests (go_core_race_tests) , Core Tests (go_core_fuzz) , SonarQube Scan

1. Duplicate service name error: Core Tests (go_core_ccip_deployment_tests)

Source of Error:

Print Filtered Test Results	2024-10-21T07:43:53.9768518Z [DEBUG] freeport: Test "Test_ActiveCandidateMigration" returned ports [28001 28002 28003 28004 28005 28006]
Print Filtered Test Results	2024-10-21T07:43:53.9769885Z ^[[0;31mpanic: duplicate name "EVM.90000001.Relayer.ChainReader": service names must be unique: types *evm.chainReader & *evm.chainReader [recovered] ^[[0m
Print Filtered Test Results	2024-10-21T07:43:53.9769885Z 	panic: duplicate name "EVM.90000001.Relayer.ChainReader": service names must be unique: types *evm.chainReader & *evm.chainReader
Print Filtered Test Results	2024-10-21T07:43:53.9770662Z goroutine 397 [running]:
Print Filtered Test Results	2024-10-21T07:43:53.9771086Z testing.tRunner.func1.2({0x8174d40, 0xc001747920})
Print Filtered Test Results	2024-10-21T07:43:53.9771724Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1631 +0x24a
Print Filtered Test Results	2024-10-21T07:43:53.9772256Z testing.tRunner.func1()
Print Filtered Test Results	2024-10-21T07:43:53.9772780Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1634 +0x377
Print Filtered Test Results	2024-10-21T07:43:53.9773474Z panic({0x8174d40?, 0xc001747920?})
Print Filtered Test Results	2024-10-21T07:43:53.9774018Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/panic.go:770 +0x132
Print Filtered Test Results	2024-10-21T07:43:53.9775190Z github.com/smartcontractkit/chainlink-common/pkg/services.(*HealthChecker).Register(0xc00207ed00, {0x7f54ccae0ac8, 0xc0092c1b00})
Print Filtered Test Results	2024-10-21T07:43:53.9776701Z 	/home/runner/go/pkg/mod/github.com/smartcontractkit/[email protected]/pkg/services/health.go:195 +0x2d3
Print Filtered Test Results	2024-10-21T07:43:53.9778216Z github.com/smartcontractkit/chainlink/v2/core/services/job.(*spawner).StartService(_, {_, _}, {0x2, {0x4b, 0xe2, 0xb9, 0xb5, 0x34, 0xae, ...}, ...})
Print Filtered Test Results	2024-10-21T07:43:53.9779329Z 	/home/runner/work/chainlink/chainlink/core/services/job/spawner.go:230 +0x910
Print Filtered Test Results	2024-10-21T07:43:53.9780591Z github.com/smartcontractkit/chainlink/v2/core/services/job.(*spawner).CreateJob(0xc00288e8c0, {0xa3fffc8, 0xc001c93180}, {0x0?, 0x0?}, 0xc003689448)
Print Filtered Test Results	2024-10-21T07:43:53.9781743Z 	/home/runner/work/chainlink/chainlink/core/services/job/spawner.go:264 +0x430
Print Filtered Test Results	2024-10-21T07:43:53.9783033Z github.com/smartcontractkit/chainlink/v2/core/services/chainlink.(*ChainlinkApplication).AddJobV2(0xb5b9e24b00000000?, {0xa3fffc8?, 0xc001c93180?}, 0x0?)
Print Filtered Test Results	2024-10-21T07:43:53.9784281Z 	/home/runner/work/chainlink/chainlink/core/services/chainlink/application.go:855 +0x2d
Print Filtered Test Results	2024-10-21T07:43:53.9785900Z github.com/smartcontractkit/chainlink/integration-tests/deployment/memory.JobClient.ProposeJob({0x0?}, {0xa3fffc8, 0xc001c93180}, 0xc003bf6780, {0x60?, 0x88ae2a0?, 0xc001c93101?})
Print Filtered Test Results	2024-10-21T07:43:53.9787424Z 	/home/runner/work/chainlink/chainlink/integration-tests/deployment/memory/job_client.go:170 +0x207
Print Filtered Test Results	2024-10-21T07:43:53.9789378Z github.com/smartcontractkit/chainlink/integration-tests/deployment/ccip.updateJobSpecsAndSendRequest(0xc0031ac4e0, {{0x8ca0f6b, 0x6}, 0xc00337e270, {0xa434540, 0xc0013729e8}, {0xc00246a980, 0x6, 0x8}, {0x7f54cc8575d8, ...}}, ...)
Print Filtered Test Results	2024-10-21T07:43:53.9791416Z 	/home/runner/work/chainlink/chainlink/integration-tests/deployment/ccip/test_helpers.go:542 +0x2cd
Print Filtered Test Results	2024-10-21T07:43:53.9792708Z github.com/smartcontractkit/chainlink/integration-tests/deployment/ccip.Test_ActiveCandidateMigration(0xc0031ac4e0)
Print Filtered Test Results	2024-10-21T07:43:53.9793998Z 	/home/runner/work/chainlink/chainlink/integration-tests/deployment/ccip/active_candidate_test.go:108 +0x1859
Print Filtered Test Results	2024-10-21T07:43:53.9794804Z testing.tRunner(0xc0031ac4e0, 0x9a0e568)
Print Filtered Test Results	2024-10-21T07:43:53.9795384Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1689 +0xfb
Print Filtered Test Results	2024-10-21T07:43:53.9795964Z created by testing.(*T).Run in goroutine 1
Print Filtered Test Results	2024-10-21T07:43:53.9796557Z 	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1742 +0x390

Why: The error occurs because there are two services with the same name "EVM.90000001.Relayer.ChainReader" being registered. Service names must be unique to avoid conflicts and ensure proper identification and management.

Suggested fix: Ensure that each service has a unique name. Check the service registration logic and modify the names to be unique, possibly by including additional identifiers or context-specific information.

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

[github-actions]

AER Report: CI Core

aer_workflow , commit , Detect Changes , Clean Go Tidy & Generate , Core Tests (go_core_tests) , Core Tests (go_core_ccip_deployment_tests) , lint , Core Tests (go_core_race_tests) , Core Tests (go_core_fuzz) , SonarQube Scan

1. Segmentation violation error in launcher.go:

Source of Error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5c2eeac]
goroutine 4002 [running]:
github.com/smartcontractkit/chainlink/v2/core/capabilities/ccip/launcher.(*launcher).processUpdate(0xc006a83760, 0xc002dd2270)
	/home/runner/work/chainlink/chainlink/core/capabilities/ccip/launcher/launcher.go:200 +0x22c
github.com/smartcontractkit/chainlink/v2/core/capabilities/ccip/launcher.(*launcher).processDiff(0xc006a83760, {0xc002dd21e0, 0xc002dd2210, 0xc002dd2270})
	/home/runner/work/chainlink/chainlink/core/capabilities/ccip/launcher/launcher.go:180 +0x10e
github.com/smartcontractkit/chainlink/v2/core/capabilities/ccip/launcher.(*launcher).tick(0xc006a83760)
	/home/runner/work/chainlink/chainlink/core/capabilities/ccip/launcher/launcher.go:163 +0x1f7
github.com/smartcontractkit/chainlink/v2/core/capabilities/ccip/launcher.(*launcher).monitor(0xc006a83760)
	/home/runner/work/chainlink/chainlink/core/capabilities/ccip/launcher/launcher.go:139 +0xdd
created by github.com/smartcontractkit/chainlink/v2/core/capabilities/ccip/launcher.(*launcher).Start.func1 in goroutine 426
	/home/runner/work/chainlink/chainlink/core/capabilities/ccip/launcher/launcher.go:126 +0xa5

Why: This error is caused by a nil pointer dereference in the processUpdate method of the launcher struct. It indicates that the code attempted to access a memory address that is not allocated, likely due to an uninitialized or nil pointer.

Suggested fix: Ensure that all pointers are properly initialized before use. Add nil checks before dereferencing pointers in the processUpdate method and other related methods.

2. Incorrect format specifier in fmt.Printf:

Source of Error:

core/capabilities/ccip/launcher/launcher.go:190:3: printf: fmt.Printf format %s has arg donID of wrong type github.com/smartcontractkit/chainlink/v2/core/services/registrysyncer.DonID (govet)
	fmt.Printf("updating id: %s", donID)
	^
core/capabilities/ccip/launcher/launcher.go:266:3: printf: fmt.Printf format %s has arg id of wrong type github.com/smartcontractkit/chainlink/v2/core/services/registrysyncer.DonID (govet)
	fmt.Printf("removing id: %s", id)
	^

Why: The fmt.Printf function is using the %s format specifier for arguments of type DonID, which is not a string type. This causes a type mismatch error.

Suggested fix: Convert the DonID type to a string before passing it to fmt.Printf. For example, use donID.String() if DonID has a String method, or use a different format specifier that matches the type of DonID.

fmt.Printf("updating id: %v", donID)
fmt.Printf("removing id: %v", id)

3. Failed to process diff error:

Source of Error:

2024-10-21T15:47:18.506Z	ERROR	launcher/launcher.go:140	Failed to tick	{"version": "unset@unset", "err": "failed to process diff: failed to handle active-candidate deployment for CCIP DON 2: close base oracle: can only close a started oracle\nchain-writer has already been stopped: already stopped\nChainReader has already been stopped: already stopped\nChainReader has already been stopped: already stopped\nChainReader has already been stopped: already stopped\nchain-writer has already been stopped: already stopped\nchain-writer has already been stopped: already stopped"}

Why: This error occurs because the code attempts to close or stop components that are already stopped, leading to redundant operations and potential errors.

Suggested fix: Add checks to ensure that components are only closed or stopped if they are currently running. This can be done by maintaining a state flag for each component and verifying its state before attempting to close or stop it.

if !component.isStopped() {
    component.stop()
}

4. Golang Lint errors:

Source of Error:

core/capabilities/ccip/launcher/launcher.go:190:3: printf: fmt.Printf format %s has arg donID of wrong type github.com/smartcontractkit/chainlink/v2/core/services/registrysyncer.DonID (govet)
	fmt.Printf("updating id: %s", donID)
	^
core/capabilities/ccip/launcher/launcher.go:266:3: printf: fmt.Printf format %s has arg id of wrong type github.com/smartcontractkit/chainlink/v2/core/services/registrysyncer.DonID (govet)
	fmt.Printf("removing id: %s", id)
	^

Why: The govet linter detects that the format specifier %s is used incorrectly for a non-string type.

Suggested fix: Convert the DonID type to a string or use the appropriate format specifier as suggested in the previous fix.

fmt.Printf("updating id: %v", donID)
fmt.Printf("removing id: %v", id)

These fixes should address the errors and improve the stability and correctness of the code.

AER Report: Operator UI CI ran successfully ✅

aer_workflow , commit

[cl-sonarqube-production]

Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint