Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RELEASE] MRC v24.10.00 #507

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

[RELEASE] MRC v24.10.00 #507

wants to merge 16 commits into from

Conversation

dagardner-nv
Copy link
Contributor

❄️ Code freeze for branch-24.10 and v24.10 release

What does this mean?

Only critical/hotfix level issues should be merged into branch-24.10 until release (merging of this PR).

All other development PRs should be retargeted towards the next release branch: branch-25.02.

What is the purpose of this PR?

  • Update documentation
  • Allow testing for the new release
  • Enable a means to merge branch-24.10 into main for the release

dagardner-nv and others added 11 commits July 3, 2024 10:18
@dagardner-nv
Creating branch for v24.10
f754c78
@dagardner-nv
Updating versions for v24.10.00
246ac97
@dagardner-nv
Merge pull request #484 from nv-morpheus/branch-24.06
8377f6a 
Forward-merge branch-24.06 into branch-24.10
@dagardner-nv
Ensure proper initialization of CMAKE_INSTALL_PREFIX if needed (#485)
bceb7ef 
* Invoke `morpheus_utils_initialize_install_prefix` function after calling `project`
* Related to nv-morpheus/Morpheus#1776

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #485
@dagardner-nv
Stop a python source once the subscriber is no longer subscribed (#493)
ca8a73f 
* When a Python generator source yields a value, and the subscriber is no longer subscribed, stop the source.
* Fix out of date docstring comment.

This is a partial fix for nv-morpheus/Morpheus#1838

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)

URL: #493
@dagardner-nv
Define a Python source which receives a reference to a subscriber (#496)
8489b45 
* Allows a Python generator source to check if the subscriber is still subscribed.
* Define a class `SubscriberFuncWrapper`  for Python sources rather than just a lambda. The reason is that python objects captured by the lambda need to be destroyed while the gil is held, which causes a problem if the lambda is destroyed unexpectedly.
* Update `conftest.py` to set the loglevel to `DEBUG` if the `GLOG_v` environment variable is defined.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #496
@dagardner-nv
Change LOG(WARNING) to VLOG(1) when no GPUs are detected (#497)
ccbcd76 
* Since CPU-only mode will become a supported feature we want to avoid unnecessary warnings.

Relates to nv-morpheus/Morpheus#1851

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #497
@dagardner-nv
Pass a mrc.Subscription object to sources rather than a `mrc.Subscr…
48d17a1 
…iber` (#499)

* Remove the `make_source_subscriber` method in favor of inspecting the Python function signature.
* Since the `make_source_subscriber` method was never part of a release I think this can still be considered a non-breaking change.

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #499
@cwharris
Update to RAPIDS 24.10 (#494)
cef7f0b 
Closes #478
Requires nv-morpheus/utilities#75

Authors:
  - Christopher Harris (https://github.com/cwharris)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)
  - David Gardner (https://github.com/dagardner-nv)

URL: #494
@mdemoret-nv
Adding environment variable to allow skip NUMA node check (#505)
43f200e 
If the NUMA node check fails for a GPU, the entire pipeline cannot start. This allows setting `MRC_SKIP_NUMA_CHECK=1` to bypass the assertion if the check fails.

Closes #504

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Devin Robison (https://github.com/drobison00)
  - David Gardner (https://github.com/dagardner-nv)

URL: #505
@dagardner-nv
Fix Incorrect docstring in mrc.core.coro (#503)
4f23470 
* Include MRC libs with quotes not <>
* Add MRC version string to coro module (all the other Python modules have this)
* Fix spelling mistake in a comment in `python/mrc/core/executor.cpp`

Closes [#492](#492)

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #503
@dagardner-nv dagardner-nv added the ! - Release Related to a Major Release label Oct 21, 2024
@dagardner-nv dagardner-nv requested review from a team as code owners October 21, 2024 18:08
@dagardner-nv dagardner-nv self-assigned this Oct 21, 2024
cwharris and others added 5 commits October 21, 2024 23:15
@cwharris
Fix get-pr-info gha reference (#500)
4acef2b 
`rapidsai/shared-action-workflows/get-pr-info` was deleted in favor of `nv-gha-runners/get-pr-info`

rapidsai/shared-actions#11

Authors:
  - Christopher Harris (https://github.com/cwharris)
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #500
@jjacobelli
devcontainer: replace VAULT_HOST with AWS_ROLE_ARN (#506)
61862b5 
This PR is replacing the `VAULT_HOST` variable with `AWS_ROLE_ARN`. This is required to use the new token service to get AWS credentials

Authors:
  - Jordan Jacobelli (https://github.com/jjacobelli)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: #506
@mdemoret-nv
Add Router nodes to support single input, multi output routing. (#502)
a4b2ca1 
Overview of new classes added in this PR:

- `DynamicRouterBase`
  - Sink
    - `WritableProvider`, `ReadableAcceptor`, `SinkChannelOwner`
  - Source
    - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner`
  - Runnable with ability to change number of outputs at runtime
  - Cannot be added to a Segment
  - Good backpressure support. Can still process requests if downstream blocks
  - `LambdaDynamicRouter`
    - Provide key function via lambda
- `DynamicRouterComponentBase`
  - Sink
    - `ForwardingWritableProvider`
  - Source
    - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner`
  - Non-Runnable with ability to change number of outputs at runtime
  - Limited backpressure support. Will block if downstream blocks
  - `LambdaDynamicRouterComponent`
    - Provide key function via lambda
- `StaticRouterRunnableBase`
  - Sink
    - `WritableProvider`, `ReadableAcceptor`, `SinkChannelOwner`
  - Source
    - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner`
  - Runnable with fixed number of outputs
  - Can be added to a Segment since it supports child outputs
  - Good backpressure support. Can still process requests if downstream blocks
  - `LambdaStaticRouterRunnable`
    - Provide key function via lambda
- `StaticRouterComponentBase`
  - Sink
    - `ForwardingWritableProvider`
  - Source
    - `MultiWritableAcceptor`, `MultiReadableProvider`, `MultiSourceChannelOwner`
  - Non-Runnable with fixed number of outputs
  - Can be added to a Segment since it supports child outputs
  - Limited backpressure support. Will block if downstream blocks
  - `LambdaStaticRouterComponent`
    - Provide key function via lambda

Closes #491

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - David Gardner (https://github.com/dagardner-nv)

URL: #502
@mdemoret-nv
Fix CMake issue which installed headers incorrectly during conda build (
7abfdf7 
#511)

Fixes a few issues:
- Headers were being installed incorrectly with new `HEADER_SET`
- Fixed packages for pymrc component
- Standardized the component naming

Closes #509

Authors:
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah)
  - Christopher Harris (https://github.com/cwharris)

URL: #511
@cwharris
fix build interface includes (#514)
06d4ea1 
build interface includes was accidentally using
```
.../build/conda/cuda-12.5/release/autogenerated/include
```
instead of 
```
.../build/conda/cuda-12.5/release/cpp/mrc/autogenerated/include
```
this PR fixes that, such when depending on mrc from down-stream libraries, the following error does not occur:
```
CMake Error in python/morpheus_llm/morpheus_llm/_lib/CMakeLists.txt:
  Imported target "mrc::pymrc" includes non-existent path

    "/home/coder/mrc/build/conda/cuda-12.5/release/autogenerated/include"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.
```

Authors:
  - Christopher Harris (https://github.com/cwharris)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #514
@codecov Codecov
Copy link

codecov bot commented Oct 24, 2024

Codecov Report

Attention: Patch coverage is 77.05128% with 179 lines in your changes missing coverage. Please review.

Project coverage is 74.17%. Comparing base (cc89f5f) to head (06d4ea1).
Report is 24 commits behind head on main.

Files with missing lines Patch % Lines
cpp/mrc/include/mrc/segment/object.hpp 80.48% 24 Missing ⚠️
cpp/mrc/include/mrc/node/source_properties.hpp 12.00% 22 Missing ⚠️
python/mrc/core/node.cpp 84.95% 17 Missing ⚠️
cpp/mrc/include/mrc/channel/status.hpp 0.00% 15 Missing ⚠️
cpp/mrc/include/mrc/node/operators/router.hpp 85.71% 14 Missing ⚠️
...pp/mrc/src/internal/segment/builder_definition.cpp 81.03% 11 Missing ⚠️
cpp/mrc/include/mrc/edge/edge_builder.hpp 44.44% 10 Missing ⚠️
cpp/mrc/include/mrc/segment/builder.hpp 38.46% 8 Missing ⚠️
python/mrc/_pymrc/src/segment.cpp 80.00% 8 Missing ⚠️
cpp/mrc/include/mrc/node/operators/zip.hpp 90.14% 7 Missing ⚠️
... and 12 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #507      +/-   ##
==========================================
+ Coverage   74.09%   74.17%   +0.08%     
==========================================
  Files         403      407       +4     
  Lines       14401    15050     +649     
  Branches     1132     1191      +59     
==========================================
+ Hits        10670    11163     +493     
- Misses       3731     3887     +156
Flag Coverage Δ
cpp 70.71% <67.55%> (-0.14%) ⬇️
py 44.23% <73.59%> (+2.41%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
cpp/mrc/include/mrc/core/fiber_pool.hpp 100.00% <100.00%> (ø)
cpp/mrc/include/mrc/core/task_queue.hpp 94.73% <100.00%> (ø)
cpp/mrc/include/mrc/core/utils.hpp 45.16% <ø> (ø)
cpp/mrc/include/mrc/edge/deferred_edge.hpp 100.00% <ø> (ø)
cpp/mrc/include/mrc/edge/edge.hpp 91.56% <100.00%> (+0.54%) ⬆️
cpp/mrc/include/mrc/edge/edge_channel.hpp 85.00% <100.00%> (+0.78%) ⬆️
cpp/mrc/include/mrc/edge/edge_connector.hpp 33.33% <ø> (-4.17%) ⬇️
cpp/mrc/include/mrc/edge/edge_readable.hpp 84.84% <ø> (ø)
cpp/mrc/include/mrc/edge/edge_writable.hpp 90.62% <ø> (ø)
...mrc/experimental/modules/mirror_tap/mirror_tap.hpp 100.00% <100.00%> (ø)
... and 48 more

... and 33 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cc89f5f...06d4ea1. Read the comment docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@dagardner-nv dagardner-nv

Labels
! - Release Related to a Major Release
Projects
Morpheus Boards
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dagardner-nv @cwharris @mdemoret-nv @jjacobelli