Skip to content

Commit

Permalink
[apache#1161] improvement(netty): Reduce the data copy when accessing…
Browse files Browse the repository at this point in the history
… data buffer len (apache#1162)

### What changes were proposed in this pull request?

If we use the off heap memory and then we use the method `getData`, we will copy the off heap memory to heap data memory. So we should avoid using it in the Netty mode.

### Why are the changes needed?

Fix: apache#1161

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?
Code Review
  • Loading branch information
jerqi authored Aug 22, 2023
1 parent 76559ce commit f864156
Show file tree
Hide file tree
Showing 2 changed files with 5 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -442,8 +442,9 @@ public void handleGetLocalShuffleData(TransportClient client, GetLocalShuffleDat
length);
long readTime = System.currentTimeMillis() - start;
ShuffleServerMetrics.counterTotalReadTime.inc(readTime);
ShuffleServerMetrics.counterTotalReadDataSize.inc(sdr.getData().length);
ShuffleServerMetrics.counterTotalReadLocalDataFileSize.inc(sdr.getData().length);
int dataLength = sdr.getDataBuffer().remaining();
ShuffleServerMetrics.counterTotalReadDataSize.inc(dataLength);
ShuffleServerMetrics.counterTotalReadLocalDataFileSize.inc(dataLength);
shuffleServer
.getNettyMetrics()
.recordProcessTime(GetLocalShuffleDataRequest.class.getName(), readTime);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,10 @@ public ShuffleDataResult readShuffleData(ShuffleDataSegment shuffleDataSegment)
+ " due to "
+ e.getMessage());
}
if (result.getData().length != expectedLength) {
if (result.getDataBuffer().remaining() != expectedLength) {
throw new RssException(
"Wrong data length expect "
+ result.getData().length
+ result.getDataBuffer().remaining()
+ " but actual is "
+ expectedLength);
}
Expand Down

0 comments on commit f864156

Please sign in to comment.