-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Improve] Binary transmission statistics #6946
base: dev
Are you sure you want to change the base?
Conversation
…s the count file size as the byte
Do I need to resubmit? |
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest we can introduce new metrics in file connector. Like readFileCount
, readFileBytes
. Not use metrics of zeta engine. You can refer
Line 88 in c140178
this.recordFetchDelay = context.getMetricsContext().counter(CDC_RECORD_FETCH_DELAY); |
package org.apache.seatunnel.api.table.type; | ||
|
||
public class BinaryRowType implements SeaTunnelDataType<BinaryRow> { | ||
|
||
public static final BinaryRowType INSTANCE = new BinaryRowType(); | ||
|
||
@Override | ||
public Class<BinaryRow> getTypeClass() { | ||
return BinaryRow.class; | ||
} | ||
|
||
@Override | ||
public SqlType getSqlType() { | ||
return SqlType.BINARY; | ||
} | ||
|
||
@Override | ||
public boolean equals(Object obj) { | ||
if (obj == this) { | ||
return true; | ||
} | ||
return obj instanceof PrimitiveByteArrayType; | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When we add a new type, we have to be careful because it affects how all Sink/Transform should handle it. But the only purpose of this PR to add this new type is just to make the metrics more accurate, I think we need to discuss it in depth. cc @hailin0 @EricJoy2048
I think use metrics of zeta engine is feasible,Just adding an extra meaning |
Count
counts the number of files (Excluding folders)
Bytes
counts the size of files (Excluding folders)
Example Results
Start Time : 2024-06-05 15:22:21
End Time : 2024-06-05 15:22:24
Total Time(s) : 2
Total Read Count : 4
Total Read Bytes : 3153492
Total Write Count : 4
Total Write Bytes : 3153492
Total Failed Count : 0