Response Time - > P90,P95 , Min and Max are not matching

[vasireddi78]

Hi,

Good Day !

When I compared Response Time data in Premethues with jmeter jtl file, P90,P95 , Min and Max are not matching and they are 2 to 3 times higher than the data we have in jtl file.

I am using Summary metric to write Response time data into Premethues and using following quantiles:
0.0,0.5|0.90,0.01|0.95,0.01 | 1.0,0.01

Can you please let know anything I am missing in the configuration / resolution for this issue ?

Thanks in advance,
Ramu

[johrstrom]

I would guess the issue is this: Aggregate Report (that's what you're using to check against?) is using the entire dataset to calculate against. These metrics are only using the last 10 minutes of data to calculate against.

From these docs:
Optionally, after a ; separator the length of the window used to calculate the quantile can be specified.

However, you can change this to say 60 minutes (if you're running a test for an hour) by adding ;60 to the end of each bucket.

0.0,0.5;60|0.90,0.01;60|0.95,0.01;60|1.0,0.01;60

[vasireddi78]

Thanks a lot for you reply.
Though I have added additional quantiles and length of time window for Response Time Summary metric, its taking only default values.

content from Jmeter Script:

0,0.1;15|0.90,0.1;15|0.95,0.1;15|1.0,0.01;15 ResponseTime

Content from scraper:

JM_ResponseTime_Summary{label="/contacts.php",quantile="0.75",} 299.0
JM_ResponseTime_Summary{label="/contacts.php",quantile="0.95",} 508.0
JM_ResponseTime_Summary{label="/contacts.php",quantile="0.99",} 899.0

Jar version I am using for testing is -> jmeter-prometheus-plugin-0.6.0.jar
Can you please look into it ?

Regards,
Ramu

[johrstrom]

Can you turn debug logging on and see if there's any issues it's printing?

Are you saying that if you have a configuration like this it does work? (i.e., without changing the window length)

0.90,0.01|0.95,0.01|0.99,0.01

[vasireddi78]

Hi @johrstrom ,
Thanks a lot for your reply.
(1) I did not observed any errors in debug logs, following INFO is printed in the log at the end of the test.

2023-03-14 23:32:11,977 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_ResponseTime_Summary
2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_RequestCount
2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_PassCount
2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_FailCount
2023-03-14 23:32:11,978 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, local)

(2) config 0.90,0.01|0.95,0.01|0.99,0.01 is working functionally, but value are not matching.

I am using data in Prometheus to prepare a custom summary report once the test is completed and comparing data with Jmeter JTL file for correctness.
all the metrics matching except P90,P95, min and max response times. I am using Summary to measure response time.

Please let me know if you need any additional information.

Regards,
Ramu

[johrstrom]

There could be a bug here. I'll have to check to see if there are tests for the same, but cursorily looking at this logic, it doesn't appear to be correct.

jmeter-prometheus-plugin/src/main/java/com/github/johrstrom/collector/BaseCollectorConfig.java

Lines 423 to 444 in 6109f9f

	public static QuantileDefinition[] parseQuantilesFromString(String fullQuantileString) {
	String quantileOnlyString = fullQuantileString.split(QUANTILE_LENGHT_SEPERATOR_REGEX)[0];
	String[] quantileDefStrings = quantileOnlyString.split(QUANTILE_DEFINITION_SEPERATOR_REGEX);
	List<QuantileDefinition> quantiles = new ArrayList<QuantileDefinition>();
	for(String quantile : quantileDefStrings) {
	try {
	QuantileDefinition q = new QuantileDefinition(quantile.split(QUANTILE_ERROR_SEPERATOR));
	quantiles.add(q);
	}catch(Exception e) {
	log.warn("couldn't parse {} because of error {}:{}. It wont be included in quantiles for the metric",
	quantile, e.getClass().toString(), e.getMessage());
	}
	}
	if(quantiles.isEmpty()) {
	log.warn("Did not parse any quantiles, returning defaults.");
	return DEFAULT_QUANTILES;
	}else {
	return quantiles.toArray(new QuantileDefinition[quantiles.size()]);
	}
	}

[johrstrom]

Try this config. The window length applies to the entire config, so you can only have 1. I've verified the logic and added a test case for the same in #120, so I believe from the test case that already worked on the main line that this should work.

0.90,0.01|0.95,0.01|0.99,0.01;60

[codegirlpower]

I am facing same issue. Above line is not working for me. Any other suggestion to see same values of Aggregate Report on Promethues.

[johrstrom]

🤦‍♂️ I'm just now realizing that time is in seconds, so 60 is 1 minute, less than the default 10 minutes.

A value of 3600 would be for an hour, if you're running an hour long test.

0.90,0.01|0.95,0.01|0.99,0.01;3600

[codegirlpower]

I am using below in quantiles for 11min window. Can you please help me identify what is wrong as it is still going with default 10 min and also its just showing series for 0.9 and 0.95 only

0.75,0.5|0.9,0.01|0.95,0.01|0.99,0.001;660

[johrstrom]

Can you please help me identify what is wrong as it is still going with default 10 min and also its just showing series for 0.9 and 0.95 only

There must be a bug somewhere that I'm not seeing in unit tests. In unit tests, this config should work.

I don't even have jmeter on my machines anymore, so I'll have to set it all up and actually fire it up to see what may be breaking.

[codegirlpower]

Hi @johrstrom , Thanks a lot for your reply. (1) I did not observed any errors in debug logs, following INFO is printed in the log at the end of the test.

2023-03-14 23:32:11,977 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test 2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_ResponseTime_Summary 2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_RequestCount 2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_PassCount 2023-03-14 23:32:11,977 DEBUG c.g.j.c.JMeterCollectorRegistry: unregistering JM_FailCount 2023-03-14 23:32:11,978 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, local)

(2) config 0.90,0.01|0.95,0.01|0.99,0.01 is working functionally, but value are not matching.

I am using data in Prometheus to prepare a custom summary report once the test is completed and comparing data with Jmeter JTL file for correctness. all the metrics matching except P90,P95, min and max response times. I am using Summary to measure response time.

Please let me know if you need any additional information.

Regards, Ramu

@vasireddi78 did you find any solution....i have exact usecase

[vasireddi78]

@codegirlpower,
apologies for delay in response. no resolution yet on this issue.
Did you find any solution ?