Memory leak observed. #646
Comments
|
Hello, Are those 2 million counters you actual use case or did you only use such a big number to make the "leak" visible? What you are seeing here is a pretty bad case of memory fragmentation. If you add a It looks like the glibc memory allocator (in its default configuration) is doing a bad job here. If I replace it with tcmalloc I don't experience any growth: A run with There are multiple ways how to deal with the problem. The easiest would be to omit the intermediate representation and directly stream into non-contiguous I/O buffers (or directly into the HTTP response stream). This change would also get rid of the memory spikes during the requests. But this change will take some time to apply. Thanks, |
Modified the sample_server program to add 2M counters.
During addition, the memory seems constant which is ok.
During curl, there is some memory being allocated as Private and not getting cleared. This keeps increasing with each curl and then the container runs out of memory.
SAMPLE SERVER code:
Exposer exposer{"127.0.0.1:8080"}; // create a metrics registry // @note it's the users responsibility to keep the object alive auto registry = std::make_shared<Registry>(); // add a new counter family to the registry (families combine values with the // same name, but distinct label dimensions) // // @note please follow the metric-naming best-practices: // https://prometheus.io/docs/practices/naming/ auto& test_gauge = BuildGauge() .Name("test_gauge") .Help("Test gauge") .Register(*registry); const char *labels[] = { "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten" }; printf("Start adding data\n"); clock_t start_time = clock(); for(int i=0;i<200000;i++) { for(int j=0;j<10;j++) { std::string str = std::to_string(i); auto &tg = test_gauge.Add({{"label1",str}, {"label2",labels[j]}}); tg.Set(i); } } printf("End adding data\n"); clock_t end_time = clock(); double duration = (double)(end_time - start_time) / CLOCKS_PER_SEC; printf("duration: %f\n",duration); // ask the exposer to scrape the registry on incoming HTTP requests exposer.RegisterCollectable(registry);MEMORY LEAK:
1. Memory, After metrics data is added (Run sample_server):
2. Memory, during curl(1st) execution:
3. Memory, after curl(1st) completed:
4. Memory, during curl(2nd) execution:
5. Memory, after curl(2nd) completed:
6. Memory, during curl(3rd) execution:
7. Memory, after curl(3rd) completed:
8. Memory, during curl(4th) execution:
9. Memory, after curl(4th) completed:
The text was updated successfully, but these errors were encountered: