How to override Azure Application Insights - Traceparent HTTP header?

[foliv57]

Expected Behavior

When an incoming HTTP request contains a "traceparent" HTTP header, this header should not be transferred to downstream. Like this, Application Insights will inject a valid traceparent header to link the downstream request to Ocelot by creating a dependency.

Actual Behavior

If the HTTP request contains a "traceperent" HTTP header, this header is transferred in the request done to the downstream endpoint. Application Insights will not create a dependency for the call done to downstream, attaching the targeted endpoint to the wrong parent.

Steps to Reproduce the Problem

1. In Ocelot project, inject Application Insights Telemetry in Startup
   
2. Send an HTTP request containing a "traceparent" HTTP header
3. Check in Application Insights. The 2 requests items, for Upstream and Downstream, will be attached to the same parent
4. Send an HTTP request without "traceparent" HTTP header
5. Check in Application Insights. You will see: Upstream request -> Downstream dependency -> Downstream request, properly linked.

Specifications

• Version:
  • Ocelot: 18.0.0
  • Microsoft.ApplicationInsights.AspNetCore: 2.17.0
• Platform: Ocelot hosted in a Linux P1V2 Azure App Service Plan (Can be also reproduced in debug)

[DamianUrbaszewski]

I could confirm that issue still occurs for version 18.0.0. I my case scenario looks like:

1. When send traceparent header with traceId-parentId defined, then next call has same traceId and same parentId (according to standard should be parentId = spanId from previous call.

2. When there is no header then next call have it genereated correctly but with wrong parentId (somehow genereted but it's not a value from first call spanId)

3. When send traceparent header with null parentId, then next call has the same null parentId (but should be spanId from previous call).

I fix the issue overrideing traceparent header value manualy, but it probably should work automatically ?

[qrdemole]

Could you possibly share your manual solution? I am trying to overcome same issue with no luck.

[voltcode]

can you share the details/code of your exact fix @DamianUrbaszewski ? how did you override traceparent and in which place?

[raman-m]

https://ocelot.readthedocs.io/en/latest/features/headerstransformation.html
Hope it will help you!

[raman-m]

See discussions below. 👇
Seems our current Headers Transformation has a lack of functionality to solve the problem.
One way to resolve the problem is usage of Delegating Handlers feature to control headers life time.

[raman-m]

@DamianUrbaszewski commented on Mar 14:

I fix the issue overrideing traceparent header value manually, but it probably should work automatically?

Hi Demian!
Why do you think so?
Ocelot routes all headers which it caught, otherwise downstream service could fail. Request should have all headers & artifacts of upstream request as direct connection should have.

If you need to hide / introduce headers to allow downstream to work well, you have to use Ocelot Headers Transformation feature.

Finally, I would say,

You have to override headers to recover your downstream service! ⚠️

[raman-m]

@foliv57 commented on Jul 18, 2022

Welcome to Ocelot world! 🐯
This is not an issue of Ocelot, as I understood your user scenario correctly.
You need to control your headers by your own.

I see a lack of usage of Ocelot Headers Transformation feature. Please use this feature to hide /override /control headers.
And let us know your testing results please!

[foliv57]

Hi @raman-m ,

I already try with header transformation without success.

The header must be injected by the package « 
Microsoft.ApplicationInsights.AspNetCore ».

But ocelot overrides it.

If I’m not wrong, header transformation cannot be configures to just ignore or filter out a specific header.

This is what I need to make it work. Having a « Ignore header XX »

By overriding the traceparent or just copying from upstream, it brakes the end to end tracing.

Regards

[raman-m]

Please show us how does Ocelot override your lib Microsoft.ApplicationInsights.AspNetCore functionality?
Maybe you need to use Custom Builder of the Dependency Injection feature?
We've developed this DI feature this summer. So, you could try it...

[foliv57]

Already demonstrated in my original message.

The Microsoft Application Insights package automatically add traceparent header to an HTTP request. But it does it only if the request doesn’t already contains this header. It does it to maintain an end to end tracing chain between the service that requested the current node and the next node (displayed as a dependency in application insights)

But because ocelot moves the traceparent header sent by the previous node, it brakes the chain. The application insight package will not create the proper dependency because the request sent to downstream already contains the traceparent header.

ocelot must not move this header to not prevent the tranceparent injection.

[raman-m]

How did you solve this problem?
With success or failed?

Because Ocelot breaks the package functionality, seems the only one way to fix it is writing custom delegating handler to remove appropriate header(s).

As a team, we can support this package because this is the product from Microsoft, ASP.NET and Visual Studio teams, and seems we have to...
Will you help us with development of this feature?

[foliv57]

For now, not solved. My traces are broken.

Delegating handler is a good idea. It can do the trick. I will give it a try and keep you posted.

Thanks for your support. Wish you a happy Christmas.

[raman-m]

Well... I will help you after holidays and the current release. And, a custom delegating handler should not have complex implemention.

I'm interested in this discussion to convert to real feature because:

• we need more support of Azure environment tools by Ocelot
• we can develop a sample solution to catch the bug first, and redevelop as Azure package support solution
• review current Trace Id feature
• introduce "header removing" feature by route JSON config

Happy Holidays! 🎄 ☃️
And Merry Christmas! 🎅 🎁

[foliv57]

@raman-m I agree with you that this is not an ocelot bug. It’s not up to ocelot to decide which header must be moved or not. It really depends on the situation. Creating an issue was a mistake sorry.

Does it make more sense if I create a feature request for header filtering/ignore ? Or is it already possible to configure ocelot to not move or override a specific header ?

Please note that hiding it by overriding it with blank value is not the expected behavior here. In my case the header must not be added by ocelot at all.

Thank you

[raman-m]

Or is it already possible to configure ocelot to not move or override a specific header ?

Impossible to remove any headers by Route JSON config! There is no such feature. Ocelot is able to add/convert some headers aka Find and Replace, but not to remove.

[raman-m]

Please note that hiding it by overriding it with blank value is not the expected behavior here. In my case the header must not be added by ocelot at all.

Right, you are able to "clean up" the header's value, but not to remove the header.
As a quick fix you should be able to control this header by Delegating Handler feature by analyzing HttpRequestMessage request object: detect the case and the header, remove it from the collection.
Have you tried this approach?

[raman-m]

Creating an issue was a mistake sorry.

Not a problem!
We can consider "header removing" feature from route JSON... But we need to understand how often will developers use the feature?
And how useful is it for the community at all?..
Also we can discuss support of the Microsoft.ApplicationInsights.AspNetCore package too...

[raman-m]

@foliv57
Would you like to continue the discussion here?