Instrumenting build retrieval and unpacking times

[vitorguidi]

Motivation

We currently lack metrics for build retrieval and unpacking times. This PR adds that, with granularity by fuzz target and job type.

There are two different implementations for build downloading/unpacking:

• In the Build class, from which RegularBuild, SplitTargetBuild, FuchsiaBuild and SymbolizedBuild inherit the downloading/unpacking behavior
• In the CustomBuild class, which implements its own logic

There are two possible cases for downloading/unpacking: clusterfuzz either downloads the whole build and unpacks it locally, or unpacks it remotely. This is the case for all build types except CustomBuild, which does not perform remote unpacking.

For build retrieval over http, we do not track download time. For all the other cases, it suffices to keep track of start/finish time for download and unpacking.

Part of #4271

[oliverchang]

Rather than propagating this variable everywhere, can we just use the JOB_NAME env var? i.e.

clusterfuzz/src/clusterfuzz/_internal/build_management/build_manager.py

Line 83 in ff71dd2

job_name = environment.get_value('JOB_NAME')

This should be set in the uworkers at this point right @jonathanmetzman ?

[vitorguidi]

'JOB_NAME'

Rather than propagating this variable everywhere, can we just use the JOB_NAME env var? i.e.

clusterfuzz/src/clusterfuzz/_internal/build_management/build_manager.py

Line 83 in ff71dd2

job_name = environment.get_value('JOB_NAME')

This should be set in the uworkers at this point right @jonathanmetzman ?

That simplifies things a lot, thanks for the suggestion!

It seems like it does propagate to uworkers

clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py

Line 141 in a190492

uworker_env['JOB_NAME'] = environment.get_value('JOB_NAME')

[jonathanmetzman]

Rather than propagating this variable everywhere, can we just use the JOB_NAME env var? i.e.

clusterfuzz/src/clusterfuzz/_internal/build_management/build_manager.py

Line 83 in ff71dd2

job_name = environment.get_value('JOB_NAME')

This should be set in the uworkers at this point right @jonathanmetzman ?

Yes!

[jonathanmetzman]

'JOB_NAME'

Rather than propagating this variable everywhere, can we just use the JOB_NAME env var? i.e.

clusterfuzz/src/clusterfuzz/_internal/build_management/build_manager.py

Line 83 in ff71dd2

job_name = environment.get_value('JOB_NAME')

This should be set in the uworkers at this point right @jonathanmetzman ?

That simplifies things a lot, thanks for the suggestion!

It seems like it does propagate to uworkers

clusterfuzz/src/clusterfuzz/_internal/bot/tasks/commands.py

Line 141 in a190492

uworker_env['JOB_NAME'] = environment.get_value('JOB_NAME')

yep env vars definitely propagate to workers.

[jonathanmetzman]

lgtm

[oliverchang]

is this comment correct here? we are setting fuzz_target here.

[oliverchang]

nit: it's more that custom builds don't support fuzz targets today, so a better comment would be:

"Custom builds don't support fuzz targets."

(We could probably fix this if needed -- I think we did support this in the past right @jonathanmetzman ?)

[oliverchang]

Looks good with some nits.