feat: allow to override parquet_prefix and include idle costs

[burahimu]

Hello there 👋🏼

This PR introduces several enhancements. The main changes are:

• Allow to override the full parquet prefix. We wanted to add a by hour partitioning
• Retrieve idle costs in order to know what we're wasting (we're not sure if the Glue table can handle idle costs)

Note

This PR is in draft until we confirm the latest point.

[lmello]

thanks for your contribution.
left some comments on it.

[lmello]

@burahimu We just merged a PR that implemented support for azure and removed the config from the code to external files to allow parametrization.

Can you have a look at the new code in v0.2.0-rc3 (which is in the main branch) and rebase your pr.

Thank you.

[burahimu]

@burahimu We just merged a PR that implemented support for azure and removed the config from the code to external files to allow parametrization.

Can you have a look at the new code in v0.2.0-rc3 (which is in the main branch) and rebase your pr.

Thank you.

Hi @lmello , it's rebased 👍🏼