You can't schedule refresh for this dataset because the following data sources currently don't support refresh

[J4YDH-73]

Hi Team,

We have published the CCO Power BI Governance Dashboard v2.2 report into a workspace for sharing. Unfortunately, it is now apparent that the connection refresh cannot be scheduled, due to the custom connector. Can you confirm the best approach to resolving this issue. Gateway or other?

Please note that the Power BI application licence has different credentials to the elevated privileges required to connect to the data.

[email protected] (Power BI Lic) + [email protected] (Connect to data)

Please let me know next steps to enable scheduled data refresh

Thank you and Best Regards
Jay

[J4YDH-73]

Answer: #151

Recommended learning paths for this requirement: https://docs.microsoft.com/en-gb/learn/certifications/exams/sc-900?tab=tab-learning-paths (Particularly: https://docs.microsoft.com/en-gb/learn/paths/describe-concepts-of-security-compliance-identity)

[thefunny1983]

Never mind, I read #151

[J4YDH]

Ok, so if you look at the Power BI appsource you will find the same reports which have the key requirement for explicit authorisation from your Administrator. The ccodashboard custom connection appeared to give access to all API data including the recently deprecated Defender URLs. - The tutorials supplied highlighted Microsofts requirement for security. Otherwise why spend years deploying expensive, configuration intensive infra when you can circumnavigate it all and drain all details without administrator consent? - It is vital that everyone involved follows the latest knowledge and training on docs.microsoft.com and preferably build internal secure data pipelines rather than using a quick fix that may be old or worse, exposing sensitive data. - Good Luck

servicetrust.microsoft.com would be a better place to start if you are going to review your security posture with your head of security.

[b-b-work]

We are having the same issue, but it is not regarding access rights.
"some datasources may not be listed because of hand-authored queries"
Is there any way to overcome this?

[annapurnaK2]

I you open Data source settings in Power BI Desktop, and you get warning (some data sources may not be listed because of hand-authored queries), it is due to the Custom Functions added to the Power BI file CCOInfradashboard v9.1

[J4YDH]

Yes, this is the case. If i think back the user privileges / RBAC permissions have to be carefully considered. An image is also embedded in the report that could send network details to an unknown developer/location. I believe that Microsoft closed off some deprecated APIs at the time but my main concern was that the report I evaluate (introduced by another party) was using a deprecated method of user credential and access. I refer to the main PowerBI appsource location for approved dashboards. This also may have been updated and changed.

[annapurnaK2]

What I like about the CCO dashboard solution is that you don't need to push data to Log Analytics Workspaces, which causes a latency (1-7 days). You get data straight via the API. The API's in CCO dashboard, seem correct. If I point Power Bi to the API's, I get the data, only catch is it does not give me all the data (presents the NextLink), where CCO dashboard resolves NextLink issue. So an awesome way to get data from Defender/Security/Advisor tables of Azure. Only issue I find is the refresh. Is there not a way to use functions to pull the data into Azure SQL database instead of Power BI using this solution. Then I can use Azure SQL database as the data source for power BI. Else, is there not a way to get around the NextLink issue when pulling data from API via Power BI.

[J4YDH]

What I didn't like was the ability to access security and infrastructure data through the APIs when my user account was blocked to view this very data via portals within the organisation and for valid security reasons. The potential for the report images to act as beacons was another concern and the credentials appeared to be founded on a plain text approach with all security alerts suppressed by the user during initial setup. - If you are working with sensitive organisational details I would recommend that the reports are fully reviewed first. I am aware that Microsoft closed off some of the old APIs for this reason, however, you should not be able to circumvent integration or policies and access information that your user account wouldn't ordinarily allow using a custom connector that doesn't fully explain its approach or answer security concerns. I believe an approved version exists with limitations following the recommended access best practices ...however, you will not be able to retrieve all details without the correct permissions.

[J4YDH]

Hi, Check Microsoft AppSource – destination for business apps<https://appsource.microsoft.com/en-gb/?product=power-bi> for the latest approved templates. From what I remember the ccodashboard used a deprecated approach for global access which highlighted security concerns. Old APIs such as Defender URLs have been deprecated and removed by Microsoft, but I would still audit access and do full due diligence before running your information through this approach. You can find more information using docs.microsoft.com (learn paths) I used the Monitor Logs (KQL) and Graph explorer to retrieve only the information explicitly required. However, the report and dashboard had to be rebuilt as you can imagine. Good Luck Regards Jay

…

________________________________ From: b-b-work ***@***.***> Sent: 10 February 2022 07:20 To: Azure/ccodashboard ***@***.***> Cc: J4YDH ***@***.***>; Comment ***@***.***> Subject: Re: [Azure/ccodashboard] You can't schedule refresh for this dataset because the following data sources currently don't support refresh (Discussion #150) We are having the same issue, but it is not regarding access rights. "some datasources may not be listed because of hand-authored queries" Is there any way to overcome this? — Reply to this email directly, view it on GitHub<#150 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AOKHHRLRQQMWDHW2OE2NVD3U2NRLRANCNFSM4YCWRAUA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[annapurnaK2]

Hi Jason David Hill, you mentioned:
I used the Monitor Logs (KQL) and Graph explorer to retrieve only the information explicitly required.

If you run Graph Explorer (using KQL queries), how do you get that info into Power BI report ?

[J4YDH]

Hi annapurnaK2

Reference Monitor Logs using KQL, Queries can be exported to M and used in Power BI directly. For some datasets including datasets derived from Azure Resource Graph you will have to create a data integration to bring it all together.

I hope this helps

Regards
Jay

[annapurnaK2]

With CCO I do get accurate data for Azure Advisor page, like the graphs, etc.
Current Score and Score History Visuals match Score by Category and Score History of Advisor Score (preview) page of Azure.
The count advisor recommendations by category graph was not giving correct data, changed relationship and get value from problem column, now gives correct value which matches bottom part (recommendations) of Advisor Score (preview) page.
Only issue I have is that I can't refresh from Power BI service (dynamic data source), need to refresh from Power BI Desktop.

[annapurnaK2]

If you run KQL queries on Log Analytics Workspaces, yes, then KQL queries can be exported to M. But if you run KQL queries on Azure Resource Graph, its not available.
If I use continuous export of subscriptions to Log Analytics Workspaces, some subscription values on Log Analytics Workspaces match those of Azure Resource Graph (Defender), a few don't match.
Also for some the data in Log Analytics Workspaces matches Azure Resource Graph Explorer if using 7 days as date period for KQL query, for other you need to use 14 days for KQL query. That's due to using snapshot (week delay).
So getting data from Log Analytics Workspaces for Defender data is not a good idea if you want accurate data.
Is there another way to get Defender data (Azure Resource Graph Explorer) ?

[J4YDH]

Regarding Refresh ...This was found to be a limitation and led to the findings that the approach was deprecated / no longer recommended by Microsoft and could lead to potential security concerns.

[J4YDH]

refresh cadence for accurate data. Yes indeed. You are correct. In my particular scenario I used the reporting refresh after receipt of incoming discovery information and logs. This was done on a Sunday, ready for Monday kick off as a snapshot (not great for social life) :)