Goaccess log reporting

[dshook]

Frontend side of the goaccess log reporting.

Allows you to

• Turn on/off goaccess reporting
• Configure log rotation and report generation frequency
• Catch up report processing frequency (how often to refresh the report with all the logs since last rotation)
• Log and Report retention period
• View the access log reports for each app and each configured domain

caprover/caprover#2182

First of all, thank you for your contribution! 😄

Please make sure you have read contribution guidelines.

Most important items

• Make sure to communicate your proposed changes ob our Slack channel beforehand.
• Large PRs (50+ lines of code) will get rejected due to increased difficulty for review - unless they have been communicated beforehand with project maintainers.
• Refactoring work will get rejected unless it's been communicated with project's maintainers beforehand. There is a thousand ways to write the same code. Every time a code is changed, there is a potential for a new bug. We don't want to refactor the code just for the sake of refactoring.

These rules are strictly enforced to make sure that we can maintain the project moving forward.

[githubsaturn]

Thanks for the PR! Very exciting! 😍
I am overall very aligned with integration with goaccess. But the proposed UI feels a bit off. Having separate links (assuming for each day) is a bit weird in the UI.

This looks like a series of "snapshot" graphs - which while useful - is not really aligned with everything else in CapRover which is realtime. If I am reading the implementation correctly, even the "current" link is not realtime and it uses a 10 snapshots, right?

Here is what I recommend:

• The current link should just process the logs ideally live and realtime
• If live/realtime not possible, at very least, the snapshot should be taken when the URL is fetched. That way, with every "refresh" of the browser, the report gets updated.

There is a bunch of other suggestions that I have - but first let's get the core right, then we get to the beautification of it :)

[dshook]

Thanks for the PR! Very exciting! 😍 I am overall very aligned with integration with goaccess. But the proposed UI feels a bit off. Having separate links (assuming for each day) is a bit weird in the UI.

This looks like a series of "snapshot" graphs - which while useful - is not really aligned with everything else in CapRover which is realtime. If I am reading the implementation correctly, even the "current" link is not realtime and it uses a 10 snapshots, right?

Here is what I recommend:

* The current link should just process the logs ideally live and realtime

* If live/realtime not possible, at very least, the snapshot should be taken when the URL is fetched. That way, with every "refresh" of the browser, the report gets updated.

There is a bunch of other suggestions that I have - but first let's get the core right, then we get to the beautification of it :)

The snapshot reports will only be generated for whatever the log rotation the users set in the settings (defaulting to once a month) only for the app/domain combo's that get any traffic in that period.

You can set GoAccess to serve a real time report with the --real-time-html flag, but this hosts its own web socket server. I'm not sure how we could dynamically run that for each app/domain combo inside one container that also rotates the logs. We could create a new container for each app/domain but that also seems very wasteful.

So in lieu of that, I went with the simpler solution to just have a catch up log that's updated more frequently (default every 10 minutes) than the monthly rotation. There's only one catch up report per app/domain combo that gets updated each time it runs.

It would be nice to update that only when a request is made for the report, but that is dependent on the app handling the request to then be able to kick off a container to update the report as part of the request lifecycle. For that, let's try to figure out a way in the backend PR.

I do think some more tooltips / messaging would be helpful around the settings to explain some of this but did want to get further in the process before adding those in case anything changes :)

[githubsaturn]

We could create a new container for each app/domain but that also seems very wasteful.

Agreed.

It would be nice to update that only when a request is made for the report, but that is dependent on the app handling the request to then be able to kick off a container to update the report as part of the request lifecycle. For that, let's try to figure out a way in the backend PR.

Yes, ideally on the "GET" path, we can generate the report. If that's too difficult to implement, we can have a "Generate Report" button with a new endpoint that kicks off the "catch-up" process. That way, you can update your report on demand if you need to see the last few minutes when catchupLog hasn't caught up yet.

Some more questions:
Q1- Do the monthly log reports only contain that specific month? Or they run monthly, but they contain everything up to the point they are kicked off?
Q2- Same question, but what about catchup log? Does it only contain last 10min? The comment on the catchupLog.sh file suggests otherwise, but wanted to double check.
Q3- Are graphs on GoAccess interactive or not? From their live demo, they seem static. In that case, I think it'd be great to have a few options for report resolution (last 15min, last 2h, last 12h, last 2d, last 7d, last 30d, last 180d) - would something like that be possible?

[dshook]

We could create a new container for each app/domain but that also seems very wasteful.

Agreed.

It would be nice to update that only when a request is made for the report, but that is dependent on the app handling the request to then be able to kick off a container to update the report as part of the request lifecycle. For that, let's try to figure out a way in the backend PR.

Yes, ideally on the "GET" path, we can generate the report. If that's too difficult to implement, we can have a "Generate Report" button with a new endpoint that kicks off the "catch-up" process. That way, you can update your report on demand if you need to see the last few minutes when catchupLog hasn't caught up yet.

Some more questions: Q1- Do the monthly log reports only contain that specific month? Or they run monthly, but they contain everything up to the point they are kicked off? Q2- Same question, but what about catchup log? Does it only contain last 10min? The comment on the catchupLog.sh file suggests otherwise, but wanted to double check. Q3- Are graphs on GoAccess interactive or not? From their live demo, they seem static. In that case, I think it'd be great to have a few options for report resolution (last 15min, last 2h, last 12h, last 2d, last 7d, last 30d, last 180d) - would something like that be possible?

Yeah GoAccess reports are static once they're generated (it's just a standalone html file) with no ability to change the date range which is why we have to do this extra work to give some options on the granularity.

Q1 - Yes, if set to monthly they'll only contain one months worth of data. If we didn't do the log rotation then it would be cumulative.

Q2 - The catchup log is whatever the whole access log has in it since the last log rotation. So if the user has it set to monthly, it will be logs from the 1st of the month till the current day of the month. However if they've disabled log rotation (with a really long custom cron period) then they'll be the whole cumulative logs.

Q3 - Since they're static this is what the whole setting for "Log Rotation and Report Generation Frequency" is for. We can easily add some more presets there. Of course, having that frequency be short means there will be a bunch of reports generated (especially with a CapRover instance with lots of apps and domains) over a long period of time. So this is where the catch up log comes in.

[dshook]

Ok I've updated it to dynamically build the live report and also pull them through the API. Since the API requires auth of course you can't just go to the page since the auth token will be missing so I'm pulling the reports through the API and then displaying the contents in a modal with an iframe.

Thinking of adding a refresh button in the modal for the live one but wanted to check in here and see what you think

Screen.Recording.2024-11-08.at.9.51.18.AM.mov

[githubsaturn]

iframe solution looks surprisingly great! I'll test out soon.

PS:
I don't think we need this, but CapRover also supports cookie auth (generally for less sensitive actions). Currently only used for netdata:
https://github.com/caprover/caprover/blob/badc8da3bdfe42294381a3f9467c5b5fb7c79d4d/src/app.ts#L126-L138