Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Statistics Page #38

Closed
wants to merge 234 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
234 commits
Select commit Hold shift + click to select a range
6977e34
remove javascript
Alextopher Nov 25, 2021
cc45a7a
this is now a go project with a todo list
Alextopher Dec 30, 2021
2dfe94b
parse nginx logs
Alextopher Jan 3, 2022
a929eaf
Basic error handling thing, not hooked up yet
Spud304 Jan 3, 2022
29a37b1
config file parsing and validation
Alextopher Jan 3, 2022
d283cbb
fix go mod
Alextopher Jan 3, 2022
1e04141
changes to config system
Alextopher Jan 11, 2022
b61f68b
changes to config system
Alextopher Jan 11, 2022
804e028
Proof of concept submitting data to influxdb
Alextopher Jan 14, 2022
963a730
refactor log parsing channels
Alextopher Jan 14, 2022
b7bd109
Create map.go
Alextopher Jan 14, 2022
2b0960d
added geo2ip db, added country codes to ngnix.go parsing
Spud304 Jan 14, 2022
c72cf5b
Update README.md
Spud304 Jan 14, 2022
3edab8a
Update README.md
Spud304 Jan 14, 2022
8d02b76
Merge branch 'master' of https://github.com/COSI-Lab/Mirror
Alextopher Jan 14, 2022
918d75d
NGINX stats retrives old data before appending new values to the data…
Alextopher Jan 17, 2022
5bca136
update libraries
Alextopher Jan 17, 2022
62f632a
Most of our mirrors config has been writen (thx sophia)
Alextopher Jan 19, 2022
2b1716c
map.go should be close to done
Spud304 Jan 19, 2022
6796510
skip if DB is unavailable
Spud304 Jan 19, 2022
3dd8a58
refactor webserver to use subrouters
Alextopher Jan 21, 2022
426f658
added missing templates and refactored static files folder
Alextopher Jan 21, 2022
1da5a70
basic webhook functionality
Spud304 Jan 21, 2022
60606ef
Added link in footer
wyantda Jan 21, 2022
206e2f7
first round of having errors and logging all in one file
Spud304 Jan 22, 2022
fb9aa83
Added footer and header details, style.css
wyantda Jan 23, 2022
f55bcbf
fixed some mistyped errors, webhook itegration now in a complete enou…
Spud304 Jan 23, 2022
a361318
Initial rsync support
Alextopher Jan 26, 2022
45bcfd8
update readme
Alextopher Jan 26, 2022
2cf6cf8
No longer breaks if hook isnt set
Spud304 Jan 26, 2022
3526499
Update map.go
Spud304 Jan 26, 2022
57e67bb
Basic format put together, style section enhanced. Added favicon and …
wyantda Jan 27, 2022
c799f26
my first comment
PMahar Jan 29, 2022
8154e7a
New Branch: Website
wyantda Jan 29, 2022
2a959fc
my first commit
layan00 Jan 29, 2022
e633170
Change logo, links
PMahar Jan 29, 2022
147480d
Ordered Style.css for communal use
wyantda Jan 29, 2022
dedb8df
Add distrotemplate.gohtml
PMahar Jan 29, 2022
4e6aa56
move queue, update packages and pass project data to webserver
Alextopher Jan 29, 2022
f9c7c80
merge changes
Alextopher Jan 29, 2022
71a0c04
missing discord token does not crash the program
Alextopher Jan 29, 2022
d919395
Add distrotemplate.gohtml
PMahar Jan 29, 2022
f8c9d2d
Rudimentary version of homepage
wyantda Jan 29, 2022
e768d19
add rsync_schedule.go
Alextopher Jan 29, 2022
353f27e
Create codeql-analysis.yml
Alextopher Jan 30, 2022
4b7dd90
Create go.yml
Alextopher Jan 30, 2022
7d3b8cf
Merge 'website' into 'master'
Jan 30, 2022
ea2161e
Create LICENSE
Alextopher Jan 30, 2022
65f0262
fix NPE in circular queue
Alextopher Jan 30, 2022
d690d28
Update issue templates
Alextopher Jan 31, 2022
92e2460
Update download-or-update-issue.md
Alextopher Jan 31, 2022
ac9c2d1
Update download-or-update-issue.md
Alextopher Jan 31, 2022
0055451
Update issue templates
Alextopher Jan 31, 2022
0b9e83a
Merge pull request #3 from COSI-Lab/add-license-1
Alextopher Jan 31, 2022
ab60637
Merge pull request #5 from COSI-Lab/Alextopher-patch-1
Alextopher Jan 31, 2022
79ac169
Merge pull request #6 from COSI-Lab/Alextopher-patch-2
Alextopher Jan 31, 2022
4a5c062
update issue templates
Alextopher Jan 31, 2022
b1e9f5e
final issue template
Alextopher Jan 31, 2022
c79b6ae
update logging
Alextopher Feb 2, 2022
0f599da
update config file
Alextopher Jan 31, 2022
098e686
to pull changes
layan00 Jan 31, 2022
910e249
Uncringe Distributions page
PMahar Jan 31, 2022
72c043a
Add HTTP Mirror links
PMahar Feb 1, 2022
e7f32eb
large refactor of the frontend
Alextopher Feb 1, 2022
cb1067b
Update the frontend todo list
Alextopher Feb 1, 2022
cb417e1
Mirror map generated from project config
Alextopher Feb 2, 2022
51465c1
Add gin-bin to gitignore
Alextopher Feb 2, 2022
a72f616
Add green background
PMahar Feb 2, 2022
f749f09
Finished Draft of index page
wyantda Feb 2, 2022
01701fb
Added content to statistics and navigation to map
wyantda Feb 2, 2022
0614b0f
changed scheduler structures
Feb 6, 2022
07294d9
added initial scheduling function
Feb 6, 2022
f638d65
go fmt
Feb 6, 2022
1939a5a
fixed small mistake
Feb 6, 2022
0c6544e
added simple test and fixed bug
Feb 6, 2022
3854d5f
added style class for history.gohtml
layan00 Feb 7, 2022
12c1dad
Merge branch 'master' of https://github.com/COSI-Lab/Mirror
layan00 Feb 7, 2022
0a377f3
refactor main and nginx log parsing
Alextopher Feb 13, 2022
78d03ae
Merge branch 'master' into scheduler
Alextopher Feb 13, 2022
7555a10
Rsync scheduling now used
Alextopher Feb 13, 2022
9d95503
improve scheduling algorithm, logging, and rsync status api
Alextopher Feb 14, 2022
7d95b3e
update config style and finish rsync proof of concept
Alextopher Feb 18, 2022
484d455
Up to date mirror config
Alextopher Feb 19, 2022
cc452e4
change database bucket and fix some issues from integration testing
Alextopher Feb 19, 2022
17868cc
add a env variable to disable rsync
Alextopher Feb 19, 2022
9163194
Merge pull request #8 from COSI-Lab/scheduler
Alextopher Feb 19, 2022
2526af2
Update errors.go
Spud304 Feb 19, 2022
3d61b27
Update README.md
Spud304 Feb 19, 2022
f9817dc
order projects by human names
Alextopher Feb 19, 2022
b2bf153
add live updating of geoip database
Alextopher Feb 19, 2022
e0b077c
refactor influxdb nginx stats
Alextopher Feb 20, 2022
f91f4c4
fix id ordering bug in map and make it more stable
Alextopher Feb 20, 2022
0b259b8
add a pie chart to starts and a caching layer to the webserver
Alextopher Feb 20, 2022
329c0fe
make caching less aggressive
Alextopher Feb 20, 2022
78d2c3e
new header, distributions work, and group mirror map responses
Alextopher Feb 21, 2022
718aad6
make map look more realistic
Alextopher Feb 21, 2022
9af0edb
send Hook function handles the interface better
Spud304 Feb 21, 2022
38dafc0
Merge branch 'master' of https://github.com/COSI-Lab/Mirror
Spud304 Feb 21, 2022
448ec76
refactor project pages and remove charts
Alextopher Feb 22, 2022
2a6c17c
merge software and disro pages
Alextopher Feb 25, 2022
ede64ee
open maxmind db file with correct flags
Alextopher Mar 11, 2022
22ff106
track total values for nginx stats
Alextopher Mar 11, 2022
2ca19ac
fix bugs in the config file
Alextopher Mar 14, 2022
f358166
centos rsync src is CentOS
Alextopher Mar 14, 2022
d152746
add --no-owner flag to and remove --numeric-ids from all projects. Ad…
Alextopher Mar 15, 2022
df9e4d9
Label centos as being streams 8 and below and change to ipv4
Alextopher Mar 15, 2022
070ad07
tweak eclipse?
Alextopher Mar 15, 2022
8a995ca
sleep for 1 extra second between rsync tasks
Alextopher Mar 15, 2022
d6186c0
fix of errors mapping for webhooks
Spud304 Mar 15, 2022
d8c4aa6
add no-group config to all syncs
Alextopher Mar 15, 2022
9dae229
remove --delete from eclipse multistage sync
Alextopher Mar 16, 2022
0f04d04
Quickly add lacking support for multistage syncs
Alextopher Mar 16, 2022
292ccff
Update to go 1.18, make circular queue data structure generic, add su…
Alextopher Mar 16, 2022
e635d96
Update go.yml
Alextopher Mar 16, 2022
1a9f376
Update mirror-request.md
Alextopher Mar 16, 2022
f1c4c8c
support username parameter for syncs
Alextopher Mar 16, 2022
7df9417
support username parameter for syncs
Alextopher Mar 16, 2022
79ea900
logging can now send files along with it
Spud304 Mar 16, 2022
165abc3
Merge branch 'master' of https://github.com/COSI-Lab/Mirror
Spud304 Mar 16, 2022
c7243e0
rsync scheduler sleep until first rsync jobs
Alextopher Mar 16, 2022
ae0b051
tempfile for logging, warn for startup. goroutines for the logging
Spud304 Mar 16, 2022
6471862
spaces so errors look nicer
Spud304 Mar 16, 2022
8b880b2
fixed use of wrong lib, removed error call
Spud304 Mar 16, 2022
bfbffb2
rysnc sends attachment now
Spud304 Mar 16, 2022
46284c9
removed ioutils lib, handle errors for sendfiles
Spud304 Mar 16, 2022
d1fe1b1
Update errors.go
Spud304 Mar 16, 2022
817c2b5
print errors all on one line as to not race condition
Spud304 Mar 16, 2022
ed9cffa
removed me being dumb function. Hopefully fixed the byte array issue
Spud304 Mar 17, 2022
f1023e7
error with attachment called now
Spud304 Mar 17, 2022
299a954
trim whitespace from password files and send rsync errors as attachments
Alextopher Mar 17, 2022
8cd62db
add centos stream 9
Alextopher Mar 24, 2022
0779e1b
make logging more resilient to network errors
Alextopher Mar 24, 2022
c1cea0c
enough webdev I'm sending it
Alextopher Mar 24, 2022
23cb8cd
move index to /home
Alextopher Mar 24, 2022
da7efc0
update navbar
Alextopher Mar 24, 2022
dca551d
unlock after geoip lookup
Alextopher Mar 24, 2022
dfc2895
skip proxy pass on the map
Alextopher Mar 24, 2022
212424c
fix map uri
Alextopher Mar 24, 2022
9214190
remove deadlinks and order projects correctly
Alextopher Mar 26, 2022
beecbc1
improve the mirror map legend
Alextopher Mar 26, 2022
32e286c
add a panic handler around geoip lookups
Alextopher Mar 30, 2022
607480f
fix the mirror map
Alextopher Apr 1, 2022
6511d5b
Should stop the panic if failed geoip lookup happens
Spud304 Apr 14, 2022
be008ec
add AiryxOS
Alextopher Apr 15, 2022
e283e83
prevent undercounting by scanning the nginx log file to find the firs…
Alextopher Apr 16, 2022
ae49fe3
add an endpoint that allows for manual 'pushed based syncs' at the re…
Alextopher Apr 16, 2022
b8ce0f9
add some error responses to manual sync handler
Alextopher Apr 16, 2022
c40d555
refactor logging again and add some more reporting to discord
Alextopher Apr 16, 2022
5f25d48
improve log messages
Alextopher Apr 16, 2022
c9605ed
Wait for project to be unlocked before running a manual sync
Alextopher Apr 16, 2022
5402663
fix bug in mirror map
Alextopher Apr 16, 2022
83e436f
Update README.md
Alextopher Apr 19, 2022
11397e7
improve projects page
Alextopher Apr 22, 2022
efd92e1
refactor statistics tracking and added rsyncd log file tracking
Alextopher Apr 22, 2022
29e60e5
Merge branch 'master' of github.com:COSI-Lab/Mirror
Alextopher Apr 22, 2022
ae6714d
track rsyncd requests
Alextopher Apr 23, 2022
7e4a1b9
add alternative sources for all projects and fix linuxmint-images
Alextopher Apr 25, 2022
3606e9f
add --no-perms flag to all syncs
Alextopher Apr 26, 2022
f5765e0
AAHHHH I MISSED A SPACE
Alextopher Apr 26, 2022
05c8f20
change ubuntu to RIT mirror
Alextopher Apr 27, 2022
cb2054f
Merge branch 'master' of github.com:COSI-Lab/Mirror
Alextopher Apr 27, 2022
334b76a
add a pull token so we can manually trigger pulls
Alextopher Apr 27, 2022
f35aa79
fix manual sync token handler
Alextopher Apr 27, 2022
786c5ef
fix manual sync token handler??
Alextopher Apr 27, 2022
1211eb4
it's a PULL_TOKEN not a PUSH_TOKEN :facepalm:
Alextopher Apr 27, 2022
cca0b73
revert some of the ubuntu changes
Alextopher Apr 28, 2022
da695b8
change name of AiryxOS to ravynos
Alextopher May 3, 2022
e449b7e
remove leaked secret from README.md and enable public rsync on all pr…
Alextopher May 18, 2022
9c3f645
errors should no longer trigger discord pings
Alextopher May 18, 2022
8c0f072
add wikimedia xml dumps
Alextopher May 19, 2022
7899c0b
cosmetic changes to home page and history
Alextopher May 23, 2022
63df836
Update README.md
Alextopher May 23, 2022
ba772c1
improve site for mobile
Alextopher May 24, 2022
1ac6189
stop sending rsync errors 23 and 24 to discord
Alextopher May 24, 2022
d110f8d
further improve mobile experience
Alextopher May 24, 2022
978d742
give more mobile love to the projects page
Alextopher May 24, 2022
92665c1
remove the table of contents from mobile
Alextopher May 24, 2022
4eb572b
AH
sophiacarlone May 25, 2022
4016f11
tiny change to add a health endpoint
Alextopher May 26, 2022
4ac00c1
update depencies
Alextopher May 26, 2022
36df77c
Add infrastructure to sync projects using shell commands
Alextopher Jul 19, 2022
69c93d4
Major refactor. Move MirrorMap into package main and move Logging, Da…
Alextopher Jul 28, 2022
8b05eb7
change influxdb url
Alextopher Jul 28, 2022
0aeff6f
Tidy and update dependencies
Alextopher Jul 28, 2022
a814c9e
send map messages
Alextopher Jul 28, 2022
f8c5076
proof of concept daily checkup
Alextopher Jul 28, 2022
d6acf4d
switch fedora to enchilada and epel
Alextopher Jul 28, 2022
292000a
remove mediawiki
Alextopher Jul 28, 2022
a59d087
fix the title for daily_sent
Alextopher Jul 28, 2022
2582cff
.
Alextopher Jul 28, 2022
776df4e
https for influxdb with insecure skip verify
Alextopher Jul 30, 2022
ee1ca14
stop creating nginx redirect config
Alextopher Jul 30, 2022
e7677ae
Project now only works on *nix systems and does not run as root
Alextopher Jul 31, 2022
e403d0e
Add today's date to health checkups to stop discord from caching the …
Alextopher Aug 3, 2022
542ffe8
Stop dumping failed command outputs to the terminal
Alextopher Aug 7, 2022
fdc5530
update logging to v1.0.2
Alextopher Aug 13, 2022
d8df44f
chore: increase VERBOSITY in quick fedora mirror
Alextopher Aug 18, 2022
078c22a
remove limit on rsyncd connections
Alextopher Aug 18, 2022
fbde41e
change daily health time from 8am to 7am
Alextopher Aug 19, 2022
c24dab7
fix: typo
Alextopher Aug 21, 2022
b75759d
fix typo
Alextopher Aug 21, 2022
250783b
fix parrot rsync upstream
Alextopher Aug 23, 2022
317f7d5
add archvsync submodule for debian
Alextopher Aug 25, 2022
561fc30
nit: typos
Alextopher Aug 25, 2022
a8c7aca
Debian work
sophiacarlone Aug 25, 2022
ec8cf1e
fix submodule
Alextopher Aug 26, 2022
2ecc11e
change fedora upstream
Alextopher Aug 26, 2022
dc7c9f6
change fedora upstream again
Alextopher Aug 26, 2022
288f175
almalinux mirror is official
Alextopher Aug 31, 2022
6fff228
mark gentoo as pending
Alextopher Sep 1, 2022
c9855ab
Fedora is now pending official
Alextopher Sep 1, 2022
537e4fb
update fedora
Alextopher Sep 10, 2022
325fe78
refactor stats.go into tracking.go. start tracking clarkson usage of …
Alextopher Sep 14, 2022
a70e135
update readme
Alextopher Sep 15, 2022
8a0240b
ADM_GROUP no longer _required_
Alextopher Sep 15, 2022
b4d9c99
debian official
Alextopher Sep 17, 2022
7efe706
centos stream 9 official
Alextopher Sep 17, 2022
87b1a5a
update logging
Alextopher Sep 30, 2022
f7769a7
remove *nix requirement and fmt
Alextopher Oct 1, 2022
06ee9f8
Sticky table of contents (#31)
carykees98 Oct 22, 2022
ad24a50
removed old comment in projects.js
carykees98 Oct 23, 2022
a38cf11
fully functional darkmode and toggle switch (#32)
carykees98 Oct 29, 2022
05d15b0
changed slackware to a distribution from software
carykees98 Oct 29, 2022
42f0081
Project logos (#34)
carykees98 Oct 29, 2022
def2001
checkOldLogs (#33)
paytonshafer Oct 30, 2022
0c7b294
css refactor and mobile nav (#36)
carykees98 Nov 4, 2022
b44463b
rebornos arm testing
Alextopher Dec 1, 2022
810cc11
RebornOS-ARM
Alextopher Dec 1, 2022
c9d47d8
RebornOS-ARM alt
Alextopher Dec 1, 2022
e207a9e
RebornOS-ARM fix dest
Alextopher Dec 1, 2022
8981066
added logo for rebornOS-arm
carykees98 Dec 1, 2022
62671ed
removed unused svg
carykees98 Dec 8, 2022
3935cf2
start of statistics page
carykees98 Dec 8, 2022
8c35b6e
removed unnecessary whitespace
carykees98 Dec 8, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/ISSUE_TEMPLATE/download-or-update-issue.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: Project Download or Update Issue
about: If you're having trouble with some software we're mirroring
title: '[project issue] <name of project>'
labels: project bug
assignees: ''

---

**What project were you trying to use? (e.g. Blender, Arch Linux, Gentoo):**

**What failed? (e.g. ISO download failed, `apt upgrade` failed, project is out of sync, Blender download failed):**

**When did you first experience the issue:**

**Please provide any other information:**
16 changes: 16 additions & 0 deletions .github/ISSUE_TEMPLATE/mirror-request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: Mirror Request
about: Suggest a project we should mirror
title: '[mirror-request] <name of project>'
labels: enhancement
assignees: ''

---

## Include full project name, a short name, and the home page url

## Are you an official maintainer of this project? How would we become an "Official" mirror

## Is there a link to documentation on how to mirror the project

## (Optional) Around how much disk space is required to host the project
70 changes: 70 additions & 0 deletions .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# For most projects, this workflow file will not need changing; you simply need
# to commit it to your repository.
#
# You may wish to alter this file to override the set of languages analyzed,
# or to provide custom queries or build logic.
#
# ******** NOTE ********
# We have attempted to detect the languages in your repository. Please check
# the `language` matrix defined below to confirm you have the correct set of
# supported CodeQL languages.
#
name: "CodeQL"

on:
push:
branches: [ master ]
pull_request:
# The branches below must be a subset of the branches above
branches: [ master ]
schedule:
- cron: '24 16 * * 1'

jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write

strategy:
fail-fast: false
matrix:
language: [ 'go', 'javascript' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ]
# Learn more about CodeQL language support at https://git.io/codeql-language-support

steps:
- name: Checkout repository
uses: actions/checkout@v2

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v1
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
# By default, queries listed here will override any specified in a config file.
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main

# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@v1

# ℹ️ Command-line programs to run using the OS shell.
# 📚 https://git.io/JvXDl

# ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
# and modify them (or add more) to build your code if your project
# uses a compiled language

#- run: |
# make bootstrap
# make release

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v1
25 changes: 25 additions & 0 deletions .github/workflows/go.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
name: Go

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:

build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Set up Go
uses: actions/setup-go@v2
with:
go-version: 1.18

- name: Build
run: go build -v ./...

- name: Test
run: go test -v ./...
11 changes: 9 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
dist/
node_modules/
scripts
.env
last_modified

src/secrets.json
mirror
Mirror
gin-bin

access.log
rsyncd.log
GeoLite2-City.mmdb
9 changes: 9 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
[submodule "scripts/quick-fedora-mirror"]
path = scripts/quick-fedora-mirror
url = https://pagure.io/quick-fedora-mirror.git
[submodule "scripts/archvsync"]
path = scripts/archvsync
url = https://github.com/COSI-Lab/archvsync.git
[submodule "scripts/raspbian-tools"]
path = scripts/raspbian-tools
url = https://github.com/plugwash/raspbian-tools.git
Loading