-
-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(drips): download csvs (#58) #60
Changes from all commits
6d4d84e
ee50322
d21d948
9300328
54227c3
cd5a945
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
name: funding-splits-content - Nightly | ||
|
||
on: | ||
schedule: | ||
- cron: 59 23 * * * | ||
|
||
jobs: | ||
build: | ||
|
||
runs-on: ubuntu-latest | ||
|
||
defaults: | ||
run: | ||
working-directory: funding-splits-content | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
|
||
- name: Use Node.js 20.x | ||
uses: actions/setup-node@v4 | ||
with: | ||
node-version: 20.x | ||
cache: 'npm' | ||
cache-dependency-path: funding-splits-content/package-lock.json | ||
|
||
- run: npm ci | ||
- run: npx tsc | ||
|
||
- run: node download.js | ||
- run: node dripify.js | ||
|
||
- name: Git Config | ||
run: | | ||
git config user.name 'Nya Ξlimu' | ||
git config user.email '[email protected]' | ||
|
||
- name: Git Commit | ||
run: | | ||
git add *.csv | ||
git commit -m 'chore(funding-splits-content): nightly build' --allow-empty | ||
|
||
- name: Git Push | ||
run: | | ||
git push |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
name: funding-splits-content | ||
|
||
on: | ||
push: | ||
branches: [ "main" ] | ||
pull_request: | ||
branches: [ "main" ] | ||
|
||
jobs: | ||
build: | ||
|
||
runs-on: ubuntu-latest | ||
|
||
strategy: | ||
matrix: | ||
node-version: [18.x, 20.x, 22.x] | ||
# See supported Node.js release schedule at https://nodejs.org/en/about/releases/ | ||
|
||
defaults: | ||
run: | ||
working-directory: funding-splits-content | ||
|
||
steps: | ||
- uses: actions/checkout@v4 | ||
- name: Use Node.js ${{ matrix.node-version }} | ||
uses: actions/setup-node@v4 | ||
with: | ||
node-version: ${{ matrix.node-version }} | ||
cache: 'npm' | ||
cache-dependency-path: funding-splits-content/package-lock.json | ||
- run: npm ci | ||
- run: npx tsc |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
node_modules | ||
*.js |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# funding-splits-content | ||
|
||
> Drip List funding splits for Content Creation 🎶🎙️ | ||
|
||
## Install Dependencies | ||
|
||
```bash | ||
npm install | ||
``` | ||
|
||
## Run the TypeScript Compiler | ||
|
||
```bash | ||
npx tsc | ||
``` | ||
|
||
## Run the JavaScripts | ||
|
||
```bash | ||
node download.js | ||
node dripify.js | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
import fs from 'node:fs' | ||
import http from 'http' | ||
|
||
const languages = ['ENG', 'HIN', 'TGL'] | ||
console.log('languages:', languages) | ||
for (const language of languages) { | ||
console.log() | ||
console.log('language:', language) | ||
|
||
const downloadUrl = `http://${language.toLowerCase()}.elimu.ai/contributor/list/contributors.csv` | ||
console.log('downloadUrl:', downloadUrl) | ||
|
||
const languageDir = `lang-${language}` | ||
console.log('languageDir:', languageDir) | ||
if (!fs.existsSync(languageDir)) { | ||
fs.mkdirSync(languageDir) | ||
} | ||
const filePath = `${languageDir}/contributors.csv` | ||
console.log('filePath:', filePath) | ||
|
||
const file = fs.createWriteStream(filePath) | ||
http.get(downloadUrl, (response) => { | ||
response.pipe(file) | ||
file.on('finish', () => { | ||
file.close(() => { | ||
console.log('File downloaded successfully:', file.path) | ||
}) | ||
}) | ||
}) | ||
Comment on lines
+21
to
+29
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Improve error handling and consider security implications. While the download logic is functional, it lacks error handling for potential issues during the HTTP request or file writing process. Additionally, there are security considerations to keep in mind when downloading files from external sources. Consider implementing the following improvements:
http.get(downloadUrl, (response) => {
if (response.statusCode !== 200) {
console.error(`Failed to download file: ${response.statusCode} ${response.statusMessage}`);
file.close();
return;
}
// ... rest of the code
}).on('error', (err) => {
console.error('Error during download:', err.message);
file.close();
});
const request = http.get(downloadUrl, { timeout: 10000 }, (response) => {
// ... existing code
});
request.on('timeout', () => {
console.error('Request timed out');
request.abort();
file.close();
});
let downloadedBytes = 0;
const MAX_FILE_SIZE = 10 * 1024 * 1024; // 10 MB limit
response.on('data', (chunk) => {
downloadedBytes += chunk.length;
if (downloadedBytes > MAX_FILE_SIZE) {
console.error('File too large, aborting download');
response.destroy();
file.close();
}
}); These changes will make the download process more robust and secure. |
||
} | ||
Comment on lines
+1
to
+30
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion Consider restructuring for better modularity and error handling. While the current implementation is functional, there are several areas where it could be improved for better modularity, error handling, and potentially performance. Consider the following suggestions:
async function downloadContributors(language: string): Promise<void> {
// ... existing download logic
}
async function main() {
for (const language of languages) {
try {
await downloadContributors(language);
} catch (error) {
console.error(`Error processing ${language}:`, error);
}
}
}
main().catch(console.error);
async function main() {
const downloadPromises = languages.map(downloadContributors);
await Promise.all(downloadPromises);
}
import { promisify } from 'util';
const writeFile = promisify(fs.writeFile);
const mkdir = promisify(fs.mkdir);
async function downloadContributors(language: string): Promise<void> {
// ... construct URL and file path
try {
await mkdir(languageDir, { recursive: true });
const response = await fetch(downloadUrl);
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
const buffer = await response.arrayBuffer();
await writeFile(filePath, Buffer.from(buffer));
console.log(`File downloaded successfully for ${language}`);
} catch (error) {
console.error(`Error downloading file for ${language}:`, error);
throw error; // re-throw to be caught in main
}
} These changes will make the code more robust, easier to maintain, and potentially more efficient. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,47 @@ | ||
import fs from 'node:fs' | ||
import { parse } from 'csv-parse' | ||
import { stringify } from 'csv-stringify' | ||
|
||
const languages = ['ENG', 'HIN', 'TGL'] | ||
console.log('languages:', languages) | ||
for (const language of languages) { | ||
console.log() | ||
console.log('language:', language) | ||
|
||
const languageDir = `lang-${language}` | ||
console.log('languageDir:', languageDir) | ||
const filePath = `${languageDir}/contributors.csv` | ||
console.log('filePath:', filePath) | ||
const outputPath = `${languageDir}/FUNDING_SPLITS.csv` | ||
console.log('outputPath:', outputPath) | ||
Comment on lines
+7
to
+16
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add error handling for file system operations. The current implementation doesn't handle potential file system errors, such as missing directories or permission issues. Consider wrapping the file system operations in try-catch blocks to gracefully handle errors. Here's an example of how you could add error handling: try {
const languageDir = `lang-${language}`
console.log('languageDir:', languageDir)
const filePath = `${languageDir}/contributors.csv`
console.log('filePath:', filePath)
const outputPath = `${languageDir}/FUNDING_SPLITS.csv`
console.log('outputPath:', outputPath)
// Check if directory exists
if (!fs.existsSync(languageDir)) {
throw new Error(`Directory ${languageDir} does not exist`)
}
} catch (error) {
console.error(`Error processing ${language}:`, error.message)
continue // Skip to the next language
} |
||
|
||
const columns = [ 'id', 'ethereum_address', 'impact_percentage' ] | ||
const stringifier = stringify({ header: true, columns: [ 'ethereum_address', 'impact_percentage' ] }) | ||
const writeStream = fs.createWriteStream(outputPath) | ||
let impactPercentageZeroAddress = 0 | ||
fs.createReadStream(filePath) | ||
.pipe(parse({ from_line: 2 })) | ||
.on('data', (row) => { | ||
console.log('row:', row) | ||
const ethereumAddress = row[columns.indexOf('ethereum_address')] | ||
const impactPercentage = Number(row[columns.indexOf('impact_percentage')]) | ||
if (impactPercentage > 0) { | ||
if (ethereumAddress == '0x0000000000000000000000000000000000000000') { | ||
impactPercentageZeroAddress += impactPercentage | ||
return | ||
} | ||
const row_dripified = [ ethereumAddress, impactPercentage ] | ||
console.log('row_dripified:', row_dripified) | ||
stringifier.write(row_dripified) | ||
} | ||
}) | ||
.on('end', () => { | ||
console.log('end:', filePath) | ||
if (impactPercentageZeroAddress > 0) { | ||
const row_dripified = [ '0x0000000000000000000000000000000000000000', impactPercentageZeroAddress ] | ||
stringifier.write(row_dripified) | ||
} | ||
stringifier.pipe(writeStream) | ||
console.log('Finished writing data:', filePath) | ||
}) | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
ethereum_address,impact_percentage | ||
0x9f954bcd8bc1b114dfa0296a68a4744603c8e649,0.48426150121065376 | ||
0x6951e11782fb77296d3666ae5a6e1b240f7c0244,9.927360774818402 | ||
0x5bb16ef826311a832c610bbed241c0e1f1784abd,0.7263922518159807 | ||
0x8edb50ab74b9ac3f83ba4c1ceb265f43ae50f9f9,0.24213075060532688 | ||
0x88aef1cebf7b6b2abdb16f883f159edfd11166b5,31.234866828087167 | ||
0x0000000000000000000000000000000000000000,57.38498789346247 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,153 @@ | ||
contributor_id,ethereum_address,impact_percentage | ||
4,0x0000000000000000000000000000000000000000,0.0 | ||
5,0x0000000000000000000000000000000000000000,0.0 | ||
11,0x0000000000000000000000000000000000000000,0.0 | ||
52,0x0000000000000000000000000000000000000000,0.0 | ||
76,0x0000000000000000000000000000000000000000,0.0 | ||
78,0x0000000000000000000000000000000000000000,0.0 | ||
80,0x0000000000000000000000000000000000000000,0.0 | ||
113,0x0000000000000000000000000000000000000000,0.0 | ||
180,0x0000000000000000000000000000000000000000,0.0 | ||
124,0x0000000000000000000000000000000000000000,0.0 | ||
152,0x0000000000000000000000000000000000000000,0.0 | ||
169,0x0000000000000000000000000000000000000000,56.416464891041166 | ||
173,0x0000000000000000000000000000000000000000,0.0 | ||
174,0x0000000000000000000000000000000000000000,0.0 | ||
175,0x0000000000000000000000000000000000000000,0.0 | ||
176,0x0000000000000000000000000000000000000000,0.0 | ||
177,0x0000000000000000000000000000000000000000,0.0 | ||
178,0x0000000000000000000000000000000000000000,0.0 | ||
179,0x0000000000000000000000000000000000000000,0.0 | ||
181,0x0000000000000000000000000000000000000000,0.0 | ||
182,0x0000000000000000000000000000000000000000,0.0 | ||
183,0x0000000000000000000000000000000000000000,0.0 | ||
184,0x0000000000000000000000000000000000000000,0.0 | ||
185,0x0000000000000000000000000000000000000000,0.0 | ||
186,0x0000000000000000000000000000000000000000,0.0 | ||
187,0x0000000000000000000000000000000000000000,0.0 | ||
188,0x0000000000000000000000000000000000000000,0.0 | ||
189,0x0000000000000000000000000000000000000000,0.0 | ||
190,0x0000000000000000000000000000000000000000,0.0 | ||
191,0x0000000000000000000000000000000000000000,0.0 | ||
192,0x0000000000000000000000000000000000000000,0.0 | ||
193,0x0000000000000000000000000000000000000000,0.0 | ||
194,0x0000000000000000000000000000000000000000,0.0 | ||
195,0x0000000000000000000000000000000000000000,0.0 | ||
196,0x0000000000000000000000000000000000000000,0.0 | ||
197,0x0000000000000000000000000000000000000000,0.0 | ||
198,0x0000000000000000000000000000000000000000,0.0 | ||
199,0x0000000000000000000000000000000000000000,0.0 | ||
200,0x0000000000000000000000000000000000000000,0.0 | ||
201,0x4709ebf314c6492d57f4c6d4f57357d5b2bb074e,0.0 | ||
202,0x0000000000000000000000000000000000000000,0.0 | ||
203,0x361dd9c707325d40abfb3ec72518585fd7b76199,0.0 | ||
204,0x7eaf6289281d72f87389bf4615e52429adf849ab,0.0 | ||
205,0x1e8ee48d0621289297693fc98914da2efdce1477,0.0 | ||
206,0x824e0a1e0bee6fe60f591d9a95767411dd60edf3,0.0 | ||
207,0x9f954bcd8bc1b114dfa0296a68a4744603c8e649,0.48426150121065376 | ||
208,0x0000000000000000000000000000000000000000,0.0 | ||
209,0x0000000000000000000000000000000000000000,0.0 | ||
210,0x08379a244ab375c32033578b78a1512fb25a5062,0.0 | ||
211,0xda0bc9ecd14d0346ee41085edbc46a950bebd73d,0.0 | ||
212,0x0000000000000000000000000000000000000000,0.0 | ||
213,0xe0945b7d8f1c5061857201191de442e8124b56c2,0.0 | ||
214,0x6951e11782fb77296d3666ae5a6e1b240f7c0244,9.927360774818402 | ||
215,0x0000000000000000000000000000000000000000,0.24213075060532688 | ||
216,0x0000000000000000000000000000000000000000,0.0 | ||
217,0xe7576177bcfaa67bf06c75f5572fd95e62899bac,0.0 | ||
218,0xb77c19804f102312c8cf66221da8886c1bf2c431,0.0 | ||
219,0x91a394c9e0e997aa6123b8bb76a84edc7694f430,0.0 | ||
220,0x0000000000000000000000000000000000000000,0.0 | ||
221,0x34df2dfb3f98f1d78152fec765cadd77fd84f2ba,0.0 | ||
222,0xf56e5050ceaecc97572be376696cb548f697a6c8,0.0 | ||
223,0x0466468f6223e5c5f2d02533f686dc1c5f936455,0.0 | ||
224,0x44302299d9d5e2245122a698df6a2090a37773f1,0.0 | ||
225,0x6afa2cdea65bd55a47cbb459aa69aa244ca3cdc8,0.0 | ||
226,0xc835bd6d29642e687a9ac024788ff94419aa57c9,0.0 | ||
227,0x0000000000000000000000000000000000000000,0.24213075060532688 | ||
228,0x0000000000000000000000000000000000000000,0.0 | ||
229,0x0000000000000000000000000000000000000000,0.0 | ||
230,0x2088bc94eee73faf12352fc2fd50f508253c8e36,0.0 | ||
231,0x0000000000000000000000000000000000000000,0.0 | ||
232,0x0000000000000000000000000000000000000000,0.0 | ||
233,0x0000000000000000000000000000000000000000,0.0 | ||
234,0xdb133b63829c927591d0847f7cde371b1d51772c,0.0 | ||
235,0x120e47109817c8d5c521cb5088566ccd94277c74,0.0 | ||
236,0x4757ea76c9e7356ab460b4373b4448e6bfb97aaf,0.0 | ||
237,0x0000000000000000000000000000000000000000,0.0 | ||
238,0x0000000000000000000000000000000000000000,0.0 | ||
239,0x0000000000000000000000000000000000000000,0.0 | ||
240,0x0000000000000000000000000000000000000000,0.0 | ||
241,0x467082758ce92b21efe75332f66d92c0e7c54d65,0.0 | ||
242,0xc2b6490597e4ceb788444fed1dadc0e10939607a,0.0 | ||
243,0xd766d89d3492df23f2b5a6df1720184c725d9f48,0.0 | ||
244,0x0000000000000000000000000000000000000000,0.0 | ||
245,0x0000000000000000000000000000000000000000,0.0 | ||
246,0x0000000000000000000000000000000000000000,0.0 | ||
247,0x5bb16ef826311a832c610bbed241c0e1f1784abd,0.7263922518159807 | ||
248,0x0000000000000000000000000000000000000000,0.0 | ||
249,0x572356ffe42f4f3c237a6b1a3e1dec04baef8b20,0.0 | ||
250,0x0000000000000000000000000000000000000000,0.0 | ||
251,0x0000000000000000000000000000000000000000,0.0 | ||
252,0x0000000000000000000000000000000000000000,0.0 | ||
253,0x0000000000000000000000000000000000000000,0.0 | ||
254,0x670e29b181e32c03b12da0c644e78a11216827a2,0.0 | ||
255,0x0000000000000000000000000000000000000000,0.0 | ||
256,0x0b187278bfc6c9af997d7a8b4ac0f81b1ed4c9a9,0.0 | ||
257,0x0000000000000000000000000000000000000000,0.0 | ||
258,0x083672b8bfead44ea00b7d3d1b9100ce0ffbcc50,0.0 | ||
259,0x6321286f9b73f427c72e1f9f1bc6b3d25ef06605,0.0 | ||
260,0x12348bbd6d384574e9522d3c5ca8555038699339,0.0 | ||
261,0x0000000000000000000000000000000000000000,0.0 | ||
262,0x0000000000000000000000000000000000000000,0.0 | ||
263,0xe646f6b98439ddadee6ffb6849c35ba3076c4574,0.0 | ||
264,0xf080d5b40c370a5148a9848a869eb3aaf7d5e146,0.0 | ||
265,0x341a56496b1f1c2c12d71e01cf587755433ef1fc,0.0 | ||
266,0x3fe9e6b5349cf92fd9dbb5c8aea9e154d66778f5,0.0 | ||
267,0xd116039d282019b9a8a71ff117306d24f3f30548,0.0 | ||
268,0x784629ec6e41688f7ff1590a7c16386197c5a111,0.0 | ||
269,0x267de1c4d3017a047c5a5c19aaca46a08d6374c7,0.0 | ||
270,0x27017cf5fc54a41de8dc76e16efc5aee4ab3a1ba,0.0 | ||
271,0x0000000000000000000000000000000000000000,0.0 | ||
272,0xed5a2a1476f5d43b7bfeebc5c160c38934de1846,0.0 | ||
273,0x010106a0a7f9087408412f37c15e769352eea280,0.0 | ||
274,0x0000000000000000000000000000000000000000,0.0 | ||
275,0x0000000000000000000000000000000000000000,0.0 | ||
276,0x0000000000000000000000000000000000000000,0.0 | ||
277,0x0000000000000000000000000000000000000000,0.0 | ||
278,0x0000000000000000000000000000000000000000,0.0 | ||
279,0x0000000000000000000000000000000000000000,0.0 | ||
280,0x0000000000000000000000000000000000000000,0.0 | ||
281,0x0000000000000000000000000000000000000000,0.0 | ||
282,0x0000000000000000000000000000000000000000,0.0 | ||
283,0x0000000000000000000000000000000000000000,0.0 | ||
284,0x0000000000000000000000000000000000000000,0.0 | ||
285,0x0000000000000000000000000000000000000000,0.0 | ||
286,0x0000000000000000000000000000000000000000,0.0 | ||
287,0x0000000000000000000000000000000000000000,0.0 | ||
288,0x0000000000000000000000000000000000000000,0.0 | ||
289,0x0000000000000000000000000000000000000000,0.0 | ||
290,0x0000000000000000000000000000000000000000,0.0 | ||
291,0x0000000000000000000000000000000000000000,0.0 | ||
292,0x0000000000000000000000000000000000000000,0.0 | ||
293,0x0000000000000000000000000000000000000000,0.0 | ||
294,0x0000000000000000000000000000000000000000,0.0 | ||
295,0x0000000000000000000000000000000000000000,0.0 | ||
296,0x0000000000000000000000000000000000000000,0.0 | ||
297,0x0000000000000000000000000000000000000000,0.0 | ||
298,0x93e67f0e310055501daf14db3526d83bd6c6c70a,0.0 | ||
299,0x0000000000000000000000000000000000000000,0.0 | ||
300,0x0000000000000000000000000000000000000000,0.0 | ||
301,0x0000000000000000000000000000000000000000,0.0 | ||
302,0x0000000000000000000000000000000000000000,0.0 | ||
303,0x0000000000000000000000000000000000000000,0.0 | ||
304,0x0000000000000000000000000000000000000000,0.0 | ||
305,0xcec0e2214d43f8245c1302904c02d666fc65e6bc,0.0 | ||
306,0x0000000000000000000000000000000000000000,0.0 | ||
307,0x0000000000000000000000000000000000000000,0.48426150121065376 | ||
308,0x0000000000000000000000000000000000000000,0.0 | ||
309,0x0000000000000000000000000000000000000000,0.0 | ||
310,0x8edb50ab74b9ac3f83ba4c1ceb265f43ae50f9f9,0.24213075060532688 | ||
311,0x88aef1cebf7b6b2abdb16f883f159edfd11166b5,31.234866828087167 | ||
312,0x2d7bc3cd7a1f4b804b8c0c7a1bea44de4f6505f5,0.0 | ||
313,0x08f9e71d658696e4ecefe8b95f82a09933a05e98,0.0 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
ethereum_address,impact_percentage | ||
0x754a9162678eaae9b8949eebd85ed1936b5abce3,4.299889746416759 | ||
0xec35e628dbf08ae096addb187a485e01d39e7560,0.2205071664829107 | ||
0xda0bc9ecd14d0346ee41085edbc46a950bebd73d,2.6460859977949283 | ||
0x37fcb9a179247bcea473ac27404a6e50e22ebed9,2.9768467475192946 | ||
0x95ba792b6eeb140625625083fa045e766c78173b,6.284454244762955 | ||
0x9f954bcd8bc1b114dfa0296a68a4744603c8e649,0.9922822491730982 | ||
0x8f866f04efdf1f1427625bffb0fa8677b08ee04d,20.61742006615215 | ||
0xd9b290431c259d0cb3e3ea55c477ea3d62f8e6fd,0.2205071664829107 | ||
0x7cba4b76fe2d68816bef9ba64f1f2d10907834ca,0.11025358324145534 | ||
0x74f8423e4935c2d988be41306d874f8bd5080403,0.2205071664829107 | ||
0x3ca0882a262cf7d5b38b31697371b7f4480a9385,0.2205071664829107 | ||
0xb0dea5cef77956cf3cb6d3bbdb6af244b357d9f9,0.2205071664829107 | ||
0xfcf315d7ca229733acbcafcad2b7c96cd5c51da7,0.7717750826901875 | ||
0x0000000000000000000000000000000000000000,60.19845644983462 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🛠️ Refactor suggestion
Consider making the language list configurable.
The current implementation hard-codes the language list. To improve flexibility and maintainability, consider moving this list to a configuration file or environment variable.
You could implement this change as follows:
This approach allows for easy modification of the language list without changing the code.