-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ISSUE-21: Index all pages (not just blog pages) #27
Conversation
Hey @etagwerker could you confirm that for fastruby we want to include all these pages from now on?
|
Gemfile.lock
Outdated
@@ -434,6 +434,7 @@ PLATFORMS | |||
aarch64-linux | |||
x86_64-darwin-20 | |||
x86_64-darwin-21 | |||
x86_64-darwin-22 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Automatically added when I bundled
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
["https://fastruby.io/blog",
"https://fastruby.io/blog/10-strategies-for-upgrading-ruby-or-rails-applications-with-low-test-coverage.html",
"https://fastruby.io/blog/bundler/memory-leaks/introducing-bundler-leak.html",
"https://fastruby.io/blog/case-study/doxo-upgrades-ruby-on-rails-increases-application-performance.html",
"https://fastruby.io/blog/case-study/power-hrg-tune-report-increases-performance.html",
"https://fastruby.io/blog/case-study/smile.io-upgrades-ruby-on-rails-increases-speed-agility-and-revenue.html",
"https://fastruby.io/blog/case-study/soundcloud-upgrades-monolithic-ruby-and-rails-system.html",
"https://fastruby.io/blog/code-quality/an-intro-to-software-quality.html",
"https://fastruby.io/blog/code-quality/calculate-tech-debt-using-skunk-on-github-actions.html",
"https://fastruby.io/blog/code-quality/churn-vs-complexity-vs-coverage.html",
"https://fastruby.io/blog/code-quality/code-coverage/rubycritic-4-2-0-simplecov-support.html",
"https://fastruby.io/blog/code-quality/escaping-the-tar-pit-at-rubyconf.html",
"https://fastruby.io/blog/code-quality/intruducing-skunk-stink-score-calculator.html",
"https://fastruby.io/blog/custom-deprecation-behavior.html",
"https://fastruby.io/blog/dealing-with-schema-changes-in-rails-7.html",
"https://fastruby.io/blog/docker/docker-ssh-keys.html",
"https://fastruby.io/blog/esbuild/webpacker/javascript/migrate-from-webpacker-to-esbuild.html",
"https://fastruby.io/blog/exploring-ruby-warnings.html",
"https://fastruby.io/blog/fix-sneaky-argument-error-when-upgrading-ruby.html",
"https://fastruby.io/blog/fortify-rails-security-webinar.html",
"https://fastruby.io/blog/gem/release/automate-the-gem-release-process.html",
"https://fastruby.io/blog/gem/release/how-to-manually-release-a-gem-in-rubygems.html",
"https://fastruby.io/blog/gems/dry/ruby/gemifying-your-styleguide.html",
"https://fastruby.io/blog/heroku/upgrades/Heroku-22-Stack-Upgrade-Guide.html",
"https://fastruby.io/blog/how-to-improve-debugging-skills-in-rails.html",
"https://fastruby.io/blog/how-to-run-multiple-versions-of-rails-on-heroku.html",
"https://fastruby.io/blog/index.html",
"https://fastruby.io/blog/introduction-to-rails-engines.html",
"https://fastruby.io/blog/monthly-maintenance-services.html",
"https://fastruby.io/blog/navigating-asset-management-madness.html",
"https://fastruby.io/blog/open-source/guide-for-unmaintained-open-source-projects.html",
"https://fastruby.io/blog/open-source/introducing-dotenv-validator.html",
"https://fastruby.io/blog/open-source/ombulabs-open-source-guidelines.html",
"https://fastruby.io/blog/performance/optimizing-images-2.html",
"https://fastruby.io/blog/performance/optimizing-images-part-3.html",
"https://fastruby.io/blog/performance/optimizing-images.html",
"https://fastruby.io/blog/performance/rails/writing-fast-rails.html",
"https://fastruby.io/blog/rails/code-quality/An-introduction-to-race-condition.html",
"https://fastruby.io/blog/rails/code-quality/how-we-estimate-rails-application-size.html",
"https://fastruby.io/blog/rails/how-to-extend-rails-associations.html",
"https://fastruby.io/blog/rails/how-to-split-rails-routes.html",
"https://fastruby.io/blog/rails/javascript/code-coverage/js-code-coverage-in-rails.html",
"https://fastruby.io/blog/rails/naming-things-is-hard.html",
"https://fastruby.io/blog/rails/performance/ruby/best-ec2-instance-for-rails-apps.html",
"https://fastruby.io/blog/rails/performance/ruby/hows-the-performance-of-ruby-3.0.0-preview1.html",
"https://fastruby.io/blog/rails/performance/writing-fast-rails-part-2.html",
"https://fastruby.io/blog/rails/performance/writing-fast-rails-part-3.html",
"https://fastruby.io/blog/rails/rspec/testing-ajax-controller-requests.html",
"https://fastruby.io/blog/rails/ruby/performance/how-fast-is-ruby-3-on-rails.html",
"https://fastruby.io/blog/rails/ruby/rubygems/announcing-audit-tool.html",
"https://fastruby.io/blog/rails/security/dangerous-query-method-deprecation.html",
"https://fastruby.io/blog/rails/security/ruby-security-toolkit.html",
"https://fastruby.io/blog/rails/simplecov/upgrades/merging-multiple-simpleCov-coverage-results.html",
"https://fastruby.io/blog/rails/tips-for-upgrading-rails-3-2-to-4.html",
"https://fastruby.io/blog/rails/upgrade/dual-booting-with-conditionals.html",
"https://fastruby.io/blog/rails/upgrade/how-to-pitch-an-upgrade-to-your-boss.html",
"https://fastruby.io/blog/rails/upgrade/our-rails-upgrade-process.html",
"https://fastruby.io/blog/rails/upgrade/prepare-for-rails-upgrade.html",
"https://fastruby.io/blog/rails/upgrade/rails-upgrade-series.html",
"https://fastruby.io/blog/rails/upgrade/testing/how-to-upgrade-rails-without-tests.html",
"https://fastruby.io/blog/rails/upgrade/zeitwerk/upgrading-to-zeitwerk.html",
"https://fastruby.io/blog/rails/upgrades/7-common-mistakes-made-while-doing-rails-upgrades.html",
"https://fastruby.io/blog/rails/upgrades/active-record-5-1-api-changes.html",
"https://fastruby.io/blog/rails/upgrades/assessing-rails-upgrades.html",
"https://fastruby.io/blog/rails/upgrades/case-study/procore-case-study.html",
"https://fastruby.io/blog/rails/upgrades/case-study/upgrading-a-large-rails-application-from-4.2-to-5.2.html",
"https://fastruby.io/blog/rails/upgrades/case-study/upgrading-a-monolith.html",
"https://fastruby.io/blog/rails/upgrades/case-study/upgrading-a-rails-app.html",
"https://fastruby.io/blog/rails/upgrades/deprecation-warnings-rails-guide.html",
"https://fastruby.io/blog/rails/upgrades/how-to-migrate-from-capybara-webkit-to-webdrivers.html",
"https://fastruby.io/blog/rails/upgrades/how-to-stay-up-to-date.html",
"https://fastruby.io/blog/rails/upgrades/our-rails-upgrade-process.html",
"https://fastruby.io/blog/rails/upgrades/rails-merge-deprecation.html",
"https://fastruby.io/blog/rails/upgrades/rails-upgrade-shims.html",
"https://fastruby.io/blog/rails/upgrades/rails-upgrade-strategies.html",
"https://fastruby.io/blog/rails/upgrades/simplecov/rails-upgrades-simplecov.html",
"https://fastruby.io/blog/rails/upgrades/southeast-ruby-upgrade-rails-workshop.html",
"https://fastruby.io/blog/rails/upgrades/strong-parameters-migration-guide.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-6-1-to-7-0.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-3-2-to-4-0.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-4-0-to-4-1.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-4-1-to-4-2.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-4-2-to-5-0.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-5-0-to-5-1.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-5-1-to-5-2.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-5-2-to-6-0.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-rails-from-6-0-to-6-1.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-to-rails-3-1.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-to-rails-3-2.html",
"https://fastruby.io/blog/rails/upgrades/upgrade-to-rails-3.html",
"https://fastruby.io/blog/rails/upgrades/what-i-learned-first-rails-upgrade.html",
"https://fastruby.io/blog/rails/upgrades/why-upgrade-your-rails-application.html",
"https://fastruby.io/blog/rails/webpack/from-sprockets-to-webpacker.html",
"https://fastruby.io/blog/rails/what-does-load-defaults-do.html",
"https://fastruby.io/blog/rails/what-is-code-coverage-ruby-on-rails.html",
"https://fastruby.io/blog/railsconf/upgrade-rails-101-the-roadmap-to-smooth-upgrades.html",
"https://fastruby.io/blog/railsconf-2023-upgrade-workshop.html",
"https://fastruby.io/blog/readonly-associations-upgrading-rails-4.html",
"https://fastruby.io/blog/rspec/debug/how-to-debug-non-deterministic-specs.html",
"https://fastruby.io/blog/rubocop/code-quality/create-a-custom-rubocop-cop.html",
"https://fastruby.io/blog/ruby/code-quality/how-we-use-rubocop-and-standardrb.html",
"https://fastruby.io/blog/ruby/dual-boot/dual-booting-gems.html",
"https://fastruby.io/blog/ruby/dual-boot/generating-gemfile-next-lock.html",
"https://fastruby.io/blog/ruby/performance/how-fast-are-ractors.html",
"https://fastruby.io/blog/ruby/performance/why-wasnt-ruby-3-faster.html",
"https://fastruby.io/blog/ruby/quality/code-quality-ruby-gems.html",
"https://fastruby.io/blog/ruby/rails/versions/compatibility-table.html",
"https://fastruby.io/blog/ruby/upgrades/upgrade-ruby-from-2.7-to-3.0.html",
"https://fastruby.io/blog/ruby/uuid/friendlier-uuid-urls-in-ruby.html",
"https://fastruby.io/blog/ruby-3-i18n-gem-and-beyond.html",
"https://fastruby.io/blog/rubyconf/rubyconf-scholars-and-guides.html",
"https://fastruby.io/blog/testing/javascript/mocking-js-requests.html",
"https://fastruby.io/blog/testing/the-aaa-pattern-writing-robust-tests-for-any-project-with-confidence.html",
"https://fastruby.io/blog/the-assets-pipeline-history.html",
"https://fastruby.io/blog/upgrade-rails/dual-boot/dual-boot-with-rails-6-0-beta.html",
"https://fastruby.io/blog/upgrade-rails/legacy-rails-silently-judging-you.html",
"https://fastruby.io/blog/upgrade-rails/workshop/workshop-rails-upgrade.html",
"https://fastruby.io/blog/upgrades/active-storage/gradually-migrate-from-paperclip.html",
"https://fastruby.io/blog/upgrades/two-different-approaches.html",
"https://fastruby.io/blog/why-you-might-not-want-to-run-rails-app-update.html",
"https://fastruby.io",
"https://fastruby.io/team",
"https://fastruby.io/blog",
"https://fastruby.io/roadmap",
"https://fastruby.io/twelve-factor",
"https://fastruby.io/security-risks",
"https://fastruby.io/static-code-analysis",
"https://fastruby.io/minimum-viable-product",
"https://fastruby.io/monolith",
"https://fastruby.io/newsletter",
"https://fastruby.io/rails-3-0-vulnerabilities",
"https://fastruby.io/rails-3-1-vulnerabilities",
"https://fastruby.io/rails-3-2-vulnerabilities",
"https://fastruby.io/rails-4-0-vulnerabilities",
"https://fastruby.io/rails-4-1-vulnerabilities",
"https://fastruby.io/rails-4-2-vulnerabilities",
"https://fastruby.io/rails-5-0-vulnerabilities",
"https://fastruby.io/rails-5-1-vulnerabilities",
"https://fastruby.io/rails-5-2-vulnerabilities",
"https://fastruby.io/newsletter/70-n-1-problems-railsconf-talks-puma-6-1-and-more",
"https://fastruby.io/newsletter/1-escaping-tar-pit",
"https://fastruby.io/newsletter/2-audit-tool-improvements",
"https://fastruby.io/newsletter/3-security-vulnerability-tools",
"https://fastruby.io/newsletter/41-latest-updates-from-ruby-on-rails-rails-6-0-4-encrypted-attributes-in-rails-7",
"https://fastruby.io/newsletter/50-javascript-test-code-coverage-in-rails-business-logic-in-rails-with-operators-enum-in-rails-explained",
"https://fastruby.io/newsletter/51-ruby-3-1-0-preview-1-under-the-hood-of-the-includes-method-in-rails",
"https://fastruby.io/newsletter/5-importance-of-rails-upgrades",
"https://fastruby.io/newsletter/6-open-source-projects",
"https://fastruby.io/newsletter/9-open-source-guide",
"https://fastruby.io/newsletter/13-rails-upgrade-strategies",
"https://fastruby.io/newsletter/23-ruby-3-changes",
"https://fastruby.io/newsletter/14-rails-performance",
"https://fastruby.io/newsletter/15-rails-upgrade-case-study",
"https://fastruby.io/newsletter/17-rails-assets-precompile-performance",
"https://fastruby.io/newsletter/19-rails-6-1-changes",
"https://fastruby.io/newsletter/21-news-rails-releases",
"https://fastruby.io/newsletter/45-hey-inspired-pop-ups-using-hotwire-mastering-rails-time-operations",
"https://fastruby.io/newsletter/46-rails-7-moves-activerecord-base-logger-to-a-class_attribute-responsible-monkeypatching-in-ruby",
"https://fastruby.io/newsletter/47-vanillanested-gem-friendlier-uuid-urls-in-ruby",
"https://fastruby.io/newsletter/24-rails-6-1-changes",
"https://fastruby.io/newsletter/26-ruby-ractors",
"https://fastruby.io/newsletter/27-rails-6-1-preview",
"https://fastruby.io/newsletter/29-ruby-3-release",
"https://fastruby.io/newsletter/28-ruby-3-0-rc1-released",
"https://fastruby.io/newsletter/48-gradually-migrate-from-paperclip-to-active-storage-how-to-reduce-memory-bloat-and-rails-7-news",
"https://fastruby.io/newsletter/49-upgrading-to-zeitwerk-rails-7-supports-nullsfirst-for-all-databases",
"https://fastruby.io/newsletter/30-how-fast-is-ruby-3",
"https://fastruby.io/newsletter/31-solving-dual-booting-issues",
"https://fastruby.io/newsletter/32-why-wasnt-ruby-3-faster",
"https://fastruby.io/newsletter/34-upgrade-ruby",
"https://fastruby.io/newsletter/37-building-with-hotwire-rails-6-1-new-defaults",
"https://fastruby.io/newsletter/52-ruby-code-quality-with-ernesto-tagwerker-design-patterns-in-ruby",
"https://fastruby.io/newsletter/54-railsconf2022-schedule-generating-the-gemfile-next-lock-custom-changes-in-ruby-and-more",
"https://fastruby.io/newsletter/60-build-a-ruby-compiler-learn-about-ruby-symbols-ruby-ractors-and-more",
"https://fastruby.io/newsletter/61-learn-about-fastruby-io-make-2d-games-with-ruby-jit-compilers-and-more",
"https://fastruby.io/newsletter/57-the-results-are-in-2022-ruby-on-rails",
"https://fastruby.io/newsletter/84-rails-versions-7-0-7-2-6-1-7-6-released-yjit-actioncable-and-optimizing-code",
"https://fastruby.io/newsletter/59-a-tool-to-help-rerun-ci-failures-how-to-track-memory-leaks-and-more",
"https://fastruby.io/newsletter/4-disallowed-deprecations",
"https://fastruby.io/newsletter/7-vulnerability-puma",
"https://fastruby.io/newsletter/8-paperclip-is-back",
"https://fastruby.io/newsletter/10-concurrency-in-ruby",
"https://fastruby.io/newsletter/11-ruby-new-syntax",
"https://fastruby.io/newsletter/12-code-quality-metrics",
"https://fastruby.io/newsletter/16-rails-active-storage-changes",
"https://fastruby.io/newsletter/18-rails-deprecations",
"https://fastruby.io/newsletter/20-ruby-3-0",
"https://fastruby.io/newsletter/22-rails-6-1-deprecations",
"https://fastruby.io/newsletter/25-rails-shims",
"https://fastruby.io/newsletter/33-ruby-code-coverage",
"https://fastruby.io/newsletter/76-save-the-date-new-security-webinar-with-expedited-security-on-june-12th-12pm-est",
"https://fastruby.io/newsletter/38-behind-the-scenes-of-rails-ujs-best-http-clients-for-2021-rails-7-add-enumerable-in_order_of",
"https://fastruby.io/newsletter/39-railsconf-2021-talks-solid-principles-rails-7-and-ruby-3-1-additions",
"https://fastruby.io/newsletter/40-upgrade-rails-from-6-0-to-6-1-upgrading-from-sprockets-to-webpacker",
"https://fastruby.io/newsletter/42-ruby-3-0-2-released-the-includes-method-in-rails",
"https://fastruby.io/newsletter/43-rails-6-1-4-has-been-released-the-i-in-solid",
"https://fastruby.io/newsletter/44-webpack-all-the-assets-using-activerecord-s-update_counters",
"https://fastruby.io/newsletter/53-rails-server-side-analytics-support-for-deferrable-foreign-key-constraints-more",
"https://fastruby.io/newsletter/55-check-vulnerabilities-quickly-building-with-rails-7-and-more",
"https://fastruby.io/newsletter/56-icymi-here-s-what-happened-at-railsconf2022",
"https://fastruby.io/newsletter/58-migrate-from-webpacker-to-esbuild-add-feature-flags-and-more",
"https://fastruby.io/newsletter/62-sidekiq-7-0-beta-ruby-3-2-s-data-class-pitchfork-a-new-rack-http-server-by-shopify-and-more",
"https://fastruby.io/newsletter/63-soundcloud-s-ruby-rails-upgrade-project-puma-6-release-rack-3-0-and-more",
"https://fastruby.io/newsletter/64-rutie-scientist-dns-inquiries-and-more",
"https://fastruby.io/newsletter/65-rails-foundation-twitter-drama-heroku-pricing-and-more",
"https://fastruby.io/newsletter/66-render-deployment-ruby-3-2-0-multi-threaded-web-crawlers-and-more",
"https://fastruby.io/newsletter/67-ruby-3-2-benchmarking-2022-review-upcoming-camps-conferences-and-more",
"https://fastruby.io/newsletter/68-rails-7-1-features-new-rails-versions-better-mailers-and-more",
"https://fastruby.io/newsletter/69-using-openai-with-ruby-the-geocoder-gem-and-a-1-million-user-one-woman-dev-team",
"https://fastruby.io/newsletter/35-test-rails-5-2-5-6-0-3-6-and-6-1-3-1-have-been-released-docker-basics-for-rails-developers",
"https://fastruby.io/newsletter/77-reminder-ruby-on-rails-security-webinar-happens-on-monday-at-12pm-est",
"https://fastruby.io/newsletter/78-in-2-hours-ruby-on-rails-security-webinar-happening-soon",
"https://fastruby.io/newsletter/79-icymi-our-first-webinar-on-ruby-on-rails-security-was-action-packed-with-information-you-might-need",
"https://fastruby.io/newsletter/71-a-railscasts-retrospective-new-rubygems-command-ractors-and-more",
"https://fastruby.io/newsletter/81-work-through-that-technical-debt-sooner-than-later",
"https://fastruby.io/newsletter/82-database-triggers-asset-management-activerecord-and-the-sunken-cost-fallacy",
"https://fastruby.io/newsletter/83-rails-7-0-7-released",
"https://fastruby.io/newsletter/72-new-ruby-versions-metaprogramming-render-and-more",
"https://fastruby.io/newsletter/36-ruby-3-0-1-released-rails-7-introduces-active-record-encryption",
"https://fastruby.io/newsletter/85-ruby-3-3-s-yjit-performance-improvements-fixed-cost-upgrade-services-for-javascript-and-deploying-with-kamal",
"https://fastruby.io/newsletter/73-railsconf-2023-workshops-easy-performance-wins-rack-deflater-and-more",
"https://fastruby.io/newsletter/74-ruby-3-2-s-yjit-rails-7-1-tune-report-a-new-front-end-language",
"https://fastruby.io/newsletter/75-new-fastruby-now-offering-fixed-cost-monthly-maintenance-services",
"https://fastruby.io/monthly-rails-maintenance",
"https://fastruby.io/security-audit"]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fbuys This list looks good to me
app/models/link.rb
Outdated
@@ -8,7 +8,7 @@ class Link < ApplicationRecord | |||
|
|||
class << self | |||
def find_urls(sitemap) | |||
sitemap.to_a.select {|x| x.end_with? ".html" }.reject {|x| x.include?("page") || x.include?("/tags/") || x.include?("author") } | |||
sitemap.to_a.reject {|x| x.include?("page") || x.include?("/tags/") || x.include?("author") } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can add more specific pages to exclude here if needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we need Ernesto's input to confirm. To me, this seems correct so far.
@fbuys The list is accurate but it has an error by including these:
Those are not new pages, they are sections of https://fastruby.io/ |
3dd7a4d
to
3fc04fa
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
cb7aa80
to
bac9419
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fbuys I think there is going to be a problem with this PR because of this: https://github.com/fastruby/librarian/blob/main/app/controllers/links_controller.rb#L11
We are using published_at to list links later in a sorted way.
You didn't do any changes to
Lines 29 to 33 in 43d91e8
def find_published_time(page) | |
element = page.at('meta[property="article:published_time"]') | |
element.nil? ? "" : element[:content] | |
end |
So what's the plan with that?
Have you tested this locally? What happens to the list of links?
We want to see the latest articles up top, so I don't know if this PR will affect that.
bac9419
to
910934f
Compare
Previously we only included links that end with `.html` from the different sitemaps. Now we include all pages. The same conditions used to exclude some specifig pages from our index has been left in place. We also exclude URLs with a hash param. Reject condition: `x.match?(/#.+\z/) || x.include?('page') || x.include?('/tags/') || x.include?('author')` We removed the where condition in the links_controller. Now links without published_at will show. This also means that links without published_at will appear at the top of the links page. See: https://stackoverflow.com/a/44912964 See: #21
910934f
to
0628caf
Compare
Hey @etagwerker that is true, thanks for highlighting it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@fbuys Looks good, thanks! 👍🏼
Previously we only included links that end with
.html
from the different sitemaps. Now we include all pages.The same conditions used to exclude some specifig pages from our index has been left in place.
Code:
.reject {|x| x.include?("page") || x.include?("/tags/") || x.include?("author") }
This comment show all the links that will now be included for fastruby.io for example.
See: #21