How we choose technologies

[andyvan-ph]

Summary

Write a short paragraph on what this article is about. If applicable, what's the opinion or point we want to make in this article?

This has tested decently as an idea, but could do with a bit of validation around what we'd actually say / cover.

Headline options

suggest a few angles

How we choose technologies

What (if any) keywords are we targeting?

list any that apply

choose technology (20/m) 🤷

Will it need custom art?

• Yes
• No

Outline (optional)

draft headings / questions you want to answer

• Why does choosing technology matter?
  • Technology is the foundation. It is your constraints.
  • It is the abstractions you choose to work within.
  • Every technology, as simple as it seems to setup initially, is a long term investment. You need to update it, maintain it, learn about it.
• Start with a problem first
  • Like hiring, you need to have a burning problem for a technology to solve. For example, we want this functionality and building it ourselves would take months.
  • For our infra team, this might be saving money. Temporal came from batch exports.
  • Maybe you are reaching the limits of what your current stack can do.
  • Have other people used this technology to solve the problem you are facing?
  • We are reaching the reasonable limits. Don’t pre-maturely optimize. We use Django. You can rewrite later.
• Understand what you are good at
  • What technologies are you familiar with? Lots of the technologies we adopt came from what we’ve used in the past.
  • Have you used this technology to solve the problem you are currently facing?
  • The technologies you choose effect. For example, although you might like a specific programming language, the talent pool might be small.
  • There is a lot of pitfalls you can avoid if you have already fallen in them once.
• Try before you buy
  • For example, teams often spend weeks or months implementing certain tools
  • Pipeline team's Q3 plan includes: Test Warpstream as PoC and decide whether to do it or not
  • Canary release. Mimic traffic. Build a proof of concept.
  • This helps you get a full picture of what the technology will be like.
• Cost is a lot more than what you pay for something
  • Maintenance, onboarding, etc.
  • A lot of guides about this will say “consider your budget,” but who is going to say no to a cheaper option that accomplishes your problem?
• Consider multiple options
  • There is no one perfect tool
  • Even if you do pick a tool, you might grow out of it and move away from it in the future
  • All technology stacks have tradeoffs
• Help people understand why you are choosing a technology
  • Write an RFC
  • What is the problem you are trying to solve?
  • What are the options you considered?
  • Why this one?
• What’s the support structure for the technology
  • Technologies are supported by organizations. Will the organization be around to support it for the long run? How much so?
  • For example, Warpstream was acquired by Confluence. Clickhouse spun out into its own company. Some open source projects become abandoned.
• What are examples of technologies we have adopted (or not)?
  • One section for technologies we choose, another for ones we didn’t. PostHog’s technology beauty pageant.
  • ClickHouse
    • https://posthog.com/blog/how-we-turned-clickhouse-into-our-eventmansion
  • C++
    • https://www.youtube.com/watch?v=LrmK0vAG1ek
  • Rust
    • Blog: How we captured billions of events in Django (and why we rewrote in Rust) #8118
  • Temporal
    • https://posthog.com/blog/temporal-exports
    • RFC: Temporal at PostHog meta#99
  • Iceberg
    • https://github.com/PostHog/product-internal/pull/648/files
    • https://posthog.slack.com/files/U01403VS4MQ/F07MVEM7SAE/delta_vs_iceberg.md
  • Node
    • https://posthog.com/blog/how-we-built-an-app-server
  • Warpstream/Pulsar
    • https://github.com/PostHog/product-internal/pull/642
    • Alternative S3 backed Kafka
  • Hog
    • https://posthog.com/docs/hog#why-hog
  • EFS
    • https://github.com/PostHog/product-internal/pull/572
  • dlt
    • https://dlthub.com/case-studies/posthog
    • https://github.com/PostHog/product-internal/pull/534
  • SAML
    • https://github.com/PostHog/product-internal/pull/653
• More inspo
  • https://thoughtbot.com/blog/how-we-help-startups-choose-a-tech-stack#moving-quickly-or-moving-safely
  • https://www.vinaysahni.com/how-to-choose-the-right-technology
  • https://emcarrio.medium.com/choosing-technologies-things-to-consider-c9864afebb15
  • https://mcfunley.com/choose-boring-technology
  • https://flowerinthenight.com/blog/2023-04-01-cto-diaries-on-choosing-tech-stack/

[ivanagas]

Added the outline