Tehnical Brainstorming

Today I'll try and brainstorm the technical details.

I did do a little research already here and there while thinking about this idea and have a general sense of what it will take to build this.

The tech

For the tech, I'll bootstrap the api, landing and web app with hatch, which gives me a solid foundation with Go on the backend, React for the web app, and Astro for the landing page and marketing. It will also bootstrap the auth and a postgres database.

What matters the most for this project is the database. I did a lot of research. For the MVP, and much long after, a simple Postgres database will suffice.

I will probably never reach the scale that a single Postgres instance cannot handle. We're talking tens of thousands of events per second until it starts to struggle. Even then, I can introduce batching and various optimizations. I will learn everything there is to learn about Postgres and squeeze out every bit of performance out of it before I seriously consider anything else. If I ever get to this point I will consider the options, but I won't get ahead of myself. The chances are slim to none.

I didn't do these calculations myself, I looked them up. I'll do my own benchmarking if and when the need arises.

Having said that, I did snoop around a bit and toyed with the idea of Kafka on the ingestion engine and Clickhouse for the read DB. A man can dream 🤩

The MVP will have a couple of main features.

Web Analytics

I need a javascript snippet that will capture and send metadata on each site visit. I'll provide the users with a random public key, which they will send with their requests. I can't possibly secure this so we're just hoping nobody bothers to abuse it.

• First visit creates a user id so we can track across sessions.
• One session = one session id.
• One visit = one request. No batching necessary here.

I need an api endpoint that will accept the data and save it into the database.

To display data, I'll have to write a few sql queries that aggregate these, and a few api endpoints to serve them. No caching is necessary for now.

Session Replays

The heavy lifting here will be done by rrweb.

I will batch data and send it to an ingestion endpoint that will save it into the db.

Goals

The goal tracking will be achieved mainly by capturing custom events. I'm not planning to add autocapturing of events in the MVP.

I'll need a crud api for the goals, and a few endpoints that will calculate and serve the stats. The logic is going to be mostly sql, with the api gluing stuff together. No caching at this stage yet.