Commit Graph

8 Commits

Author SHA1 Message Date
Aleksei Voronov 8ddca6303d Bluesky blew through max int number of messages
Well, end of an era I guess. Time to use a bigger (bigint) cursor.

Interestingly enough, some of Rust parts were already using an i64,
but others weren't. Huh.
2024-10-18 20:04:27 +02:00
Aleksei Voronov 149cd44227 Add an index on Profile.has_been_processed
Who made the oldest mistake in database management history?

This guy.
2024-01-15 15:17:13 +01:00
Aleksei Voronov 419f72f3bb Store Bluesky firehose host as part of subscription state
The host has changed recently and the cursors are, apparently, incompatible with each
other, so we need to migrate to the new one, and this seems like the easiest way to
do it:

1. Store the host as part of the subscription state
2. Roll out the version that uses that new column as part of all queries
3. Switch to the new host
4. Roll out the version with the new host
   - The new instance will start processing messages with a 0 cursor and so start anew
   - The old instance will die off
2023-11-05 19:01:20 +01:00
Aleksei Voronov dec35a867c Fix SQL syntax
Good God, Aleksei.
2023-10-05 20:38:22 +02:00
Aleksei Voronov 93c4979c71 Keep subscription state in order to not lose messages
This isn't a good way to do it though, because opreations processor is only called for each operation,
so we end up not updating the cursor as often as we realistically should be.

I'll refactor this slightly later
2023-09-21 12:33:17 +02:00
Aleksei Voronov 46974a9d8c Detect profiles' likely country of residence through chatgpt
In a separate thread
2023-09-06 08:26:03 +02:00
Aleksei Voronov a18d448c38 Store profiles in the database as we see them
No processing yet, but it's coming soon I guess
2023-08-31 15:11:14 +02:00
Aleksei Voronov e5b3db1470 Refactor streaming and start inserting Russian posts into db 2023-08-31 09:42:56 +02:00