642a3d57cc
Remove ciborium in favor of custom deserialization logic
...
Unfortunately, looks like serde is not flexible enough to support everything CBOR does,
so a lot of messages cannot be deserialized properly. Other serde-based CBOR libraries
suffer from the same problem.
So now we have a bunch of boring deserialization logic supported by sk-cbor
2023-09-24 20:06:20 +02:00
ffccdc40fe
Update the roadmap a little bit to mention everything that's needed to get to v1
2023-09-23 20:42:35 +02:00
2268f9ca14
Limit language detector to only use cyrillic script
...
The makes the memory consumption like 100Mb, which is much more reasonable than 1Gb that
it was using previously
2023-09-23 20:39:51 +02:00
658996d5d5
Delete posts from the database when they are deleted from bluesky
2023-09-23 20:29:56 +02:00
dd33333649
Rewrite streaming processing in a more sane way
...
And also add support for likes and follows
2023-09-23 20:25:26 +02:00
3a54e04bf4
Upgrade atrium-api dependency
2023-09-22 18:21:08 +02:00
aa17ece012
Fix clippy lints
...
Nothing major here tbh
2023-09-22 17:15:48 +02:00
83bede52ce
Remove dead code
2023-09-22 17:12:49 +02:00
e95c4923d6
Add some untested version of publishing a feed
...
Also adjust names of different env vars, and also adjust setup instructions
2023-09-22 13:33:13 +02:00
5128bf9d4a
Refactor streaming stuff
...
Now we call the processor once per commit, and it's also now
a commit processor, not an operation processor, so that we can
update the cursor properly
2023-09-22 12:37:10 +02:00
08dc55b2cd
Rejiggle things a bit to make it possible to have multiple binaries here for publishing
2023-09-21 15:01:43 +02:00
901c4b6e97
Make Algo.should_index_post a fallible async function, for maximum extensibility
...
We may want to perform some more complicated operations here in the future
2023-09-21 13:31:27 +02:00
c02bded6f8
Formatting
2023-09-21 13:28:22 +02:00
2fd1474647
Don't crash when unable to classify a profile due to some random problem
...
Random problems include: deleted profiles.
Also always wait 10 seconds between runs, we don't need to do it so often
2023-09-21 13:25:36 +02:00
93c4979c71
Keep subscription state in order to not lose messages
...
This isn't a good way to do it though, because opreations processor is only called for each operation,
so we end up not updating the cursor as often as we realistically should be.
I'll refactor this slightly later
2023-09-21 12:33:17 +02:00
62b00ceed7
Replace random print statements with proper logging setup
2023-09-21 11:22:18 +02:00
f4ee482ce7
Use Arcs to pass stuff around to avoid dealing with lifetimes
...
And also implement proper language detection through lingua-rs,
because Bluesky's detection is really bad
2023-09-21 10:36:47 +02:00
9a2a88dc6b
Mark serving feed as done 🎉
2023-09-16 21:08:32 +02:00
14b9f846da
Refactor things
...
Make the overall design a little more flexible if one wants
to easily implement multiple feeds, and also kinda closer to
the original template provided by Bluesky devs
2023-09-16 21:07:00 +02:00
fb17aa3e6a
Formatting
2023-09-16 17:40:07 +02:00
a11c5446d5
Upgrade dependencies
2023-09-16 17:39:09 +02:00
944be0cd5a
Convert into more modern Rust module structure
...
And also use re-exports liberally for shorter imports
2023-09-16 17:21:18 +02:00
227118dfb1
Rename PostSaver into PostIndexer, which is a better name
2023-09-16 17:16:01 +02:00
b4250e12cd
Serve the feed, according to all the Atproto endpoints
2023-09-16 17:13:57 +02:00
c2899951f6
Read chatgpt and postgresql credentials from .env
2023-09-07 19:20:38 +02:00
832bdf6e92
Simplify lifetimes
...
I think I may have overdone it with lifetime-per-dependency, I doubt that'll ever become necessary
2023-09-07 19:06:39 +02:00
9d3cd75db9
Refactor things a little
...
Split into proper domain components, and 3rd party service glue
2023-09-07 19:02:36 +02:00
3180db4779
Upgrade dependencies, part 2
2023-09-07 19:02:12 +02:00
ba3fbf524d
Upgrade dependencies
2023-09-06 08:30:21 +02:00
46974a9d8c
Detect profiles' likely country of residence through chatgpt
...
In a separate thread
2023-09-06 08:26:03 +02:00
262cd707b4
Rejiggle the imports a bit
2023-08-31 15:16:28 +02:00
a18d448c38
Store profiles in the database as we see them
...
No processing yet, but it's coming soon I guess
2023-08-31 15:11:14 +02:00
6365007fd8
Upgrade dependencies
2023-08-31 15:09:42 +02:00
6394321599
Formatting
2023-08-31 09:56:14 +02:00
e68fa95b97
Update the README a little bit
2023-08-31 09:45:48 +02:00
e5b3db1470
Refactor streaming and start inserting Russian posts into db
2023-08-31 09:42:56 +02:00
1b80bf6ab5
Add a little readme explaining what this project does and what the status is
2023-08-26 18:15:18 +02:00
7f403657aa
Don't crash when cid isn't there
...
Although I don't know why that would happen
2023-08-26 18:01:24 +02:00
a7e5384dfb
Refactor things a tiny little bit
2023-08-19 20:20:27 +02:00
13cef8786c
Initial version of things: just reads the stream of messages and prints it out and that's that
2023-08-18 21:11:49 +02:00
Aleksei Voronov