I was using it’s APIs. But new restrictions have effectively been put in place that prevent me from using them for what I need. Similar API calls were being made that were causing DDOS attacks on lemmy.world.
As for running a lemmy instance itself. That’s a thought but I need the data in a different format to do efficient searches. It’s a tricky problem.
Yep that’s the new idea. The sad part is that with this method there’s no way to get historical data. Only new posts. So if a server goes down, gets DDOSd etc… I’ll lose posts forever.
Also building an ActivityPub implementation from scratch isn’t trivial either. So that’ll take some time.
I’ve got a few other ideas I’m playing with as well. Like just assuming that internal post IDs are all sequential and literally fetching them one by one. Or maybe some combination of both?
Thanks for the shout-out.
But FYI I’ve run into some bugs that’s preventing new content from being indexed. So you won’t see anything new (from about a week ago) until I can find a new method to fetch new posts.
Ya if it’s not federated already, I’m thinking the link will just remain as-is? Then comes the question of what happens after federation is complete? Ideally I’m thinking to do these rewrites as the comment containing the link gets federated so it only happens once, but that prevents updates… If I do it during someone looking at the post, then that becomes a lot of network traffic…
Still a lot to plan out initially and then propose to the devs for feedback… So it’ll be awhile still.
I just got a PR merged today that might help with this. I’ll start experimenting with it more over the next week or so.
Basic theory is that I can detect at least lemmy posts in the comment bodies and then rewrite those to your local instance. Primarily question is going to be performance, as remote network calls will be necessary.
Fair enough. Sometimes you can’t help but go down these rabbit holes though.
How do you account for the duplicate Riker in TNG? Who’s the real one and where did the extra matter come from then to assemble William vs Tom?
(It’s been a long time since I’ve seen that episode so I don’t remember if they covered that but on-screen)
A similar question could be raised for the Rascals episode…
Not sure if I entirely understand what you’re asking but here’s my setup that sounds similar-ish that might help.
I’ve got essentially 3 machines
The download machine has a network share to download directly to the NAS in a special /downloads/ folder. Once a download completes Sonarr, etc… move it to it’s correct media folder.
Finally the Jellyfin machine is monitoring the media folders for changes.
I assume you could set up something similar with Plex instead of jellyfin and then store the fully downloaded files on a separate machine with a network drive, so Plex can see it. Essentially the NAS for you would be two machines one (the seedbox) for the partial downloads and a local NAS for the fully downloaded files?
Anyway, not sure if that’s what you’re looking for.
Think of Lemmy as email. Each post or comment is just an email sent to a distribution group (a community). If your email server goes down, all of those users and distribution groups are gone. Now I’ll still have the emails I sent to you in my email box but you won’t be able to see them as your email server is offline. Sure you could create a new account on a new server but you’d have to tell everyone about your new address (federate) but there’s nothing to associate your old user with your new one and there’s no way to backfill data. I could reply-all or forward (comment) on to your new address but there’s still no way to associate those old posts with your new account.
If an instance goes down (permanently), federation of all of the communities hosted by that instance essentially stop. The content that has already been posted remains but anything new added to those communities only remain on your home instance. The only way for federation to resume is for that instance to come back online with the same domain it started with.
Same. Honestly I need to create a community just for this tool IMO. But I don’t have the time to moderate it.
https://www.search-lemmy.com/ ?
I’m open to feedback though if the search results seem out of order etc…
With ActivityPub all of the primary ids contain the domain of the hosting server. So if you lose your domain none of the other instances know that you’re the authority on those communities, posts, comments or users. So essentially federation breaks with all of the old data.
I’m also running Ubuntu as my main machine at home. (I have a Mac and do Android development for my day job).
But at home, I do a lot of website and backend dev.
docker buildxdocker compose up -dUnless you have an account there’s no easy way to get access to the content on the page. Once you have an account there’s technically nothing stopping you from just saving the HTML file to your computer.
Something else you can try though, assuming you don’t have an account, is to just turn off JavaScript. If the site lets you partially load the content and then asks you to create an account to read more, they usually just block the content by having JavaScript add an opaque overlay. With JavaScript disabled, obviously it’s not there to add the overlay and you’re able to keep reading.
It searches the title and body. It also automatically searches for similar words like. Like ‘bike’, ‘biking’, ‘bikes’ (aka stemming). Granted though, I’m still improving the page ranking as time goes on.
Lookup Overseerr. https://overseerr.dev/
There is a public API now. While I won’t support sorting, you can process and do what you will with the results as-is. Currently I only support Posts and Communities for now.
When you search for posts you’re just matching against the title or body. For communities it’s searching the posts within that community.
There’s also more filters now with: instance/community/author/since/until and a safe-search option.
So I’m not sure how close this comes to your idea but I thought I’d share.
Thank for sharing the results.
I can say one of the most requested features I keep getting requests for is a global search as well. I’m just blocked by a bug in Lemmy at the moment.
A tool to see better recommendations on who to follow could be interesting as well. I’ve recently added a new feature that allows for users to search communities (on Lemmy and Kbin) about what posts are actually talking about. I wonder if something similar could be built for users…
But again thanks for sharing.
Correct. As I can only provide links to posts that are on your selected home instance. Eventually I’ll change this but you’ll get a 404 page for links that aren’t on your home instance, but see my P.S. below.
P.s. there have been changes to the Lemmy API that have prevented me from getting updates for about a month now. So most of the results you’re seeing are from old posts only. Until I can rebuild the crawler or find a new API there won’t be any new content.