Also removes bibliogram from the auto-updater
Bibliogram is discontinued, and many instances are going offline as a
result. This clears out the ones that have already been deactivated, but
the better solution would probably be to stop supporting bibliogram.
The cloudflare filter, when performed as part of the github action
workflow, doesn't seem to work nearly as reliably when run on an actual
machine.
The farside server will instead run the un-cloudflare script whenever it
pulls in new changes to services-full.json, which should be a much more
reliable approach to filtering out cloudflare instances.
If dig returns exit code 9 (no reply from server) when checking an
instance for cloudflare records, it shouldn't fail the CI build but
rather just skip adding the instance to the non-cloudflare services
list.
This should be re-evaluated soon to see if the CI build routinely has
issues with getting a server reply. If so, a different approach needs to
be taken to check if an instance is using cloudflare.
The filtered service list isn't guaranteed to have more than one
instance per service, which fails a test that was designed to ensure
back-to-back requests don't use the same instance. Using the full
services json file more accurately tests the intended functionality for
Farside.
This updates the services json file to exclude all instances that are
detected to be using Cloudflare nameservers.
A separate "services-full.json" file will continue to be tracked in the
repo, which will include the full list of all instances for each
service and can be used with the `FARSIDE_SERVICES_JSON` environment
variable for anyone wanting to access the full instance list for each
service.
See #43
Wikiless updated their instance json with a couple of changes that broke
Farside's auto update workflow:
- The protocol for each instance is now included by default (no need to prepend
"https://")
- The instances are differentiated between regular, onion, and i2p (no need to
check for ".onion" in regular instance URLs)
Added a new seperate service for only redirecting to SearXNG instances.
Note that plain "searx" redirects will use both SearX and SearXNG
instances for those who don't have a preference between the two.
Closes#23
- teddit (https://teddit.net/about)
A free and open source alternative Reddit front-end focused on
privacy. Inspired by the Nitter project.
- Piped (https://github.com/TeamPiped/Piped)
An alternative privacy-friendly YouTube frontend which is efficient
by design.
- SimplyTranslate (https://simple-web.org/projects/simplytranslate.html)
We aim to provide fast and private translations to the user without
wasting much overhead for extensive styling or JavaScript
___
Also adds SimplyTranslate to the github pipeline, since they provide a
list of the service's public instances.
Closes#4
The steps taken to commit the changes to services.json were overly complicated.
This simplifies the steps to just `exit 0` if there are no changes to commit.
* Create nightly update workflow for instances
A nightly GitHub Actions CI workflow has been added to fetch new
instances of supported services within Farside.
Currently only Searx is supported, but obviously others could be added
if there are similarly easy ways to fetch and filter instances
programmatically.
services.json has also been updated with the initial results of the
workflow script.
* Set headers for every HTTPoison request
This serves as a workaround for bot blocking via filtron.
* Expand filtering of searx instances
New filter enforces:
- No Cloudflare
- Good TLS config
- Good HTTP header config
- Vanilla instances or forks
- Instances with 100% search success
Rather than requiring a traditional crontab install, the app now
leverages quantum-core (link below) to schedule the instance update/sync
task every 5 minutes. Some updates as a result:
- The new job is scheduled at runtime in server.ex.
- The update.exs script was refactored to be compiled along with the
rest of the app as instances.ex.
- Scheduler and Server modules were added for creating and executing
the new update task
- All shell scripts were removed, as they are no longer needed
https://github.com/quantum-elixir/quantum-core
Now allows setting FARSIDE_TEST to skip individually fetching each
instance, and instead just adds all of them to redis instantly. This
allows for an easier time in CI builds, for both the sake of speed and
to prevent a scenario where many simultaneous builds have a noticeable
impact on actual instances.
