helpers | ||
search_engines | ||
static | ||
templates | ||
website_blueprints | ||
__init__.py | ||
admin_operations.py | ||
crawler.py | ||
crawler_pid.txt | ||
Endpoints.md | ||
flask_admin_server.py | ||
flask_app_server.py | ||
LICENSE | ||
README.md | ||
requirements.txt | ||
requirements_strict.txt | ||
robots_parser.py | ||
session_cookie.txt | ||
sites_to_crawl.txt | ||
url_analysis.py | ||
wsgi.py |
Infinity Decentralized
Infinity Decentralized is a decentralized search engine and crawler that can get results from its own instance and from any other Infinity instance on the internet.
How To Find Other Infinity Instances
On our GitLab repository at Endpoints.md, we try to keep it as updated as possible with instances that are currently running. If you know of more, please let us know and we'll add them to this list!
Prerequisites
A computer/server that can run Python3, ElasticSearch, and MongoDB. If you want to use Elastic Search and MongoDB from a cloud location, then all that you would need is Python3.
Setup
Install:
Then when you are in this directory, run the following command in your terminal:
pip3 install -r requirements.txt
Usage
The User Interface
To start the user interface, run
python3 -m wsgi
and you should be able to access the interface at http://localhost:5000.
Note: If you are getting package errors when you run this, you may need to run this command first and then try running the program again:
pip3 install -r requirements_strict.txt
The crawler
There are two ways to manage the crawler and how it works.
Option 1
For an easy to use web interface, just run
python3 -m flask_admin_server
and you should be able to access an admin interface at http://localhost:5001 where you can control the crawler and update some of its settings.
Option 2
If you want to control the crawler directly from the command line, then all you will need to do is manually edit the site_to_crawl.txt file with a new url on each line and then run crawler.py:
# To just run it in your terminal
python3 -m crawler
# If you want to run it in the background, use this one below instead
nohup python3 -u -m crawler > /dev/null &
You would only really need to run the "python3 - m crawler" command to make sure that the crawler is actually working. Then, the second one will run in the background without any output. when you want to stop the crawler, you'll need to run the "stop_crawler" function from the admin_operations.py file.
The admin_operations.py file has many other functions that you can use to handle the crawler and index like adding and removing urls, getting the crawler status, and resetting the index.
Deployment
Since this is only a Flask app, you would want to deploy it with a web server like Nginx or Apache. The Flask documentation and tutorials about this can be found here.
If you are using Ubuntu, this Digital Ocean Article has start to finish documentation for deploying a Flask app with Nginx. It also includes (optional) instructions on how to secure your server with HTTPS.
Monetization
Infinity Decentralized has default support for monetizing through Coil with the Web Monetization Standard.
Once you have a web monetization payment pointer ready, the only thing you need to change in this repository is the web_monetization_payment_pointer variable at the top of the results_api.py file and then you'll be all set to receive micropayments!
When your Infinity Decentralized instance is live on the internet, Coil members will be able to stream micropayments to your payment pointer whenever they visit your site.