* refactor harmonic centrality into separate centrality module
* betweenness centrality
* betweenness speedup by not using hashmaps
* [WIP] trust centrality
* more robust warc download?
* unit test for trust centrality calculation
* refactor centrality store and also save trust centrality in store
* approx harmonic centrality working
* Order trusted nodes by betweenness, if the user specified too many, merge the worse trusted nodes into some of the better and update their weight
* dislike sites
* re-enable harmonic centrality calculation
* calculate betweenness on full graph
* added nodeid to schema
* use personal centrality during search
* sort centrality values in csv files
* liked and disliked sites in goggles syntax
* Move settings into a subfolder
This ensures that the paths are layed out the same on the frontend and
on the backend.
* Format goggles and sites .astro
* Add Alpine.js and use `ServeDir` instead of `Spa`
The `ServeDir` allows us the use the public dir and remove the extra
/assets/ also solving the issue with loading scripts without inline.
Currently only a few computes are ported to use Alpine.js, but the rest
should be doable!
* gui
* forgot to implement bang in search_server
* added opensearch.xml file
* reduce indexer memory by commiting more often
* fix query parser bug with special characters
* change autosuggest browser url
* remove '#' from url during indexing
* pre-hashed domain field
* we will also have a subscription model without ads
* site query should not match partial domains from tokenization
* ttl cache
* send site rankings from js to backend during search
* apply site rankings during search
* sort postings by a pre-computed field
* Limit the number of webpages considered for ranking during search.
At the moment we rank top 10 million pages. This can be done since we have sorted the postinglists based on
all pre-computable signals. The chance of "the best" webpage being at > 10 mil is extremely low, and this will allow us to speedup search significantly.
* Use faker data in askama templates during dev
By adding a `$ {{lorem.limes}}` for example to askama`...` expressions,
fake data is inserted during development.
Additionally all askama specific text (such as {% for x in xs %}) is
not produced during development for cleaner pages.
* Add astro-icons/heroicons and cleanup some styling (#45)
* Add astro-icons/heroicons and cleanup some styling
* Make the header a component
By adding a `$ {{lorem.limes}}` for example to askama`...` expressions,
fake data is inserted during development.
Additionally all askama specific text (such as {% for x in xs %}) is
not produced during development for cleaner pages.
This commit removes the Montserrat font stored in public and replaces
them with the fonts installed with @fontsource/montserrat.
This also streamlines the font import process, and ensures that correct
typeface formats are loaded.
One thing to consider, is using variable fonts instead. I failed to get
loading with this setup however, but it seems to be supported.
* goggles parser
* support weird urls in goggle
* quite significant speedboost during indexing (approx x1.7)
* merge index explicitly in indexer
* query benchmark
* search performance improvements primarily by not using hash during search
* turn goggle into tantivy query
* goggle benchmark
* Goggles are working!!
* fixed bug where goggle would enforce that sitename must be A and B and C etc. instead of A or B or C
* document some more goggle syntax
* less bold search highlights
* Build frontend using Astro
This commit replaces the prior pure Askama templates, into templates
statically generated by Astro.
The purpose of this is to enable features such as MD/MDX,
minimization, better Tailwind integration, and JSX'ish component syntax.
In the future a frontend framework, like React, could also be added, while
still compiling the frontend templates statically.
* Delete README.md
* Make the searchbar more round
* Run `npm install` in build.rs
* Fix some image paths
* Make search bar suggestion visibility CSS only
The suggestions are only shown when the input has focus and is not empty
* Add prettier astro and tailwind plugin
* Remove meta astro tag
* Move privacy.astro to privacy-and-happy-lawyers.astro
* Convert privacy and about page to MDX
* Remove old about page
Co-authored-by: Mikkel Denker <Mikkeldenker@gmail.com>
