An insane amount of time was spent decompressing (about 80%) when fetching inbound links.
This made it impractical to index backlink texts, which of course is very usefull information to have in the index.
Each service should handle their required mutability with atomic/locking mechanisms.
Otherwise, there would be no way for the sonic server to accept multiple incoming requests even if the server had available capacity to handle the request.
Now the server spawns a tokio task for each incoming request.
This should allow us to disable lingering for the socket since the client will only close the connection once it has received adequate data to satisfy the user request. No stray tcp packets should therefore hit the socket.
* Begin overview documentation in mdbook format
* Overview of the different docs
* Move overview documentation to mkdocs
* Reduce webgraph segment merges by introducing a webgraph commit mode that commits the live segment directly to the stored segment
* Parallel harmonic centrality calculations
* Even more parallelism in harmonic centrality calculations
* Way faster hyperloglog but also less accurate
* Dynamic exact counting threshold proportional to size of graph
* improve inbound similarity speed and fix hyperloglog out-of-bounds bug
* no need to load all nodes into memory for harmonic centrality
* Use rayon directly in indexer.
Hopefully this fixes the bug where the indexer takes a new job before it has finished the first one. I think what happened was that the indexer thread took a new job when hitting the webgraph executor.
* single threaded webgraph when indexing
* No need for node2id anymore
* Use single thread in tantviy by default.
We introduce a method to optmize the index for search, which currently just sets the tantivy executor to be multithreaded. This should improve the indexing performance.
* Reduce memory arena in tantivy
* try jemalloc
* Revert tantivy memory arena reduction. Caused too many files to be created when indexing warc files
* deunsafication and testing of sonic protocol
Originally sonic used unsafe casts to read and write the Header for
packets. This was "safe" as it was, but relied on the code reading and
writing to make use that it used the correct types on each end of the
protocol.
This introduces bytemuck which does the same thing, but safely. Safety
comes from bytemuck asserting at compile-time that the Header is safe
to cast to bytes, and that there are enough bytes when interpreting
the received bytes.
In addition to this, this commit also introduces proptest testing of the
sonic communication, by generating arbitray messages and sending them to
a server, and asserting that the request and response is always as
expected.
* introduce type safe sonic and sonic service
The core of sonic communication now establishes a protocol that has
the request and response types determined upon creation, instead of
defered to sending and accepting. This upfront declaration makes it
harder to misuse the protocol, and more obvious what messages are
expected at each part of the protocol.
A blocker for implementing this, was in tailored request/response cycles
where a specific response type was expected based on the given request
type. Formerly, this was handeled ad-hoc, where the expected response
type was declared on the connection side to match what type the server
would send in response. This implicit coupling was brittle, and depended
opon the two call sites agreeing on what to send and when.
Based on the prior downsides, sonic::service is introduced to formalize
the request/response cycle. This is done by defining messages, which
implement a trait with an associated response type and a handler. It is
when a service connection then sends a message to a server of the same
service, the connection will statically know what response to expect,
by use of the associated type of the sent message. This provides
connections with a RPC-like interface to the service, while the service
can be expanded with new messages without encuring any additional
overhead to the call-sites of existing connections.
A new service is declared using a macro, that defined appropriate
request and response enums, used during communication. It additionally
generates the code for an ergonomic interface that uses the assiciated
response type to cover over implementation details of tagging messages
while providing a type-safe interface.
Most servers are converted to use the new sonic::service, but some still
remain at the more lowlevel sonic, which is now type-safe, since they
are used in ways incompatible with strict request/response +
predetermined types model; notably MapReduce is generic over the job,
and has non-service-like-control-flow.
* `: Clone` bye bye!
By adding a new `RequestRef<'a>` for the sending part of sonic::service
we get to take references to the send messages, and thus remove the
need for copying each message sent!
This relies on `Request` and `RequestRef` being the same shape when serialized and deserialized, since we serialize the ref version but
deserialzie the non-ref version. Since both are generated by the same
macro, we can have quite high confidence in that they are the same, but
we should perhaps add some tests to ensure that this are the case. I
don't think it would be possible to exhastiavly check, but a proptest
cycling through serialize/deserialize loop could give us some confidence
that it is correct.
* Added a proptest to cycle 'Request' and 'RequestRef'
This commit adds a proptest to the CounterService in order to increase our confidence, that a 'Request' struct can be deserialized from a 'RequestRef' struct. It's important to note while this proptest does not guarantee such the property to hold, it does increase our confidence.
This is a giant re-write to use the spec compliant 'url' crate. The parser we had before did not follow relative urls correctly, which caused many 404's when crawling.
There was probably also a bunch of other issues with the simple parser, so this should make webgraph and everything related to it more robust as well.
