Commit graph

7 commits

Author SHA1 Message Date
Mikkel Denker
303a2cf2da accept unicode-3.0 license 2024-06-27 17:12:28 +02:00
Mikkel Denker
af73d33b39 forgot to push new accepted licenses 2024-05-23 13:01:34 +02:00
Oliver Bøving
18d9d279fb
Cratify bloom and speedy-kv (#193)
* Move bloom into separate crate

* Move speedy_kv into a separate crate

* add licenses

---------

Co-authored-by: Mikkel Denker <mikkel@stract.com>
2024-04-22 21:18:44 +02:00
Mikkel Denker
3ab4f944e0
MapReduce -> AMPC (#189)
* [WIP] structure for mapreduce -> ampc and introduce tables in dht

* temporarily disable failing lints in ampc/mod.rs

* establish dht connection in ampc

* support batch get/set in dht

* ampc implementation (not tested yet)

* dht upsert

* no more todo's in ampc harmonic centrality impl

* return 'UpsertAction' instead of bool from upserts
this makes it easier to see what action was taken from the callers perspective. a bool is not particularly descriptive

* add ability to have multiple dht tables for each ampc algorithm
gives better type-safety as each table can then have their own key-value type pair

* some bundled bug/correctness fixes.
* await currently scheduled jobs after there are no more jobs to schedule.
* execute each mapper fully at a time before scheduling next mapper.
* compute centrality scores from set cardinalities.

* refactor into smaller functions

* happy path ampc dht test and split ampc into multiple files

* correct harmonic centrality calculation in ampc

* run distributed harmonic centrality worker and coordinator from cli

* stream key/values from dht using range queries in batches

* benchmark distributed centrality calculation

* faster hash in shard selection and drop table in background thread

* Move all rpc communication to bincode2. This should give a significant serilization/deserilization performance boost

* dht store copy-on-write for keys and values to make table clone faster

* fix flaky dht test and improve .set performance using entries

* dynamic batch size based on number of shards in dht cluster
2024-04-15 10:29:33 +02:00
Mikkel Denker
0b69853fa9 chore: 'cargo update' and remove some unused trait method.
also accept gplv3 licenses in libraries as this is permitted under section 13 of gplv3.
2024-02-12 13:49:20 +01:00
Mikkel Denker
e4e3044e47 finally ditch that pesky libtorch dependency! 2024-02-02 13:11:06 +01:00
Mikkel Denker
1a9f381d15
GGML Rust bindings (#122)
* move crates into a 'crates' folder

* added cargo-about to check dependency licenses

* create ggml-sys bindings and build as a static library.
simple addition sanity test passes

* update licenses

* yeet alice

* yeet qa model

* yeet fact model

* [wip] idiomatic rust bindings for ggml

* [ggml] mul, add and sub ops implemented for tensors.
i think it would be easier to try and implement a bert model in order to figure out which ops we should include in the binding. for instance, is view and concat needed?
2024-01-27 12:27:27 +01:00