Daoud Clarke
|
6ea3a95684
|
Allow batches to fail silently
|
2022-06-30 19:52:58 +01:00 |
|
Daoud Clarke
|
ddc8664c11
|
Queue the right type of batch
|
2022-06-29 22:52:12 +01:00 |
|
Daoud Clarke
|
2b52b50569
|
Queue new batches for indexing
|
2022-06-29 22:49:24 +01:00 |
|
Daoud Clarke
|
b8c495bda8
|
Correctly insert new URLs
|
2022-06-29 22:39:21 +01:00 |
|
Daoud Clarke
|
955d650cf4
|
Prevent deadlock when inserting URLs
|
2022-06-28 22:34:46 +01:00 |
|
Daoud Clarke
|
1457cba2c2
|
Cache batches; start a background process
|
2022-06-27 23:44:25 +01:00 |
|
Daoud Clarke
|
ff2312a5ca
|
Use different scores for same domain links
|
2022-06-27 22:46:06 +01:00 |
|
Daoud Clarke
|
36b168a8f6
|
Fix logic in found URL logic SQL and allow crawling URLs crawled by one user for now
|
2022-06-26 21:23:57 +01:00 |
|
Daoud Clarke
|
5e1ec9ccd5
|
Temporarily disable startup background processes; add root domains; check for empty batches.
|
2022-06-26 21:15:52 +01:00 |
|
Daoud Clarke
|
e27d749e18
|
Investigate duplication of URLs in batches
|
2022-06-26 21:11:51 +01:00 |
|
Daoud Clarke
|
eb571fc5fe
|
Add a script to count urls in the index
|
2022-06-21 21:55:38 +01:00 |
|
Daoud Clarke
|
1d9b5cb3ca
|
Make more robust
|
2022-06-21 08:44:46 +01:00 |
|
Daoud Clarke
|
30e1e19072
|
Update queued pages in the index
|
2022-06-20 23:35:44 +01:00 |
|
Daoud Clarke
|
4330551e0f
|
Tokenize documents and store pages to be added to the index
|
2022-06-20 22:54:35 +01:00 |
|
Daoud Clarke
|
9594915de1
|
WIP: index continuously. Retrieve batches and store in Postgres
|
2022-06-19 23:23:57 +01:00 |
|
Daoud Clarke
|
b8b605daed
|
Factor out connection code
|
2022-06-19 16:52:25 +01:00 |
|
Daoud Clarke
|
c31cea710f
|
CORS is handled by nginx
|
2022-06-19 13:13:36 +01:00 |
|
Daoud Clarke
|
96da534ca5
|
Don't add CORS on the python side
|
2022-06-19 11:34:54 +01:00 |
|
Daoud Clarke
|
9dbb724ba9
|
Use updated CORS settings
|
2022-06-19 11:31:55 +01:00 |
|
Daoud Clarke
|
e3baf87918
|
Remove seemingly extraneous backslashes
|
2022-06-19 11:27:37 +01:00 |
|
Daoud Clarke
|
c245be775b
|
Use an updated template
|
2022-06-19 11:25:38 +01:00 |
|
Daoud Clarke
|
01772517da
|
Remove problematic SSL_DIRECTIVES line
|
2022-06-19 11:23:01 +01:00 |
|
Daoud Clarke
|
a67ca7b298
|
Enable CORS in nginx
|
2022-06-19 11:16:03 +01:00 |
|
Daoud Clarke
|
866c17f2dc
|
Use the dokku app storage
|
2022-06-19 09:53:19 +01:00 |
|
Daoud Clarke
|
16c2692099
|
Start processing historical data on startup
|
2022-06-19 08:56:55 +01:00 |
|
Daoud Clarke
|
d400950689
|
Add script to process historical data
|
2022-06-18 15:31:35 +01:00 |
|
Daoud Clarke
|
eb1c59990c
|
Expose the port
|
2022-06-17 23:57:58 +01:00 |
|
Daoud Clarke
|
d7c6dcb5c2
|
Use the correct port for dokku
|
2022-06-17 23:54:22 +01:00 |
|
Daoud Clarke
|
77088a8a1b
|
Use a database URL env var
|
2022-06-17 23:39:24 +01:00 |
|
Daoud Clarke
|
476481c5f8
|
Put the resources in the package
|
2022-06-17 23:32:43 +01:00 |
|
Daoud Clarke
|
505e7521d4
|
Copy the resources
|
2022-06-17 23:29:04 +01:00 |
|
Daoud Clarke
|
5ea9efcfa2
|
Fix relative path
|
2022-06-17 23:19:30 +01:00 |
|
Daoud Clarke
|
1c7420e5fb
|
Don't depend on existing data
|
2022-06-17 23:12:22 +01:00 |
|
Daoud Clarke
|
a003914e91
|
Fix boto3 dependency
|
2022-06-17 22:14:55 +01:00 |
|
Daoud Clarke
|
363103468e
|
Update Dockerfile for changes
|
2022-06-17 21:26:21 +01:00 |
|
Daoud Clarke
|
e2eb405083
|
Combine crawler and search servers
|
2022-06-16 22:49:41 +01:00 |
|
Daoud Clarke
|
7771657684
|
Merge pull request #53 from mwmbl/record-historical-batches
Record historical batches
|
2022-06-16 22:09:12 +01:00 |
|
Daoud Clarke
|
14107acc75
|
Use new server
|
2022-06-09 22:24:54 +01:00 |
|
Daoud Clarke
|
aaca8b2b6e
|
Record historical batches via the API
|
2022-06-05 09:15:04 +01:00 |
|
Daoud Clarke
|
617666e3b7
|
Merge pull request #51 from mwmbl/learning-to-rank
Learning to rank
|
2022-06-04 12:36:15 +01:00 |
|
Daoud Clarke
|
770b4b945b
|
Refactor feature extraction
|
2022-05-07 22:52:36 +01:00 |
|
Daoud Clarke
|
87d8b40cad
|
Make order_results public
|
2022-05-06 23:15:50 +01:00 |
|
Daoud Clarke
|
229819e57e
|
Refactor to allow LTR ranker
|
2022-03-27 22:32:44 +01:00 |
|
Daoud Clarke
|
94287cec01
|
Get features for each string separately
|
2022-03-21 21:49:10 +00:00 |
|
Daoud Clarke
|
4740d89c6a
|
Add domain score feature
|
2022-03-21 21:13:20 +00:00 |
|
Daoud Clarke
|
af6a28fac3
|
Implement learning to rank feature extraction and thresholding
|
2022-03-20 22:01:45 +00:00 |
|
Daoud Clarke
|
2d334074af
|
Make get_results() public for learning to rank
|
2022-03-20 17:25:54 +00:00 |
|
Daoud Clarke
|
ee5ca6bcf6
|
Experiment with score variations (best is simple weighted domain score)
|
2022-02-27 21:24:16 +00:00 |
|
Daoud Clarke
|
6fb310c363
|
Use addition instead of multiplication
|
2022-02-25 22:19:26 +00:00 |
|
Daoud Clarke
|
4e6516ccf1
|
Scale by 0.99
|
2022-02-25 22:14:49 +00:00 |
|