beenull/mwmbl
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity
mwmbl/pyproject.toml

79 lines
1.8 KiB
TOML
Raw Permalink Normal View History

Add common crawl extract script and dependency management with poetry
2021-12-05 20:31:49 +00:00
[tool.poetry]
renamed package to mwmbl - renamed package to mwmbl in pyproject.toml - tinysearchengine and indexer modules have been moved into mwmbl package folder - analyse module has been left as is in the root of the repo - import statements in tinysearchengine now use mwmbl.tinysearchengine - import statements in indexer now use mwmbl.indexer or mwmbl.tinysearchengine or relative imports like .paths - import statements in analyse now use mwmbl.indexer or mwmbl.tinysearchengine - final CMD in Dockerfile now uses updated path mwmbl.tinysearchengine.app - fixed a couple of import statement errors in tinysearchengine/indexer.py
2021-12-28 11:02:48 +00:00
name = "mwmbl"
Add common crawl extract script and dependency management with poetry
2021-12-05 20:31:49 +00:00
version = "0.1.0"
description = ""
authors = ["Daoud Clarke <daoud.clarke@gmail.com>"]
[tool.poetry.dependencies]
Implement learning to rank feature extraction and thresholding
2022-03-20 22:01:45 +00:00
python = ">=3.10,<3.11"
pandas = "^1.3.5"
scipy = "^1.8.0"
scikit-learn = "^1.0.2"
Remove debug print code
2021-12-26 08:47:33 +00:00
zstandard = "^0.16.0"
mmh3 = "^3.0.0"
fastapi = "^0.70.1"
uvicorn = "^0.16.0"
added config and binary/entrypoint for mwmbl.tinysearchengine - using pydantic to validate the config - added a default bootstrap config at config/tinysearchengine.yaml - refactored app.py to include parsing CLI argument using argparse - refactored app.py to use fewer global variables - added "mwmbl-tinysearchengine" binary/entrypoint in pyproject.toml - updated Dockerfile to work with these changes and added comments to it
2021-12-29 14:18:02 +00:00
pyyaml = "==6.0"
Combine crawler and search servers
2022-06-16 21:49:41 +00:00
boto3 = "^1.20.37"
Fix boto3 dependency
2022-06-17 21:14:55 +00:00
requests = "^2.27.1"
psycopg2-binary = "^2.9.3"
Add script to process historical data
2022-06-18 14:31:35 +00:00
spacy = "==3.2.1"
Add pytest, unit tests for completer,gh actions ci
2023-02-23 05:37:10 +00:00
pytest = "^7.2.1"
pytest-mock = "^3.10.0"
Keep track of curated couments
2023-04-30 17:25:48 +00:00
jusText = "==3.0.0"
Implement learning to rank feature extraction and thresholding
2022-03-20 22:01:45 +00:00
Fixes #12, Added dependencies for indexer as extra or extra_requires - dependencies for indexer can be installed using "pip install .[indexer]" or "poetry install -E indexer"
2021-12-27 14:46:24 +00:00
# Optional dependencies do not get installed by default. Look under tool.poetry.extras section
# to see which extras to use.
ujson = {version= "==4.3.0", optional = true}
warcio = {version= "==1.7.4", optional = true}
idna = {version= "==3.3", optional = true}
beautifulsoup4 = {version= "==4.10.0", optional = true}
lxml = {version= "==4.6.4", optional = true}
langdetect = {version= "==1.0.9", optional = true}
pyarrow = {version= "==6.0.0", optional = true}
pyspark = {version= "==3.2.0", optional = true}
Levenshtein = {version= "==0.16.0", optional = true}
Add django
2023-08-27 06:37:15 +00:00
django = "^4.2.4"
Add django ninja
2023-09-22 18:56:42 +00:00
django-ninja = "^0.22.2"
Exclude blacklisted domains
2023-09-22 20:53:53 +00:00
requests-cache = "^1.1.0"
Store stats in redis
2023-09-28 16:48:29 +00:00
redis = {extras = ["hiredis"], version = "^5.0.1"}
Add login using allauth
2023-10-24 09:32:06 +00:00
django-allauth = "^0.57.0"
Add database config
2023-10-26 16:32:46 +00:00
dj-database-url = "^2.1.0"
Update query string
2023-11-05 21:45:13 +00:00
django-htmx = "^1.17.0"
Fix broken JS
2023-11-07 18:59:38 +00:00
django-vite = "^2.1.3"
Fixes #12, Added dependencies for indexer as extra or extra_requires - dependencies for indexer can be installed using "pip install .[indexer]" or "poetry install -E indexer"
2021-12-27 14:46:24 +00:00
[tool.poetry.extras]
indexer = [
"ujson",
"warcio",
"idna",
"beautifulsoup4",
"lxml",
"langdetect",
"pyarrow",
"pyspark",
"Levenshtein",
]
Remove debug print code
2021-12-26 08:47:33 +00:00
[tool.poetry.dev-dependencies]
WIP: implement docker image. TODO: copy index and set the correct index path using env var
2021-12-22 23:21:23 +00:00
# botocore = "^1.23.20"
# boto3 = "^1.20.20"
# ujson = "^4.3.0"
# warcio = "^1.7.4"
# idna = "^3.3"
# beautifulsoup4 = "^4.10.0"
# lxml = "^4.6.4"
# jusText = "^3.0.0"
# pyspark = "^3.2.0"
# langdetect = "^1.0.9"
# spacy = "^3.2.1"
# Levenshtein = "^0.16.0"
WIP: include metadata in index - using struct approach
2022-02-18 22:12:22 +00:00
pytest = "^7.0.1"
Add common crawl extract script and dependency management with poetry
2021-12-05 20:31:49 +00:00
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
added config and binary/entrypoint for mwmbl.tinysearchengine - using pydantic to validate the config - added a default bootstrap config at config/tinysearchengine.yaml - refactored app.py to include parsing CLI argument using argparse - refactored app.py to use fewer global variables - added "mwmbl-tinysearchengine" binary/entrypoint in pyproject.toml - updated Dockerfile to work with these changes and added comments to it
2021-12-29 14:18:02 +00:00
[tool.poetry.scripts]
Update Dockerfile for changes
2022-06-17 20:26:21 +00:00
mwmbl-tinysearchengine = "mwmbl.main:run"
Reference in a new issue Copy permalink