Compare commits

...

4 Commits

Author SHA1 Message Date
Anonymous
37866ae727 docs
Some checks are pending
Deploy Jekyll with GitHub Pages dependencies preinstalled / build (push) Waiting to run
Deploy Jekyll with GitHub Pages dependencies preinstalled / deploy (push) Blocked by required conditions
2024-07-06 12:09:00 +08:00
Anonymous
862699a524 feat: end to end test, self-sign ssl 2024-07-06 11:56:51 +08:00
Anonymous
098b06a58b feat: go proxy 2024-07-06 11:30:15 +08:00
Anonymous
c9ebbac425 feat: direct_proxy with retry 2024-07-06 11:29:24 +08:00
26 changed files with 359 additions and 100 deletions

View File

@ -7,6 +7,15 @@ BASE_DOMAIN=local.homeinfra.org
# for aria2 and mirrors # for aria2 and mirrors
RPC_SECRET=changeit RPC_SECRET=changeit
# for all, if you want set proxy for all services # in case any other service need the uppercase or lowercase one
# aria2 need the lowercase one, FYI https://aria2.github.io/manual/en/html/aria2c.html#environment # aria2 need the lowercase one, FYI https://aria2.github.io/manual/en/html/aria2c.html#environment
no_proxy=lightmirrors,aria2
NO_PROXY=lightmirrors,aria2
# all_proxy= # all_proxy=
# ALL_PROXY=
# http_proxy=
# HTTP_PROXY=
# https_proxy=
# HTTPS_PROXY=

View File

@ -12,7 +12,16 @@
LightMirrors是一个开源的缓存镜像站服务用于加速软件包下载和镜像拉取。 LightMirrors是一个开源的缓存镜像站服务用于加速软件包下载和镜像拉取。
目前支持**DockerHub**、**K8s**、**GitHub Container Registry**、**Quay.io**、PyPI、PyTorch、NPM等镜像缓存服务。 当前项目仍处于早期阶段。 目前支持
**DockerHub**、
**K8s**、
**GitHub Container Registry**、
**Quay.io**、
PyPI、
PyTorch、
NPM、
GoProxy
等镜像缓存服务。 当前项目仍处于早期阶段。
欢迎提交Pull Request和Issue我们非常期待您的宝贵建议和意见。 欢迎提交Pull Request和Issue我们非常期待您的宝贵建议和意见。
@ -41,17 +50,20 @@ docker-compose up
``` ```
并尝试通过控制台输出的地址进行访问http://aria2.local.homeinfra.org/aria2/index.html 并尝试通过控制台输出的地址进行访问https://aria2.local.homeinfra.org/aria2/index.html
为aria2的管理界面用于查看下载状态`。 为aria2的管理界面用于查看下载状态`。
可以使用以下命令进行测试镜像站是否正常工作: 可以使用以下命令进行测试镜像站是否正常工作:
```bash ```bash
docker pull docker.local.homeinfra.org/alpine docker pull docker.local.homeinfra.org/alpine
pip3 download -i http://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org pip3 download -i https://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org pip3 download -i https://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
``` ```
也可以通过查看 `./test/scripts` 下的测试脚本验证其他镜像站是否正常工作。
### Deployment ### Deployment
@ -59,19 +71,16 @@ pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torc
- docker + docker-compose. - docker + docker-compose.
- 一个域名,设置 `*.yourdomain` 的A记录指向您服务器的IP. - 一个域名,设置 `*.yourdomain` 的A记录指向您服务器的IP.
- `*.local.homeinfra.org` 默认指向 `127.0.0.1`,本地测试可以直接使用。
- 代理服务器(如有必要). - 代理服务器(如有必要).
> 如果需要使用HTTPS可以在外层新增一个HTTP网关如Caddy请参考后续章节。 安全起见我们默认开启了HTTPS通过 `docker-compose.yml` 中默认使用自签名证书。
> **对于DockerHub镜像我们强烈建议启用HTTPS** 我们也提供了一个基于 Caddy 的 HTTPS 部署方案,具体请参考下一节。
修改 `.env` 文件,设置下列参数: 修改 `.env` 文件,设置下列参数:
- `BASE_DOMAIN`: 基础域名,如 `local.homeinfra.org`,可以通过 `*.local.homeinfra.org` 访问镜像站。 - `BASE_DOMAIN`: 基础域名,如 `local.homeinfra.org`,可以通过 `*.local.homeinfra.org` 访问镜像站。
- `RPC_SECRET`Aria2的RPC密钥。 - `RPC_SECRET`Aria2的RPC密钥。
- `all_proxy`:代理服务器地址,如有必要。 - `*_proxy`:代理服务器地址,如有必要。
- `SCHEME``http``https`
配置完成之后,执行以下命令: 配置完成之后,执行以下命令:
@ -81,7 +90,7 @@ docker-compose up
#### HTTPS #### HTTPS
在 .env 中配置 `SCHEME=https`CLOUDFLARE_DNS_API_TOKEN。 在 .env 中配置 CLOUDFLARE_DNS_API_TOKEN。
本项目提供了一个基于Cloudflare DNS的 `Caddyfile``Dockerfile`。如果您希望使用其他DNS Provider或者LB请自行修改。 本项目提供了一个基于Cloudflare DNS的 `Caddyfile``Dockerfile`。如果您希望使用其他DNS Provider或者LB请自行修改。
配置完成后,使用以下命令代替上述的`docker-compose up` (多了 `-f docker-compose-caddy.yml`) 配置完成后,使用以下命令代替上述的`docker-compose up` (多了 `-f docker-compose-caddy.yml`)
@ -109,22 +118,18 @@ LightMirrors依赖于两个组件
docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。 docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
> 请注意:当 `SCHEME=http``DOCKER_BUILDKIT=1` 时,
> Dockerfile 中的 `FROM docker.local.homeinfra.org/xxx` 语法默认将从 https 站点拉取镜像,
> 此时将会出现错误。请使用 `docker pull`代替,或者尝试设置环境变量 `DOCKER_BUILDKIT=0`
### PyPI ### PyPI
- https: `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/` - `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
- http: `pip install jinja2 --index-url http://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
> 当使用自签名证书时,需要添加 `--trusted-host`
### PyTorch ### PyTorch
- https: `pip install torch --index-url https://torch.local.homeinfra.org/whl/` - `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
- http: `pip install torch --index-url http://torch.local.homeinfra.org/whl/ --trusted-host torch.local.homeinfra.org`
`download.pytorch.org`替换为 `torch.local.homeinfra.org` `download.pytorch.org`替换为 `torch.local.homeinfra.org`
如果使用的是http还需添加 `--trusted-host torch.local.homeinfra.org`
> 可以根据不同的硬件类型,切换不同的索引,如 https://download.pytorch.org/whl/cpu ,其中 `cpu` > 可以根据不同的硬件类型,切换不同的索引,如 https://download.pytorch.org/whl/cpu ,其中 `cpu`
> 可以替换为cu116/cu118/cu121/rocm5.4.2 等等。 > 可以替换为cu116/cu118/cu121/rocm5.4.2 等等。
@ -134,8 +139,7 @@ docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
npm 命令后加上 `--registry https://npm.local.homeinfra.org` 即可。 npm 命令后加上 `--registry https://npm.local.homeinfra.org` 即可。
- https: `npm install -S express --registry https://npm.local.homeinfra.org` - `npm install -S express --registry https://npm.local.homeinfra.org`
- http: `npm install -S express --registry http://npm.local.homeinfra.org`
## Star History ## Star History

View File

@ -8,21 +8,21 @@ services:
- ./data/cache:/app/cache - ./data/cache:/app/cache
env_file: env_file:
- .env - .env
environment:
- SSL_SELF_SIGNED=false
networks: networks:
- app - app
restart: unless-stopped restart: unless-stopped
# for linux # for linux
extra_hosts: extra_hosts:
- "host.docker.internal:host-gateway" - "host.docker.internal:host-gateway"
ports:
- "80:80"
caddy: caddy:
image: lightmirrors/caddy image: lightmirrors/caddy
build: build:
context: caddy context: caddy
dockerfile: Dockerfile dockerfile: Dockerfile
ports: ports:
- "80:80"
- "443:443" - "443:443"
volumes: volumes:
- ./caddy/Caddyfile:/etc/caddy/Caddyfile - ./caddy/Caddyfile:/etc/caddy/Caddyfile

View File

@ -8,6 +8,8 @@ services:
- ./data/cache:/app/cache - ./data/cache:/app/cache
env_file: env_file:
- .env - .env
environment:
- SSL_SELF_SIGNED=true
networks: networks:
- app - app
restart: unless-stopped restart: unless-stopped
@ -16,6 +18,7 @@ services:
- "host.docker.internal:host-gateway" - "host.docker.internal:host-gateway"
ports: ports:
- "80:80" - "80:80"
- "443:443"
aria2: aria2:
image: lightmirrors/aria2 image: lightmirrors/aria2
build: ./aria2 build: ./aria2

View File

@ -3,8 +3,7 @@ FROM python:3.11-alpine
RUN mkdir -p /wwwroot RUN mkdir -p /wwwroot
WORKDIR /wwwroot WORKDIR /wwwroot
# Optimization for China as the project is aimed at Chinese users ADD https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
ADD https://hub.gitmirror.com/https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
RUN unzip AriaNg-1.3.7.zip && rm AriaNg-1.3.7.zip RUN unzip AriaNg-1.3.7.zip && rm AriaNg-1.3.7.zip

19
src/certs/certificate.crt Normal file
View File

@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
-----END CERTIFICATE-----

47
src/certs/certificate.pem Normal file
View File

@ -0,0 +1,47 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
1UswIYVccfrbHGd7X5vxE4yY
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
-----END CERTIFICATE-----

15
src/certs/csr.pem Normal file
View File

@ -0,0 +1,15 @@
-----BEGIN CERTIFICATE REQUEST-----
MIICZTCCAU0CAQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMIIB
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzStc22z67EbnI7YuBtcQEtEb
XSOE2YM4RLIqrF7c1M1Eqv+Ekp51uq3NCreBi17T4+8/Iw7aNSmts3Et8AFFq8kh
ehdwvwH0ZcuoVmkGQlFU5KeLH38Sgp/2pCWi8XjX5VVv67tcYGzVy3ATqeGRMU2y
EyJvFMy3Uc1aMKw4hofSBfiGLT0Kb93jarny753QynSuLSzixvQil/+6eenagzOz
VUNeqVBfJUIovGaeDIqMQfNPC0LaWEV+w0Us+TxYCYRa06xz5wq8JGbirx3quU+B
mRiKPnd16p28m+V4LZiu5OWVfGo3+X7Xxizke0hcCyc1bP671xssA383EAyR6wID
AQABoAAwDQYJKoZIhvcNAQELBQADggEBAAU7hmM0/R8jyZLlG8D2JMdakzcy62HW
QYBC9nivStBN7R2oZFZ63egRiCBC4sqjL2epr7QAuixLaDwLrlzq7nJczRqyw5/f
X83Qo7+tLs4/4LXfVheWfDYYmKJja7lp72EmDPGBAnO4gb1bFLkYlrJU734gdtA+
tOzNbwmD9pSLPIBGrI9it7gY4666TnwgtlYs6kIzhW9m3fh4XnhArAFWAmU1Z/Ud
1ewnNCIsuHVa3vzPiL2ncN/IcIfDNdjkuk8kIh0QSFKHGRPk2wB02V8tWvITD4Rw
kXIX/H30cdBQTsgdNDAm7xxXumcO2f2meiyfBWjO1nd1mDVIAvPsX/Q=
-----END CERTIFICATE REQUEST-----

4
src/certs/generate.sh Normal file
View File

@ -0,0 +1,4 @@
openssl genpkey -algorithm RSA -out private.key
openssl req -new -key private.key -out csr.pem -config san.cnf
openssl x509 -req -days 3650 -in csr.pem -signkey private.key -out certificate.crt -extensions v3_req -extfile san.cnf
cat private.key certificate.crt > certificate.pem

28
src/certs/private.key Normal file
View File

@ -0,0 +1,28 @@
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
1UswIYVccfrbHGd7X5vxE4yY
-----END PRIVATE KEY-----

12
src/certs/san.cnf Normal file
View File

@ -0,0 +1,12 @@
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[v3_req]
subjectAltName = @alt_names
[alt_names]
DNS.1 = local.homeinfra.org
DNS.2 = *.local.homeinfra.org

View File

@ -4,8 +4,9 @@ ARIA2_RPC_URL = os.environ.get("ARIA2_RPC_URL", "http://aria2:6800/jsonrpc")
RPC_SECRET = os.environ.get("RPC_SECRET", "") RPC_SECRET = os.environ.get("RPC_SECRET", "")
BASE_DOMAIN = os.environ.get("BASE_DOMAIN", "local.homeinfra.org") BASE_DOMAIN = os.environ.get("BASE_DOMAIN", "local.homeinfra.org")
SCHEME = os.environ.get("SCHEME", "http").lower() SCHEME = "https"
assert SCHEME in ["http", "https"]
SSL_SELF_SIGNED = os.environ.get("SSL_SELF_SIGNED", "true") == "true"
CACHE_DIR = os.environ.get("CACHE_DIR", "/app/cache/") CACHE_DIR = os.environ.get("CACHE_DIR", "/app/cache/")
EXTERNAL_HOST_ARIA2 = f"aria2.{BASE_DOMAIN}" EXTERNAL_HOST_ARIA2 = f"aria2.{BASE_DOMAIN}"

View File

@ -6,6 +6,7 @@ import httpx
from httpx import Request as HttpxRequest from httpx import Request as HttpxRequest
from starlette.requests import Request from starlette.requests import Request
from starlette.responses import Response from starlette.responses import Response
from tenacity import retry, retry_if_exception_type, stop_after_attempt
SyncPreProcessor = Callable[[Request, HttpxRequest], HttpxRequest] SyncPreProcessor = Callable[[Request, HttpxRequest], HttpxRequest]
@ -54,6 +55,14 @@ async def post_process_response(
return response return response
@retry(
stop=stop_after_attempt(6),
retry=retry_if_exception_type(Exception),
reraise=True,
after=lambda retry_state: logger.warning(
f"retry {retry_state.attempt_number} {retry_state.outcome}"
),
)
async def direct_proxy( async def direct_proxy(
request: Request, request: Request,
target_url: str, target_url: str,
@ -74,6 +83,7 @@ async def direct_proxy(
request.method, request.method,
target_url, target_url,
headers=req_headers, headers=req_headers,
timeout=30,
) )
httpx_req = await pre_process_request(request, httpx_req, pre_process) httpx_req = await pre_process_request(request, httpx_req, pre_process)

View File

@ -21,7 +21,7 @@ from mirrorsrun.config import (
RPC_SECRET, RPC_SECRET,
EXTERNAL_URL_ARIA2, EXTERNAL_URL_ARIA2,
EXTERNAL_HOST_ARIA2, EXTERNAL_HOST_ARIA2,
SCHEME, SCHEME, SSL_SELF_SIGNED,
) )
from mirrorsrun.sites.npm import npm from mirrorsrun.sites.npm import npm
@ -29,6 +29,7 @@ from mirrorsrun.sites.pypi import pypi
from mirrorsrun.sites.torch import torch from mirrorsrun.sites.torch import torch
from mirrorsrun.sites.docker import dockerhub, k8s, quay, ghcr, nvcr from mirrorsrun.sites.docker import dockerhub, k8s, quay, ghcr, nvcr
from mirrorsrun.sites.common import common from mirrorsrun.sites.common import common
from mirrorsrun.sites.goproxy import goproxy
subdomain_mapping = { subdomain_mapping = {
"mirrors": common, "mirrors": common,
@ -40,6 +41,7 @@ subdomain_mapping = {
"ghcr": ghcr, "ghcr": ghcr,
"quay": quay, "quay": quay,
"nvcr": nvcr, "nvcr": nvcr,
"goproxy": goproxy,
} }
logging.basicConfig( logging.basicConfig(
@ -129,8 +131,10 @@ if __name__ == "__main__":
uvicorn.run( uvicorn.run(
app="server:app", app="server:app",
host="0.0.0.0", host="0.0.0.0",
port=port, ssl_keyfile='/app/certs/private.key' if SSL_SELF_SIGNED else None,
ssl_certfile='/app/certs/certificate.pem' if SSL_SELF_SIGNED else None,
port=443 if SSL_SELF_SIGNED else 80,
reload=True, # TODO: reload only in dev mode reload=True, # TODO: reload only in dev mode
proxy_headers=True, # trust x-forwarded-for etc. proxy_headers=not SSL_SELF_SIGNED, # trust x-forwarded-for etc.
forwarded_allow_ips="*", forwarded_allow_ips="*",
) )

View File

@ -0,0 +1,29 @@
from starlette.requests import Request
from mirrorsrun.proxy.direct import direct_proxy
from starlette.responses import Response
async def goproxy(request: Request):
path = request.url.path
sumdb_prefix = "/sumdb/sum.golang.org"
if path.startswith(sumdb_prefix):
sumdb_path = path.removeprefix(sumdb_prefix)
if sumdb_path.startswith("/supported"):
return Response(
content=b"",
)
target_url = "https://sum.golang.org" + sumdb_path
return await direct_proxy(
request,
target_url,
)
target_url = "https://proxy.golang.org" + path
return await direct_proxy(
request,
target_url,
)

19
src/poetry.lock generated
View File

@ -1,4 +1,4 @@
# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand. # This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
[[package]] [[package]]
name = "annotated-types" name = "annotated-types"
@ -498,6 +498,21 @@ typing-extensions = {version = ">=3.10.0", markers = "python_version < \"3.10\""
[package.extras] [package.extras]
full = ["httpx (>=0.22.0)", "itsdangerous", "jinja2", "python-multipart", "pyyaml"] full = ["httpx (>=0.22.0)", "itsdangerous", "jinja2", "python-multipart", "pyyaml"]
[[package]]
name = "tenacity"
version = "8.4.2"
description = "Retry code until it succeeds"
optional = false
python-versions = ">=3.8"
files = [
{file = "tenacity-8.4.2-py3-none-any.whl", hash = "sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2"},
{file = "tenacity-8.4.2.tar.gz", hash = "sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef"},
]
[package.extras]
doc = ["reno", "sphinx"]
test = ["pytest", "tornado (>=4.5)", "typeguard"]
[[package]] [[package]]
name = "tomli" name = "tomli"
version = "2.0.1" version = "2.0.1"
@ -542,4 +557,4 @@ standard = ["colorama (>=0.4)", "httptools (>=0.5.0)", "python-dotenv (>=0.13)",
[metadata] [metadata]
lock-version = "2.0" lock-version = "2.0"
python-versions = "^3.9" python-versions = "^3.9"
content-hash = "95fe42214f7987d04a55af59e60128b0a3e224b1b2abbdf962053703379a7160" content-hash = "5e3dd2b39fd601cef175cb5768d11ea71f4583d815cbe8d97e4b2214d0ee1e97"

View File

@ -10,6 +10,7 @@ python = "^3.9"
fastapi = "^0.109.0" fastapi = "^0.109.0"
uvicorn = "^0.27.0.post1" uvicorn = "^0.27.0.post1"
httpx = "^0.26.0" httpx = "^0.26.0"
tenacity = "^8.4.2"
[tool.poetry.group.dev.dependencies] [tool.poetry.group.dev.dependencies]

View File

@ -120,6 +120,9 @@ sniffio==1.3.0 ; python_version >= "3.9" and python_version < "4.0" \
starlette==0.35.1 ; python_version >= "3.9" and python_version < "4.0" \ starlette==0.35.1 ; python_version >= "3.9" and python_version < "4.0" \
--hash=sha256:3e2639dac3520e4f58734ed22553f950d3f3cb1001cd2eaac4d57e8cdc5f66bc \ --hash=sha256:3e2639dac3520e4f58734ed22553f950d3f3cb1001cd2eaac4d57e8cdc5f66bc \
--hash=sha256:50bbbda9baa098e361f398fda0928062abbaf1f54f4fadcbe17c092a01eb9a25 --hash=sha256:50bbbda9baa098e361f398fda0928062abbaf1f54f4fadcbe17c092a01eb9a25
tenacity==8.4.2 ; python_version >= "3.9" and python_version < "4.0" \
--hash=sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2 \
--hash=sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef
typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0" \ typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0" \
--hash=sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783 \ --hash=sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783 \
--hash=sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd --hash=sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd

View File

@ -1,25 +1,25 @@
version: "3.3"
services: services:
test:
image: lightmirrors/test docker_test:
build: image: docker:dind
context: .
dockerfile: Dockerfile
volumes: volumes:
- .:/app - ./scripts:/scripts
- /var/run/docker.sock:/var/run/docker.sock command: sh /scripts/docker.sh
networks: network_mode: host
- lightmirrors_app privileged: true
external_links:
- lightmirrors:aria2.local.homeinfra.org python_test:
- lightmirrors:docker.local.homeinfra.org image: python:3-alpine
- lightmirrors:pypi.local.homeinfra.org volumes:
- lightmirrors:torch.local.homeinfra.org - ./scripts:/scripts
- lightmirrors:npm.local.homeinfra.org command: sh /scripts/python.sh
- lightmirrors:ubuntu.local.homeinfra.org network_mode: host
- lightmirrors:debian.local.homeinfra.org
- lightmirrors:proxy.local.homeinfra.org
- lightmirrors:github.local.homeinfra.org golang_test:
- lightmirrors:alpine.local.homeinfra.org image: golang:alpine
networks: volumes:
lightmirrors_app: - ./scripts:/scripts
external: true command: sh /scripts/golang.sh
network_mode: host

View File

@ -1,32 +0,0 @@
import unittest
from utils import call
PYPI_HOST = "pypi.local.homeinfra.org"
PYPI_INDEX = f"http://{PYPI_HOST}/simple"
TORCH_HOST = "torch.local.homeinfra.org"
TORCH_INDEX = f"http://{TORCH_HOST}/whl"
class TestPypi(unittest.TestCase):
def test_pypi_http(self):
call(f"pip download -i {PYPI_INDEX} django --trusted-host {PYPI_HOST} --dest /tmp/pypi/")
def test_torch_http(self):
call(f"pip download -i {TORCH_INDEX} tqdm --trusted-host {TORCH_HOST} --dest /tmp/torch/")
def test_dockerhub_pull(self):
call(f"docker pull docker.local.homeinfra.org/alpine:3.12")
def test_ghcr_pull(self):
call(f"docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine")
def test_quay_pull(self):
call(f"docker pull quay.local.homeinfra.org/quay/busybox")
def test_k8s_pull(self):
call(f"docker pull k8s.local.homeinfra.org/pause:3.5")
def test_nvcr_pull(self):
call(f"docker pull nvcr.local.homeinfra.org/nvidia/cuda")

29
test/utils.py → test/runner.py Normal file → Executable file
View File

@ -1,3 +1,4 @@
#!/usr/bin/env python3
import os import os
import subprocess import subprocess
from pathlib import Path from pathlib import Path
@ -9,19 +10,29 @@ root_dir = Path(__file__).parent.parent
def call(cmd): def call(cmd):
print(f">> {cmd}") print(f">> {cmd}")
p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate() stdout, stderr = p.communicate()
assert p.returncode == 0, f"Error: {stderr.decode()}" assert p.returncode == 0, f"Error: {stderr.decode()}"
print(">>", stdout.decode()) print(">>", stdout.decode())
return stdout.decode(), stderr.decode() return stdout.decode(), stderr.decode()
class SetupMirrors(): services = [
def __enter__(self): "python_test",
os.chdir(root_dir) "docker_test",
call("docker-compose up -d") "golang_test"
return self ]
def __exit__(self, exc_type, exc_val, exc_tb): os.chdir(root_dir)
call("docker-compose down") call("docker-compose up -d --force-recreate --wait")
os.chdir(test_dir)
return False os.chdir(test_dir)
try:
for service in services:
call(f'docker-compose up --force-recreate --exit-code-from {service} {service}')
except Exception as e:
raise e
finally:
os.chdir(root_dir)
call("docker-compose down")

View File

@ -0,0 +1,19 @@
-----BEGIN CERTIFICATE-----
MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
-----END CERTIFICATE-----

37
test/scripts/docker.sh Normal file
View File

@ -0,0 +1,37 @@
set -ex
export DOCKER_HOST=unix:///var/run/docker.sock
cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
dockerd&
docker_ready() {
docker version >/dev/null 2>&1
}
max_wait_time=5
elapsed_time=0
# Wait for Docker to be ready
while [ true ]; do
if docker_ready; then
echo "Docker is ready!"
break
else
echo "Waiting for Docker to start..."
sleep 1
elapsed_time=$((elapsed_time + 1))
if [ $elapsed_time -gt $max_wait_time ]; then
echo "Docker failed to start in $max_wait_time seconds!"
exit 1
fi
fi
done
docker pull docker.local.homeinfra.org/busybox
docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine
docker pull quay.local.homeinfra.org/quay/busybox
docker pull k8s.local.homeinfra.org/pause
# https is required
echo 'FROM docker.local.homeinfra.org/alpine' | docker build -

9
test/scripts/golang.sh Normal file
View File

@ -0,0 +1,9 @@
set -ex
cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
go env -w GOPROXY=https://goproxy.local.homeinfra.org,direct
go clean -modcache
go mod init test
go get golang.org/x/sys@v0.22.0

0
test/scripts/npm.sh Normal file
View File

12
test/scripts/python.sh Normal file
View File

@ -0,0 +1,12 @@
set -ex
cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
pip config set global.index-url https://pypi.local.homeinfra.org/simple
pip config set global.trusted-host pypi.local.homeinfra.org
pip download jinja2 --dest /tmp/pypi/
pip config set global.index-url https://torch.local.homeinfra.org/whl
pip config set global.trusted-host torch.local.homeinfra.org
pip download tqdm --dest /tmp/torch/