mirror of
https://github.com/NoCLin/LightMirrors
synced 2025-08-02 23:47:15 +08:00
Compare commits
4 Commits
e6e2eae600
...
37866ae727
Author | SHA1 | Date | |
---|---|---|---|
![]() |
37866ae727 | ||
![]() |
862699a524 | ||
![]() |
098b06a58b | ||
![]() |
c9ebbac425 |
11
.env.example
11
.env.example
@ -7,6 +7,15 @@ BASE_DOMAIN=local.homeinfra.org
|
|||||||
# for aria2 and mirrors
|
# for aria2 and mirrors
|
||||||
RPC_SECRET=changeit
|
RPC_SECRET=changeit
|
||||||
|
|
||||||
# for all, if you want set proxy for all services
|
# in case any other service need the uppercase or lowercase one
|
||||||
# aria2 need the lowercase one, FYI https://aria2.github.io/manual/en/html/aria2c.html#environment
|
# aria2 need the lowercase one, FYI https://aria2.github.io/manual/en/html/aria2c.html#environment
|
||||||
|
|
||||||
|
no_proxy=lightmirrors,aria2
|
||||||
|
NO_PROXY=lightmirrors,aria2
|
||||||
|
|
||||||
# all_proxy=
|
# all_proxy=
|
||||||
|
# ALL_PROXY=
|
||||||
|
# http_proxy=
|
||||||
|
# HTTP_PROXY=
|
||||||
|
# https_proxy=
|
||||||
|
# HTTPS_PROXY=
|
48
README.md
48
README.md
@ -12,7 +12,16 @@
|
|||||||
|
|
||||||
|
|
||||||
LightMirrors是一个开源的缓存镜像站服务,用于加速软件包下载和镜像拉取。
|
LightMirrors是一个开源的缓存镜像站服务,用于加速软件包下载和镜像拉取。
|
||||||
目前支持**DockerHub**、**K8s**、**GitHub Container Registry**、**Quay.io**、PyPI、PyTorch、NPM等镜像缓存服务。 当前项目仍处于早期阶段。
|
目前支持
|
||||||
|
**DockerHub**、
|
||||||
|
**K8s**、
|
||||||
|
**GitHub Container Registry**、
|
||||||
|
**Quay.io**、
|
||||||
|
PyPI、
|
||||||
|
PyTorch、
|
||||||
|
NPM、
|
||||||
|
GoProxy
|
||||||
|
等镜像缓存服务。 当前项目仍处于早期阶段。
|
||||||
|
|
||||||
欢迎提交Pull Request和Issue,我们非常期待您的宝贵建议和意见。
|
欢迎提交Pull Request和Issue,我们非常期待您的宝贵建议和意见。
|
||||||
|
|
||||||
@ -41,17 +50,20 @@ docker-compose up
|
|||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
并尝试通过控制台输出的地址进行访问,http://aria2.local.homeinfra.org/aria2/index.html
|
并尝试通过控制台输出的地址进行访问,https://aria2.local.homeinfra.org/aria2/index.html
|
||||||
为aria2的管理界面,用于查看下载状态`。
|
为aria2的管理界面,用于查看下载状态`。
|
||||||
|
|
||||||
可以使用以下命令进行测试镜像站是否正常工作:
|
可以使用以下命令进行测试镜像站是否正常工作:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
docker pull docker.local.homeinfra.org/alpine
|
docker pull docker.local.homeinfra.org/alpine
|
||||||
pip3 download -i http://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
|
pip3 download -i https://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
|
||||||
pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
|
pip3 download -i https://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
|
||||||
```
|
```
|
||||||
|
|
||||||
|
也可以通过查看 `./test/scripts` 下的测试脚本验证其他镜像站是否正常工作。
|
||||||
|
|
||||||
|
|
||||||
### Deployment
|
### Deployment
|
||||||
|
|
||||||
|
|
||||||
@ -59,19 +71,16 @@ pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torc
|
|||||||
|
|
||||||
- docker + docker-compose.
|
- docker + docker-compose.
|
||||||
- 一个域名,设置 `*.yourdomain` 的A记录指向您服务器的IP.
|
- 一个域名,设置 `*.yourdomain` 的A记录指向您服务器的IP.
|
||||||
- `*.local.homeinfra.org` 默认指向 `127.0.0.1`,本地测试可以直接使用。
|
|
||||||
- 代理服务器(如有必要).
|
- 代理服务器(如有必要).
|
||||||
|
|
||||||
> 如果需要使用HTTPS,可以在外层新增一个HTTP网关(如Caddy),请参考后续章节。
|
安全起见,我们默认开启了HTTPS,通过 `docker-compose.yml` 中默认使用自签名证书。
|
||||||
> **对于DockerHub镜像,我们强烈建议启用HTTPS**。
|
我们也提供了一个基于 Caddy 的 HTTPS 部署方案,具体请参考下一节。
|
||||||
|
|
||||||
|
|
||||||
修改 `.env` 文件,设置下列参数:
|
修改 `.env` 文件,设置下列参数:
|
||||||
|
|
||||||
- `BASE_DOMAIN`: 基础域名,如 `local.homeinfra.org`,可以通过 `*.local.homeinfra.org` 访问镜像站。
|
- `BASE_DOMAIN`: 基础域名,如 `local.homeinfra.org`,可以通过 `*.local.homeinfra.org` 访问镜像站。
|
||||||
- `RPC_SECRET`:Aria2的RPC密钥。
|
- `RPC_SECRET`:Aria2的RPC密钥。
|
||||||
- `all_proxy`:代理服务器地址,如有必要。
|
- `*_proxy`:代理服务器地址,如有必要。
|
||||||
- `SCHEME`:`http` 或 `https`。
|
|
||||||
|
|
||||||
配置完成之后,执行以下命令:
|
配置完成之后,执行以下命令:
|
||||||
|
|
||||||
@ -81,7 +90,7 @@ docker-compose up
|
|||||||
|
|
||||||
#### HTTPS
|
#### HTTPS
|
||||||
|
|
||||||
在 .env 中配置 `SCHEME=https` 与 CLOUDFLARE_DNS_API_TOKEN。
|
在 .env 中配置 CLOUDFLARE_DNS_API_TOKEN。
|
||||||
本项目提供了一个基于Cloudflare DNS的 `Caddyfile` 和 `Dockerfile`。如果您希望使用其他DNS Provider或者LB,请自行修改。
|
本项目提供了一个基于Cloudflare DNS的 `Caddyfile` 和 `Dockerfile`。如果您希望使用其他DNS Provider或者LB,请自行修改。
|
||||||
|
|
||||||
配置完成后,使用以下命令代替上述的`docker-compose up` (多了 `-f docker-compose-caddy.yml`) :
|
配置完成后,使用以下命令代替上述的`docker-compose up` (多了 `-f docker-compose-caddy.yml`) :
|
||||||
@ -109,22 +118,18 @@ LightMirrors依赖于两个组件:
|
|||||||
|
|
||||||
docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
|
docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
|
||||||
|
|
||||||
> 请注意:当 `SCHEME=http` 且 `DOCKER_BUILDKIT=1` 时,
|
|
||||||
> Dockerfile 中的 `FROM docker.local.homeinfra.org/xxx` 语法默认将从 https 站点拉取镜像,
|
|
||||||
> 此时将会出现错误。请使用 `docker pull`代替,或者尝试设置环境变量 `DOCKER_BUILDKIT=0`
|
|
||||||
|
|
||||||
### PyPI
|
### PyPI
|
||||||
|
|
||||||
- https: `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/`
|
- `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
|
||||||
- http: `pip install jinja2 --index-url http://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
|
|
||||||
|
> 当使用自签名证书时,需要添加 `--trusted-host`。
|
||||||
|
|
||||||
|
|
||||||
### PyTorch
|
### PyTorch
|
||||||
|
|
||||||
- https: `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
|
- `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
|
||||||
- http: `pip install torch --index-url http://torch.local.homeinfra.org/whl/ --trusted-host torch.local.homeinfra.org`
|
|
||||||
|
|
||||||
把`download.pytorch.org`替换为 `torch.local.homeinfra.org` ,
|
把`download.pytorch.org`替换为 `torch.local.homeinfra.org` ,
|
||||||
如果使用的是http,还需添加 `--trusted-host torch.local.homeinfra.org`。
|
|
||||||
|
|
||||||
> 可以根据不同的硬件类型,切换不同的索引,如 https://download.pytorch.org/whl/cpu ,其中 `cpu`
|
> 可以根据不同的硬件类型,切换不同的索引,如 https://download.pytorch.org/whl/cpu ,其中 `cpu`
|
||||||
> 可以替换为cu116/cu118/cu121/rocm5.4.2 等等。
|
> 可以替换为cu116/cu118/cu121/rocm5.4.2 等等。
|
||||||
@ -134,8 +139,7 @@ docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
|
|||||||
|
|
||||||
npm 命令后加上 `--registry https://npm.local.homeinfra.org` 即可。
|
npm 命令后加上 `--registry https://npm.local.homeinfra.org` 即可。
|
||||||
|
|
||||||
- https: `npm install -S express --registry https://npm.local.homeinfra.org`
|
- `npm install -S express --registry https://npm.local.homeinfra.org`
|
||||||
- http: `npm install -S express --registry http://npm.local.homeinfra.org`
|
|
||||||
|
|
||||||
## Star History
|
## Star History
|
||||||
|
|
||||||
|
@ -8,21 +8,21 @@ services:
|
|||||||
- ./data/cache:/app/cache
|
- ./data/cache:/app/cache
|
||||||
env_file:
|
env_file:
|
||||||
- .env
|
- .env
|
||||||
|
environment:
|
||||||
|
- SSL_SELF_SIGNED=false
|
||||||
networks:
|
networks:
|
||||||
- app
|
- app
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
# for linux
|
# for linux
|
||||||
extra_hosts:
|
extra_hosts:
|
||||||
- "host.docker.internal:host-gateway"
|
- "host.docker.internal:host-gateway"
|
||||||
ports:
|
|
||||||
- "80:80"
|
|
||||||
caddy:
|
caddy:
|
||||||
image: lightmirrors/caddy
|
image: lightmirrors/caddy
|
||||||
build:
|
build:
|
||||||
context: caddy
|
context: caddy
|
||||||
dockerfile: Dockerfile
|
dockerfile: Dockerfile
|
||||||
|
|
||||||
ports:
|
ports:
|
||||||
|
- "80:80"
|
||||||
- "443:443"
|
- "443:443"
|
||||||
volumes:
|
volumes:
|
||||||
- ./caddy/Caddyfile:/etc/caddy/Caddyfile
|
- ./caddy/Caddyfile:/etc/caddy/Caddyfile
|
||||||
|
@ -8,6 +8,8 @@ services:
|
|||||||
- ./data/cache:/app/cache
|
- ./data/cache:/app/cache
|
||||||
env_file:
|
env_file:
|
||||||
- .env
|
- .env
|
||||||
|
environment:
|
||||||
|
- SSL_SELF_SIGNED=true
|
||||||
networks:
|
networks:
|
||||||
- app
|
- app
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
@ -16,6 +18,7 @@ services:
|
|||||||
- "host.docker.internal:host-gateway"
|
- "host.docker.internal:host-gateway"
|
||||||
ports:
|
ports:
|
||||||
- "80:80"
|
- "80:80"
|
||||||
|
- "443:443"
|
||||||
aria2:
|
aria2:
|
||||||
image: lightmirrors/aria2
|
image: lightmirrors/aria2
|
||||||
build: ./aria2
|
build: ./aria2
|
||||||
|
@ -3,8 +3,7 @@ FROM python:3.11-alpine
|
|||||||
RUN mkdir -p /wwwroot
|
RUN mkdir -p /wwwroot
|
||||||
WORKDIR /wwwroot
|
WORKDIR /wwwroot
|
||||||
|
|
||||||
# Optimization for China as the project is aimed at Chinese users
|
ADD https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
|
||||||
ADD https://hub.gitmirror.com/https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
|
|
||||||
|
|
||||||
RUN unzip AriaNg-1.3.7.zip && rm AriaNg-1.3.7.zip
|
RUN unzip AriaNg-1.3.7.zip && rm AriaNg-1.3.7.zip
|
||||||
|
|
||||||
|
19
src/certs/certificate.crt
Normal file
19
src/certs/certificate.crt
Normal file
@ -0,0 +1,19 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
|
||||||
|
BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
|
||||||
|
MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
|
||||||
|
cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
|
||||||
|
dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
|
||||||
|
8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
|
||||||
|
7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
|
||||||
|
vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
|
||||||
|
/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
|
||||||
|
ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
|
||||||
|
ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
|
||||||
|
m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
|
||||||
|
rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
|
||||||
|
nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
|
||||||
|
0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
|
||||||
|
NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
|
||||||
|
xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
|
||||||
|
-----END CERTIFICATE-----
|
47
src/certs/certificate.pem
Normal file
47
src/certs/certificate.pem
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
-----BEGIN PRIVATE KEY-----
|
||||||
|
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
|
||||||
|
PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
|
||||||
|
xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
|
||||||
|
7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
|
||||||
|
rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
|
||||||
|
XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
|
||||||
|
5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
|
||||||
|
2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
|
||||||
|
53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
|
||||||
|
EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
|
||||||
|
h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
|
||||||
|
S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
|
||||||
|
eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
|
||||||
|
drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
|
||||||
|
z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
|
||||||
|
sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
|
||||||
|
0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
|
||||||
|
2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
|
||||||
|
JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
|
||||||
|
q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
|
||||||
|
Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
|
||||||
|
tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
|
||||||
|
BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
|
||||||
|
029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
|
||||||
|
BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
|
||||||
|
1UswIYVccfrbHGd7X5vxE4yY
|
||||||
|
-----END PRIVATE KEY-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
|
||||||
|
BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
|
||||||
|
MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
|
||||||
|
cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
|
||||||
|
dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
|
||||||
|
8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
|
||||||
|
7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
|
||||||
|
vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
|
||||||
|
/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
|
||||||
|
ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
|
||||||
|
ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
|
||||||
|
m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
|
||||||
|
rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
|
||||||
|
nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
|
||||||
|
0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
|
||||||
|
NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
|
||||||
|
xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
|
||||||
|
-----END CERTIFICATE-----
|
15
src/certs/csr.pem
Normal file
15
src/certs/csr.pem
Normal file
@ -0,0 +1,15 @@
|
|||||||
|
-----BEGIN CERTIFICATE REQUEST-----
|
||||||
|
MIICZTCCAU0CAQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMIIB
|
||||||
|
IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzStc22z67EbnI7YuBtcQEtEb
|
||||||
|
XSOE2YM4RLIqrF7c1M1Eqv+Ekp51uq3NCreBi17T4+8/Iw7aNSmts3Et8AFFq8kh
|
||||||
|
ehdwvwH0ZcuoVmkGQlFU5KeLH38Sgp/2pCWi8XjX5VVv67tcYGzVy3ATqeGRMU2y
|
||||||
|
EyJvFMy3Uc1aMKw4hofSBfiGLT0Kb93jarny753QynSuLSzixvQil/+6eenagzOz
|
||||||
|
VUNeqVBfJUIovGaeDIqMQfNPC0LaWEV+w0Us+TxYCYRa06xz5wq8JGbirx3quU+B
|
||||||
|
mRiKPnd16p28m+V4LZiu5OWVfGo3+X7Xxizke0hcCyc1bP671xssA383EAyR6wID
|
||||||
|
AQABoAAwDQYJKoZIhvcNAQELBQADggEBAAU7hmM0/R8jyZLlG8D2JMdakzcy62HW
|
||||||
|
QYBC9nivStBN7R2oZFZ63egRiCBC4sqjL2epr7QAuixLaDwLrlzq7nJczRqyw5/f
|
||||||
|
X83Qo7+tLs4/4LXfVheWfDYYmKJja7lp72EmDPGBAnO4gb1bFLkYlrJU734gdtA+
|
||||||
|
tOzNbwmD9pSLPIBGrI9it7gY4666TnwgtlYs6kIzhW9m3fh4XnhArAFWAmU1Z/Ud
|
||||||
|
1ewnNCIsuHVa3vzPiL2ncN/IcIfDNdjkuk8kIh0QSFKHGRPk2wB02V8tWvITD4Rw
|
||||||
|
kXIX/H30cdBQTsgdNDAm7xxXumcO2f2meiyfBWjO1nd1mDVIAvPsX/Q=
|
||||||
|
-----END CERTIFICATE REQUEST-----
|
4
src/certs/generate.sh
Normal file
4
src/certs/generate.sh
Normal file
@ -0,0 +1,4 @@
|
|||||||
|
openssl genpkey -algorithm RSA -out private.key
|
||||||
|
openssl req -new -key private.key -out csr.pem -config san.cnf
|
||||||
|
openssl x509 -req -days 3650 -in csr.pem -signkey private.key -out certificate.crt -extensions v3_req -extfile san.cnf
|
||||||
|
cat private.key certificate.crt > certificate.pem
|
28
src/certs/private.key
Normal file
28
src/certs/private.key
Normal file
@ -0,0 +1,28 @@
|
|||||||
|
-----BEGIN PRIVATE KEY-----
|
||||||
|
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
|
||||||
|
PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
|
||||||
|
xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
|
||||||
|
7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
|
||||||
|
rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
|
||||||
|
XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
|
||||||
|
5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
|
||||||
|
2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
|
||||||
|
53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
|
||||||
|
EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
|
||||||
|
h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
|
||||||
|
S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
|
||||||
|
eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
|
||||||
|
drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
|
||||||
|
z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
|
||||||
|
sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
|
||||||
|
0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
|
||||||
|
2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
|
||||||
|
JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
|
||||||
|
q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
|
||||||
|
Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
|
||||||
|
tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
|
||||||
|
BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
|
||||||
|
029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
|
||||||
|
BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
|
||||||
|
1UswIYVccfrbHGd7X5vxE4yY
|
||||||
|
-----END PRIVATE KEY-----
|
12
src/certs/san.cnf
Normal file
12
src/certs/san.cnf
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
[req]
|
||||||
|
req_extensions = v3_req
|
||||||
|
distinguished_name = req_distinguished_name
|
||||||
|
|
||||||
|
[req_distinguished_name]
|
||||||
|
|
||||||
|
[v3_req]
|
||||||
|
subjectAltName = @alt_names
|
||||||
|
|
||||||
|
[alt_names]
|
||||||
|
DNS.1 = local.homeinfra.org
|
||||||
|
DNS.2 = *.local.homeinfra.org
|
@ -4,8 +4,9 @@ ARIA2_RPC_URL = os.environ.get("ARIA2_RPC_URL", "http://aria2:6800/jsonrpc")
|
|||||||
RPC_SECRET = os.environ.get("RPC_SECRET", "")
|
RPC_SECRET = os.environ.get("RPC_SECRET", "")
|
||||||
BASE_DOMAIN = os.environ.get("BASE_DOMAIN", "local.homeinfra.org")
|
BASE_DOMAIN = os.environ.get("BASE_DOMAIN", "local.homeinfra.org")
|
||||||
|
|
||||||
SCHEME = os.environ.get("SCHEME", "http").lower()
|
SCHEME = "https"
|
||||||
assert SCHEME in ["http", "https"]
|
|
||||||
|
SSL_SELF_SIGNED = os.environ.get("SSL_SELF_SIGNED", "true") == "true"
|
||||||
|
|
||||||
CACHE_DIR = os.environ.get("CACHE_DIR", "/app/cache/")
|
CACHE_DIR = os.environ.get("CACHE_DIR", "/app/cache/")
|
||||||
EXTERNAL_HOST_ARIA2 = f"aria2.{BASE_DOMAIN}"
|
EXTERNAL_HOST_ARIA2 = f"aria2.{BASE_DOMAIN}"
|
||||||
|
@ -6,6 +6,7 @@ import httpx
|
|||||||
from httpx import Request as HttpxRequest
|
from httpx import Request as HttpxRequest
|
||||||
from starlette.requests import Request
|
from starlette.requests import Request
|
||||||
from starlette.responses import Response
|
from starlette.responses import Response
|
||||||
|
from tenacity import retry, retry_if_exception_type, stop_after_attempt
|
||||||
|
|
||||||
SyncPreProcessor = Callable[[Request, HttpxRequest], HttpxRequest]
|
SyncPreProcessor = Callable[[Request, HttpxRequest], HttpxRequest]
|
||||||
|
|
||||||
@ -54,6 +55,14 @@ async def post_process_response(
|
|||||||
return response
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
@retry(
|
||||||
|
stop=stop_after_attempt(6),
|
||||||
|
retry=retry_if_exception_type(Exception),
|
||||||
|
reraise=True,
|
||||||
|
after=lambda retry_state: logger.warning(
|
||||||
|
f"retry {retry_state.attempt_number} {retry_state.outcome}"
|
||||||
|
),
|
||||||
|
)
|
||||||
async def direct_proxy(
|
async def direct_proxy(
|
||||||
request: Request,
|
request: Request,
|
||||||
target_url: str,
|
target_url: str,
|
||||||
@ -74,6 +83,7 @@ async def direct_proxy(
|
|||||||
request.method,
|
request.method,
|
||||||
target_url,
|
target_url,
|
||||||
headers=req_headers,
|
headers=req_headers,
|
||||||
|
timeout=30,
|
||||||
)
|
)
|
||||||
|
|
||||||
httpx_req = await pre_process_request(request, httpx_req, pre_process)
|
httpx_req = await pre_process_request(request, httpx_req, pre_process)
|
||||||
|
@ -21,7 +21,7 @@ from mirrorsrun.config import (
|
|||||||
RPC_SECRET,
|
RPC_SECRET,
|
||||||
EXTERNAL_URL_ARIA2,
|
EXTERNAL_URL_ARIA2,
|
||||||
EXTERNAL_HOST_ARIA2,
|
EXTERNAL_HOST_ARIA2,
|
||||||
SCHEME,
|
SCHEME, SSL_SELF_SIGNED,
|
||||||
)
|
)
|
||||||
|
|
||||||
from mirrorsrun.sites.npm import npm
|
from mirrorsrun.sites.npm import npm
|
||||||
@ -29,6 +29,7 @@ from mirrorsrun.sites.pypi import pypi
|
|||||||
from mirrorsrun.sites.torch import torch
|
from mirrorsrun.sites.torch import torch
|
||||||
from mirrorsrun.sites.docker import dockerhub, k8s, quay, ghcr, nvcr
|
from mirrorsrun.sites.docker import dockerhub, k8s, quay, ghcr, nvcr
|
||||||
from mirrorsrun.sites.common import common
|
from mirrorsrun.sites.common import common
|
||||||
|
from mirrorsrun.sites.goproxy import goproxy
|
||||||
|
|
||||||
subdomain_mapping = {
|
subdomain_mapping = {
|
||||||
"mirrors": common,
|
"mirrors": common,
|
||||||
@ -40,6 +41,7 @@ subdomain_mapping = {
|
|||||||
"ghcr": ghcr,
|
"ghcr": ghcr,
|
||||||
"quay": quay,
|
"quay": quay,
|
||||||
"nvcr": nvcr,
|
"nvcr": nvcr,
|
||||||
|
"goproxy": goproxy,
|
||||||
}
|
}
|
||||||
|
|
||||||
logging.basicConfig(
|
logging.basicConfig(
|
||||||
@ -129,8 +131,10 @@ if __name__ == "__main__":
|
|||||||
uvicorn.run(
|
uvicorn.run(
|
||||||
app="server:app",
|
app="server:app",
|
||||||
host="0.0.0.0",
|
host="0.0.0.0",
|
||||||
port=port,
|
ssl_keyfile='/app/certs/private.key' if SSL_SELF_SIGNED else None,
|
||||||
|
ssl_certfile='/app/certs/certificate.pem' if SSL_SELF_SIGNED else None,
|
||||||
|
port=443 if SSL_SELF_SIGNED else 80,
|
||||||
reload=True, # TODO: reload only in dev mode
|
reload=True, # TODO: reload only in dev mode
|
||||||
proxy_headers=True, # trust x-forwarded-for etc.
|
proxy_headers=not SSL_SELF_SIGNED, # trust x-forwarded-for etc.
|
||||||
forwarded_allow_ips="*",
|
forwarded_allow_ips="*",
|
||||||
)
|
)
|
||||||
|
29
src/mirrorsrun/sites/goproxy.py
Normal file
29
src/mirrorsrun/sites/goproxy.py
Normal file
@ -0,0 +1,29 @@
|
|||||||
|
from starlette.requests import Request
|
||||||
|
|
||||||
|
from mirrorsrun.proxy.direct import direct_proxy
|
||||||
|
from starlette.responses import Response
|
||||||
|
|
||||||
|
|
||||||
|
async def goproxy(request: Request):
|
||||||
|
path = request.url.path
|
||||||
|
|
||||||
|
sumdb_prefix = "/sumdb/sum.golang.org"
|
||||||
|
if path.startswith(sumdb_prefix):
|
||||||
|
sumdb_path = path.removeprefix(sumdb_prefix)
|
||||||
|
if sumdb_path.startswith("/supported"):
|
||||||
|
return Response(
|
||||||
|
content=b"",
|
||||||
|
)
|
||||||
|
target_url = "https://sum.golang.org" + sumdb_path
|
||||||
|
return await direct_proxy(
|
||||||
|
request,
|
||||||
|
target_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
target_url = "https://proxy.golang.org" + path
|
||||||
|
|
||||||
|
return await direct_proxy(
|
||||||
|
request,
|
||||||
|
target_url,
|
||||||
|
)
|
||||||
|
|
19
src/poetry.lock
generated
19
src/poetry.lock
generated
@ -1,4 +1,4 @@
|
|||||||
# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand.
|
# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "annotated-types"
|
name = "annotated-types"
|
||||||
@ -498,6 +498,21 @@ typing-extensions = {version = ">=3.10.0", markers = "python_version < \"3.10\""
|
|||||||
[package.extras]
|
[package.extras]
|
||||||
full = ["httpx (>=0.22.0)", "itsdangerous", "jinja2", "python-multipart", "pyyaml"]
|
full = ["httpx (>=0.22.0)", "itsdangerous", "jinja2", "python-multipart", "pyyaml"]
|
||||||
|
|
||||||
|
[[package]]
|
||||||
|
name = "tenacity"
|
||||||
|
version = "8.4.2"
|
||||||
|
description = "Retry code until it succeeds"
|
||||||
|
optional = false
|
||||||
|
python-versions = ">=3.8"
|
||||||
|
files = [
|
||||||
|
{file = "tenacity-8.4.2-py3-none-any.whl", hash = "sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2"},
|
||||||
|
{file = "tenacity-8.4.2.tar.gz", hash = "sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef"},
|
||||||
|
]
|
||||||
|
|
||||||
|
[package.extras]
|
||||||
|
doc = ["reno", "sphinx"]
|
||||||
|
test = ["pytest", "tornado (>=4.5)", "typeguard"]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tomli"
|
name = "tomli"
|
||||||
version = "2.0.1"
|
version = "2.0.1"
|
||||||
@ -542,4 +557,4 @@ standard = ["colorama (>=0.4)", "httptools (>=0.5.0)", "python-dotenv (>=0.13)",
|
|||||||
[metadata]
|
[metadata]
|
||||||
lock-version = "2.0"
|
lock-version = "2.0"
|
||||||
python-versions = "^3.9"
|
python-versions = "^3.9"
|
||||||
content-hash = "95fe42214f7987d04a55af59e60128b0a3e224b1b2abbdf962053703379a7160"
|
content-hash = "5e3dd2b39fd601cef175cb5768d11ea71f4583d815cbe8d97e4b2214d0ee1e97"
|
||||||
|
@ -10,6 +10,7 @@ python = "^3.9"
|
|||||||
fastapi = "^0.109.0"
|
fastapi = "^0.109.0"
|
||||||
uvicorn = "^0.27.0.post1"
|
uvicorn = "^0.27.0.post1"
|
||||||
httpx = "^0.26.0"
|
httpx = "^0.26.0"
|
||||||
|
tenacity = "^8.4.2"
|
||||||
|
|
||||||
|
|
||||||
[tool.poetry.group.dev.dependencies]
|
[tool.poetry.group.dev.dependencies]
|
||||||
|
@ -120,6 +120,9 @@ sniffio==1.3.0 ; python_version >= "3.9" and python_version < "4.0" \
|
|||||||
starlette==0.35.1 ; python_version >= "3.9" and python_version < "4.0" \
|
starlette==0.35.1 ; python_version >= "3.9" and python_version < "4.0" \
|
||||||
--hash=sha256:3e2639dac3520e4f58734ed22553f950d3f3cb1001cd2eaac4d57e8cdc5f66bc \
|
--hash=sha256:3e2639dac3520e4f58734ed22553f950d3f3cb1001cd2eaac4d57e8cdc5f66bc \
|
||||||
--hash=sha256:50bbbda9baa098e361f398fda0928062abbaf1f54f4fadcbe17c092a01eb9a25
|
--hash=sha256:50bbbda9baa098e361f398fda0928062abbaf1f54f4fadcbe17c092a01eb9a25
|
||||||
|
tenacity==8.4.2 ; python_version >= "3.9" and python_version < "4.0" \
|
||||||
|
--hash=sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2 \
|
||||||
|
--hash=sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef
|
||||||
typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0" \
|
typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0" \
|
||||||
--hash=sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783 \
|
--hash=sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783 \
|
||||||
--hash=sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd
|
--hash=sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd
|
||||||
|
@ -1,25 +1,25 @@
|
|||||||
|
version: "3.3"
|
||||||
services:
|
services:
|
||||||
test:
|
|
||||||
image: lightmirrors/test
|
docker_test:
|
||||||
build:
|
image: docker:dind
|
||||||
context: .
|
|
||||||
dockerfile: Dockerfile
|
|
||||||
volumes:
|
volumes:
|
||||||
- .:/app
|
- ./scripts:/scripts
|
||||||
- /var/run/docker.sock:/var/run/docker.sock
|
command: sh /scripts/docker.sh
|
||||||
networks:
|
network_mode: host
|
||||||
- lightmirrors_app
|
privileged: true
|
||||||
external_links:
|
|
||||||
- lightmirrors:aria2.local.homeinfra.org
|
python_test:
|
||||||
- lightmirrors:docker.local.homeinfra.org
|
image: python:3-alpine
|
||||||
- lightmirrors:pypi.local.homeinfra.org
|
volumes:
|
||||||
- lightmirrors:torch.local.homeinfra.org
|
- ./scripts:/scripts
|
||||||
- lightmirrors:npm.local.homeinfra.org
|
command: sh /scripts/python.sh
|
||||||
- lightmirrors:ubuntu.local.homeinfra.org
|
network_mode: host
|
||||||
- lightmirrors:debian.local.homeinfra.org
|
|
||||||
- lightmirrors:proxy.local.homeinfra.org
|
|
||||||
- lightmirrors:github.local.homeinfra.org
|
golang_test:
|
||||||
- lightmirrors:alpine.local.homeinfra.org
|
image: golang:alpine
|
||||||
networks:
|
volumes:
|
||||||
lightmirrors_app:
|
- ./scripts:/scripts
|
||||||
external: true
|
command: sh /scripts/golang.sh
|
||||||
|
network_mode: host
|
||||||
|
@ -1,32 +0,0 @@
|
|||||||
import unittest
|
|
||||||
|
|
||||||
from utils import call
|
|
||||||
|
|
||||||
PYPI_HOST = "pypi.local.homeinfra.org"
|
|
||||||
PYPI_INDEX = f"http://{PYPI_HOST}/simple"
|
|
||||||
TORCH_HOST = "torch.local.homeinfra.org"
|
|
||||||
TORCH_INDEX = f"http://{TORCH_HOST}/whl"
|
|
||||||
|
|
||||||
|
|
||||||
class TestPypi(unittest.TestCase):
|
|
||||||
|
|
||||||
def test_pypi_http(self):
|
|
||||||
call(f"pip download -i {PYPI_INDEX} django --trusted-host {PYPI_HOST} --dest /tmp/pypi/")
|
|
||||||
|
|
||||||
def test_torch_http(self):
|
|
||||||
call(f"pip download -i {TORCH_INDEX} tqdm --trusted-host {TORCH_HOST} --dest /tmp/torch/")
|
|
||||||
|
|
||||||
def test_dockerhub_pull(self):
|
|
||||||
call(f"docker pull docker.local.homeinfra.org/alpine:3.12")
|
|
||||||
|
|
||||||
def test_ghcr_pull(self):
|
|
||||||
call(f"docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine")
|
|
||||||
|
|
||||||
def test_quay_pull(self):
|
|
||||||
call(f"docker pull quay.local.homeinfra.org/quay/busybox")
|
|
||||||
|
|
||||||
def test_k8s_pull(self):
|
|
||||||
call(f"docker pull k8s.local.homeinfra.org/pause:3.5")
|
|
||||||
|
|
||||||
def test_nvcr_pull(self):
|
|
||||||
call(f"docker pull nvcr.local.homeinfra.org/nvidia/cuda")
|
|
29
test/utils.py → test/runner.py
Normal file → Executable file
29
test/utils.py → test/runner.py
Normal file → Executable file
@ -1,3 +1,4 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
import os
|
import os
|
||||||
import subprocess
|
import subprocess
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
@ -9,19 +10,29 @@ root_dir = Path(__file__).parent.parent
|
|||||||
def call(cmd):
|
def call(cmd):
|
||||||
print(f">> {cmd}")
|
print(f">> {cmd}")
|
||||||
p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||||
|
|
||||||
stdout, stderr = p.communicate()
|
stdout, stderr = p.communicate()
|
||||||
assert p.returncode == 0, f"Error: {stderr.decode()}"
|
assert p.returncode == 0, f"Error: {stderr.decode()}"
|
||||||
print(">>", stdout.decode())
|
print(">>", stdout.decode())
|
||||||
return stdout.decode(), stderr.decode()
|
return stdout.decode(), stderr.decode()
|
||||||
|
|
||||||
|
|
||||||
class SetupMirrors():
|
services = [
|
||||||
def __enter__(self):
|
"python_test",
|
||||||
os.chdir(root_dir)
|
"docker_test",
|
||||||
call("docker-compose up -d")
|
"golang_test"
|
||||||
return self
|
]
|
||||||
|
|
||||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
os.chdir(root_dir)
|
||||||
call("docker-compose down")
|
call("docker-compose up -d --force-recreate --wait")
|
||||||
os.chdir(test_dir)
|
|
||||||
return False
|
os.chdir(test_dir)
|
||||||
|
|
||||||
|
try:
|
||||||
|
for service in services:
|
||||||
|
call(f'docker-compose up --force-recreate --exit-code-from {service} {service}')
|
||||||
|
except Exception as e:
|
||||||
|
raise e
|
||||||
|
finally:
|
||||||
|
os.chdir(root_dir)
|
||||||
|
call("docker-compose down")
|
19
test/scripts/certs/certificate.crt
Normal file
19
test/scripts/certs/certificate.crt
Normal file
@ -0,0 +1,19 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
|
||||||
|
BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
|
||||||
|
MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
|
||||||
|
cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
|
||||||
|
dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
|
||||||
|
8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
|
||||||
|
7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
|
||||||
|
vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
|
||||||
|
/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
|
||||||
|
ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
|
||||||
|
ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
|
||||||
|
m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
|
||||||
|
rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
|
||||||
|
nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
|
||||||
|
0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
|
||||||
|
NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
|
||||||
|
xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
|
||||||
|
-----END CERTIFICATE-----
|
37
test/scripts/docker.sh
Normal file
37
test/scripts/docker.sh
Normal file
@ -0,0 +1,37 @@
|
|||||||
|
set -ex
|
||||||
|
export DOCKER_HOST=unix:///var/run/docker.sock
|
||||||
|
|
||||||
|
cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
|
||||||
|
|
||||||
|
dockerd&
|
||||||
|
|
||||||
|
docker_ready() {
|
||||||
|
docker version >/dev/null 2>&1
|
||||||
|
}
|
||||||
|
|
||||||
|
max_wait_time=5
|
||||||
|
elapsed_time=0
|
||||||
|
|
||||||
|
# Wait for Docker to be ready
|
||||||
|
while [ true ]; do
|
||||||
|
if docker_ready; then
|
||||||
|
echo "Docker is ready!"
|
||||||
|
break
|
||||||
|
else
|
||||||
|
echo "Waiting for Docker to start..."
|
||||||
|
sleep 1
|
||||||
|
elapsed_time=$((elapsed_time + 1))
|
||||||
|
if [ $elapsed_time -gt $max_wait_time ]; then
|
||||||
|
echo "Docker failed to start in $max_wait_time seconds!"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
docker pull docker.local.homeinfra.org/busybox
|
||||||
|
docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine
|
||||||
|
docker pull quay.local.homeinfra.org/quay/busybox
|
||||||
|
docker pull k8s.local.homeinfra.org/pause
|
||||||
|
|
||||||
|
# https is required
|
||||||
|
echo 'FROM docker.local.homeinfra.org/alpine' | docker build -
|
9
test/scripts/golang.sh
Normal file
9
test/scripts/golang.sh
Normal file
@ -0,0 +1,9 @@
|
|||||||
|
set -ex
|
||||||
|
|
||||||
|
cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
|
||||||
|
|
||||||
|
go env -w GOPROXY=https://goproxy.local.homeinfra.org,direct
|
||||||
|
|
||||||
|
go clean -modcache
|
||||||
|
go mod init test
|
||||||
|
go get golang.org/x/sys@v0.22.0
|
0
test/scripts/npm.sh
Normal file
0
test/scripts/npm.sh
Normal file
12
test/scripts/python.sh
Normal file
12
test/scripts/python.sh
Normal file
@ -0,0 +1,12 @@
|
|||||||
|
set -ex
|
||||||
|
|
||||||
|
cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
|
||||||
|
|
||||||
|
pip config set global.index-url https://pypi.local.homeinfra.org/simple
|
||||||
|
pip config set global.trusted-host pypi.local.homeinfra.org
|
||||||
|
pip download jinja2 --dest /tmp/pypi/
|
||||||
|
|
||||||
|
|
||||||
|
pip config set global.index-url https://torch.local.homeinfra.org/whl
|
||||||
|
pip config set global.trusted-host torch.local.homeinfra.org
|
||||||
|
pip download tqdm --dest /tmp/torch/
|
Loading…
x
Reference in New Issue
Block a user