.env.example

@@ -7,15 +7,6 @@ BASE_DOMAIN=local.homeinfra.org
 # for aria2 and mirrors
 RPC_SECRET=changeit
 
-# in case any other service need the uppercase or lowercase one
+# for all, if you want set proxy for all services
 # aria2 need the lowercase one, FYI https://aria2.github.io/manual/en/html/aria2c.html#environment
-
-no_proxy=lightmirrors,aria2
-NO_PROXY=lightmirrors,aria2
-
 # all_proxy=
-# ALL_PROXY=
-# http_proxy=
-# HTTP_PROXY=
-# https_proxy=
-# HTTPS_PROXY=

README.md

@@ -12,16 +12,7 @@
 
 
 LightMirrors是一个开源的缓存镜像站服务，用于加速软件包下载和镜像拉取。
-目前支持
+目前支持**DockerHub**、**K8s**、**GitHub Container Registry**、**Quay.io**、PyPI、PyTorch、NPM等镜像缓存服务。 当前项目仍处于早期阶段。
-**DockerHub**、
-**K8s**、
-**GitHub Container Registry**、
-**Quay.io**、
-PyPI、
-PyTorch、
-NPM、
-GoProxy
-等镜像缓存服务。 当前项目仍处于早期阶段。
 
 欢迎提交Pull Request和Issue，我们非常期待您的宝贵建议和意见。
 
@@ -50,20 +41,17 @@ docker-compose up
 
 ```
 
-并尝试通过控制台输出的地址进行访问，https://aria2.local.homeinfra.org/aria2/index.html
+并尝试通过控制台输出的地址进行访问，http://aria2.local.homeinfra.org/aria2/index.html
 为aria2的管理界面，用于查看下载状态`。
 
 可以使用以下命令进行测试镜像站是否正常工作：
 
 ```bash
 docker pull docker.local.homeinfra.org/alpine
-pip3 download -i https://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
+pip3 download -i http://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
-pip3 download -i https://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
+pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
 ```
 
-也可以通过查看 `./test/scripts` 下的测试脚本验证其他镜像站是否正常工作。
-
-
 ### Deployment
 
 
@@ -71,16 +59,19 @@ pip3 download -i https://torch.local.homeinfra.org/whl/ torch --trusted-host tor
 
 - docker + docker-compose.
 - 一个域名，设置 `*.yourdomain` 的A记录指向您服务器的IP.
+    - `*.local.homeinfra.org` 默认指向 `127.0.0.1`，本地测试可以直接使用。
 - 代理服务器（如有必要）.
 
-安全起见，我们默认开启了HTTPS，通过 `docker-compose.yml` 中默认使用自签名证书。
+> 如果需要使用HTTPS，可以在外层新增一个HTTP网关（如Caddy），请参考后续章节。
-我们也提供了一个基于 Caddy 的 HTTPS 部署方案，具体请参考下一节。
+> **对于DockerHub镜像，我们强烈建议启用HTTPS**。
+
 
 修改 `.env` 文件，设置下列参数：
 
 - `BASE_DOMAIN`: 基础域名，如 `local.homeinfra.org`，可以通过 `*.local.homeinfra.org` 访问镜像站。
 - `RPC_SECRET`：Aria2的RPC密钥。
-- `*_proxy`：代理服务器地址，如有必要。
+- `all_proxy`：代理服务器地址，如有必要。
+- `SCHEME`：`http` 或 `https`。
 
 配置完成之后，执行以下命令：
 
@@ -90,7 +81,7 @@ docker-compose up
 
 #### HTTPS
 
-在 .env 中配置 CLOUDFLARE_DNS_API_TOKEN。
+在 .env 中配置 `SCHEME=https` 与 CLOUDFLARE_DNS_API_TOKEN。
 本项目提供了一个基于Cloudflare DNS的 `Caddyfile` 和 `Dockerfile`。如果您希望使用其他DNS Provider或者LB，请自行修改。
 
 配置完成后，使用以下命令代替上述的`docker-compose up` (多了 `-f docker-compose-caddy.yml`) ：
@@ -118,18 +109,22 @@ LightMirrors依赖于两个组件：
 
 docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
 
+> 请注意：当 `SCHEME=http` 且 `DOCKER_BUILDKIT=1` 时，
+> Dockerfile 中的 `FROM docker.local.homeinfra.org/xxx` 语法默认将从 https 站点拉取镜像，
+> 此时将会出现错误。请使用 `docker pull`代替，或者尝试设置环境变量 `DOCKER_BUILDKIT=0`
+
 ### PyPI
 
-- `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
+- https: `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/`
-
+- http: `pip install jinja2 --index-url http://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
-> 当使用自签名证书时，需要添加 `--trusted-host`。
-
 
 ### PyTorch
 
-- `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
+- https: `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
+- http: `pip install torch --index-url http://torch.local.homeinfra.org/whl/ --trusted-host torch.local.homeinfra.org`
 
 把`download.pytorch.org`替换为 `torch.local.homeinfra.org` ，
+如果使用的是http，还需添加 `--trusted-host torch.local.homeinfra.org`。
 
 > 可以根据不同的硬件类型，切换不同的索引，如 https://download.pytorch.org/whl/cpu ，其中 `cpu`
 > 可以替换为cu116/cu118/cu121/rocm5.4.2 等等。
@@ -139,7 +134,8 @@ docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
 
 npm 命令后加上 `--registry https://npm.local.homeinfra.org` 即可。
 
-- `npm install -S express --registry https://npm.local.homeinfra.org`
+- https: `npm install -S express --registry https://npm.local.homeinfra.org`
+- http: `npm install -S express --registry http://npm.local.homeinfra.org`
 
 ## Star History
 

docker-compose-caddy.yml

@@ -8,21 +8,21 @@ services:
       - ./data/cache:/app/cache
     env_file:
       - .env
-    environment:
-      - SSL_SELF_SIGNED=false
     networks:
       - app
     restart: unless-stopped
     # for linux
     extra_hosts:
       - "host.docker.internal:host-gateway"
+    ports:
+      - "80:80"
   caddy:
     image: lightmirrors/caddy
     build:
       context: caddy
       dockerfile: Dockerfile
+
     ports:
-      - "80:80"
       - "443:443"
     volumes:
       - ./caddy/Caddyfile:/etc/caddy/Caddyfile

docker-compose.yml

@@ -8,8 +8,6 @@ services:
       - ./data/cache:/app/cache
     env_file:
       - .env
-    environment:
-      - SSL_SELF_SIGNED=true
     networks:
       - app
     restart: unless-stopped
@@ -18,7 +16,6 @@ services:
       - "host.docker.internal:host-gateway"
     ports:
       - "80:80"
-      - "443:443"
   aria2:
     image: lightmirrors/aria2
     build: ./aria2

src/Dockerfile

@@ -3,7 +3,8 @@ FROM python:3.11-alpine
 RUN mkdir -p /wwwroot
 WORKDIR /wwwroot
 
-ADD https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
+# Optimization for China as the project is aimed at Chinese users
+ADD https://hub.gitmirror.com/https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
 
 RUN unzip AriaNg-1.3.7.zip && rm AriaNg-1.3.7.zip
 

src/certs/certificate.crt

@@ -1,19 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
-BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
-MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
-cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
-dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
-8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
-7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
-vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
-/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
-ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
-ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
-m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
-rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
-nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
-0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
-NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
-xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
------END CERTIFICATE-----

src/certs/certificate.pem

@@ -1,47 +0,0 @@
------BEGIN PRIVATE KEY-----
-MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
-PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
-xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
-7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
-rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
-XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
-5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
-2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
-53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
-EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
-h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
-S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
-eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
-drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
-z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
-sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
-0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
-2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
-JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
-q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
-Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
-tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
-BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
-029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
-BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
-1UswIYVccfrbHGd7X5vxE4yY
------END PRIVATE KEY-----
------BEGIN CERTIFICATE-----
-MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
-BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
-MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
-cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
-dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
-8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
-7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
-vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
-/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
-ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
-ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
-m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
-rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
-nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
-0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
-NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
-xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
------END CERTIFICATE-----

src/certs/csr.pem

@@ -1,15 +0,0 @@
------BEGIN CERTIFICATE REQUEST-----
-MIICZTCCAU0CAQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMIIB
-IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzStc22z67EbnI7YuBtcQEtEb
-XSOE2YM4RLIqrF7c1M1Eqv+Ekp51uq3NCreBi17T4+8/Iw7aNSmts3Et8AFFq8kh
-ehdwvwH0ZcuoVmkGQlFU5KeLH38Sgp/2pCWi8XjX5VVv67tcYGzVy3ATqeGRMU2y
-EyJvFMy3Uc1aMKw4hofSBfiGLT0Kb93jarny753QynSuLSzixvQil/+6eenagzOz
-VUNeqVBfJUIovGaeDIqMQfNPC0LaWEV+w0Us+TxYCYRa06xz5wq8JGbirx3quU+B
-mRiKPnd16p28m+V4LZiu5OWVfGo3+X7Xxizke0hcCyc1bP671xssA383EAyR6wID
-AQABoAAwDQYJKoZIhvcNAQELBQADggEBAAU7hmM0/R8jyZLlG8D2JMdakzcy62HW
-QYBC9nivStBN7R2oZFZ63egRiCBC4sqjL2epr7QAuixLaDwLrlzq7nJczRqyw5/f
-X83Qo7+tLs4/4LXfVheWfDYYmKJja7lp72EmDPGBAnO4gb1bFLkYlrJU734gdtA+
-tOzNbwmD9pSLPIBGrI9it7gY4666TnwgtlYs6kIzhW9m3fh4XnhArAFWAmU1Z/Ud
-1ewnNCIsuHVa3vzPiL2ncN/IcIfDNdjkuk8kIh0QSFKHGRPk2wB02V8tWvITD4Rw
-kXIX/H30cdBQTsgdNDAm7xxXumcO2f2meiyfBWjO1nd1mDVIAvPsX/Q=
------END CERTIFICATE REQUEST-----

src/certs/generate.sh

@@ -1,4 +0,0 @@
-openssl genpkey -algorithm RSA -out private.key
-openssl req -new -key private.key -out csr.pem -config san.cnf
-openssl x509 -req -days 3650 -in csr.pem -signkey private.key -out certificate.crt -extensions v3_req -extfile san.cnf
-cat private.key certificate.crt > certificate.pem

src/certs/private.key

@@ -1,28 +0,0 @@
------BEGIN PRIVATE KEY-----
-MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
-PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
-xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
-7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
-rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
-XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
-5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
-2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
-53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
-EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
-h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
-S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
-eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
-drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
-z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
-sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
-0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
-2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
-JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
-q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
-Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
-tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
-BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
-029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
-BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
-1UswIYVccfrbHGd7X5vxE4yY
------END PRIVATE KEY-----

src/certs/san.cnf

@@ -1,12 +0,0 @@
-[req]
-req_extensions = v3_req
-distinguished_name = req_distinguished_name
-
-[req_distinguished_name]
-
-[v3_req]
-subjectAltName = @alt_names
-
-[alt_names]
-DNS.1 = local.homeinfra.org
-DNS.2 = *.local.homeinfra.org

src/mirrorsrun/config.py

@@ -4,9 +4,8 @@ ARIA2_RPC_URL = os.environ.get("ARIA2_RPC_URL", "http://aria2:6800/jsonrpc")
 RPC_SECRET = os.environ.get("RPC_SECRET", "")
 BASE_DOMAIN = os.environ.get("BASE_DOMAIN", "local.homeinfra.org")
 
-SCHEME = "https"
+SCHEME = os.environ.get("SCHEME", "http").lower()
-
+assert SCHEME in ["http", "https"]
-SSL_SELF_SIGNED = os.environ.get("SSL_SELF_SIGNED", "true") == "true"
 
 CACHE_DIR = os.environ.get("CACHE_DIR", "/app/cache/")
 EXTERNAL_HOST_ARIA2 = f"aria2.{BASE_DOMAIN}"

src/mirrorsrun/proxy/direct.py

@@ -6,7 +6,6 @@ import httpx
 from httpx import Request as HttpxRequest
 from starlette.requests import Request
 from starlette.responses import Response
-from tenacity import retry, retry_if_exception_type, stop_after_attempt
 
 SyncPreProcessor = Callable[[Request, HttpxRequest], HttpxRequest]
 
@@ -55,14 +54,6 @@ async def post_process_response(
         return response
 
 
-@retry(
-    stop=stop_after_attempt(6),
-    retry=retry_if_exception_type(Exception),
-    reraise=True,
-    after=lambda retry_state: logger.warning(
-        f"retry {retry_state.attempt_number} {retry_state.outcome}"
-    ),
-)
 async def direct_proxy(
     request: Request,
     target_url: str,
@@ -83,7 +74,6 @@ async def direct_proxy(
             request.method,
             target_url,
             headers=req_headers,
-            timeout=30,
         )
 
         httpx_req = await pre_process_request(request, httpx_req, pre_process)

src/mirrorsrun/server.py

@@ -21,7 +21,7 @@ from mirrorsrun.config import (
     RPC_SECRET,
     EXTERNAL_URL_ARIA2,
     EXTERNAL_HOST_ARIA2,
-    SCHEME, SSL_SELF_SIGNED,
+    SCHEME,
 )
 
 from mirrorsrun.sites.npm import npm
@@ -29,7 +29,6 @@ from mirrorsrun.sites.pypi import pypi
 from mirrorsrun.sites.torch import torch
 from mirrorsrun.sites.docker import dockerhub, k8s, quay, ghcr, nvcr
 from mirrorsrun.sites.common import common
-from mirrorsrun.sites.goproxy import goproxy
 
 subdomain_mapping = {
     "mirrors": common,
@@ -41,7 +40,6 @@ subdomain_mapping = {
     "ghcr": ghcr,
     "quay": quay,
     "nvcr": nvcr,
-    "goproxy": goproxy,
 }
 
 logging.basicConfig(
@@ -131,10 +129,8 @@ if __name__ == "__main__":
     uvicorn.run(
         app="server:app",
         host="0.0.0.0",
-        ssl_keyfile='/app/certs/private.key' if SSL_SELF_SIGNED else None,
+        port=port,
-        ssl_certfile='/app/certs/certificate.pem' if SSL_SELF_SIGNED else None,
-        port=443 if SSL_SELF_SIGNED else 80,
         reload=True,  # TODO: reload only in dev mode
-        proxy_headers=not SSL_SELF_SIGNED,  # trust x-forwarded-for etc.
+        proxy_headers=True,  # trust x-forwarded-for etc.
         forwarded_allow_ips="*",
     )

src/mirrorsrun/sites/goproxy.py

@@ -1,29 +0,0 @@
-from starlette.requests import Request
-
-from mirrorsrun.proxy.direct import direct_proxy
-from starlette.responses import Response
-
-
-async def goproxy(request: Request):
-    path = request.url.path
-
-    sumdb_prefix = "/sumdb/sum.golang.org"
-    if path.startswith(sumdb_prefix):
-        sumdb_path = path.removeprefix(sumdb_prefix)
-        if sumdb_path.startswith("/supported"):
-            return Response(
-                content=b"",
-            )
-        target_url = "https://sum.golang.org" + sumdb_path
-        return await direct_proxy(
-            request,
-            target_url,
-        )
-
-    target_url = "https://proxy.golang.org" + path
-
-    return await direct_proxy(
-        request,
-        target_url,
-    )
-

src/poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand.
 
 [[package]]
 name = "annotated-types"
@@ -498,21 +498,6 @@ typing-extensions = {version = ">=3.10.0", markers = "python_version < \"3.10\""
 [package.extras]
 full = ["httpx (>=0.22.0)", "itsdangerous", "jinja2", "python-multipart", "pyyaml"]
 
-[[package]]
-name = "tenacity"
-version = "8.4.2"
-description = "Retry code until it succeeds"
-optional = false
-python-versions = ">=3.8"
-files = [
-    {file = "tenacity-8.4.2-py3-none-any.whl", hash = "sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2"},
-    {file = "tenacity-8.4.2.tar.gz", hash = "sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef"},
-]
-
-[package.extras]
-doc = ["reno", "sphinx"]
-test = ["pytest", "tornado (>=4.5)", "typeguard"]
-
 [[package]]
 name = "tomli"
 version = "2.0.1"
@@ -557,4 +542,4 @@ standard = ["colorama (>=0.4)", "httptools (>=0.5.0)", "python-dotenv (>=0.13)",
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.9"
-content-hash = "5e3dd2b39fd601cef175cb5768d11ea71f4583d815cbe8d97e4b2214d0ee1e97"
+content-hash = "95fe42214f7987d04a55af59e60128b0a3e224b1b2abbdf962053703379a7160"

src/pyproject.toml

@@ -10,7 +10,6 @@ python = "^3.9"
 fastapi = "^0.109.0"
 uvicorn = "^0.27.0.post1"
 httpx = "^0.26.0"
-tenacity = "^8.4.2"
 
 
 [tool.poetry.group.dev.dependencies]

src/requirements.txt

@@ -120,9 +120,6 @@ sniffio==1.3.0 ; python_version >= "3.9" and python_version < "4.0" \
 starlette==0.35.1 ; python_version >= "3.9" and python_version < "4.0" \
     --hash=sha256:3e2639dac3520e4f58734ed22553f950d3f3cb1001cd2eaac4d57e8cdc5f66bc \
     --hash=sha256:50bbbda9baa098e361f398fda0928062abbaf1f54f4fadcbe17c092a01eb9a25
-tenacity==8.4.2 ; python_version >= "3.9" and python_version < "4.0" \
-    --hash=sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2 \
-    --hash=sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef
 typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0" \
     --hash=sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783 \
     --hash=sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd

test/docker-compose.yml

@@ -1,25 +1,25 @@
-version: "3.3"
 services:
-
+  test:
-  docker_test:
+    image: lightmirrors/test
-    image: docker:dind
+    build:
+      context: .
+      dockerfile: Dockerfile
     volumes:
-      - ./scripts:/scripts
+      - .:/app
-    command: sh /scripts/docker.sh
+      - /var/run/docker.sock:/var/run/docker.sock
-    network_mode: host
+    networks:
-    privileged: true
+      - lightmirrors_app
-
+    external_links:
-  python_test:
+      - lightmirrors:aria2.local.homeinfra.org
-    image: python:3-alpine
+      - lightmirrors:docker.local.homeinfra.org
-    volumes:
+      - lightmirrors:pypi.local.homeinfra.org
-      - ./scripts:/scripts
+      - lightmirrors:torch.local.homeinfra.org
-    command: sh /scripts/python.sh
+      - lightmirrors:npm.local.homeinfra.org
-    network_mode: host
+      - lightmirrors:ubuntu.local.homeinfra.org
-
+      - lightmirrors:debian.local.homeinfra.org
-
+      - lightmirrors:proxy.local.homeinfra.org
-  golang_test:
+      - lightmirrors:github.local.homeinfra.org
-    image: golang:alpine
+      - lightmirrors:alpine.local.homeinfra.org
-    volumes:
+networks:
-      - ./scripts:/scripts
+  lightmirrors_app:
-    command: sh /scripts/golang.sh
+    external: true
-    network_mode: host

test/mirrors_test.py

@@ -0,0 +1,32 @@
+import unittest
+
+from utils import call
+
+PYPI_HOST = "pypi.local.homeinfra.org"
+PYPI_INDEX = f"http://{PYPI_HOST}/simple"
+TORCH_HOST = "torch.local.homeinfra.org"
+TORCH_INDEX = f"http://{TORCH_HOST}/whl"
+
+
+class TestPypi(unittest.TestCase):
+
+    def test_pypi_http(self):
+        call(f"pip download -i {PYPI_INDEX} django --trusted-host {PYPI_HOST}  --dest /tmp/pypi/")
+
+    def test_torch_http(self):
+        call(f"pip download -i {TORCH_INDEX} tqdm --trusted-host {TORCH_HOST} --dest /tmp/torch/")
+
+    def test_dockerhub_pull(self):
+        call(f"docker pull docker.local.homeinfra.org/alpine:3.12")
+
+    def test_ghcr_pull(self):
+        call(f"docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine")
+
+    def test_quay_pull(self):
+        call(f"docker pull quay.local.homeinfra.org/quay/busybox")
+
+    def test_k8s_pull(self):
+        call(f"docker pull k8s.local.homeinfra.org/pause:3.5")
+
+    def test_nvcr_pull(self):
+        call(f"docker pull nvcr.local.homeinfra.org/nvidia/cuda")

test/scripts/certs/certificate.crt

@@ -1,19 +0,0 @@
------BEGIN CERTIFICATE-----
-MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
-BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
-MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
-cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
-dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
-8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
-7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
-vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
-/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
-ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
-ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
-m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
-rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
-nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
-0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
-NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
-xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
------END CERTIFICATE-----

test/scripts/docker.sh

@@ -1,37 +0,0 @@
-set -ex
-export DOCKER_HOST=unix:///var/run/docker.sock
-
-cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
-
-dockerd&
-
-docker_ready() {
-  docker version >/dev/null 2>&1
-}
-
-max_wait_time=5
-elapsed_time=0
-
-# Wait for Docker to be ready
-while [ true ]; do
-  if docker_ready; then
-    echo "Docker is ready!"
-    break
-  else
-    echo "Waiting for Docker to start..."
-    sleep 1
-    elapsed_time=$((elapsed_time + 1))
-    if [ $elapsed_time -gt $max_wait_time ]; then
-      echo "Docker failed to start in $max_wait_time seconds!"
-      exit 1
-  fi
-  fi
-done
-
-docker pull docker.local.homeinfra.org/busybox
-docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine
-docker pull quay.local.homeinfra.org/quay/busybox
-docker pull k8s.local.homeinfra.org/pause
-
-# https is required
-echo 'FROM docker.local.homeinfra.org/alpine' | docker build -

test/scripts/golang.sh

@@ -1,9 +0,0 @@
-set -ex
-
-cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
-
-go env -w GOPROXY=https://goproxy.local.homeinfra.org,direct
-
-go clean -modcache
-go mod init test
-go get golang.org/x/sys@v0.22.0

test/scripts/python.sh

@@ -1,12 +0,0 @@
-set -ex
-
-cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
-
-pip config set global.index-url https://pypi.local.homeinfra.org/simple
-pip config set global.trusted-host pypi.local.homeinfra.org
-pip download jinja2 --dest /tmp/pypi/
-
-
-pip config set global.index-url https://torch.local.homeinfra.org/whl
-pip config set global.trusted-host torch.local.homeinfra.org
-pip download tqdm --dest /tmp/torch/

test/utils.py

@@ -1,4 +1,3 @@
-#!/usr/bin/env python3
 import os
 import subprocess
 from pathlib import Path
@@ -10,29 +9,19 @@ root_dir = Path(__file__).parent.parent
 def call(cmd):
     print(f">>  {cmd}")
     p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-
     stdout, stderr = p.communicate()
     assert p.returncode == 0, f"Error: {stderr.decode()}"
     print(">>", stdout.decode())
     return stdout.decode(), stderr.decode()
 
 
-services = [
+class SetupMirrors():
-    "python_test",
+    def __enter__(self):
-    "docker_test",
-    "golang_test"
-]
-
         os.chdir(root_dir)
-call("docker-compose up -d --force-recreate --wait")
+        call("docker-compose up -d")
+        return self
 
-os.chdir(test_dir)
+    def __exit__(self, exc_type, exc_val, exc_tb):
-
-try:
-    for service in services:
-        call(f'docker-compose up --force-recreate --exit-code-from {service} {service}')
-except Exception as e:
-    raise e
-finally:
-    os.chdir(root_dir)
         call("docker-compose down")
+        os.chdir(test_dir)
+        return False