docs

.env.example

@@ -7,6 +7,15 @@ BASE_DOMAIN=local.homeinfra.org
 # for aria2 and mirrors
 RPC_SECRET=changeit
 
-# for all, if you want set proxy for all services
+# in case any other service need the uppercase or lowercase one
 # aria2 need the lowercase one, FYI https://aria2.github.io/manual/en/html/aria2c.html#environment
+
+no_proxy=lightmirrors,aria2
+NO_PROXY=lightmirrors,aria2
+
 # all_proxy=
+# ALL_PROXY=
+# http_proxy=
+# HTTP_PROXY=
+# https_proxy=
+# HTTPS_PROXY=

README.md

@@ -12,7 +12,16 @@
 
 
 LightMirrors是一个开源的缓存镜像站服务，用于加速软件包下载和镜像拉取。
-目前支持**DockerHub**、**K8s**、**GitHub Container Registry**、**Quay.io**、PyPI、PyTorch、NPM等镜像缓存服务。 当前项目仍处于早期阶段。
+目前支持
+**DockerHub**、
+**K8s**、
+**GitHub Container Registry**、
+**Quay.io**、
+PyPI、
+PyTorch、
+NPM、
+GoProxy
+等镜像缓存服务。 当前项目仍处于早期阶段。
 
 欢迎提交Pull Request和Issue，我们非常期待您的宝贵建议和意见。
 
@@ -41,17 +50,20 @@ docker-compose up
 
 ```
 
-并尝试通过控制台输出的地址进行访问，http://aria2.local.homeinfra.org/aria2/index.html
+并尝试通过控制台输出的地址进行访问，https://aria2.local.homeinfra.org/aria2/index.html
 为aria2的管理界面，用于查看下载状态`。
 
 可以使用以下命令进行测试镜像站是否正常工作：
 
 ```bash
 docker pull docker.local.homeinfra.org/alpine
-pip3 download -i http://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
-pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
+pip3 download -i https://pypi.local.homeinfra.org/simple/ jinja2 --trusted-host pypi.local.homeinfra.org
+pip3 download -i https://torch.local.homeinfra.org/whl/ torch --trusted-host torch.local.homeinfra.org
 ```
 
+也可以通过查看 `./test/scripts` 下的测试脚本验证其他镜像站是否正常工作。
+
+
 ### Deployment
 
 
@@ -59,19 +71,16 @@ pip3 download -i http://torch.local.homeinfra.org/whl/ torch --trusted-host torc
 
 - docker + docker-compose.
 - 一个域名，设置 `*.yourdomain` 的A记录指向您服务器的IP.
-    - `*.local.homeinfra.org` 默认指向 `127.0.0.1`，本地测试可以直接使用。
 - 代理服务器（如有必要）.
 
-> 如果需要使用HTTPS，可以在外层新增一个HTTP网关（如Caddy），请参考后续章节。
-> **对于DockerHub镜像，我们强烈建议启用HTTPS**。
-
+安全起见，我们默认开启了HTTPS，通过 `docker-compose.yml` 中默认使用自签名证书。
+我们也提供了一个基于 Caddy 的 HTTPS 部署方案，具体请参考下一节。
 
 修改 `.env` 文件，设置下列参数：
 
 - `BASE_DOMAIN`: 基础域名，如 `local.homeinfra.org`，可以通过 `*.local.homeinfra.org` 访问镜像站。
 - `RPC_SECRET`：Aria2的RPC密钥。
-- `all_proxy`：代理服务器地址，如有必要。
-- `SCHEME`：`http` 或 `https`。
+- `*_proxy`：代理服务器地址，如有必要。
 
 配置完成之后，执行以下命令：
 
@@ -81,7 +90,7 @@ docker-compose up
 
 #### HTTPS
 
-在 .env 中配置 `SCHEME=https` 与 CLOUDFLARE_DNS_API_TOKEN。
+在 .env 中配置 CLOUDFLARE_DNS_API_TOKEN。
 本项目提供了一个基于Cloudflare DNS的 `Caddyfile` 和 `Dockerfile`。如果您希望使用其他DNS Provider或者LB，请自行修改。
 
 配置完成后，使用以下命令代替上述的`docker-compose up` (多了 `-f docker-compose-caddy.yml`) ：
@@ -109,22 +118,18 @@ LightMirrors依赖于两个组件：
 
 docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
 
-> 请注意：当 `SCHEME=http` 且 `DOCKER_BUILDKIT=1` 时，
-> Dockerfile 中的 `FROM docker.local.homeinfra.org/xxx` 语法默认将从 https 站点拉取镜像，
-> 此时将会出现错误。请使用 `docker pull`代替，或者尝试设置环境变量 `DOCKER_BUILDKIT=0`
-
 ### PyPI
 
-- https: `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/`
-- http: `pip install jinja2 --index-url http://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
+- `pip install jinja2 --index-url https://pypi.local.homeinfra.org/simple/ --trusted-host pypi.local.homeinfra.org`
+
+> 当使用自签名证书时，需要添加 `--trusted-host`。
+
 
 ### PyTorch
 
-- https: `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
-- http: `pip install torch --index-url http://torch.local.homeinfra.org/whl/ --trusted-host torch.local.homeinfra.org`
+- `pip install torch --index-url https://torch.local.homeinfra.org/whl/`
 
 把`download.pytorch.org`替换为 `torch.local.homeinfra.org` ，
-如果使用的是http，还需添加 `--trusted-host torch.local.homeinfra.org`。
 
 > 可以根据不同的硬件类型，切换不同的索引，如 https://download.pytorch.org/whl/cpu ，其中 `cpu`
 > 可以替换为cu116/cu118/cu121/rocm5.4.2 等等。
@@ -134,8 +139,7 @@ docker pull 的时候添加前缀 `docker.local.homeinfra.org` 即可。
 
 npm 命令后加上 `--registry https://npm.local.homeinfra.org` 即可。
 
-- https: `npm install -S express --registry https://npm.local.homeinfra.org`
-- http: `npm install -S express --registry http://npm.local.homeinfra.org`
+- `npm install -S express --registry https://npm.local.homeinfra.org`
 
 ## Star History
 

docker-compose-caddy.yml

@@ -8,21 +8,21 @@ services:
       - ./data/cache:/app/cache
     env_file:
       - .env
+    environment:
+      - SSL_SELF_SIGNED=false
     networks:
       - app
     restart: unless-stopped
     # for linux
     extra_hosts:
       - "host.docker.internal:host-gateway"
-    ports:
-      - "80:80"
   caddy:
     image: lightmirrors/caddy
     build:
       context: caddy
       dockerfile: Dockerfile
-
     ports:
+      - "80:80"
       - "443:443"
     volumes:
       - ./caddy/Caddyfile:/etc/caddy/Caddyfile

docker-compose.yml

@@ -8,6 +8,8 @@ services:
       - ./data/cache:/app/cache
     env_file:
       - .env
+    environment:
+      - SSL_SELF_SIGNED=true
     networks:
       - app
     restart: unless-stopped
@@ -16,6 +18,7 @@ services:
       - "host.docker.internal:host-gateway"
     ports:
       - "80:80"
+      - "443:443"
   aria2:
     image: lightmirrors/aria2
     build: ./aria2

src/Dockerfile

@@ -3,8 +3,7 @@ FROM python:3.11-alpine
 RUN mkdir -p /wwwroot
 WORKDIR /wwwroot
 
-# Optimization for China as the project is aimed at Chinese users
-ADD https://hub.gitmirror.com/https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
+ADD https://github.com/mayswind/AriaNg/releases/download/1.3.7/AriaNg-1.3.7.zip /wwwroot/
 
 RUN unzip AriaNg-1.3.7.zip && rm AriaNg-1.3.7.zip
 

src/certs/certificate.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
+BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
+MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
+cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
+dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
+8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
+7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
+vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
+/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
+ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
+ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
+m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
+rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
+nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
+0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
+NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
+xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
+-----END CERTIFICATE-----

src/certs/certificate.pem

@@ -0,0 +1,47 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
+PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
+xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
+7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
+rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
+XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
+5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
+2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
+53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
+EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
+h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
+S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
+eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
+drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
+z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
+sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
+0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
+2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
+JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
+q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
+Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
+tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
+BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
+029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
+BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
+1UswIYVccfrbHGd7X5vxE4yY
+-----END PRIVATE KEY-----
+-----BEGIN CERTIFICATE-----
+MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
+BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
+MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
+cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
+dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
+8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
+7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
+vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
+/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
+ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
+ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
+m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
+rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
+nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
+0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
+NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
+xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
+-----END CERTIFICATE-----

src/certs/csr.pem

@@ -0,0 +1,15 @@
+-----BEGIN CERTIFICATE REQUEST-----
+MIICZTCCAU0CAQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMIIB
+IjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzStc22z67EbnI7YuBtcQEtEb
+XSOE2YM4RLIqrF7c1M1Eqv+Ekp51uq3NCreBi17T4+8/Iw7aNSmts3Et8AFFq8kh
+ehdwvwH0ZcuoVmkGQlFU5KeLH38Sgp/2pCWi8XjX5VVv67tcYGzVy3ATqeGRMU2y
+EyJvFMy3Uc1aMKw4hofSBfiGLT0Kb93jarny753QynSuLSzixvQil/+6eenagzOz
+VUNeqVBfJUIovGaeDIqMQfNPC0LaWEV+w0Us+TxYCYRa06xz5wq8JGbirx3quU+B
+mRiKPnd16p28m+V4LZiu5OWVfGo3+X7Xxizke0hcCyc1bP671xssA383EAyR6wID
+AQABoAAwDQYJKoZIhvcNAQELBQADggEBAAU7hmM0/R8jyZLlG8D2JMdakzcy62HW
+QYBC9nivStBN7R2oZFZ63egRiCBC4sqjL2epr7QAuixLaDwLrlzq7nJczRqyw5/f
+X83Qo7+tLs4/4LXfVheWfDYYmKJja7lp72EmDPGBAnO4gb1bFLkYlrJU734gdtA+
+tOzNbwmD9pSLPIBGrI9it7gY4666TnwgtlYs6kIzhW9m3fh4XnhArAFWAmU1Z/Ud
+1ewnNCIsuHVa3vzPiL2ncN/IcIfDNdjkuk8kIh0QSFKHGRPk2wB02V8tWvITD4Rw
+kXIX/H30cdBQTsgdNDAm7xxXumcO2f2meiyfBWjO1nd1mDVIAvPsX/Q=
+-----END CERTIFICATE REQUEST-----

src/certs/generate.sh

@@ -0,0 +1,4 @@
+openssl genpkey -algorithm RSA -out private.key
+openssl req -new -key private.key -out csr.pem -config san.cnf
+openssl x509 -req -days 3650 -in csr.pem -signkey private.key -out certificate.crt -extensions v3_req -extfile san.cnf
+cat private.key certificate.crt > certificate.pem

src/certs/private.key

@@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDgSLxIjn71kp11
+PgXXixvFKKJVG/u+o0spx62+wUtl/imqKCRLliWEBP5BeF0JO7Ru6YRGpTfot1bx
+xvoTkO9pNWdEZkiM4KXljsbcLDefft/K6taGELwucN6uLkWtTjdupE0t0fWZyhjv
+7xoWt07Oh5PHgUQkwfGL80Sasm9fq3fttEtsuIP6DWumCc+wIBSkMSfqmxI3tJ68
+rEV6qdx2QeNLlLtSPWBaVxXbVKJrgXZ2QtAGt8yDjIvfC76rptxOlP0TULVgHPz+
+XbcJ/auYtPRzwbxI0xulf2cEHlXvLRCQBTelIvwvUJGekuYp+xE6wgfXCjehZQpm
+5Y0hSUENAgMBAAECggEAFUWGE1JDI6Kv1mJ3MWCF1hGAMUGXLcHVAMsUZRwgvN/5
+2aPhqrLnL0897L2wAo6BaBIzs8jltZJLPQvEb2ihRuwutXkHS9KJSW1c2khDT5g7
+53CVv19aSa6gyrT5chy8pdIQE9bETBKz13Bo8VtErn0t/I+mWTTKZkiV6soeRVy+
+EiOzCMVae7vKtCs6OuStfaVMGsoR5lOQPs0AzUUYslwxA6lpfet+ydNDYSVUOlep
+h0D2GJTzpYgg3lb70D/JPmjS4N/Qs4MthkqwVQkpyCqR1UelLc01pEn0CpR3/lt0
+S/3hmddz2truT0mIavZeoKPQXoMxNnOtGaBk+5JwDQKBgQD39zKG4iaMe1XQ4oQB
+eembxJeO8KZAYq4p+dzFcC6qUY5EzaV5suiLeLj/rEFVkWYxAHxDg7Bn0vtyxLE1
+drbSYl67G/nXVx0YXjMeFSVrZUuw/mT0Dcv/vBIpn6FdKY/avwjTEngAlx5I68iV
+z+w+DiD55bAtZekJAQ1eYJLblwKBgQDnjRtmJobLeh7XKHyaw+IwdGaiDCrM2JNb
+sOXk9qW6miDUYiFKkta3scxUk/PYkburbUxq0BYdZ2NZ4iP4usStQmJwTKYO+nc/
+0fOoI1ZbW0BTC/l9nDKVXP+PmxSDd4izMkYzoKtFFefah6k524BWXansQLHKrtL+
+2/lFcqIs+wKBgQCg4OkPSjroejy7QbuTmZ1PSsVqIAg9TcjhwxLRpm8gYbmEOkha
+JRFrK/AbMY9SDPvYXxBAXXheZCxv0d2HanZDGT+M9iGq7QIbhUJ1ChM1EouBUgv9
+q9XnHgZKqx7uasDCk58OErV47JLhjNifGj+UI2aRu8wKKEyBYwVhbhbaEQKBgQC/
+Twt7K4ulJCHE/LNunzcLd7CwZEMcejfNwus+cDZfZSaIswb0eTukC+dZXeBTZ5zZ
+tvUzLveGlbw1MC0uPQlwaO+x/vIYYAIUnZ+lpISd+Fe+BZ+9k0cLtwmDGcsqD6HH
+BFuSHkZDTyT/naRIpD11ne9dvdnnLugvH4nlQiAR7wKBgC2N9cRlyng4K9hBfBTU
+029T3pGKHaXSWnW6tS30HBDSVlkdb4sDpjHo1IfOcP9C3errP7Xq2j5V4C6k1Z1y
+BnVKcB0v17ZbCoRwxCzpnVGS4cyTVIQClTWlBHvdwm5SfBlD9tnLTiq65y+2TLJu
+1UswIYVccfrbHGd7X5vxE4yY
+-----END PRIVATE KEY-----

src/certs/san.cnf

@@ -0,0 +1,12 @@
+[req]
+req_extensions = v3_req
+distinguished_name = req_distinguished_name
+
+[req_distinguished_name]
+
+[v3_req]
+subjectAltName = @alt_names
+
+[alt_names]
+DNS.1 = local.homeinfra.org
+DNS.2 = *.local.homeinfra.org

src/mirrorsrun/config.py

@@ -4,8 +4,9 @@ ARIA2_RPC_URL = os.environ.get("ARIA2_RPC_URL", "http://aria2:6800/jsonrpc")
 RPC_SECRET = os.environ.get("RPC_SECRET", "")
 BASE_DOMAIN = os.environ.get("BASE_DOMAIN", "local.homeinfra.org")
 
-SCHEME = os.environ.get("SCHEME", "http").lower()
-assert SCHEME in ["http", "https"]
+SCHEME = "https"
+
+SSL_SELF_SIGNED = os.environ.get("SSL_SELF_SIGNED", "true") == "true"
 
 CACHE_DIR = os.environ.get("CACHE_DIR", "/app/cache/")
 EXTERNAL_HOST_ARIA2 = f"aria2.{BASE_DOMAIN}"

src/mirrorsrun/proxy/direct.py

@@ -6,6 +6,7 @@ import httpx
 from httpx import Request as HttpxRequest
 from starlette.requests import Request
 from starlette.responses import Response
+from tenacity import retry, retry_if_exception_type, stop_after_attempt
 
 SyncPreProcessor = Callable[[Request, HttpxRequest], HttpxRequest]
 
@@ -54,6 +55,14 @@ async def post_process_response(
         return response
 
 
+@retry(
+    stop=stop_after_attempt(6),
+    retry=retry_if_exception_type(Exception),
+    reraise=True,
+    after=lambda retry_state: logger.warning(
+        f"retry {retry_state.attempt_number} {retry_state.outcome}"
+    ),
+)
 async def direct_proxy(
     request: Request,
     target_url: str,
@@ -74,6 +83,7 @@ async def direct_proxy(
             request.method,
             target_url,
             headers=req_headers,
+            timeout=30,
         )
 
         httpx_req = await pre_process_request(request, httpx_req, pre_process)

src/mirrorsrun/server.py

@@ -21,7 +21,7 @@ from mirrorsrun.config import (
     RPC_SECRET,
     EXTERNAL_URL_ARIA2,
     EXTERNAL_HOST_ARIA2,
-    SCHEME,
+    SCHEME, SSL_SELF_SIGNED,
 )
 
 from mirrorsrun.sites.npm import npm
@@ -29,6 +29,7 @@ from mirrorsrun.sites.pypi import pypi
 from mirrorsrun.sites.torch import torch
 from mirrorsrun.sites.docker import dockerhub, k8s, quay, ghcr, nvcr
 from mirrorsrun.sites.common import common
+from mirrorsrun.sites.goproxy import goproxy
 
 subdomain_mapping = {
     "mirrors": common,
@@ -40,6 +41,7 @@ subdomain_mapping = {
     "ghcr": ghcr,
     "quay": quay,
     "nvcr": nvcr,
+    "goproxy": goproxy,
 }
 
 logging.basicConfig(
@@ -129,8 +131,10 @@ if __name__ == "__main__":
     uvicorn.run(
         app="server:app",
         host="0.0.0.0",
-        port=port,
+        ssl_keyfile='/app/certs/private.key' if SSL_SELF_SIGNED else None,
+        ssl_certfile='/app/certs/certificate.pem' if SSL_SELF_SIGNED else None,
+        port=443 if SSL_SELF_SIGNED else 80,
         reload=True,  # TODO: reload only in dev mode
-        proxy_headers=True,  # trust x-forwarded-for etc.
+        proxy_headers=not SSL_SELF_SIGNED,  # trust x-forwarded-for etc.
         forwarded_allow_ips="*",
     )

src/mirrorsrun/sites/goproxy.py

@@ -0,0 +1,29 @@
+from starlette.requests import Request
+
+from mirrorsrun.proxy.direct import direct_proxy
+from starlette.responses import Response
+
+
+async def goproxy(request: Request):
+    path = request.url.path
+
+    sumdb_prefix = "/sumdb/sum.golang.org"
+    if path.startswith(sumdb_prefix):
+        sumdb_path = path.removeprefix(sumdb_prefix)
+        if sumdb_path.startswith("/supported"):
+            return Response(
+                content=b"",
+            )
+        target_url = "https://sum.golang.org" + sumdb_path
+        return await direct_proxy(
+            request,
+            target_url,
+        )
+
+    target_url = "https://proxy.golang.org" + path
+
+    return await direct_proxy(
+        request,
+        target_url,
+    )
+

src/poetry.lock

@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.7.1 and should not be changed by hand.
+# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
 
 [[package]]
 name = "annotated-types"
@@ -498,6 +498,21 @@ typing-extensions = {version = ">=3.10.0", markers = "python_version < \"3.10\""
 [package.extras]
 full = ["httpx (>=0.22.0)", "itsdangerous", "jinja2", "python-multipart", "pyyaml"]
 
+[[package]]
+name = "tenacity"
+version = "8.4.2"
+description = "Retry code until it succeeds"
+optional = false
+python-versions = ">=3.8"
+files = [
+    {file = "tenacity-8.4.2-py3-none-any.whl", hash = "sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2"},
+    {file = "tenacity-8.4.2.tar.gz", hash = "sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef"},
+]
+
+[package.extras]
+doc = ["reno", "sphinx"]
+test = ["pytest", "tornado (>=4.5)", "typeguard"]
+
 [[package]]
 name = "tomli"
 version = "2.0.1"
@@ -542,4 +557,4 @@ standard = ["colorama (>=0.4)", "httptools (>=0.5.0)", "python-dotenv (>=0.13)",
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.9"
-content-hash = "95fe42214f7987d04a55af59e60128b0a3e224b1b2abbdf962053703379a7160"
+content-hash = "5e3dd2b39fd601cef175cb5768d11ea71f4583d815cbe8d97e4b2214d0ee1e97"

src/pyproject.toml

@@ -10,6 +10,7 @@ python = "^3.9"
 fastapi = "^0.109.0"
 uvicorn = "^0.27.0.post1"
 httpx = "^0.26.0"
+tenacity = "^8.4.2"
 
 
 [tool.poetry.group.dev.dependencies]

src/requirements.txt

@@ -120,6 +120,9 @@ sniffio==1.3.0 ; python_version >= "3.9" and python_version < "4.0" \
 starlette==0.35.1 ; python_version >= "3.9" and python_version < "4.0" \
     --hash=sha256:3e2639dac3520e4f58734ed22553f950d3f3cb1001cd2eaac4d57e8cdc5f66bc \
     --hash=sha256:50bbbda9baa098e361f398fda0928062abbaf1f54f4fadcbe17c092a01eb9a25
+tenacity==8.4.2 ; python_version >= "3.9" and python_version < "4.0" \
+    --hash=sha256:9e6f7cf7da729125c7437222f8a522279751cdfbe6b67bfe64f75d3a348661b2 \
+    --hash=sha256:cd80a53a79336edba8489e767f729e4f391c896956b57140b5d7511a64bbd3ef
 typing-extensions==4.9.0 ; python_version >= "3.9" and python_version < "4.0" \
     --hash=sha256:23478f88c37f27d76ac8aee6c905017a143b0b1b886c3c9f66bc2fd94f9f5783 \
     --hash=sha256:af72aea155e91adfc61c3ae9e0e342dbc0cba726d6cba4b6c72c1f34e47291cd

test/docker-compose.yml

@@ -1,25 +1,25 @@
+version: "3.3"
 services:
-  test:
-    image: lightmirrors/test
-    build:
-      context: .
-      dockerfile: Dockerfile
+
+  docker_test:
+    image: docker:dind
     volumes:
-      - .:/app
-      - /var/run/docker.sock:/var/run/docker.sock
-    networks:
-      - lightmirrors_app
-    external_links:
-      - lightmirrors:aria2.local.homeinfra.org
-      - lightmirrors:docker.local.homeinfra.org
-      - lightmirrors:pypi.local.homeinfra.org
-      - lightmirrors:torch.local.homeinfra.org
-      - lightmirrors:npm.local.homeinfra.org
-      - lightmirrors:ubuntu.local.homeinfra.org
-      - lightmirrors:debian.local.homeinfra.org
-      - lightmirrors:proxy.local.homeinfra.org
-      - lightmirrors:github.local.homeinfra.org
-      - lightmirrors:alpine.local.homeinfra.org
-networks:
-  lightmirrors_app:
-    external: true
+      - ./scripts:/scripts
+    command: sh /scripts/docker.sh
+    network_mode: host
+    privileged: true
+
+  python_test:
+    image: python:3-alpine
+    volumes:
+      - ./scripts:/scripts
+    command: sh /scripts/python.sh
+    network_mode: host
+
+
+  golang_test:
+    image: golang:alpine
+    volumes:
+      - ./scripts:/scripts
+    command: sh /scripts/golang.sh
+    network_mode: host

test/mirrors_test.py

@@ -1,32 +0,0 @@
-import unittest
-
-from utils import call
-
-PYPI_HOST = "pypi.local.homeinfra.org"
-PYPI_INDEX = f"http://{PYPI_HOST}/simple"
-TORCH_HOST = "torch.local.homeinfra.org"
-TORCH_INDEX = f"http://{TORCH_HOST}/whl"
-
-
-class TestPypi(unittest.TestCase):
-
-    def test_pypi_http(self):
-        call(f"pip download -i {PYPI_INDEX} django --trusted-host {PYPI_HOST}  --dest /tmp/pypi/")
-
-    def test_torch_http(self):
-        call(f"pip download -i {TORCH_INDEX} tqdm --trusted-host {TORCH_HOST} --dest /tmp/torch/")
-
-    def test_dockerhub_pull(self):
-        call(f"docker pull docker.local.homeinfra.org/alpine:3.12")
-
-    def test_ghcr_pull(self):
-        call(f"docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine")
-
-    def test_quay_pull(self):
-        call(f"docker pull quay.local.homeinfra.org/quay/busybox")
-
-    def test_k8s_pull(self):
-        call(f"docker pull k8s.local.homeinfra.org/pause:3.5")
-
-    def test_nvcr_pull(self):
-        call(f"docker pull nvcr.local.homeinfra.org/nvidia/cuda")

test/runner.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 import os
 import subprocess
 from pathlib import Path
@@ -9,19 +10,29 @@ root_dir = Path(__file__).parent.parent
 def call(cmd):
     print(f">>  {cmd}")
     p = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+
     stdout, stderr = p.communicate()
     assert p.returncode == 0, f"Error: {stderr.decode()}"
     print(">>", stdout.decode())
     return stdout.decode(), stderr.decode()
 
 
-class SetupMirrors():
-    def __enter__(self):
-        os.chdir(root_dir)
-        call("docker-compose up -d")
-        return self
+services = [
+    "python_test",
+    "docker_test",
+    "golang_test"
+]
+
+os.chdir(root_dir)
+call("docker-compose up -d --force-recreate --wait")
 
-    def __exit__(self, exc_type, exc_val, exc_tb):
-        call("docker-compose down")
 os.chdir(test_dir)
-        return False
+
+try:
+    for service in services:
+        call(f'docker-compose up --force-recreate --exit-code-from {service} {service}')
+except Exception as e:
+    raise e
+finally:
+    os.chdir(root_dir)
+    call("docker-compose down")

test/scripts/certs/certificate.crt

@@ -0,0 +1,19 @@
+-----BEGIN CERTIFICATE-----
+MIIDJjCCAg6gAwIBAgIUPL5HwTzU1jkc1C8mPpvLTmhFreYwDQYJKoZIhvcNAQEL
+BQAwIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5mcmEub3JnMB4XDTI0MDcwNjAy
+MzMwMloXDTM0MDcwNDAyMzMwMlowIDEeMBwGA1UEAwwVKi5sb2NhbC5ob21laW5m
+cmEub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA4Ei8SI5+9ZKd
+dT4F14sbxSiiVRv7vqNLKcetvsFLZf4pqigkS5YlhAT+QXhdCTu0bumERqU36LdW
+8cb6E5DvaTVnRGZIjOCl5Y7G3Cw3n37fyurWhhC8LnDeri5FrU43bqRNLdH1mcoY
+7+8aFrdOzoeTx4FEJMHxi/NEmrJvX6t37bRLbLiD+g1rpgnPsCAUpDEn6psSN7Se
+vKxFeqncdkHjS5S7Uj1gWlcV21Sia4F2dkLQBrfMg4yL3wu+q6bcTpT9E1C1YBz8
+/l23Cf2rmLT0c8G8SNMbpX9nBB5V7y0QkAU3pSL8L1CRnpLmKfsROsIH1wo3oWUK
+ZuWNIUlBDQIDAQABo1gwVjA1BgNVHREELjAsghNsb2NhbC5ob21laW5mcmEub3Jn
+ghUqLmxvY2FsLmhvbWVpbmZyYS5vcmcwHQYDVR0OBBYEFPbvlkTvPGfym8Yaz70W
+m8XC54kqMA0GCSqGSIb3DQEBCwUAA4IBAQADhsPOxB3jq/w3ZcZdN+ur0oOoRAK1
+rDG+5BeJx3EN3sTHBYYBVYmPNQI4KcSrgNw8kMGZSa4VrHOoiqDT0OgXYcfXsfmM
+nGZQBvDmOrTMdEauY1oQLCvZxnu4GYSAnTQIim4j59qV8IfOcXw6l1JEtGxvJEbG
+0BUV4aOg52+W155ov4gZh+lnl5Yzc6qpUcj11yC9PNThXAifMoN/AqdtauKOpPmF
+NiHiOXNnMJfxh5QsJkWbLsfrFKQ4KMJRt5mdVCdsAKOQpISxu5TzVmgQsXgnGZc1
+xUavylzHxSg7lzJLs7DPFa4H4/xtJB5fwHGQz8QGFCdNeipyxtqA9zJo
+-----END CERTIFICATE-----

test/scripts/docker.sh

@@ -0,0 +1,37 @@
+set -ex
+export DOCKER_HOST=unix:///var/run/docker.sock
+
+cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
+
+dockerd&
+
+docker_ready() {
+  docker version >/dev/null 2>&1
+}
+
+max_wait_time=5
+elapsed_time=0
+
+# Wait for Docker to be ready
+while [ true ]; do
+  if docker_ready; then
+    echo "Docker is ready!"
+    break
+  else
+    echo "Waiting for Docker to start..."
+    sleep 1
+    elapsed_time=$((elapsed_time + 1))
+    if [ $elapsed_time -gt $max_wait_time ]; then
+      echo "Docker failed to start in $max_wait_time seconds!"
+      exit 1
+  fi
+  fi
+done
+
+docker pull docker.local.homeinfra.org/busybox
+docker pull ghcr.local.homeinfra.org/linuxcontainers/alpine
+docker pull quay.local.homeinfra.org/quay/busybox
+docker pull k8s.local.homeinfra.org/pause
+
+# https is required
+echo 'FROM docker.local.homeinfra.org/alpine' | docker build -

test/scripts/golang.sh

@@ -0,0 +1,9 @@
+set -ex
+
+cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
+
+go env -w GOPROXY=https://goproxy.local.homeinfra.org,direct
+
+go clean -modcache
+go mod init test
+go get golang.org/x/sys@v0.22.0

test/scripts/python.sh

@@ -0,0 +1,12 @@
+set -ex
+
+cat /scripts/certs/certificate.crt >> /etc/ssl/certs/ca-certificates.crt
+
+pip config set global.index-url https://pypi.local.homeinfra.org/simple
+pip config set global.trusted-host pypi.local.homeinfra.org
+pip download jinja2 --dest /tmp/pypi/
+
+
+pip config set global.index-url https://torch.local.homeinfra.org/whl
+pip config set global.trusted-host torch.local.homeinfra.org
+pip download tqdm --dest /tmp/torch/