ansible-test: CentOS 7: lots of 502s

[felixfontein]

Summary

Since CentOS 7 is using the PyPI proxy (#83226), there are a lot of failing pip installs with 502 errors, at least for CentOS 7 tests on AZP. (I don't use CentOS 7 on GHA, so no idea whether it's AZP specific.)

Examples:

• https://dev.azure.com/ansible/community.general/_build/results?buildId=111717&view=logs&j=1c0ec436-f679-5cbc-b174-d65c38486113&t=f2641880-6819-5fee-79e5-3b357eafd18e&l=7040
• https://dev.azure.com/ansible/community.docker/_build/results?buildId=111726&view=logs&j=76b3f021-16f2-5851-35d0-9a79204688ca&t=c9446ffc-6f39-505c-9326-74434a47398d&l=4010
• https://dev.azure.com/ansible/community.docker/_build/results?buildId=111726&view=logs&j=67891f2a-d414-5247-205a-71d1aab1d04b&t=17156c0b-ab0e-5977-e7cd-0d38a5140915

Error messages from the above (two of them are failing for the same package):

stderr:   HTTP error 502 while getting http://172.19.0.3:3141/root/pypi/+f/311/02eeb323fbe57/supervisor-3.4.0.tar.gz#sha256=31102eeb323fbe57384d6a3746beab8f9ae9db705cdbd04a0861d21e38907a45 (from http://172.19.0.3:3141/root/pypi/+simple/supervisor/)
  Could not install requirement supervisor<4.0.0 from http://172.19.0.3:3141/root/pypi/+f/311/02eeb323fbe57/supervisor-3.4.0.tar.gz#sha256=31102eeb323fbe57384d6a3746beab8f9ae9db705cdbd04a0861d21e38907a45 because of error 502 Server Error: Bad Gateway for url: http://172.19.0.3:3141/root/pypi/+f/311/02eeb323fbe57/supervisor-3.4.0.tar.gz
Could not install requirement supervisor<4.0.0 from http://172.19.0.3:3141/root/pypi/+f/311/02eeb323fbe57/supervisor-3.4.0.tar.gz#sha256=31102eeb323fbe57384d6a3746beab8f9ae9db705cdbd04a0861d21e38907a45 because of HTTP error 502 Server Error: Bad Gateway for url: http://172.19.0.3:3141/root/pypi/+f/311/02eeb323fbe57/supervisor-3.4.0.tar.gz for URL http://172.19.0.3:3141/root/pypi/+f/311/02eeb323fbe57/supervisor-3.4.0.tar.gz#sha256=31102eeb323fbe57384d6a3746beab8f9ae9db705cdbd04a0861d21e38907a45 (from http://172.19.0.3:3141/root/pypi/+simple/supervisor/)

or

  HTTP error 502 while getting http://172.24.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from http://172.24.0.3:3141/root/pypi/+simple/cffi/)
  Could not install requirement cffi!=1.14.3,>=1.14.2 from http://172.24.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from -c /tmp/ansible.8S0svP.test/constraints.txt (line 7)) because of error 502 Server Error: Bad Gateway for url: http://172.24.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz
Could not install requirement cffi!=1.14.3,>=1.14.2 from http://172.24.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from -c /tmp/ansible.8S0svP.test/constraints.txt (line 7)) because of HTTP error 502 Server Error: Bad Gateway for url: http://172.24.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz for URL http://172.24.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from http://172.24.0.3:3141/root/pypi/+simple/cffi/)

I'm not sure what exactly is happening, in particular since I cannot see the error log from the PyPI proxy container. Since this is happening a lot since that PR was merged, I wanted to report it here.

Issue Type

Bug Report

Component Name

ansible-test

Ansible Version

2.15
2.16

Configuration

-

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

-

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[ansibot]

Files identified in the description:

• bin/ansible-test

If these files are incorrect, please update the component name section of the description or use the component bot command.

[felixfontein]

Latest failures:

• https://dev.azure.com/ansible/community.general/_build/results?buildId=111827&view=logs&j=1c0ec436-f679-5cbc-b174-d65c38486113&t=f2641880-6819-5fee-79e5-3b357eafd18e&l=7044
• https://dev.azure.com/ansible/community.docker/_build/results?buildId=111839&view=logs&j=67891f2a-d414-5247-205a-71d1aab1d04b&t=17156c0b-ab0e-5977-e7cd-0d38a5140915
• https://dev.azure.com/ansible/community.docker/_build/results?buildId=111839&view=logs&j=76b3f021-16f2-5851-35d0-9a79204688ca&t=c9446ffc-6f39-505c-9326-74434a47398d

[webknjaz]

I retrieved some logs from the CI's pip install output:

stdout:

Requirement already satisfied (use --upgrade to upgrade): cryptography<3.4,>=1.3.0 in /usr/lib64/python2.7/site-packages (from -c /tmp/ansible.w1fvZJ.test/constraints.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): setuptools<45 in /usr/lib/python2.7/site-packages (from -c /tmp/ansible.w1fvZJ.test/constraints.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /usr/lib/python2.7/site-packages (from cryptography<3.4,>=1.3.0->-c /tmp/ansible.w1fvZJ.test/constraints.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography<3.4,>=1.3.0->-c /tmp/ansible.w1fvZJ.test/constraints.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/lib/python2.7/site-packages (from cryptography<3.4,>=1.3.0->-c /tmp/ansible.w1fvZJ.test/constraints.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/lib/python2.7/site-packages (from cryptography<3.4,>=1.3.0->-c /tmp/ansible.w1fvZJ.test/constraints.txt (line 11))
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/lib/python2.7/site-packages (from cryptography<3.4,>=1.3.0->-c /tmp/ansible.w1fvZJ.test/constraints.txt (line 11))
Collecting cffi!=1.14.3,>=1.14.2 (from -c /tmp/ansible.w1fvZJ.test/constraints.txt (line 7))

stderr:

Ignoring coverage: markers u"python_version > '3.7'" don't match your environment
Ignoring cryptography: markers u"python_version < '2.7'" don't match your environment
Ignoring urllib3: markers u"python_version < '2.7'" don't match your environment
Ignoring wheel: markers u"python_version < '2.7'" don't match your environment
Ignoring paramiko: markers u"python_version < '2.7'" don't match your environment
Ignoring requests: markers u"python_version < '2.7'" don't match your environment
Ignoring virtualenv: markers u"python_version < '2.7'" don't match your environment
Ignoring pyopenssl: markers u"python_version < '2.7'" don't match your environment
  HTTP error 502 while getting http://172.25.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from http://172.25.0.3:3141/root/pypi/+simple/cffi/)
  Could not install requirement cffi!=1.14.3,>=1.14.2 from http://172.25.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from -c /tmp/ansible.w1fvZJ.test/constraints.txt (line 7)) because of error 502 Server Error: Bad Gateway for url: http://172.25.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz
Could not install requirement cffi!=1.14.3,>=1.14.2 from http://172.25.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from -c /tmp/ansible.w1fvZJ.test/constraints.txt (line 7)) because of HTTP error 502 Server Error: Bad Gateway for url: http://172.25.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz for URL http://172.25.0.3:3141/root/pypi/+f/bcb/3ef43e58665bb/cffi-1.16.0.tar.gz#sha256=bcb3ef43e58665bbda2fb198698fcae6776483e0c4a631aa5647806c25e02cc0 (from http://172.25.0.3:3141/root/pypi/+simple/cffi/)

This suggests that our PyPI proxy is erroring out internally, somehow.

[sivel]

After looking through devpi, I think we need to add --replica-max-retries to the container.

When that flag was added, it came with this:

- add a new command line option ``--replica-max-retries``

  Under certain network conditions, it's possible for a connection from devpi
  to replicas (such as pypi) to be dropped, resulting in a 502 bad gateway
  being returned from devpi. When replica-max-retries is set to a number > 0,
  devpi will attempt to retry connections until the retry limit is reached.

Looking at how the HTTP requests are made, --replica-max-retries is respected by the mirroring capability as well.

[felixfontein]

I created a PR for that: ansible/pypi-test-container#9