http mode should not be always using $HTTP_PROXY

[DrDaveD]

I saw a squid log where stashcp is using the squid.

172.28.23.112 - - [21/Apr/2023:11:28:26.372 -0500] "GET http://128.142.248.156/stashservers.dat?list=xroot HTTP/1.1" 200 1091 TCP_REFRESH_MODIFIED:HIER_DIRECT 107 "- -" "-" "stashcp/6.10.0"
172.28.23.112 - - [21/Apr/2023:11:28:26.388 -0500] "GET http://osg-kansas-city-stashcache.nrp.internet2.edu:8000/osgconnect/public/osg/testfile.txt HTTP/1.1" 200 289 TCP_REFRESH_MODIFIED:HIER_DIRECT 15 "- -" "-" "grab"

It must be honoring $HTTP_PROXY. That's fine for stashservers.dat and small, frequently used files such as testfile.txt, but larger files and low cache re-use can potentially thrash a squid, depending on usage. The OSDF is not supposed to be going through squid. I request that HTTP_PROXY be unset when doing data transfers.

[DrDaveD]

This is important, it needs to get fixed. It is overloading squids.

[matyasselmeci]

We added environment variables to disable the proxy in #38, we just need to modify our pilots to make use of it.

[rynge]

I can get that configured on Jetstream, but how would a regular site know that they need to set these variables?

[matyasselmeci]

We could modify the pilot startup scripts to have it be the default.

[bbockelm]

I'm fine with changing the default but we should really think about how to do this loudly -- and how to give sites tools to manage this.

In the past, we've had sites (SLAC comes to mind) where use of $http_proxy is mandatory for any HTTP traffic -- site security policy. To go from complying with policy to suddenly bypassing it (even though I've got strong opinions on the policy in the first place...) would be surprising for sites.

Within the OSG context, how about we do something like:

• Write a documentation page about how to manage this setting from a CE (or a section, perhaps here,
• Update to ternary logic (enabled / disabled / auto),
• Give notice as a prominent item in the release notes,
• Subsequent release switches the default.

[DrDaveD]

I think that it would be best to both modify the pilot and to change the default. Changing the pilot would be the most immediate impact, since it will take a lot longer for everybody's stashcp to get updated. On the other hand, there are probably also a lot of different pilot scripts in different VOs so it will also take a while for that to propagate.