Reverse proxy encoding troubles

#1
Hi all,

I am trying to reverse proxy an InfluxDB for Telegraf agents to connect to. When directing the Telegraf agents directly to the InfluxDB url (<ip>:8086) everything works as intended. When I put Openlitespeed inbetween as a reverse proxy I get this error back in the Telegraf agent:

Code:
2020-07-06T09:30:20Z E! [outputs.influxdb] When writing to [http://<REDACTED>]: 400 Bad Request: invalid byte in chunk length
I tried both http and https, same result.
When sniffing the traffic I can see the agent start with a GET request with a defined content length - this is accepted by the InfluxDB webserver:

Code:
User-Agent: Telegraf/1.14.5
Content-Length: 32
Authorization: Basic aW5mbHV4OmxhYXRtZWZsdXhlbg==
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip
All the next requests use chunked encoding and then the error appears as described above.

Can you help me figuring out what's going wrong here?

Thanks!
 
#3
Hi Pong,

Thank you, but it seems Openlitespeed modifies the headers somehow thereby breaking the communication between the InfluxDB client and server. Without Openlitespeed inbetween it works without problems.
 
#6
Hi Wackym, thanks a lot for confirming this issue! I did not respond to this thread because I know this issue exists for a long time and I do not want to give external parties root access to my servers.
Anyway, I downloaded the source and did a grep on 'X' and found the following code that might be of interest:

src/http/accesslog.cpp:
Code:
case REF_CONN_STATE:
            if (pSession->getStream()->isAborted())
                *pBuf++ = 'X';
            else if (pSession->getReq()->isKeepAlive())
                *pBuf++ = '+';
            else
                *pBuf++ = '-';
            break;
But this code is relevant for logging and I would have expected to find something in a http-handler. Can you verify that it is always an 'X' that is appended to the broken JSON?
 
#7
Yes, always an X. I also looked at the source. It seems 'X' is often used as a placeholder/debugging. My conclusion was that somewhere a memcopy is done with wrong length...but I couldn't pinpoint it as I'm not familiar with the code.
 
#8
Yes, always an X. I also looked at the source. It seems 'X' is often used as a placeholder/debugging. My conclusion was that somewhere a memcopy is done with wrong length...but I couldn't pinpoint it as I'm not familiar with the code.
I would expect that something goes wrong in proxyconn.cpp (https://github.com/litespeedtech/openlitespeed/blob/master/src/extensions/proxy/proxyconn.cpp). There are quite some disabled instructions, both for debugging purposes and seemingly unfinished procedures. The code barely contains any comments, so it's hard to debug. With regards to the 'X', the X_Forwarded* headers might cause the error? BTW the commercial version of Litespeed seems to work flawlessly. Apparently the proxy codebase is not shared between the two.
 
Top