Reverse proxy encoding troubles

[raldnor]

Hi all,

I am trying to reverse proxy an InfluxDB for Telegraf agents to connect to. When directing the Telegraf agents directly to the InfluxDB url (<ip>:8086) everything works as intended. When I put Openlitespeed inbetween as a reverse proxy I get this error back in the Telegraf agent:

2020-07-06T09:30:20Z E! [outputs.influxdb] When writing to [http://<REDACTED>]: 400 Bad Request: invalid byte in chunk length

I tried both http and https, same result.
When sniffing the traffic I can see the agent start with a GET request with a defined content length - this is accepted by the InfluxDB webserver:

User-Agent: Telegraf/1.14.5
Content-Length: 32
Authorization: Basic aW5mbHV4OmxhYXRtZWZsdXhlbg==
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip

All the next requests use chunked encoding and then the error appears as described above.

Can you help me figuring out what's going wrong here?

Thanks!

 

[Pong]

400 Bad Request error normally client side error which may break protocol. You can investigate client side. https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400

 

[raldnor]

Hi Pong,

Thank you, but it seems Openlitespeed modifies the headers somehow thereby breaking the communication between the InfluxDB client and server. Without Openlitespeed inbetween it works without problems.

 

[Pong]

You can log a ticket with us with tmp root ssh access, we can turn on debug on to check.

 

[wackym]

Hi,
I think this is related: https://github.com/litespeedtech/openlitespeed/issues/192
I also posted an easy example how to replace this...

 

[raldnor]

Hi Wackym, thanks a lot for confirming this issue! I did not respond to this thread because I know this issue exists for a long time and I do not want to give external parties root access to my servers.
Anyway, I downloaded the source and did a grep on 'X' and found the following code that might be of interest:

src/http/accesslog.cpp:

case REF_CONN_STATE:
            if (pSession->getStream()->isAborted())
                *pBuf++ = 'X';
            else if (pSession->getReq()->isKeepAlive())
                *pBuf++ = '+';
            else
                *pBuf++ = '-';
            break;

But this code is relevant for logging and I would have expected to find something in a http-handler. Can you verify that it is always an 'X' that is appended to the broken JSON?

 

[wackym]

Yes, always an X. I also looked at the source. It seems 'X' is often used as a placeholder/debugging. My conclusion was that somewhere a memcopy is done with wrong length...but I couldn't pinpoint it as I'm not familiar with the code.

 

[raldnor]

Yes, always an X. I also looked at the source. It seems 'X' is often used as a placeholder/debugging. My conclusion was that somewhere a memcopy is done with wrong length...but I couldn't pinpoint it as I'm not familiar with the code.

I would expect that something goes wrong in proxyconn.cpp (https://github.com/litespeedtech/openlitespeed/blob/master/src/extensions/proxy/proxyconn.cpp). There are quite some disabled instructions, both for debugging purposes and seemingly unfinished procedures. The code barely contains any comments, so it's hard to debug. With regards to the 'X', the X_Forwarded* headers might cause the error? BTW the commercial version of Litespeed seems to work flawlessly. Apparently the proxy codebase is not shared between the two.

 

[delonbest]

400 Bad Request: invalid byte in chunk length

The 400 (Bad Request) status code indicates that the server cannot or will not process the request because the received syntax is invalid, nonsensical, or exceeds some limitation on what the server is willing to process. It means that the request itself has somehow incorrect or corrupted and the server couldn't understand it. The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method . Therefore, it prevents the website from being properly displayed. The main thing to understand is that the 400 Bad Request error is a client-side error. The cause of a 400 error can be a wrongly written URL or a URL that contains unrecognizable characters. Another cause of the error might be an invalid or expired cookie. Also, if you try to upload a file that's too large. If the server is programmed with a file size limit, then you might encounter a 400 error.