Read Timeout TIdTCPClient

[d7d1cd]

Good day. I use the TIdTCPClient component to send requests to the server and read the response. I know the size of the response for certain requests, but not for others.

When I know the size of the response, then my data reading code looks like this:

Code: Select all

IdTCPClient1->Socket->Write(requestBuffer);
IdTCPClient1->Socket->ReadBytes(answerBuffer, expectSize);

When the size of the response is not known to me, then I use this code:

Code: Select all

IdTCPClient1->Socket->Write(requestBuffer);
IdTCPClient1->Socket->ReadBytes(answerBuffer, -1);

In both cases, I ran into problems.
In the first case, if the server does not return all the data (less than expectSize), then IdTCPClient1 will wait for ReadTimeout to finish, but there will be no data at all in the answerBuffer (even if the server sent something). Is this the logic behind TIdTCPClient? It is right?

In the second case, ReadTimeout does not work at all. That is, the ReadBytes function ends immediately and nothing is written to the answerBuffer, or several bytes from the server are written. However, I expected that since this function in this case does not know the number of bytes to read, it must wait for ReadTimeout and read the bytes, who came during this time. For the experiment, I inserted Sleep (500) between writing and reading, and then I read all the data that arrived.
May I ask you to answer why this is happening?

[rlebeau]

Good day. I use the TIdTCPClient component to send requests to the server and read the response. I know the size of the response for certain requests, but not for others.

Why do you not know the size of all of the responses? What does your protocol actually look like? TCP is a byte stream, each message MUST be framed in such a way that a receiver can know where each message begins and ends in order to read the messages correctly and preserve the integrity of the stream. As such, messages MUST either include their size in their payload, or be uniquely delimited between messages. So, which is the case in your situation? It doesn't sound like you are handling either possibility.

When the size of the response is not known to me, then I use this code:

Code: Select all

IdTCPClient1->Socket->Write(requestBuffer);
IdTCPClient1->Socket->ReadBytes(answerBuffer, -1);

When you set AByteCount to -1, that tells ReadBytes() to return whatever bytes are currently available in the IOHandler's InputBuffer. If the InputBuffer is empty, ReadBytes() waits, up to the ReadTimeout interval, for at least 1 byte to arrive, and then it returns whatever bytes were actually received into the InputBuffer, up to the maximum specified by the IOHandler's RecvBufferSize. So it may still take multiple reads to read an entire message in full.

In general, you should NEVER set AByteCount to -1 when dealing with an actual protocol. -1 is good to use only when proxying/streaming arbitrary data, where you don't care what the bytes actually are. Any other use require knowledge of the protocol's details of how messages are framed.

In the first case, if the server does not return all the data (less than expectSize), then IdTCPClient1 will wait for ReadTimeout to finish, but there will be no data at all in the answerBuffer (even if the server sent something). Is this the logic behind TIdTCPClient? It is right?

Yes. When AByteCount is > 0, ReadBytes() waits for the specified number of bytes to be available in the InputBuffer before then extracting that many bytes into your output TIdBytes. Your answerBuffer will not be modified unless all of the requested bytes are available. If the ReadTimeout elapses, an EIdReadTimeout exception is raised, and your answerBuffer is left untouched.

If that is not the behavior you want, then consider using ReadStream() instead of ReadBytes(), using a TIdMemoryBufferStream or TBytesStream to read into.

In the second case, ReadTimeout does not work at all. That is, the ReadBytes function ends immediately and nothing is written to the answerBuffer.

I have never heard of ReadBytes() not waiting for the ReadTimeout. What you describe should only happen if there are no bytes available in the InputBuffer and the ReadTimeout is set to some very small value, like 0 msecs.

or several bytes from the server are written.

That is a perfectly reasonable outcome given you are asking ReadBytes() to read an arbitrary number of bytes between 1..RecvBufferSize inclusive, or read no bytes if the timeout elapses.

However, I expected that since this function in this case does not know the number of bytes to read, it must wait for ReadTimeout and read the bytes, who came during this time.

That is how it should be working, yes. And how it has always worked. So I suggest you debug into ReadBytes() at runtime and find out why it is not working the way you are expecting. Also, are you using an up-to-date version of Indy to begin with (or at least a version from the last few years)? And what platform are you running your code on when you are having this trouble?

[d7d1cd]

Why do you not know the size of all of the responses?

Because, in fact, I'm doing a survey of an electronic device. This device has its own network IP address and port. So, the device can respond to the same request in different ways, depending on its status. Strictly speaking, there can be two answers to some queries and they have different lengths. It is in these cases, when reading, I specify AByteCount = -1 to read any device response.

I have never heard of ReadBytes() not waiting for the ReadTimeout.

You're right! I was wrong. When specifying AByteCount = -1, I get one byte. As you said, if at least one byte arrives, it returns and ReadBytes() ends.

Also, are you using an up-to-date version of Indy to begin with (or at least a version from the last few years)? And what platform are you running your code on when you are having this trouble?

I work in C ++ Builder 10.3 Community Edition, Indy version 10.6.2.5366. I create an application for Windows.

If that is not the behavior you want, then consider using ReadStream() instead of ReadBytes(), using a TIdMemoryBufferStream or TBytesStream to read into.

Is the behavior of ReadStream() different from ReadBytes()? I thought these were just different ways of reading information from a socket...
Thank you invaluable help!

[rlebeau]

Because, in fact, I'm doing a survey of an electronic device. This device has its own network IP address and port. So, the device can respond to the same request in different ways, depending on its status.

That does not necessarily mean that there is not a structure to the responses. Again, what does the actual protocol look like?

Strictly speaking, there can be two answers to some queries and they have different lengths.

And? Just because they have different lengths does not necessarily mean there is no way to know what those lengths actually are every time.

It is in these cases, when reading, I specify AByteCount = -1 to read any device response.

Without knowing what the protocol actually looks like, I'm not convinced that is the best solution. In fact, it is rarely ever the correct solution.

You're right! I was wrong. When specifying AByteCount = -1, I get one byte. As you said, if at least one byte arrives, it returns and ReadBytes() ends.

OK.

[d7d1cd]

Again, what does the actual protocol look like?

A protocol similar to MODBUS. There is no criterion for ending the sequence of answers in it. Except for the criterion of cyclic redundancy code (CRC16) at the end of each response.
Although, probably, I confused you. The protocol of exchange with the device ensures that for each request there will be a response of a specific length. However, there may be the following situation. Before sending requests to the device, you must log in to the device. That is, you need to send a special access request. If this is not done, then the device will respond to any other request with the same sequence with the access error code. Therefore, before sending requests, of course, you need to open the communication channel and then the response from the device will be expected. Thus, I misled you. Of course, knowing the request and observing the order during the exchange (first authorization, then everything else), the length of the answers will be known.
Although there is one situation where the length of the response will not be known. This is if you create an application that allows the user to form queries to the device by simply typing the command in hexadecimal numbers. Here we can say that we need to analyze the request and determine how long the device should return, but what if the application is universal? What if it is designed simply to send requests and read responses?
And? Just because they have different lengths does not necessarily mean there is no way to know what those lengths actually are every time.

Allow another question about the operation of the ReadStream function.
Wrote the following code:

Code: Select all

TIdTCPClient* IdTCPClient1 = new TIdTCPClient();
IdTCPClient1->ReadTimeout = 5000;
TByteDynArray request, answer, tmp;
TBytesStream* bytesStream = new TBytesStream(tmp);
int expect;

// Some code

expect = 8;
try {
  IdTCPClient1->Connect();
  IdTCPClient1->Socket->Write(request);
  IdTCPClient1->Socket->ReadStream(bytesStream, expect);
} catch(...) {}

answer = bytesStream->Bytes;
answer.set_length(bytesStream->Size);

A request is sent to the Write function, to which the device gives 8 bytes of response. If expect = 8, then everything works fine. If expect = -1, then the ReadStream function, as expected, waits 5 seconds, but only 4 bytes are written to bytesStream, not 8. Why is this happening?
And one more thing: why do I need to pass a parameter to the TBytesStream constructor (in my tmp code)? Why can't you create an empty stream?

[rlebeau]

There is no criterion for ending the sequence of answers in it. Except for the criterion of cyclic redundancy code (CRC16) at the end of each response.

I seriously doubt that. There HAS to be a way to know what each message length is. The actual MODBUS TCP/IP protocol does exactly that. It uses a structure where every message begins with a fixed 7-byte header:

2-byte Transaction ID
2-byte Protocol ID
2-byte Length Field
1-byte Unit ID

The Length Field is the size of the MODBUS data that follows the header (the MODBUS Address and Checksum fields are not transmitted over TCP). So, a receiver would simply read the first 6 bytes, then read the number of bytes specified in the Length Field to finish the message (the length includes the Unit ID).

I strongly suspect you need to do something similar in your situation, but I can't be sure since you have not provides ANY details about the actual protocol you are dealing with. Saying it is "similar to MODBUS" is not the same as "actually being MODBUS". So what is different in this situation? Do you have ANY documentation about your device's actual transmission protocol?

The protocol of exchange with the device ensures that for each request there will be a response of a specific length. However, there may be the following situation. Before sending requests to the device, you must log in to the device. That is, you need to send a special access request. If this is not done, then the device will respond to any other request with the same sequence with the access error code. Therefore, before sending requests, of course, you need to open the communication channel and then the response from the device will be expected.

That does not change anything I have said so far. EVERY message back and forth has a layout to it. You need to write your code to follow that layout so you know where each message begins and ends.

Of course, knowing the request and observing the order during the exchange (first authorization, then everything else), the length of the answers will be known.

It is not common for a TCP-based protocol to dictate every request has a specific fixed-length response, like you describe. Most TCP-based protocol are more generalized than that, using a single unified format for every message. So again, I ask you, what are the specific details about the ACTUAL PROTOCOL that your device is really using? I can't help you fix your Indy code without that information.

Although there is one situation where the length of the response will not be known. This is if you create an application that allows the user to form queries to the device by simply typing the command in hexadecimal numbers. Here we can say that we need to analyze the request and determine how long the device should return, but what if the application is universal? What if it is designed simply to send requests and read responses?

Even if you allow the user to format their own queries, your code is still responsible for reading the responses before presenting them to the user. So your code is responsible for reading the responses correctly.

Allow another question about the operation of the ReadStream function.
...
A request is sent to the Write function, to which the device gives 8 bytes of response. If expect = 8, then everything works fine. If expect = -1, then the ReadStream function, as expected, waits 5 seconds, but only 4 bytes are written to bytesStream, not 8. Why is this happening?

The only possibility is that only 4 bytes were actually available after the timeout elapsed.

As I said earlier, using AByteCount=-1 returns whatever bytes are available at that very moment. If there are no bytes available yet, then it waits for the ReadTimeout to elapse and then returns whatever bytes are available.

Using AByteCount>0 instead, it actually waits for all of the requested bytes to arrive, however long they may take (subject to the ReadTimeout, of course).

I strongly suspect that using AByteCount=-1 is the WRONG solution to your situation.

why do I need to pass a parameter to the TBytesStream constructor (in my tmp code)?

Because that is just the way TBytesStream works. It is a wrapper for a byte array, so you have to give it an initial byte array to work with, even if it is just an empty one. TBytesStream does not have a parameter-less constructor. If you want that, use TMemoryStream instead.

Why can't you create an empty stream?

You can, if you give it an empty byte array. That is just the way TBytesStream was coded to operate.

[d7d1cd]

Do you have ANY documentation about your device's actual transmission protocol?

Yes, of course, there is a protocol, only it is in Russian. The device has 2 communication interfaces: an infrared transceiver, as well as a wired RS-485, work through which is carried out via the COM port. Access to the device via TCP is carried out through additional equipment that creates a “transparent” communication channel, due to which data exchange with the device via TCP is no different from working with it via RS-485 or infrared channel. I can provide you with a protocol translated through a Google translator.

The only possibility is that only 4 bytes were actually available after the timeout elapsed.
As I said earlier, using AByteCount=-1 returns whatever bytes are available at that very moment. If there are no bytes available yet, then it waits for the ReadTimeout to elapse and then returns whatever bytes are available.

That is, in fact, the behavior of ReadStream is no different from ReadBytes?

That is just the way TBytesStream was coded to operate.

OK.

[rlebeau]

Yes, of course, there is a protocol, only it is in Russian.

So what? Never heard of online translators, like Google Translate, Bing Translator, etc?

The device has 2 communication interfaces: an infrared transceiver, as well as a wired RS-485, work through which is carried out via the COM port. Access to the device via TCP is carried out through additional equipment that creates a “transparent” communication channel, due to which data exchange with the device via TCP is no different from working with it via RS-485 or infrared channel.

All of that information is irrelevant to this discussion. The only thing that matters right now is the format of the messages that are being transmitted over TCP. Please only focus on that.

I can provide you with a protocol translated through a Google translator.

Please do.

[d7d1cd]

Please do.

I am sending you the exchange protocol in the original. I wanted to send in the translated version, but I could not copy the translated text from the online translator.

[rlebeau]

I am sending you the exchange protocol in the original. I wanted to send in the translated version, but I could not copy the translated text from the online translator.

Here is an English PDF that I translated from the original using this Russian-to-English translator.

I'll dig into the PDF more after Christmas. But from a cursory glance, it appears that you are likely going to have to know ahead of time which exact response you are reading in order to know how many bytes the response contains. I think (I may be wrong) that the first byte is a sequence number that is copied from the request. So, you might have to keep track of the requests you send, and how many bytes each expected response is, then when reading a response read just the 1st byte and match it to your list of requests, and then read the expected number of bytes for that response.

This is not a very well-designed protocol for TCP usage.

[d7d1cd]

I think (I may be wrong) that the first byte is a sequence number that is copied from the request.

The first byte is the so-called network address. You probably already understood that the protocol describes an electric energy meter (counter). So, there can be many of these counters in one network (RS-485). To address a request to a specific one, its network address is indicated, which are programmed different for all counters on the same network.

So, you might have to keep track of the requests you send, and how many bytes each expected response is, then when reading a response read just the 1st byte and match it to your list of requests, and then read the expected number of bytes for that response.

Information exchange with the device is carried out synchronously. That is, if a request is sent, then you must immediately, given the timeouts, read the answer.

This is not a very well-designed protocol for TCP usage.

This protocol was not designed to work through the TCP. It is necessary to work through the TCP here in the same way as through the COM port in synchronous mode.

[rlebeau]

Information exchange with the device is carried out synchronously. That is, if a request is sent, then you must immediately, given the timeouts, read the answer.

Well, then there shouldn't be any problem, since you know what kind of request you are sending each time, so you should know what kind of response to expect and how many bytes it requires to read. If a timeout occurs, all you can do is close the connection and move on. So what are you actually having trouble with exactly?

This protocol was not designed to work through the TCP.

TCP wrappers do usually require adequate message framing, though. Which this protocol does not seem to provide.

It is necessary to work through the TCP here in the same way as through the COM port in synchronous mode.

Even COM port protocols usually use adequate message framing.