PDA

View Full Version : Problem in UTF-8 conversion


Sean.Wang
August 17th, 2007, 05:40 AM
Some problem may occur in QueryReply : parseResult2. The correctness of the code depends on integer index i. Sometime it may not be correctly maintained when UTF-8 conversion is not reversible.

Here's what I understand from reading the code. When parsing Responses in a query reply, i is updated according to the number of bytes in a response, which is equals 8+filename size+1+rawMeta size+1. The filename size is given by Response:nameByte and rawMeta size is by Response:extBytes. In fact, the code determines the byte filename nameByte1 by null terminators. It should be ok to update index i using the size of nameByte1 as filename size.

However, it's not the fact in the code. It first converts nameByte1 in query reply to get a UTF-8 Stringname and then convert name back to byte array nameByte.

If the length of nameByte1 is not equal to that of nameByte, the code fails for sure. It happens upon the following filename nameByte1 with length 28:
nameByte1 = {0x72,0x69,0x63,0x6B,0x20,0x72,0x6F,0x73,0x73,0x20 ,0x96,0x20,0x61,0x6C,0x6C,0x20,0x77,0x68,0x69,0x74 ,0x65,0x20,0x35,0x32,0x2E,0x77,0x6D,0x61}

UTF-8 String name converted from nameByte1 is of length 30. Convert name back to byte array nameByte:
nameByte = {0x72,0x69,0x63,0x6B,0x20,0x72,0x6F,0x73,0x73,0x20 ,0xEF,0xBF,0xBD,0x20,0x61,0x6C,0x6C,0x20,0x77,0x68 ,0x69,0x74,0x65,0x20,0x35,0x32,0x2E,0x77,0x6D,0x61 }

The above example gives nameByte1.length == 28 but nameByte.length == 30, which causes i is not correctly updated. It's from a query reply from a Bearshare client. I'm not sure if it's a problem for replies from other types of clients. Maybe it's better to let nameByte1.length to update i instead of nameByte.length. It's a minor problem and will recover after skipping some queryhit messages, though.

Just my 2 cents.

Sean

Sam
September 11th, 2007, 11:35 PM
Thanks for pointing this out, Sean. If you're a programmer, could you write a reproducible testcase that has the failure, and possibly a solution? We'll take a look into it.

Sam
September 14th, 2007, 07:58 PM
FYI, Sean, this is being worked on at: https://www.limewire.org/jira/browse/CORE-333 .