BearShare clients no longer able to download

[RickH]

I upload a lot of files, and lately I've been noticing that there haven't been any BearShare downloaders. I turned on some logging, and noticed quite a few instances of the following error:

2007-06-30 16:55:38,070 DEBUG [NIODispatcher] gnutella.Acceptor.processSocket - Dispatching new client connecton: /xx.xx.xx.xx
2007-06-30 16:55:38,086 DEBUG [NIODispatcher] gnutella.Acceptor.handleRead - Dispatching word: GET
2007-06-30 16:55:38,101 DEBUG [NIODispatcher] gnutella.ConnectionDispatcher.delegate - Handling dispatched word: GET in same thread
2007-06-30 16:55:38,101 DEBUG [NIODispatcher] gnutella.HTTPAcceptor.fatalIOException - HTTP connection error
java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(Unknow n Source)
at org.apache.http.impl.nio.reactor.SessionInputBuffe r.readLine(SessionInputBuffer.java:149)
at org.apache.http.impl.nio.codecs.HttpMessageParser. parse(HttpMessageParser.java:131)
at org.apache.http.impl.nio.DefaultNHttpServerConnect ion.consumeInput(DefaultNHttpServerConnection.java :96)
at org.apache.http.impl.nio.DefaultServerIOEventDispa tch.inputReady(DefaultServerIOEventDispatch.java:9 4)
at org.limewire.http.HttpChannel.handleRead(HttpChann el.java:107)
at org.limewire.nio.AbstractNBSocket$1.run(AbstractNB Socket.java:178)
at org.limewire.nio.NIODispatcher$NIOExecutorService. execute(NIODispatcher.java:1035)
at org.limewire.nio.AbstractNBSocket.setReadObserver( AbstractNBSocket.java:141)
at org.limewire.http.HttpIOReactor.connectSocket(Http IOReactor.java:135)
at org.limewire.http.HttpIOReactor.acceptConnection(H ttpIOReactor.java:152)
at com.limegroup.gnutella.HTTPAcceptor$4.acceptConnec tion(HTTPAcceptor.java:292)
at com.limegroup.gnutella.ConnectionDispatcher$Delega tor.delegate(ConnectionDispatcher.java:169)
at com.limegroup.gnutella.ConnectionDispatcher.dispat ch(ConnectionDispatcher.java:119)
at com.limegroup.gnutella.Acceptor$AsyncConnectionDis patcher.handleRead(Acceptor.java:733)
at org.limewire.nio.AbstractNBSocket.handleRead(Abstr actNBSocket.java:261)
at org.limewire.nio.NIODispatcher.processRead(NIODisp atcher.java:435)
at org.limewire.nio.NIODispatcher.process(NIODispatch er.java:756)
at org.limewire.nio.NIODispatcher.process(NIODispatch er.java:664)
at org.limewire.nio.NIODispatcher.run(NIODispatcher.j ava:845)
at java.lang.Thread.run(Unknown Source)


repeated regularly from a couple different addresses. To see if this was something new, I went back and ran a stable copy of 4.13.6 I've kept around for safety, and lo, two BearShare clients with those addresses quickly connected and resumed long-interrupted downloads.

Looks like some fairly recent change has broken backwards compatibility with BearShare. The 2 clients that have reconnected with the 4.13.6 so far are using BearShare Pro 5.2.4.1 and BearShare Pro 5.2.5.3, respectively, but I don't think the version is really significant. I normally get quite a few BearShare users per day, and haven't seen any at all for quite some time. I haven't noticed any obvious lack of other clients such as gtk, eTomi, Phex, etc., but there are so many and most are uncommon enough to be easy to miss. I get enough BearShare users that the lack was noticeable.

In fact, as I've been typing this, another BearShare user running BearShare 5.2.5.6 (non-pro) has jumped on and started downloading like crazy. Looks like the outage caused some pent-up demand. I guess I'll have to keep running 4.13.6 for now.

[zab]

I'm afraid the problem is either with Bearshare or with Sun. If you look at http://svn.apache.org/repos/asf/jakarta/httpcomponents/httpcore/trunk/module-nio/src/main/java/org/apache/http/impl/nio/reactor/SessionInputBuffer.java around line 149 you'll find:


CoderResult result = this.chardecoder.decode(
this.buffer,
this.charbuffer,
true);
if (result.isError()) {
result.throwException();
}


Which basically means that whatever the bears are sending cannot be parsed as asccii text by the Sun charset decoders. Before, we parsed the headers with our own code which was very lax.

[zab]

hmm, I think this might be related to their bitfield representation of available ranges. If you really want to help them, try sniffing out (with ethereal for example) what headers they send. My guess is that they're including non-ascii characters as part of that header.

@Aaron - If you want to help address this, coordinate with RickH a debugging session.. it'd help us to know what BS sends vs. what LW sees.

[Aaron.Walkhouse]

What's your port number, Rick?

[RickH]

Got it. Caught the following (sanitized) GET request with my packet sniffer:

0x0000 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x0010 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x0020 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x0030 xx xx xx xx xx xx 47 45 54 20 2F 75 72 69 2D 72 xxxxxxGET /uri-r
0x0040 65 73 2F 4E 32 52 3F 75 72 6E 3A 73 68 61 31 3A es/N2R?urn:sha1:
0x0050 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x0060 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x0070 20 48 54 54 50 2F 31 2E-31 0D 0A 48 6F 73 74 3A HTTP/1.1..Host:
0x0080 20 xx xx 2E xx xx xx 2E-xx xx xx 2E xx xx 3A xx xx.xxx.xxx.xx:x
0x0090 xx xx xx xx 0D 0A 55 73-65 72 2D 41 67 65 6E 74 xxxx..User-Agent
0x00A0 3A 20 42 65 61 72 53 68-61 72 65 20 35 2E 32 2E : BearShare 5.2.
0x00B0 35 2E 36 0D 0A 52 61 6E-67 65 3A 20 62 79 74 65 5.6..Range: byte
0x00C0 73 3D 30 2D xx xx xx xx-xx xx 0D 0A 58 2D 47 6E s=0-xxxxxx..X-Gn
0x00D0 75 74 65 6C 6C 61 2D 43-6F 6E 74 65 6E 74 2D 55 utella-Content-U
0x00E0 52 4E 3A 20 75 72 6E 3A-73 68 61 31 3A xx xx xx RN: urn:sha1:xxx
0x00F0 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x0100 xx xx xx xx xx xx xx xx xx xx xx xx xx 0D 0A 58 xxxxxxxxxxxxx..X
0x0110 2D 43 6F 6E 6E 65 63 74-69 6F 6E 2D 54 79 70 65 -Connection-Type
0x0120 3A 20 54 31 0D 0A 46 50-2D 31 61 3A 20 31 32 38 : T1..FP-1a: 128
0x0130 2C E9 5E B0 E7 88 76 F1-CA 51 8F 6D BF C8 AA 82 ,é^°çˆvñÊQ�m¿Èª‚
0x0140 6C 33 35 25 25 86 CC BF-7E C5 EE 58 96 B6 2D 7E l35%%†Ì¿~ÅîX–¶-~
0x0150 9E 21 D5 0D 0A 46 50 2D-41 75 74 68 2D 43 68 61 ž!Õ..FP-Auth-Cha
0x0160 6C 6C 65 6E 67 65 3A 20-56 45 51 53 59 58 53 57 llenge: VEQSYXSW
0x0170 4E 47 53 4E 4E 34 32 55-43 4B 58 36 55 33 55 36 NGSNN42UCKX6U3U6
0x0180 33 42 46 43 54 4B 46 46-0D 0A 43 6F 6E 74 65 6E 3BFCTKFF..Conten
0x0190 74 2D 44 69 73 70 6F 73-69 74 69 6F 6E 3A 20 69 t-Disposition: i
0x01A0 6E 6C 69 6E 65 3B 20 66-69 6C 65 6E 61 6D 65 3D nline; filename=
0x01B0 22 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx "xxxxxxxxxxxxxxx
0x01C0 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x01D0 xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xx xxxxxxxxxxxxxxxx
0x01E0 xx xx xx xx xx xx xx xx xx xx xx xx xx xx 22 0D xxxxxxxxxxxxxx".
0x01F0 0A 58 2D 46 65 61 74 75-72 65 73 3A 20 62 72 6F .X-Features: bro
0x0200 77 73 65 2F 31 2E 30 2C-20 71 75 65 75 65 2F 30 wse/1.0, queue/0
0x0210 2E 31 2C 20 66 77 61 6C-74 2F 30 2E 31 0D 0A 58 .1, fwalt/0.1..X
0x0220 2D 4E 6F 64 65 3A 20 xx-xx xx 2E xx xx xx 2E xx -Node: xxx.xxx.x
0x0230 xx xx 2E xx xx xx 3A xx-xx xx xx 0D 0A 58 2D 51 xx.xxx:xxxx..X-Q
0x0240 75 65 75 65 3A 20 30 2E-31 0D 0A 0D 0A ueue: 0.1....


I checked the log and verified that this request did blow up in the same way previously described. Note the spectacularly non-ASCII "FP-1a" header. Looks like it's encoded in some high-radix format that can output any byte > ' '. I also note that it's not tagged "X-FP-1a", which might have let the parser ignore it.

What the hell is an FP-1a header? Never heard of it, and only found a couple non-informative Google hits for it. Presumably goes with the following FP-Auth-Challenge header. The lack of "X-" prefix should indicate that it's a standard header. Or that the author was clueless...

I've never done anything with Java's NIO system. Is there some way to set a data filter on the channel to let you clean this crap out before it parses? Or a handler or extended parser class to skip over this header? 'Cause BearShare obviously isn't gonna get updated to fix this (not the versions people use, anyway), and it'll be a huge mess if LimeWire suddenly cuts all the thousands (millions?) of established BearShare users out of the network.

[RickH]

OK, I appear to have a working fix. It's crude and ugly, but as a proof of concept it does work. I've been using it to upload to a mix of several LimeWire and BearShare clients with 4.13.9 for the past couple hours without any problems.

Since HTTP requests are a few hundred bytes, they should always fit in one TCP packet, and be received by your system in one atomic chunk. So, if you increase the size of the AsyncConnectionDispatcher's ByteBuffer to big enough to hold the whole packet, you can pre-process the whole request's contents before dispatching the request keyword and going through the HTTP parser. This shouldn't cause any performance hit, since the whole packet will be sitting ready in the TCP receive buffer anyway.

In com.limegroup.gnutella.Acceptor.AsyncConnectionDis patcher.AsyncConnectionDispatcher():
super(RouterService.getConnectionDispatcher().getM aximumWordSize() + 1);
was changed to:
super(1024);

In void com.limegroup.gnutella.Acceptor.AsyncConnectionDis patcher.handleRead():
if(LOG.isDebugEnabled())
LOG.debug("Dispatching word: " + word);
buffer.limit(buffer.position()).position(i+1);
source.interestRead(false);
RouterService.getConnectionDispatcher().dispatch(w ord, client, true);
was changed to:
if(LOG.isDebugEnabled())
LOG.debug("Dispatching word: " + word);
// If this is a GET or HEAD request...
if (word.equals("GET") || word.equals("HEAD")) {
// Check for an FP-1a header...
String sRequest = new String(buffer.array(), 0, buffer.position());
int iFP1a = sRequest.toUpperCase().indexOf("FP-1A: ");
if (iFP1a >= 0) {
// And if found, stomp the contents to death, rendering it harmless
for (iFP1a += 7; iFP1a < buffer.position(); ++iFP1a) {
byte b = buffer.get(iFP1a);
if (b == '\r' || b == '\n')
break;
buffer.put(iFP1a, (byte)'x');
}
LOG.debug("Neutralized FP-1a header");
}
}
buffer.limit(buffer.position()).position(i+1);
source.interestRead(false);
RouterService.getConnectionDispatcher().dispatch(w ord, client, true);

It needs to be generalized and prettied up to be release quality, but I think the idea's sound. And worst case, if the request packet is somehow fragmented for some reason, you're no worse off than you are now.

This makes me wonder, though; just how finicky is the HTTP parser about strict ASCII contents? I worry about that <Content-Disposition: inline; filename="fubar.txt"> header. If the filename is "José's Bar Mitzvah.avi", will the accented e blow it up? Several of my shared files have slightly non-ASCII characters like accents in their names. Maybe the parser is more tolerant inside quoted strings?

[zab]

This makes me wonder, though; just how finicky is the HTTP parser about strict ASCII contents? I worry about that <Content-Disposition: inline; filename="fubar.txt"> header. If the filename is "José's Bar Mitzvah.avi", will the accented e blow it up? Several of my shared files have slightly non-ASCII characters like accents in their names. Maybe the parser is more tolerant inside quoted strings?

In this case we use java.net.URL.getFile() which afaik encodes any non-ascii characters in a URL-friendly way. (com.limegroup.gnutella.downloader.HTTPDownloader, connectHTTP(), towards the end of the method)

As far as BS is concerned.. can anyone verify if the same thing happens with versions prior to 5.2? If it does, then this header is most likely used when establishing an encrypted BS link. Aaron can correct me if I'm wrong, but afaik when in secure mode BS will only establish connections to other BS clients in secure mode. So, it was actually a bug that any uploads to such BS client worked previously.

[RickH]

Huh? You mean:
SimpleWriteHeaderState writer = new SimpleWriteHeaderState(
"GET " + _rfd.getUrl().getFile() + " HTTP/1.1",
headers,
But that's writing the GET line with the SHA1 URN, which is always safe anyway, e.g. "GET /uri-res/N2R?urn:sha1:32CAPITALLETTERSANDNUMERICDIGITS HTTP/1.1"

I meant the Content-Disposition: header that BearShare is adding to the request, such as:
GET /uri-res/N2R?urn:sha1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1
Host: xx.xxx.xxx.xx:xxxxx
User-Agent: BearShare 4.7.3.2
Range: bytes=0-xxxxxx
X-Gnutella-Content-URN: urn:sha1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-Connection-Type: Broadband
FP-1a: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
FP-Auth-Challenge: K6E5J2S45EX5YDB322HJW5MZJJEIGYC6
Content-Disposition: inline; filename="The name of the file being downloaded.rar"
X-Features: queue/0.1
X-Queue: 0.1

The requests I've captured don't appear to be doing URL encoding on the filename, or it would be using %20 instead of spaces. I'm seeing the literal filename from my system in the filename="" field. If that request read:
GET /uri-res/N2R?urn:sha1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1
Host: xx.xxx.xxx.xx:xxxxx
User-Agent: BearShare 4.7.3.2
Range: bytes=0-xxxxxx
X-Gnutella-Content-URN: urn:sha1:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
X-Connection-Type: Broadband
FP-1a: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
FP-Auth-Challenge: K6E5J2S45EX5YDB322HJW5MZJJEIGYC6
Content-Disposition: inline; filename="José's Bar Mitzvah.avi"
X-Features: queue/0.1
X-Queue: 0.1

would it be fatal?

As for which versions do it, every BearShare request I've capture so far that contains a GET command (for a specific file, not a browse) includes the Content-Disposition header, and some requests that contain a HEAD command do (4.7.3.2) and others don't (5.2.1.1, Lite 5.2.5.1), and one sometimes does, and sometimes doesn't (5.2.5.4). Argh. But I've only captured requests from one copy of BearShare from before version 5.2 so far, from a single user running BearShare 4.7.3.2.

I don't understand why BearShare is sending the server the Content-Disposition header in the first place. All the server cares about is the SHA1 URN from the GET line at the top. Aren't HTTP Content-Disposition headers normally sent from the server to the downloader in the response packet, to suggest the filename to save the file as?

[RickH]

As far as BS is concerned.. can anyone verify if the same thing happens with versions prior to 5.2? If it does, then this header is most likely used when establishing an encrypted BS link. Aaron can correct me if I'm wrong, but afaik when in secure mode BS will only establish connections to other BS clients in secure mode. So, it was actually a bug that any uploads to such BS client worked previously.
Oh, sorry, I get you now. You meant which BearShare versions include the FP-1a header? So far, every single BearShare request I've captured has included the FP-1a header, including from version 4.7.3.2. But that's the only non-5.2 version I've captured from so far.

I don't think it's a bug for BearShare to connect despite this header. BearShare is obviously choosing to go ahead with the connection, despite the fact that LimeWire is blowing it off and not authenticating or encrypting at all. I can't imagine that they'd miss the little detail that all LimeWire connections are working (prior to 4.13.7) despite not being authenticated; they have to be allowing them on purpose. I expect there's likely a BearShare configuration option for "try to authenticate but allow connection anyway" versus "require authentication".

I think this whole authentication thing is just BearShare's non-spoofable version of "isLimeWire()". It's up to the settings in the BearShare client what happens when the check fails because LimeWire blew it off.

[zab]

I'm not sure exactly what happens when bears go into their secure mode. Last I heard that mode was turned on.

Have you tried to browse host and download something from them? If they don't allow that, we really have no motivation to fix this. Even if they allow it, there are still several factors to consider:

* BS is no longer developed
* Its only possible to download 5.2 and older from third-party websites
* BS is under the control of an entity whose status as good Gnutella citizen is questionable
* even w/o the non-ascii symbols that headers looks malformed
* there isn't a way to fix this that does not look like a horrible hack.

My personal opinion is that they're out of luck. We'll announce the team's position on the issue sometime next week.

[RickH]

I'd be all for dropping support for them if I thought the current BearShare users would then upgrade en masse to LimeWire and be back to using a supported program. But I'm pretty sure what would actually happen is that many/most of them would simply not be able to download from me and many other servers suddenly, and wouldn't have a clue why, essentially killing off a large part of my user base.

Oh well, whatever. I suppose it's just as well that I've figured out a horrible hack that works for my own use, regardless. Yay, open source!

[Sam]

It should be possible to workaround by setting a CodingErrorAction of IGNORE or REPLACE for either onUnmappableCharacter or onMalformedInput on the CharsetDecoder. Doesn't seem like too terrible of a hack, so long as it doesn't have very bad consequences. Would be nice to keep the old BearShares on the network (especially if the coding errors also happen when they're uploading, due to possibly malformed content-disposition headers). Dunno if it's possible to selectively apply the CodingErrorAction to specific headers -- that'd be the best course of action, I think.

[zab]

Assuming uploading to limewires is something they do in the first place. Can we verify that the their number of upload slots has not been reduced? I remember hearing something along those lines.

Update: the specific dump does not contain 0D0A (\r\n) until the end of the non-ascii part, so it would get parsed and ignored properly by our downloading code. So the question boils down to are they allowed to upload at all.

[Aaron.Walkhouse]

It's not that bad. BearShare does that handshake all the time but the various secure
modes are rarely used. It mostly assures users they are connected to genuine
software which hasn't been hacked. No other servent has ever had a problem with
it before and it won't be too hard to take care of before 4.13 goes to the public.
Not to do so could split the network so it's worth the effort in this case.

The reason for the extra file info going back and forth is that some diagnostics
were never commented out when betas were released. They got used to readable
headers and decided to leave it in because it was easier to just leave alone.

It should be a trivial matter to just ignore or discard unused data.
Unlikely to create a performance problem.

[zab]

@Aaron:

its actually a rather involved change because we're using a third-party library so there's more runaround required than usual. So, could we please get a definite yes or no on whether bears upload to limes?

This is like "fixing the glitch" in office space sense. ;)

[Sam]

httpcomponents (the library we use) just went into alpha5 code freeze, so it's unlikely they'd add a new feature like this very quickly. We've added modifications to third-party libraries before, though, so it's not something difficult to do.

If the CharsetDecoder is exposed, it'd be trivial to do and won't even require a patch. Otherwise we'll have to do some modifications to either expose it or enable a fault-tolerant mode in the parser, to let some http errors slide.

[Sam]

It looks like Steffen found a very good workaround that requires no changing of the library, and should work completely. Next beta should fix it. See: https://www.limewire.org/jira/browse/CORE-225

[RickH]

Confirmed. Charset fix applied, horrible hack removed, BearShare clients resumed downloading without problems.

Glad there was an easy, clean fix for this after all. Good catch, Steffen. One of those "D'oh, why didn't I think of that?" fixes that's clearly obvious in hindsight. The parser barfs because the headers contain chars not in the assumed default ASCII charset? Tell it the default charset is 8-bit ASCII instead of 7-bit. Duh. :o

[Aaron.Walkhouse]

I see uploads to LimeWire daily over here. No problem with that.