I am calling a webservice on another machine usually about 1-5 times during the execution of an asp.net page (from VB.net) This works well in DEV and I am not able to reliably reproduce in our live environment but throughout the day I get the following exception :

Method Exception ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted WWW.XXX.YYY.ZZZ
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.InternalConnect(EndPoint remoteEP) at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Int32 timeout, Exception& exception) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.GetRequestStream() at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at blah blah blah...

I have exhausted google & bing - I think - I have tried implementing a fix I found on the internet, where I'm extending the default webservice proxy with the following :
this didn't work

I've also tried increasing MaxUserPort and TcpTimedWaitDelay in the registry, but it doesn't seem like it has helped.

Can anyone help me? I don't really understand this problem. I figure that this scenario, calling a webservice from a webpage would be a fairly common practice, and that I wouldn't run into errors which look like basic plumbing.

client is asp.net 3.5 / win2k3 r2.
server isas400 / apache tomcat 4

Imports System.Net
Imports System.Net.Sockets
Imports System.Threading
Partial Public Class lookup
    Inherits originalLookup
    Private Shared m_LastBindPortUsed As Integer
    Protected Overrides Function GetWebRequest(ByVal uri As System.Uri) As System.Net.WebRequest
        Dim webreq As HttpWebRequest = CType(MyBase.GetWebRequest(uri), HttpWebRequest)
        webreq.KeepAlive = True
        webreq.ServicePoint.BindIPEndPointDelegate = New BindIPEndPoint(AddressOf BindIPEndPointCallback)
        Return webreq
    End Function
    Public Shared Function BindIPEndPointCallback(ByVal servicePoint As ServicePoint, ByVal remoteEndPoint As IPEndPoint, ByVal retryCount As Integer) As IPEndPoint
        Dim port As Integer = Interlocked.Increment(m_LastBindPortUsed) 'increment
        Interlocked.CompareExchange(m_LastBindPortUsed, 5001, 65534)
        If (remoteEndPoint.AddressFamily = AddressFamily.InterNetwork) Then
                Return New IPEndPoint(IPAddress.Any, port)
        Else
                Return New IPEndPoint(IPAddress.IPv6Any, port)
        End If
    End Function
End Class

System.Net Verbose: 0 : [2744] WebRequest::Create(http://wpg400a:8080/axis/services/CustGa.CustGaHttpSoap11Endpoint/)

System.Net Verbose: 0 : [2744] HttpWebRequest#40664061::HttpWebRequest(http://wpg400a:8080/axis/services/CustGa.CustGaHttpSoap11Endpoint/#1130302560)

System.Net Verbose: 0 : [2744] Exiting HttpWebRequest#40664061::HttpWebRequest()

System.Net Verbose: 0 : [2744] Exiting WebRequest::Create()     -> HttpWebRequest#40664061

System.Net Verbose: 0 : [2744] HttpWebRequest#40664061::GetRequestStream()

System.Net Information: 0 : [2744] Associating HttpWebRequest#40664061 with ServicePoint#45637159

System.Net Information: 0 : [2744] Associating Connection#17943449 with HttpWebRequest#40664061

System.Net.Sockets Verbose: 0 : [2744] Socket#2148853::Socket(InterNetwork#2)

System.Net.Sockets Verbose: 0 : [2744] Exiting Socket#2148853::Socket()

System.Net.Sockets Verbose: 0 : [2744] Socket#2148853::Connect(249:8080#-110661139)

System.Net.Sockets Error: 0 : [2744] Exception in the Socket#2148853::Connect - Only one usage of each socket address (protocol/network address/port) is normally permitted www.xxx.yyy.zzz:8080

System.Net.Sockets Verbose: 0 : [2744] Socket#2148853::Connect(249:8080#-110661139)

System.Net.Sockets Error: 0 : [2744] Exception in the Socket#2148853::Connect - Only one usage of each socket address (protocol/network address/port) is normally permitted www.xxx.yyy.zzz:8080

System.Net.Sockets Verbose: 0 : [2744] Socket#2148853::Dispose()

System.Net Error: 0 : [2744] Exception in the HttpWebRequest#40664061:: - Unable to connect to the remote server

System.Net Error: 0 : [2744]    at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket6, Int32 timeout)

   at System.Net.PooledStream.Activate(Object owningObject, Boolean async, Int32 timeout, GeneralAsyncDelegate asyncCallback)

   at System.Net.Connection.CompleteStartConnection(Boolean async, HttpWebRequest httpWebRequest)

System.Net Error: 0 : [2744] Exception in the HttpWebRequest#40664061::EndGetRequestStream - Unable to connect to the remote server

System.Net Error: 0 : [2744]    at System.Net.ServicePoint.GetConnection(PooledStream PooledStream, Object owner, Boolean async, IPAddress& address, Socket& abortSocket, Socket& abortSocket6, Int32 timeout)

   at System.Net.PooledStream.Activate(Object owningObject, Boolean async, Int32 timeout, GeneralAsyncDelegate asyncCallback)

   at System.Net.Connection.CompleteStartConnection(Boolean async, HttpWebRequest httpWebRequest)

Unfortunately, that error can mean two different things depending on where its thrown from, which makes finding reliable information about it harder. When thrown from Socket.Bind, it means that you are trying to Bind to a port which is currently in use by another Socket. This is the problem that the workaround with BindIPEndPointDelegate is targetting. But when thrown from Connect, the error means something slightly different.

A TCP connection is really just a mapping from the four-tuple (local address, local port, remote address, remote port) to a particular socket. The operating system uses this mapping to route incoming data to the appropriate socket. When you close a TCP connection, the operating system does not allow you to reopen another connection with the same four-tuple immediately. This time (controlled by TcpTimedWaitDelay) helps ensure that any data from the old connection that arrives after you close the connection gets dropped, and not accidentally routed to a new connection.

Getting this error from Connect means that the Socket was successfully bound to a local port which was in this TCP_TIMEWAIT state, which prevents it from being connected to that specific remote address/port for a short amount of time. This generally only becomes a problem when an application is creating lots of short-lived connections to the same remote address/port. The solution is to reuse connections rather than closing them and creating new ones. Since you are setting HttpWebRequest.KeepAlive to true, its likely the server is closing the connections.

If you cannot get the server to support keep-alive, the only recourse is to retry requests that fail with this error. Eventually the TCP_TIMEWAIT will expire and you will be able to create a new connection.

=dave