I am not sure I understand why you are using the RemotingSurrogateSelector. It would be helpful if you can share the exception you are getting. A shot in the dark: Please try nulling out SurrogateSelector before deserializing; i.e. Add
formatter.SurrogateSelector = null;
before formatter.Deserialize...
I hope this works