Even though the CLR doesn't preload assemblies until you use them, AFAIK the JIT will require the type references to be resolved once you enter the method, and therefore the 32-bit assembly will be preloaded and generate an exception in the 64-bit environment and vice versa.
What I recommend is that you force the .NET side to be 32-bit (this can be done with corflags after the build, or with the build configuration in VS), if you really want to use the same package. 32-bit applications can run very well on a 64-bit OS.