Uh, 3DSM works a little differently to Blender.
I am not familiar with the 'layer' blend type. Is that like overlay? O_o
Either way, you know when you go to the material settings, then you go over to the texture box on the right, and click to the Map To tab (where you set if its going to be a normal map, colour map, specular, etc etc). There is a drop down box just below (default value is Mix). He is saying to change that to layer (? not sure of blender equivalent? layer isnt standard). AFAIK that is what he is talking about.