Last one I saw like this was a pug hdi engine where the mounting for the HP pump was very slightly bent. This was fixed with a lump hammer and a block of wood. Even the slightest mis-alignment will cause the belt to move as you describe. Have you tried a straight edge across the pulleys to figure out if any of them are out of line? While running it may be possible to lever pulleys to see the effect it has on the run of the belt. Check end float in the crank maybe?