High-resolution microwave spectra of the ground state ²⁰Ne-¹⁴N₂, ²⁰Ne-¹⁵N₂, ²²Ne-¹⁴N₂, and ²²Ne-¹⁵N₂ Van der Waals complexes, involving rotational levels up to J = 4, are reported. Interpretation and assignment of the observed transitions were made by combining results of measurements and theoretical predictions of the MW line positions in terms of available empirical potential energy surfaces and of a new high-level ab initio potential energy surface. The deviations of the calculated MW spectra from those observed experimentally are more uniform for the ab initio potential surface than they are for the empirical potential surfaces, allowing for reduction of the deviations to within 0.07% for all isotopomers by a single-parameter scaling of the ab initio potential energy surface. The scaled Ne-N₂ interaction potential was used to predict the MW line positions for the transitions J'-J'' = 3-2, 4-3 for all species. A simple procedure is proposed to improve the ab initio results for atom-diatom systems on the basis of atom-atom interaction components.