The West defends NATO's "open-door policy" towards potential future members, while Moscow is demanding a cast-iron guarantee that the alliance will not expand further towards its territory.
The NATO allies have warned Russia that they would not compromise on the alliance's right to defend its eastern members to avoid further conflict in Ukraine.
However, the alliance invited Moscow to further talks on calming security concerns.
"There are significant differences between NATO allies and Russia on these issues," alliance Secretary General Jens Stoltenberg said on Wednesday.
Stoltenberg's statement came after talks with Russian envoys at NATO headquarters in Brussels.
Stoltenberg said that it would be impossible for the 30 NATO members to agree to Moscow's core demands for a new security order in Europe, and in particular added that Russia would have no veto on Ukraine's right to eventually join the alliance.
President Vladimir Putin's government has issued a series of demands for the West to rule out accepting new members like Ukraine, Georgia or Finland on its eastern flanks and demanded limits on allied deployments in former Soviet allies that joined NATO after the Cold War.
Stoltenberg said it was "positive" that the two sides had been able to sit down together to discuss arms control and "many other issues to prevent a new armed conflict."
Stoltenberg warned, however, that Russia would have no veto over any bid by Ukraine to join NATO and called for a "de-escalation" in Russia's military build-up on its neighbour's border.
'Moment of truth'
Russia's diplomatic mission to the alliance was withdrawn in October last year after eight of its staff were expelled on allegations of espionage.
But the former ambassador - now a deputy foreign minister - Alexander Grushko was back to confront Stoltenberg and US Deputy Secretary of State Wendy Sherman, having earlier this week described the meeting as a "moment of truth".
After the meeting, Sherman tweeted: "In today's NATO-Russia Council, I reaffirmed the fundamental principles of the international system and of European security: Every country has the sovereign right to choose its own path."
Russia denies that its massive troop build-up around already partially occupied Ukraine is a threat, but the deployment has forced Washington to engage with Moscow to head off fears of an all-out military confrontation.