Scores of fighters believed to be loyal to an Islamic State group-affiliated Boko Haram faction stormed the base in Gudumbali, in the Guzamala area of Borno state.
At least eight civilians were believed to have been killed, according to one local civilian militia member.
The attack is the latest against troops in recent weeks and comes as the government encourages the return of displaced people from the Borno state capital, Maiduguri, to Guzamala.
The authorities, which maintain the Islamist militants are severely weakened to the point of defeat, say the area is safe for people to return, despite continued attacks.
But aid agencies say people who have been living in makeshift camps are being persuaded to return to areas without basic services area -- and worse security.
President Muhammadu Buhari was elected in 2015 on a promise to defeat Boko Haram and is seeking a second term of office at polls in February next year.
An official of Guzamala local government area, of which Gudumbali is the headquarters, told AFP: "It is true Boko Haram have taken over Gudumbali this morning after pushing troops out of the military facility in the town.
"They are in full control of the town and the military facility," the official said, speaking from Maiduguri on anonymity for fear of government sanction.
A military source in Maiduguri added: "The terrorists attacked the base at about 7:50 pm (1850 GMT) and engaged soldiers in an intense battle that continued until early Saturday when troops were forced to withdraw."
The source, who asked to remain anonymous because he was not authorised to speak to the media, said he had no information about casualties.
But local civilian militia member Musa Ari said: "Most civilians were spared because the attack was targeted at the military base.
"So far eight civilians who were errand boys for troops were believed to have been killed in the attack."
The IS-backed faction led by Abu Mus'ab al-Barnawi has vowed to attack only "hard" military or government targets unlike those loyal to long-time leader Abubakar Shekau.
The Shekau faction has been behind the repeated suicide and bomb attacks on civilians as well as kidnappings in the nine-year conflict.
Ari said soldiers and residents fled Gudumbali to Damasak, some 80 kilometres (50 miles) away, on the border with Niger.
Others escaped south towards Gajiram, where nine soldiers were killed in a similar attack in June.
There was no immediate response from the Nigerian military.
Boko Haram captured Gudumbali in 2014 when swathes of territory were seized in northern Borno and declared their Islamic Caliphate.
The military reclaimed the town two years later in a regional fightback involving troops from neighbouring Chad, Niger and Cameroon.