Brigadier General Stephen was in a vulnerable position while trying to move back past Brigadier General Thomas Conway's troops of Stirling's brigade.
Stirling's men had just fallen into defensive position when Stephen's brigades withdrew from Birmingham Hill. Stirling tried to cover the brigade, in addition to Woodford's retreating troops, but himself was attacked unexpectedly. The 2nd Battalion of British Light Infantry and Jaegers who had been delayed from joining the fray due to swampy ground, now arrived to attack Stirling.
The fresh British regiments tore into the weary Stephen's flank, causing severe casualties.
The center of the American defense now lay just east of the Birmingham Road and was held by Conway's brigade who had two of the remaining cannon. Though the troops tried conspicuously to hold the center and were rallied by Sullivan, Stirling and Conway, the situation at the second line grew desperate.
Fighting here was the most intense of the day with the dreaded bayonet again coming into play.
Sullivan recalled that at Battle Hill, the second American defensive line, "The general fire of the line had lasted an hour and forty minutes." Veteran officers had never seen "so close and severe a fire." Sullivan summarized the action at Battle Hill:
On the right, where General Stephen was, it [the fighting] was long and severe, and on the left considerable. When we found the right and left oppressed by numbers and giving way on all quarters, we were obliged to abandon the hill we had so long contended for, but not till we had almost covered the ground between that and Birmingham meetinghouse with the dead bodies of the enemy.
Around 6 P.M., the Americans ammunition began to run out. The British gradually gained ground. The Americans started to crumble. A British captain summed it up Battle Hill from the Redcoat perspective:
The fire of Musquetry all this time was as Incessant & Tremendous, as ever had been Rember'd ... The Rebel Line incapable of further Resistance gave way in every part & fled with the utmost disorder — they were pursued closely but the fatigue of the day having been great & the Men encumbere'd with their Blankets, &C: it soon became necessary to halt & form...
The Americans gave way and started retreating southwest toward Dilworth. Sullivan tired to rally his lines at two more locations nearer to Dilworth, but was unsuccessful. The British kept on coming. (Fortunately for the Americans, a sizable element of Cornwallis's right wing were hampered by the dense woods in the area of Birmingham Meeting and were unable to press the attack even harder.)
The British tailed the Americans past Dilworth which was located about two miles southwest of Birmingham Meeting House and two and a half miles northeast of Chadd's Ford.