American guards fired tear gas, and palls of smoke rose over the embassy grounds.
An Associated Press reporter at the scene saw flames rising from inside the compound and U.S. soldiers on the roof of the main embassy building with their guns pointed at protesters.
A man on a loudspeaker urged the mob not to enter the compound, saying, "The message was delivered."
There were no reports of casualties. The State Department said all American personnel were safe and that there were no plans to evacuate the embassy. The government planned to send more troops to protect the compound.
The breach followed U.S. airstrikes on Sunday that killed 25 fighters of the Iran-backed militia in Iraq, the Kataeb Hezbollah. The U.S. military said the strikes were in retaliation for last week's killing of an American contractor in a rocket attack on an Iraqi military base that the U.S. blamed on the militia.
President Donald Trump blamed Iran for the embassy breach and called on Iraq to protect the diplomatic mission.
"Iran killed an American contractor, wounding many. We strongly responded, and always will. Now Iran is orchestrating an attack on the U.S. Embassy in Iraq. They will be held fully responsible. In addition, we expect Iraq to use its forces to protect the Embassy, and so notified!" he tweeted from his estate in Palm Beach, Florida.
By early evening, the protesters had retreated from the compound but set up several tents outside where they said they intended to stage a sit-in. Dozens of yellow flags belonging to Iran-backed Shiite militias fluttered atop the reception area and were plastered along the embassy's concrete wall along with anti-U.S. graffiti. American Apache helicopters flew overhead and dropped flares over the area.
Trump, who is spending the holiday week at his Florida home, is in "close touch" and receiving regular updates from his national security team, said White House press secretary Stephanie Grisham. She echoed the sentiment contained in Trump's tweet earlier Tuesday.
"As the president said, Iran is orchestrating this attack, and they will be held fully responsible," Grisham said in an emailed statement. "It will be the president's choice how and when we respond to their escalation."
The developments also represent a major downturn in Iraq-U.S. relations that could further undermine U.S. influence in the region and American troops in Iraq and weaken Washington's hand in its pressure campaign against Iran.
Iraq has long struggled to balance its ties with the U.S. and Iran, both allies of the Iraqi government. But the government's angry reaction to the U.S. airstrikes and its apparent decision not to prevent the protesters from reaching the embassy signaled a sharp deterioration of U.S.-Iraq relations.
Iraqi security forces made no effort to stop the protesters as they marched to the heavily fortified Green Zone after a funeral for those killed in the airstrikes. The demonstrators were allowed to pass through a security checkpoint leading to the area.
The marchers, many of them in militia uniforms, shouted "Down, down USA!" and "Death to America" and "Death to Israel" outside the compound, hurling water and stones over its walls. The mob set fire to three trailers used by security guards along the wall. AP journalists saw some try to scale the walls.
Others then smashed the gates used by cars to enter, and dozens pushed into the compound. The protesters stopped in a corridor after about 5 meters (16 feet), and were only about 200 meters away from the main building.
The sprawling embassy compound enjoys a prominent position on the banks of the Tigris River in the heart of the Iraqi capital. Resembling a fortified college campus, the complex is rimmed with thick blast walls and cylindrical watch towers, lending it the look of a modern-day castle.
Gates visitors use to enter the complex consist of an airlock-like vestibule fortified with heavy doors and bulletproof glass. Even if protesters breached the first set of doors, they would have to force past heavily armed military contractors and U.S. Marine guards and a second set of heavy doors before entering the main compound.
Numerous buildings are inside the walls, including dormitories for staff, well-stocked dining and recreation facilities, and a power station.
The protesters taunted the embassy's security staff, which remained behind glass windows in the gates' reception area. They hung a poster on the wall declaring "America is an aggressor" and sprayed graffiti on the wall and windows reading, "Closed in the name of the resistance."
"This is a victory in retaliation to the American airstrike. This is the initial retaliation, God willing, there will be more," said Mahmoud, a fighter with the Imam Ali Brigades who was carrying a black bag filled with electricity cables that he said he took from the reception area.
A video obtained by the AP showed militiamen trashing the reception area and taking away paperwork.
The embassy, on its Facebook page, urged American citizens not to approach the compound and "to review their personal security and emergency preparedness."
An Iraqi employee at the embassy told the AP that the embassy's security team had evacuated some local staff from a rear gate while others left by helicopters and the rest remained inside "safe" areas within the embassy. The employee spoke on condition of anonymity because of not being authorized to speak to journalists.
The U.S. ambassador was traveling outside Iraq at the time of the attack and planned to return, the State Department said.
Some commanders of militia factions loyal to Iran joined the protesters outside the embassy in a strikingly bold move. Among them was Qais al-Khizali, the head of one of the most powerful Iranian-backed Shiite militias in Iraq who is on a U.S. terror list, and Hadi al-Amiri, the head of the state-sanctioned paramilitary Popular Mobilization Units, the umbrella group for the Iran-backed militias.
Jaafar al-Husseini, a spokesman for Kataeb Hezbollah, said the protesters had no intention of storming the embassy. He told the AP that the sit-in will continue "until American troops leave Iraq and the embassy is closed."
The U.S. airstrikes — the largest targeting an Iraqi state-sanctioned militia in recent years — and the subsequent calls by the militia for retaliation, represent a new escalation in the proxy war between the U.S. and Iran playing out in the Middle East.
The attack also outraged the Iraqi government, which said it will reconsider its relationship with the U.S.-led coalition — the first time it has said it will do so since an agreement was struck to keep some U.S. troops in the country. It called the attack a "flagrant violation" of its sovereignty.