(NewsNation) — Hundreds of government employees in San Francisco are working from home due to dangerous working conditions as criminal activity surrounds the San Francisco Federal Building.

Homeless people are camping out all around the complex, formally known as the Nancy Pelosi Federal Building. Open drug markets are not far away either, with a recent drug bust going down just outside the building.

The office of Speaker of the House Nancy Pelosi tells NewsNation that the safety of all workers at the San Francisco Federal Building has always been a priority, and new safety measures have been implemented.

Despite the new attention on criminal activity in the area, those who work there say they always have their guard up.

“What are the dangers coming through here? There’s all kinds of different dangers, depending on what time of day it is, what time of night it is and depending on what type of person, you know?” said downtown San Francisco worker Daisy Graziano.

While workers with over a dozen federal agencies are coming and going through the area, a memo sent out earlier this month stated Health and Human Services employees are allowed to work from home for the foreseeable future, citing conditions at the federal building.

“The Department of Health and Human Services has a role to play in this. The optics of telling their employees that they should stay home is terrible,” said Matt Dorsey, a supervisor representing the district the building falls under. “If there are concerns about federal employees, I would like to see the U.S. Marshals and Federal Protective Service make sure that the entire block is protected. We’re not talking about a small private enterprise or a small business, we’re talking about the government of the United States of America.”

Pelosi has already asked the U.S. attorney’s office to address tenant concerns at the Federal Building while also securing more federal resources for the city to fight the fentanyl crisis, which is making the streets dangerous.

There are no details about the enhanced security measures, but there appear to be fewer people hanging around the concrete ledges outside the building.

For now, those who have to continue dealing with the danger are expressing feelings of unfairness.

Jacqueline Freeman, an employee in the area, says, “There are a lot of agencies that work in the area, that do the same things that are at the same risk, yet we don’t have that option to work from home.”