Like many small business owners around the country, Leo and Lydia Lee saw the earnings from their Los Angeles restaurant nosedive when the Covid pandemic hit.

They also had to endure anti-Asian incidents.

The Chinese-American couple had been running their Cantonese BBQ restaurant, RiceBox, in downtown L.A. since September of 2018. The majority of their customers came from businesses in the area, thanks to catering contracts and the lunch crowd.

When businesses shut their doors, the pair opted to stay open. They survived by continuing to offer take out and adding in delivery services. Still, business dropped by about 70%.

Then came the phone calls.

More from Invest in You:
Small businesses struggle to find workers as pandemic eases
PPP is out of money for most borrowers. What to know
As small businesses slowly recover, financial help becomes more targeted

"We got a lot of prank phone calls," Leo Lee said. "People would ask, 'Do you serve bats? Do you serve Covid?"

Another time, a customer pushed past the blocked front door, where orders were being taken instead of inside the establishment, coughed in the Lees' direction and walked back out.

"We were scared," Leo Lee said.

The pandemic has brought a twofold blow to Asian-American/Pacific Islander (AAPI) small business owners.

Just over 80% of small business owners reported negative effects from the pandemic and 44% have decreased the number of people they employ, according to a survey conducted from Sept. 28 to Nov. 30, 2020 by the Asian/Pacific Islander American Chamber of Commerce and Entrepreneurship. Almost 1 in 3 AAPI women business owners have experienced anti-AAPI sentiment because of the pandemic, the survey found.

"The rise in anti-Asian racism and violence has further damaged the small business owners during this global pandemic," said Chiling Tong, president and CEO of National ACE.

"We got a vaccine for Covid, but we don't have a vaccine for hate."

Hate crimes against Asians in the U.S. surged by 169% from the first quarter of 2020 to the first quarter of 2021, the Center for the Study of Hate & Extremism at California State University, San Bernardino, found. From March 19 to March 31, there were 6,603 incidents reported to Stop AAPI Hate, including verbal harassment (65%), shunning (18.1%) and physical assault (12.6%).

That has many Asian-Americans afraid: A recent Pew survey found that one-third feared someone might threaten or physically attack them.

Joanne Kwong, president of family run Pearl River Mart in New York, sees that fear every day in her employees. Known as the first Chinese-American department store when it opened 50 years ago, the business is now split into three smaller locations. They completely shuttered last March for a few months before two gradually reopened. In October, the company added a new business, Pearl River Mart Foods.

Since reopening, Kwong has had to adjust her employees' shifts so that they feel safe commuting to and from work. She has also changed store hours, closing earlier before it gets dark.

"We have all had incidents where people might have said something [racist] on the street," said Kwong, who is the daughter of Chinese immigrants from the Philippines.

"We have employees that have actually been pushed or spat upon."

In addition to the psychological toll, the pandemic has brought an economic impact. Business first fell about 90% and is now hovering at about 40% of where it was pre-pandemic, she said.

Kwong received a federal Paycheck Protection Program loan in the program's first round. Thanks to an administration error by her first bank, she has yet to be approved for the second round of PPP, which has now run out of money for most borrowers. Fortunately, Kwong also has applied for aid with a community financial institution, which is still able to submit new applications to the Small Business Administration.

"We are on pins and needles" she said. "When is that going to run out?"

Kwong said she is also fortunate that, unlike many AAPI business owners, her command of English is exellent.

In addition to a language barrier making the application process difficult, there are also cultural issues at play, particularly with two-thirds of the community being born outside of the U.S., said Bill Imada, founder and chairman of IW Group, a minority-owned advertising and marketing agency. When the pandemic hit, Imada started helping AAPI business owners navigate available aid.

"Many of these immigrants have never asked for any support from the government," said Imada, who is Japanese-American. "In some cases, they come from countries where they are worried about the government, they don't trust the banks."

Half of AAPI entrepreneurs applied for PPP last year and of those, 80% received a loan, according to the Small Business Majority.

Now that smaller banks are involved in PPP, he's seen more AAPI businesses getting loans. Plus, restaurant owners can apply for aid through the Restaurant Revitalization Fund and venues can apply for a Shuttered Venue Operators Grant. Economic Injury Disaster Loans are still available for those who qualify.

Private industry is also doing its part. For instance, GrubHub announced it is donating all its proceeds from its Donate the Change program in May, which is AAPI heritage month, to AAPI-owned restaurants. Yelp, which saw searches for Asian-owned businesses increase more than 3,000% year over year in February, has made it easier to find and support those businesses.

It's that outpouring of support and the unity within the AAPI community that has been the silver lining, said Kwong.

"The community has been very inspiring and is more cohesive in a way that I have never seen," she said.

SIGN UP: Money 101 is an 8-week learning course to financial freedom, delivered weekly to your inbox.

CHECK OUT: We paid off $250,000 in debt and grew our net worth to $800,000: Here's our best advice via Grow with Acorns+CNBC via Grow with Acorns+CNBC.

Disclosure: NBCUniversal and Comcast Ventures are investors in Acorns.