The COVID-19 pandemic significantly affected human mobility. This study examines the changes in activity-travel behavior over 23 months (from Jan 2020 to Nov 2021) and how these changes are associated with the socio-economic status (SES) at the block group level in North Carolina. We identified 5 pandemic stages with different restriction regimes: the pre-pandemic, lockdown, reopening stage, restriction, and complete opening stage. Using the block-group mobility data from SafeGraph, we quantify visits to 8 types of destinations during the 5 stages. We construct regression models with interaction terms between SES and stages and find that visit patterns during the pandemic vary for different types of destinations and SES areas. Specifically, we show that visits to retail stores have a slight decrease for low and medium SES areas, and visits to retail stores and restaurants and bars bounced back immediately after the lockdown for all SES areas. The results suggest that people in low SES areas continued traveling during the pandemic. Transportation planners and policymakers should carefully design the transportation system to satisfy travel needs of those residents. Furthermore, the results also highlight the importance of designing mitigation policies that recognize the immediate recovery of visits to retail locations, restaurants, and bars.