Preheat a smoker to 250°F. Heat the olive oil in a skillet over medium heat. Add the chopped mushrooms and cook, stirring occasionally, until tender and their released moisture has evaporated. Set them aside to cool.
In a large mixing bowl, combine the ground beef, egg, breadcrumbs, milk, salt, pepper, garlic powder, and onion powder. Add the cooled mushrooms and Swiss cheese. Mix gently until combined, taking care not to overwork the meat.
Shape the mixture into a compact loaf and place it on a smoker-safe tray. Smoke until the center reaches 160°F when checked with a meat thermometer.
While the meatloaf smokes, stir together the remaining BBQ sauce and brown sugar. Brush the glaze over the meatloaf during the last 20 minutes of cooking.
Remove the meatloaf from the smoker and let it rest for 10 minutes before slicing. This helps the slices hold together and keeps the cheese from spilling out too quickly.