In a medium bowl, gently combine crab meat, panko, mayonnaise, red pepper, scallion, teriyaki sauce and ginger. Form into 8 small patties and place on a dish; refrigerate 1 hour.
Heat oil in a large nonstick skillet over medium-high heat. Add 4 crab cakes and cook 4 to 5 minutes, without moving, until well browned. Gently turn and cook an additional 4 to 5 minutes. Remove to a plate and keep warm. Repeat with remaining crab cakes.
Place a piece of frisée on bottom of each roll and top with a crab cake. Spread cut side of each top half with 1 tsp wasabi mayonnaise and place on top of crab cake. Serve immediately with lime wedges, if desired.