Diabetes mellitus is a complex disorder characterized by insufficient insulin production or insulin resistance, which results in a lifelong dependence on glucose-lowering drugs for almost all patients. During the fight with diabetes, researchers are always thinking about what characteristics the ideal hypoglycemic drugs should have. From the point of view of the drugs, they should maintain effective control of blood sugar, have a very low risk of hypoglycemia, not increase or decrease body weight, improve β-cell function, and delay disease progression. Recently, the advent of oral peptide drugs, such as semaglutide, brings exciting hope to patients with chronic diabetes. Legumes, as an excellent source of protein, peptides, and phytochemicals, have played significant roles in human health throughout human history. Some legume-derived peptides with encouraging anti-diabetic potential have been gradually reported over the last two decades. Their hypoglycemic mechanisms have also been clarified at some classic diabetes treatment targets, such as the insulin receptor signaling pathway or other related pathways involved in the progress of diabetes, and key enzymes including α-amylase, α-glucosidase, and dipeptidyl peptidase-IV (DPP-4). This review summarizes the anti-diabetic activities and mechanisms of peptides from legumes and discusses the prospects of these peptide-based drugs in type 2 diabetes (T2D) management.