The structure of two new phases in the bismuth manganite system are reported. The phases were determined by electron diffraction studies of two oxygen-deficient bulk samples. The first phase, a minority component of bulk BiMnO2.94 forms a n = 2 Ruddlesden-Popper phase with space group Cmc21. The second phase, from bulk BiMnO2.99, is an orthorhombic structure with space group Pmn21 and a unit cell approximately equal to times the parent perovskite cell. Importantly both phases are non-centrosymmetric and offer further potential for multiferroic studies.