With the advent of deep networks and availability of open source datasets, single-view room layout estimation and planar reconstruction methods have seen significant progress in recent years. To improve accuracy and maximize generalizability, previous methods make significant simplification assumptions that might not capture the full spectrum of real-world environments. In this work, we present a framework that does not make such assumptions and can model general and complex room geometries. This is achieved by strategically segmenting layout planes using a two-step method, followed by a planar optimization with connectivity and Manhattan constraints. We evaluate against the state-of-the-art general layout estimation methods, on a dataset of diverse real-home photographs and show significant improvements, towards the goal of room layout estimation in the wild.