Bipolar disorder and schizophrenia are highly heritable conditions that are associated with structural brain abnormalities. Although brain abnormalities are found in the well relatives of people with schizophrenia, the extent to which genetic liability relates to brain structure in either disorder is still unclear. This study sought to ascertain the effects of genetic liability to schizophrenia and bipolar disorder on white and grey matter volume in patients with these diagnoses and their well relatives. Seventy-one patients and 72 unaffected relatives were recruited for the study. Patients included those with schizophrenia from families affected by schizophrenia alone, those with bipolar disorder from families affected by bipolar disorder alone and those with bipolar disorder from families affected by both bipolar disorder and schizophrenia. Samples of unaffected relatives of each patient group were also recruited. Subjects underwent an MRI scan of the brain, which was analysed using optimised voxel-based morphometry (VBM). Grey and white matter volume was then related to a continuous measure of genetic liability based on a threshold-liability model. Genetic liability to schizophrenia was associated with decreased grey matter volume in dorso- (DLPFC) and ventrolateral prefrontal (VLPFC) cortices. The relationship remained after diagnostic status had been taken into account. Complementary white matter changes were also demonstrated. No relationship was demonstrated between a genetic liability to bipolar disorder and either white or grey matter volume. Genes that raise the likelihood of developing schizophrenia may exert their effects by diminishing grey matter volume in the DLPFC and VLPFC and their associated white matter connections. Genes for bipolar illness might have subtle effects on brain structure, which may need particularly large samples to detect.