Mies van der Rohe’s plans for an office tower in the City were scuppered by Prince Charles’ criticism and Margaret Thatcher’s fear of new public space – yet this tale has as much to do with shifts in public opinion against modernist architecture