Walls will be knocked down but can be rebuilt

If they are from a dishonorable alliance or are persistently farming you and won't stop, then the are a few things to do to annoy them greatly and should stop them:

Hide your troops in sanctuary - in the castle
Keep resource levels below storehouse levels.  buy kingdoms of camelot gems A level 10 storehouse with level 10 shrinking powder keeps 2 million of each resource safe, excluding gold.buy cheap KOC gems

Keep building; buy kingdoms of camelot gems  don't leave the game - that's what they might want you to do.

Pikemen are awesome defenders so build lots of them along with heavy cavalry and archers.

Walls will be knocked down but can be rebuilt, so if you build these the attackers will lose troops attacking you and it may stop them.

If nobody likes them by the time you have grown enough and got plenty of heavy cavalry,  cheap kingdoms of camelot gems  pikes and archers they will have been farmed to oblivion so portal by them with all your cities, farm them and don't stop.