Elderly neighbours have said they are scared to go out alone after some tripped and fell on a road "pitted with potholes".