End-to-end refers to the fact that we are asking the learning algorithm to go directly from the input to the desired output i.e. the learning algorithm directly connects the input end of the system to the output end.