End to End Arguments in System Design

The main difficulty of deciding what functions should be implemented closer to the application and what at a lower level seems to originate from the problem of deciding between a leaner system at the expense of burdening the application developer and a function-rich system that may not be worth the while.

The end-to-end argument is a design principle that argues that the implementation of specific functions should be pushed towards the higher-level layers of the system where it is ultimately needed. The rationale is that the application is what is ultimately responsible for ensuring the correctness of a certain task execution and unless the function is implemented at the lower-level perfectly, which is not very likely, the application can never be free of this burden and will eventually have to deal with this possibility.

But it is necessary to implement at least some functionality at the lower-level so that the application can properly complete a task with an expectable amount of delay or error frequency. It is important to decide where to draw the line because of this trade off between performance enhancement and effort put into low-level implementation. But it was interesting to see that implementing something at the lower level doesn't necessarily mean that it will be beneficial to the performance, as services not needing that specific functionality will still have to pay for it in a way.

Generally agreeing to the end-to-end argument, one question I want to ask is if there is a function in question, why not just implement it in the next layer? If it is possible to have multiple layers of abstraction that are not strictly hierarchical, we can avoid problems like some application being subject to some unwanted feature implemented in the lower level by directly interacting with the lower level layer or a different sibling layer. So another option in my opinion is to consider increasing the granularity of the abstraction. From this perspective, I think the problem of deciding where to implement a function is not much different from the problem of deciding what kind of layers are we going to have.