In this work, we optimized both engine and platform. Our study reveals that with the growing complexity of LLM applications, the platform latency will be the major bottleneck. Our take being, instead of optimizing the local inference speed, the industrial research should focus more on simplifying the serving gateway and optimizing the platform.

In this work, we propose an easy-to-use federated learning framework for diverse computer vision tasks, including image classification, image segmentation, and object detection, dubbed FedCV.

In some examples, methods and systems to automatically create runtime environments are provided. For example, a method includes: receiving a request to create a runtime environment; automatically generating a cluster of nodes based on the request, wherein the cluster of nodes are configured to run one or more containerized applications for the runtime environment; automatically applying a manifest onto the cluster of nodes, wherein the manifest includes one or more configurations associated with the runtime environment; and automatically deploying one or more software products into the cluster of nodes.