Create a React library with high performance

I am creating a JS library that can be included into a website via CDN. I have configured code splitting but the size of the library is growing and it impacts lighthouse score. If I set up lazy loading then the app becomes slow as downloading files dynamically takes time. I looked at using service workers to precache the assets but service worker should be hosted at the same domain as the website. Looking for someone who can help in solving this.