How to add a new referency style to endnote online

You can’t.  

If you have a local EndNote online Administrators, they have the ability to update Styles for their site if EndNote online access is provided thru an institutional Web of Knowledge subscription. If you have EndNote online access through your institution, you can contact your local Administrator to update existing Styles or upload new Styles.  

The administrator (if you have one) should see here for instructions on how to add a custom output style: 

https://www.myendnoteweb.com/help/en_us/ENW/ha_styles.htm