To completely customize the #SwiftUI Toggle view's colors, you're going to have to reconstruct it. I show you one way to do this in the SwiftUI Views book.
In the example above, I'm using a custom ToggleStyle (ColoredToggleStyle) that reconstructs the Toggle view so I can customize the colors any way I want.
My patrons have access to the full code.
Full Code: https://www.patreon.com/posts/31036259
Free Book Sample: https://www.bigmountainstudio.com/free-swiftui-book