That quite frankly isn't a good idea. Thumb drives are lost all too easily. It's fine that you have confidence you'll never lose it from your shirt pocket, but if you forget to take it out and run it through the washer, then what? What happens if you lose it, or it gets corrupted? Is that the ONLY place you have them stored? At the minimum, you should have those passwords encrypted in a file, and the drive encrypted also. But really, what's the practical difference in doing all that, vs using the same security precautions on your computers and devices?
Most of us use a password manager like 1Password. All my passwords are stored in that. The database is encrypted and accessible only if you know the password for 1Password. You can only get to 1Password if you know the user password for the computer or device. The password database, which I sync via iCloud, can't be accessed unless my iCloud account gets compromised, which requires a compromise of my 2-factor authentication among other measures. That just ain't gonna happen. These measures are superior in EVERY regard to just tossing them on a thumb drive and toting that around, even with best-practice measures for the thumb drive.