You need to exclude requests that already contain /upload/index/. This is most easily done by constraining the rule to requests for /photo/filename.xtn Is /index/ the name of a directory? Does it contain the files that were formerly in /photo/ alone?
You may or may not be able to do it in mod_alias (Redirect by that name, or rather RedirectMatch) depending on exactly what URLs are involved. If you need to convert to mod_rewrite, you'll also need to change all your existing redirects, because you can't combine these two mods without unintended consequences*. Happily there's a near-identical question posted within the last 48 hours
/photo and /photo/upload/index aren't really folders, they are calls to sections of /index.php that create the pages on the fly from the css, a base template html file that is located elsewhere, and the contents of a MySQL database. So until called they don't exist, so don't show up in the url as a filename, all you see in the browser is /photo and /photo/upload/index/
Would it help if I posted the rest of my .htaccess file?
For this situation it doesn't matter if the folders physically exist; it's just a question of what the URL looks like. But if you're doing stuff with php and a database, like a hand-rolled CMS, then surely you're using mod_rewrite already? If so, all redirects have to be expressed as RewriteRule.
Either way, if the filename you want to target is just /photo and that's all, then you need to express the pattern as /photo$ using a final anchor. This remains true whether you're using mod_rewrite or mod_alias (RedirectMatch, which uses Regular Expressions), and whether or not you're capturing anyting for reuse.