Clean up robots.txt

What does this MR do?

Clean up robots.txt from repeats and unnecessary syntax. Reduced file size by 11 lines

Inspired by !31782 (merged)

I used https://www.google.com/webmasters/tools/robots-testing-tool for testing.

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content