[SM5] What other video type will be compatible?

Hello and once again, I am asking another question.
I don't know but it is an AVI type that is compatible for a video background for a song.
However, when I made an AVI video for my song, it seems like it is extremely a large file (a 2-minute video sized up to 800 MB).
Are there any video types that would be compatible for a video background in SM?
Or, if it's only AVI, how will I decrease the size of my video? (I'm using Vegas 10 for video making)

edit:
Tried the MP4 format, but SM does not recognize it even after I reloaded the songs.

From SM5 beta 2 onward, acceptable movie formats include:
.avi, .mpg, .mkv, .mp4, .mov, .flv

If you've scripted one type of video into your chart and then want to change it, you need to update your chart to use the new video.
You can do this in StepMania's editor by pressing the B key.

The alternative is to manually edit your .ssc file with a text editor, but that requires some knowledge of the file format...

Or, if it's only AVI, how will I decrease the size of my video? (I'm using Vegas 10 for video making)
You have likely exported your video file uncompressed, which is why it's so huge.

As with all file types, you should be able to change your encoding settings when you export/render your file as .avi.
I've never used Sony Vegas before so I can't give you exact instructions I'm afraid. You'll probably have to poke around or consult Sony Vegas' manual to find out how to.
But just to give an example of what settings you could use, if you've found out how to change them: I'm using XviD for the codec, and an average bitrate of 500-800 kbps depending on how long the movie is and how large I want the video's filesize to be. (There's more details to it then I've given here but this should be enough for you to get a working and small-sized file for StepMania).
Good luck :)