It's not too late to take up a sport. If rugby is something you'd really enjoy, then go for it. Just make sure to exercise and practice, get plenty of rest, and to eat a healthy diet. If you do that, you should find yourself steadily improving at the sport.