The investigation could result in Salford being charged with a breach operational rules, which if proven would expose the club to a range of potential sanctions.
A statement read: "The RFL have today contacted Salford Red Devils to confirm that a compliance investigation will be launched to consider the circumstances around their squad selection for Saturday's Betfred Super League fixture at St Helens, and whether they have brought the sport into disrepute."
Due to salary cap restrictions, the Red Devils were denied the use of a number of senior players for the match, but head coach Paul Rowley still chose not to start with other, more experienced squad members who were available.
Saints' 15-try romp was the biggest winning margin in Super League history and drew criticism from rivals including Warrington head coach Sam Burgess, who said that it had compromised the integrity of the competition.
RFL chiefs met on Monday morning and determined the club could have a case to answer over a potential breach of competition rules, which require that each club "has at all times a squad of players sufficient to meet its obligations in all competitions".
Salford announced the completion of a takeover on Friday, but it is understood that the details are still to be sufficiently finalised for the RFL to lift their current restrictions.
Barring further developments, they could find themselves under the same restrictions for their next match, against Leeds on Saturday.
Salford boss Rowley said after the Saints defeat that he had been told he could not use any of the players who had featured in the Challenge Cup victory over Midlands Hurricanes the previous Sunday, and that he chose not to name a mixture of youth and experience.
"I don't think it's the right thing to do," he said.
Salford's team selection was widely criticised, with Burgess describing the situation as "a black mark on the competition" after his own side's win at Huddersfield on Sunday.
Salford, who have seven days to respond, were not immediately available for comment.
